Controlling AI Crawlers

Tomohiro Iida · Published June 26, 2026

robots.txt lets a site tell AI crawlers such as GPTBot, OAI-SearchBot, ClaudeBot, and Google-Extended what to do, but robots.txt is a request, not an enforceable block, and treating "do not train on us" and "show up in AI answers" as the same decision is the most common mistake. This guide separates crawlers into two purposes: training crawlers (GPTBot, ClaudeBot, CCBot), which feed a foundation model's training data, and search/reference crawlers (OAI-SearchBot, PerplexityBot), which an AI references in real time to answer a user's question and cite a source. Blocking the wrong category can cost a company visibility in AI answers without anyone realizing it. The guide covers what each major bot is for, clears up the common misunderstanding about Google-Extended and search rankings, gives robots.txt syntax and copy-paste examples for three policy stances, and explains how to verify a rule is actually being followed by checking server access logs.

What AI crawlers are

An AI crawler is a bot operated by an AI company that automatically visits and fetches pages, similar in mechanism to a search-engine crawler such as Googlebot but different in what it does with the content. AI crawlers split into two categories: training crawlers, used as training data for a foundation model (large language model), and search/reference crawlers, which an AI consults in real time when answering a user's question, for use in the answer or as a citation. This distinction is the single most important point in the topic.

Blocking a training bot makes a company's content less likely to feed a model's training. Blocking a search/reference bot makes a company less likely to appear when the AI answers in real time by referencing the web. The goals can point in opposite directions, so do not treat them as one decision. Note that a single bot can sometimes serve more than one purpose, and operators update their specs, so check each vendor's current documentation.

Major AI crawlers and what they do

The three OpenAI bots need to be kept separate: GPTBot is for training, OAI-SearchBot is a reference bot for ChatGPT's search results, and ChatGPT-User is triggered by a user's own action. Blocking "all of OpenAI" as one group risks losing search-result visibility along with training exclusion, so target the block to the purpose actually intended. CCBot, run by Common Crawl, broadly collects the public web to publish a dataset rather than serving one AI company specifically; that dataset can be reused for training by multiple companies, so it is worth considering for a company that wants to avoid AI training generally. Bot names and specs change, so check each vendor's latest documentation before implementing.

Google-Extended: the common misconception

Google-Extended is not an independent crawler or user agent. Google still uses the same Googlebot to crawl the web; no separate Google-Extended bot visits a site's pages. Instead, Google-Extended is a token a site operator sets in robots.txt to indicate whether content Googlebot has already crawled may be used to train or ground (source) Google's AI models such as Gemini. Two consequences follow: blocking Google-Extended has no effect on Google Search crawling or ranking, since normal crawling, indexing, and ranking by Googlebot continue unchanged and only AI usage permission is separated out; and it therefore gives a rare option to keep Google Search visibility while declining to let content be used for Google's AI training or generation.

The setting is User-agent: Google-Extended followed by Disallow: /. Again, this has no bearing on Google Search rankings.

robots.txt basics

robots.txt is a plain-text file placed at a site's root (https://example.com/robots.txt) with instructions for crawlers. It mainly uses three fields: User-agent (which crawler the instruction targets: a specific bot, or an asterisk for all crawlers), Disallow (which paths to avoid: a single slash for the whole site, an empty value for no restriction), and Allow (an exception carved out of a broader Disallow).

• Instructions are grouped by User-agent block, separated by blank lines.
• A bot with its own explicit block follows only that block, not the wildcard block, so if a GPTBot-specific block is written, GPTBot ignores the wildcard instructions, and any rule meant to apply to it must be repeated in its own block.
• Matching of User-agent names is usually case-insensitive in practice, but it is safest to write it exactly as each vendor documents it.

Fundamentally, robots.txt is a request with no technical enforcement. Well-behaved crawlers honor it, but there is no guarantee that every crawler does. If a guaranteed block is needed, combine it with server-side access control (denying by User-Agent or IP); robots.txt is best understood as a way of communicating intent to legitimate operators.

Two separate questions: training and appearing in AI answers

This is where judgment most often goes wrong in practice: conflating "we do not want AI using our data" with "we want to appear in AI answers." They are separate axes. To avoid feeding training data, control training bots (GPTBot, ClaudeBot, CCBot) and Google-Extended. To keep appearing when ChatGPT or Perplexity answer a question, do not block the search/reference bots (OAI-SearchBot, PerplexityBot). A common mistake is blocking search bots along with everything else out of a general wish not to be used by AI, which removes a company from ChatGPT and Perplexity answers and forfeits AI-driven awareness and traffic it could otherwise have had.

For B2B companies that want AI-search inquiries but not to have their whole site trained on, Policy B is often the practical middle ground. Which policy is right depends on a company's brand and IP stance and how much weight it puts on AI-driven acquisition.

How to decide, and how to verify

• Content type: for information a company wants widely known, such as service pages, know-how articles, and case studies, keep search/reference bots allowed for AI-answer exposure; training use is a separate call. For distinctive, hard-to-replicate know-how, consider blocking training bots, though robots.txt has no enforcement. For anything that should not be public at all, manage access with authentication rather than robots.txt, which itself reveals the existence of any path it lists.
• How much you value AI-driven acquisition: to gain AI-search awareness and traffic, keep search/reference bots allowed; if that is not a priority, a broader block is a reasonable choice, decided by where AI ranks among a company's lead-generation priorities.
• IP, legal, and brand policy: whether a company is comfortable having its text or diagrams reflected in generative AI output is a business and legal decision, not just a technical one, so align with the relevant teams before setting it.
• How much enforcement is needed: a statement of intent via robots.txt is enough for well-behaved crawlers; if a guaranteed block is required, add server-side access control as well.

Verifying your settings actually work

• Check the file itself: visit the domain's robots.txt in a browser and confirm it renders as intended, with no accidental site-wide block hiding in the wildcard User-agent, and no 404 or server error, which would mean it is never read at all.
• Check server access logs to see whether requests from a blocked User-agent, such as GPTBot, have actually stopped; since robots.txt has no enforcement, this is the only way to confirm real compliance.
• Check Search Console after any change, to make sure Googlebot's crawling and indexing have not been affected. Google-Extended has no effect on ranking, but an overly broad Disallow under the wildcard User-agent would.
• Revisit the file every six months to a year, since bot names, vendor specs, and new AI crawlers keep changing.

Example robots.txt by policy

Policy A (maximize exposure, training included) needs no special AI-specific entries: leave the normal search-facing robots.txt as is, without explicitly blocking any AI bot.

Policy B (appear in AI answers, do not allow training): block GPTBot, ClaudeBot, CCBot, and Google-Extended (each with Disallow: /); allow OAI-SearchBot and PerplexityBot (Disallow: left empty); leave the wildcard User-agent unrestricted for search engines.

Policy C (minimize all AI involvement): block GPTBot, OAI-SearchBot, ChatGPT-User, ClaudeBot, PerplexityBot, CCBot, and Google-Extended (each with Disallow: /), while leaving the wildcard User-agent unrestricted so search-engine crawling, and search traffic, are not affected.

In both examples, the wildcard User-agent is never given a blanket Disallow: /, so search-engine crawling is preserved. And as before, these entries are a request to well-behaved crawlers; compliance is up to each operator. For a guaranteed block, add server-side access control and confirm compliance afterward from the access logs.