Allowing or Blocking AI Crawlers

Tomohiro Iida · Published June 26, 2026 · Updated August 6, 2026

This article separates stating a policy in robots.txt from observing what actually happens in server logs. It is written for web, IT, and engineering teams deciding their own policy. The bot specifications here come from each vendor's official documentation, checked August 6, 2026. Names and specifications change, so check the current version before configuring anything.

1. AI crawlers split into three purposes

To put the conclusion first: treating all AI crawlers as one category produces results nobody intended. The purposes split three ways, and the search-and-reference and training uses can be specified independently in robots.txt. Only user-initiated fetching may not stop at robots.txt.

OpenAI's official documentation states that the settings for OAI-SearchBot and GPTBot are independent of one another, so a site can ask to appear in search results while declining training use. Anthropic likewise documents ClaudeBot for training, Claude-User for user-initiated access, and Claude-SearchBot for improving search quality as separate identifiers. Handling this in bulk, as "block all of OpenAI" or "block Anthropic as a whole", cuts off appearing in search results at the same time.

User-initiated access is different in nature from automated crawling. OpenAI states that ChatGPT-User is initiated by a user and that robots.txt rules may therefore not apply, and Perplexity states that Perplexity-User generally does not consult robots.txt rules. Do not design on the assumption that robots.txt will stop it.

OpenAI: Overview of OpenAI CrawlersChecked August 6, 2026.

Anthropic: Does Anthropic crawl data from the web?Checked August 6, 2026.

Perplexity: Perplexity CrawlersChecked August 6, 2026.

2. The main crawlers and their official documentation

The table covers only identifiers documented by their operators. If you configure an identifier not listed here, confirm its purpose in the operator's own primary source first.

CCBot is Common Crawl's crawler. It is not dedicated to one AI company: it broadly collects the public web and publishes it as open data, and because it is provided as a public dataset it may be reused by third parties. Common Crawl states that it is aware of crawlers impersonating CCBot and recommends verification by reverse DNS and IP range.

Do not treat the version number in a User-Agent string as fixed. OpenAI notes in its own documentation that version numbers may change, and Google advises using a wildcard rather than an exact version number when searching or filtering logs. Build extraction conditions on a partial match of the bot name. Names and specifications themselves change too, so re-read the official documentation every quarter.

Common Crawl: CCBotChecked August 6, 2026.

Google: Google common crawlersChecked August 6, 2026.

3. robots.txt is a fetching policy, not access control

robots.txt is a mechanism for communicating a fetching policy to crawlers. It is a text file placed at the root of a site (https://example.com/robots.txt) and mainly uses three fields.

User-agent

Specifies which crawler an instruction targets. It can name a bot individually, as in "User-agent: GPTBot", or target every crawler with an asterisk.

Disallow

Specifies paths a crawler is asked not to fetch. A single slash covers the whole site; a path such as /private/ covers one directory. An empty value means no restriction.

Allow

Specifies a path permitted as an exception within a broader Disallow. Where Allow and Disallow conflict, the rule with the longer path match wins, and where the lengths are equal Allow takes precedence (RFC 9309 and Google's robots.txt specification).

• A group consists of one or more User-agent lines followed by rules. Several user agents can be combined into one group, and where multiple groups target the same user agent, implementations may merge them. In practice, separating each target with a blank line makes the file harder to misread.
• When a rule matching a specific user agent exists, that bot follows the rules addressed to it rather than the wildcard instructions. Write a dedicated block for GPTBot and GPTBot stops consulting the "User-agent: *" block. Anything meant to apply to both has to be written into the dedicated block as well.
• Matching of user-agent names is commonly case-insensitive in implementations, but writing it exactly as the official documentation does is safer.

A minimal example that allows search and reference while declining training: a block with "User-agent: OAI-SearchBot" and "Allow: /", and a second block with "User-agent: GPTBot" and "Disallow: /". This states an intention not to obstruct fetching by the ChatGPT search crawler while declining fetching by GPTBot for training purposes.

To be explicit about the premise: robots.txt is neither authentication nor access control. Well-behaved crawlers respect what is written, but there is no guarantee that no crawler ignores it. On top of that, a path written into robots.txt is also a public statement that such a directory exists. Protect anything that should not be public with logins, IP restrictions, and permission management. Where a block genuinely has to hold, combine it with server-side access control.

Google: Introduction to robots.txtThe robots.txt specification referenced here. Checked August 6, 2026.

4. Allowing does not guarantee that you appear

Allowing a search-and-reference crawler does not mean you will appear in an AI search answer. What OpenAI's official documentation states is two things: sites that opt out do not appear in ChatGPT search answers (though they may appear as a navigation link), and appearing in search results is helped by allowing access in robots.txt and accepting access from the published IP ranges. Allowing is the entrance to being a candidate; it does not promise an appearance.

Beyond being fetchable, the conditions below are what we check in practice. They are our own operating guideline, not criteria any vendor publishes.

• A public page exists that answers the question
• Service and company information is accurate and current
• The body text contains material that can be cited: figures, sources, dates
• Pages are fast and return 200 reliably
• A WAF or CDN is not blocking the bot in question
• Enough time has passed for a robots.txt change to take effect

The causation in the other direction does have documentary support. Block OAI-SearchBot and you will not appear in ChatGPT search answers (the possibility of appearing as a navigation link remains). Anthropic likewise states that blocking Claude-SearchBot may reduce visibility and accuracy in search responses. In other words, "block it and you will not appear" holds up well, while "allow it and you will appear" does not. Estimate the effect of a configuration change on the basis of that asymmetry.

5. What Google-Extended actually is

Google-Extended is the place where AI crawler control assumptions most often need alignment. It is not an independent crawler or user agent. The official documentation states that Google-Extended has no dedicated User-Agent string for HTTP requests, that crawling itself is performed by Google's existing user agents, and that the robots.txt token is used for control purposes only. Google-Extended is the token by which a site operator indicates whether content Google has crawled may be used for training Gemini models in Gemini Apps and Vertex AI, and for grounding, the process of passing search index content to the model at answer time to improve factuality and relevance.

• Blocking Google-Extended does not affect a site's presence in Google Search and is not used as a Google Search ranking signal. This is stated explicitly in the official documentation.
• So if the policy is to keep appearing in Google Search as before while declining to have your content used for Google's AI training and grounding, blocking Google-Extended is a coherent choice.

The robots.txt entry is a block with "User-agent: Google-Extended" and "Disallow: /". Source: Google, Google common crawlers (checked August 6, 2026).

6. Information worth considering blocking

Decide blocking by the nature of the information. Below are the angles we check when designing client sites.

Proprietary material you do not want used for training

Paid teaching material, proprietary data, and information restricted by contract are candidates for blocking training crawlers. Note that publishing it at all does not fully prevent other means of retrieval or reading by humans. Declining in robots.txt is a way of stating an intention, not a guaranteed block.

Information that should not be public

Protect it with authentication rather than by writing the path into robots.txt. robots.txt is readable by anyone, and listing a blocked path also publishes your directory structure. Manage the scope of publication itself with logins, IP restrictions, and permission management.

Sites that cannot absorb heavy traffic

If load is the problem, do not try to solve it by blocking specific bots alone: design rate limiting, a CDN, caching, and a WAF. Check overall stability first. Anthropic states that it honours Crawl-delay, so an official means of adjusting load is sometimes available.

IP, legal, and brand policy

Whether a company can accept its text and diagrams being reflected in generative AI output is a business and legal decision rather than a technical one. Align the policy with the relevant departments before configuring anything. Declining training use in robots.txt is one way of stating that policy technically.

7. What to check in server logs

After changing robots.txt, confirm the actual access in your logs. Writing a setting and the setting behaving as written are different things. This section covers only the angles to observe.

Checking AI crawlers in server logsThe concrete extraction and aggregation procedure lives in this companion article.

Fields to store

Timestamp, IP address, User-Agent, request URL, status code, response time, referrer, whether robots.txt was fetched, and request count.

Angles to check

Whether the addresses match the IP ranges each vendor publishes; whether something is merely impersonating the User-Agent name; whether robots.txt is being fetched; whether 403, 429, and 5xx responses are frequent; whether important pages are being reached; whether large numbers of parameterized URLs are being fetched unnecessarily; and whether crawl load is high relative to the site's capacity.

Do not conclude from the User-Agent string alone that a request came from an official crawler. User-Agent strings are easy to spoof. OpenAI, Perplexity, and Common Crawl publish their IP ranges as JSON, and Common Crawl also documents a reverse DNS verification procedure. Where published IP information exists, match against it.

How to obtain logs depends on the environment. Whether you are on Vercel, Cloudflare, AWS, or shared hosting determines whether you look at CDN logs, WAF logs, web server logs, hosting access logs, an APM, custom middleware, or edge functions.

Google Analytics alone will not show crawler visits. Crawlers do not normally execute JavaScript measurement, so server-side logs are required.

8. Do not infer citation from logs

Even when an AI crawler fetches a page and the page is later cited in an AI answer, the two cannot be tied together as direct cause and effect. Several routes are possible at the same time.

• The answer may have used the results of a different search backend
• It may have used an earlier crawl
• The fetch may have been user-initiated
• It may have come via a citation on another site
• Even for the same URL, the content referenced may differ from the content at fetch time

Logs show the fact of a fetch and nothing more. Citation, absorption into the answer, and referral traffic are each observed by their own method. We recommend recording these three as separate metrics.

Measuring Google's AI features: Search Console vs. external AI citation observation

Measuring AI visibility: repetitions, confidence intervals, and the limits of a score

9. A procedure for deciding your policy, with configuration examples

Here is everything above, in the order it happens in practice. Six steps.

• Classify your content: information you want widely discovered, information you want in search but not in training, information that needs authentication, and information you cannot publish under contract.
• Check the purpose per service: in each vendor's official documentation, confirm whether an identifier is for search and reference, for training, or user-initiated.
• Update robots.txt: check that no site-wide block has crept into the instructions addressed to all crawlers. Check the staging environment and any admin-side configuration as well as the production domain.
• Confirm in server logs: allowing for the time a change takes to take effect, confirm the actual access and any errors (403, 429, 5xx) in your logs.
• Observe AI answers separately: record the question, whether you were cited, the answer content, and anything wrong. Treat it as a metric distinct from what the fetch logs show.
• Re-check every quarter: user-agent names, published IP ranges, and product specifications change. Re-read the official documentation quarterly.

Policies fall into roughly three shapes. Which is right depends on your IP and legal position and on how much weight you place on acquisition through AI.

Policy A (restrict neither search and reference nor training) needs no special AI-specific entries: leave the ordinary search-facing robots.txt as it is and do not explicitly block the AI crawlers.

Policy B (allow search and reference, decline training): give GPTBot, ClaudeBot, CCBot, and Google-Extended their own blocks with "Disallow: /", with a comment noting that declining Google's training and grounding use does not affect presence or ranking in Google Search; give OAI-SearchBot, Claude-SearchBot, and PerplexityBot their own blocks with "Allow: /"; and leave search engines as they are with a "User-agent: *" block whose "Disallow:" is empty.

Policy C (broadly decline AI-related crawlers): give GPTBot, OAI-SearchBot, ChatGPT-User, ClaudeBot, Claude-SearchBot, Claude-User, PerplexityBot, Perplexity-User, CCBot, and Google-Extended their own blocks with "Disallow: /", while leaving the "User-agent: *" block with an empty "Disallow:" so that search engine crawling, and search traffic, are not stopped.

Note that in both examples "Disallow: /" is never placed under "User-agent: *". Put a site-wide block there and search engine crawling stops too, and search traffic goes with it. For Policy C, also bear in mind that vendors state the user-initiated identifiers (ChatGPT-User and Perplexity-User) may not stop at robots.txt. If they have to be stopped, handle it with server-side access control.

After writing the file, open /robots.txt on your own domain and confirm the content is what you intended and that it does not return a 404 or a server error. Re-confirm as well that once a bot has its own dedicated rules, it stops consulting the "User-agent: *" entries. This is where a gap easily opens up: you believe you blocked everything, and the bots with dedicated entries sail straight through.

10. Netsujo's own policy

Our public services, technical articles, and company information are, by default, kept in a state where search-and-reference crawlers can fetch them. Training crawlers are judged separately, according to the nature of the content and the terms of the relevant contracts.

When designing client sites, we confirm the position on acquisition through AI, whether proprietary information is involved, server load, and legal conditions, then design the robots.txt policy statement and the server-side access control separately. The point is to draw the line first between what robots.txt can protect and what it cannot.

This is our own operating policy, not a universal right answer for every site. Judge it against your own IP position and legal conditions.

Frequently asked questions

Will writing it in robots.txt reliably stop AI training?

Not reliably. robots.txt is a mechanism for communicating a fetching policy; it is not authentication or access control. OpenAI, Anthropic, Perplexity, and Common Crawl document how to control their crawlers through robots.txt, but not every crawler on the web necessarily complies. Where a block genuinely has to hold, combine it with server-side access control. Information that should not be public is protected with logins, IP restrictions, and permission management rather than robots.txt.

What is the difference between GPTBot and OAI-SearchBot?

Both are OpenAI bots, with different purposes. GPTBot fetches content that may be used to train generative AI foundation models; OAI-SearchBot fetches so a site can be shown in answers from the ChatGPT search feature. OpenAI's official documentation states that the two settings are independent of one another, so you can, for instance, ask to appear in search results while declining training use.

If I allow OAI-SearchBot, will I appear in ChatGPT answers?

Allowing it does not guarantee an appearance. OpenAI's official documentation indicates that sites which opt out do not appear in ChatGPT search answers (though they may appear as a navigation link), and that appearing in search results is helped by allowing access in robots.txt and accepting access from the published IP ranges. Allowing secures the possibility of being fetched; whether you appear is a separate question.

Will blocking Google-Extended lower my search rankings?

Google's official documentation states explicitly that Google-Extended does not affect a site's presence in Google Search and is not used as a Google Search ranking signal. Google-Extended is not an independent crawler but a robots.txt token indicating whether already-crawled content may be used for Gemini training and grounding. It is an option where you want to appear in search but not have your content used for Google's AI training.

Can robots.txt stop access opened by a user action?

Sometimes it cannot. OpenAI states that for ChatGPT-User, because access is user-initiated, robots.txt rules may not apply. Perplexity likewise states that Perplexity-User generally does not consult robots.txt rules. To control user-initiated access as well, handle it with server-side access control.

Does Anthropic have only one bot?

Anthropic's official documentation lists three: ClaudeBot for training, Claude-User for user-initiated access, and Claude-SearchBot for improving search quality. Blocking only ClaudeBot has a different effect from blocking all three. Check the difference in purpose before treating them as one.

When does a robots.txt update take effect?

For search results, OpenAI states that it can take around 24 hours for a robots.txt update to be reflected in its systems. Perplexity likewise documents up to 24 hours for a configuration change to take effect. Do not judge from logs taken immediately after a change; allow time for it to take effect first.

How do I confirm the settings are being respected?

Check in your server access logs how access from the relevant User-Agent has changed. Because User-Agent strings can be spoofed, match against the IP ranges each vendor publishes, or against reverse DNS. The concrete extraction and aggregation procedure is covered in the companion article on checking AI crawlers in server logs.

Should I match on the version number in the User-Agent?

No. OpenAI's official documentation states that version numbers in User-Agent strings can change, and Google advises using a wildcard rather than an exact version number when searching logs. Build log extraction conditions on a partial match of the bot name and design them not to pin the version number.