AI Crawlers and robots.txt: What a Blogger Should Allow
Key takeawaysAI companies run different bots for different jobs: training, search indexing, and fetching a page when a user asks. Blocking all of them with one...
19 October 2026 · 5 min read
Key takeaways
- AI companies run different bots for different jobs: training, search indexing, and fetching a page when a user asks. Blocking all of them with one rule blocks the ones that bring you readers.
- OpenAI, Anthropic and Perplexity each document a search or user-facing bot separately from their training bot.
Google-Extendedis a robots.txt token, not a crawler, and it does not affect Google Search.- robots.txt changes can take about a day to be picked up. Verify, don't assume.
Why is "block all AI bots" a bad default?
Because the bots are not the same thing. Blocking every user agent that mentions AI stops model training, but it can also remove you from the places readers now go to ask questions. Decide per bot.
What does each bot do?
| Bot | Company | Purpose | robots.txt behavior |
|---|---|---|---|
| OAI-SearchBot | OpenAI | Surfaces sites in ChatGPT's search features | Opted-out sites don't appear in ChatGPT search answers, though they can still show as navigational links |
| GPTBot | OpenAI | Crawls content that may be used to train foundation models | Disallowing signals your content should not be used for training |
| ChatGPT-User | OpenAI | Fetches a page when a user asks in ChatGPT or a custom GPT | Because users initiate it, robots.txt rules may not apply |
| ClaudeBot | Anthropic | Collects public content that may help train models | Disallow excludes future content from training |
| Claude-SearchBot | Anthropic | Indexes content to improve search results | Blocking may reduce your visibility in user search results |
| Claude-User | Anthropic | Retrieves pages when a Claude user's question needs the web | Blocking may reduce visibility in user-directed search |
| PerplexityBot | Perplexity | Surfaces and links sites in Perplexity search; documented as not used for foundation model training | Allow it to appear in results |
| Perplexity-User | Perplexity | Visits a page when a user asks | Generally ignores robots.txt because a user requested the fetch |
| Google-Extended | Product token controlling whether crawled content is used for future Gemini training and grounding | Does not affect inclusion in Google Search |
Sources for each row are listed below.
Three types of bot, three decisions
| Type | Examples | Question to ask |
|---|---|---|
| Training | GPTBot, ClaudeBot, Google-Extended | Do I want my content used to train models? |
| Search / index | OAI-SearchBot, Claude-SearchBot, PerplexityBot | Do I want to appear in AI search answers? |
| User-initiated fetch | ChatGPT-User, Claude-User, Perplexity-User | Mostly outside robots.txt; decide at the firewall if at all |
Is Google-Extended a crawler?
No. Google describes it as a standalone product token. It has no HTTP user agent of its own, because crawling happens with Google's existing user agents. Google also says it does not impact a site's inclusion in Google Search. Separately, Google states there are no additional requirements to appear in AI Overviews or AI Mode beyond being indexed and eligible to show a snippet, and that you need no special markup or AI text files.
What is a sensible robots.txt for a blog?
If your goal is to stay visible in AI answers while opting out of training:
# Allow search and answer bots
User-agent: OAI-SearchBot
Allow: /
User-agent: Claude-SearchBot
Allow: /
User-agent: PerplexityBot
Allow: /
# Opt out of training
User-agent: GPTBot
Disallow: /
User-agent: ClaudeBot
Disallow: /
User-agent: Google-Extended
Disallow: /
This is a policy choice, not a rule. If you want maximum exposure, allow all of them. If you sell content, you may block more. The point is to choose deliberately.
| Goal | Allow | Block |
|---|---|---|
| Maximum visibility | All of the above | Nothing |
| Visible but not used for training | Search bots | Training bots / tokens |
| Locked down | Googlebot only | AI bots (accept less AI visibility) |
What won't robots.txt do?
- It will not stop user-initiated fetchers that document ignoring it, such as Perplexity-User.
- It is not an access control. Anthropic specifically advises against blocking by IP, because that can prevent the crawler from reading your robots.txt at all. Anthropic also publishes its IP ranges for verification.
- It applies per host. Anthropic notes that you repeat the rules for each subdomain.
- Changes are not instant. OpenAI says search systems can take about 24 hours to adjust, and Perplexity says up to 24 hours.
How do I check what is actually hitting my blog?
- Look at server or CDN logs for these user-agent strings.
- Compare IPs against each vendor's published list (Anthropic and Perplexity publish JSON lists).
- If you use a firewall or WAF, check it is not blocking the search bots you meant to allow. Perplexity advises matching both the user agent and published IP ranges, and updating rules periodically.
- Re-check a day after any change.
A 15-minute robots.txt review
| Step | Done |
|---|---|
Open yourblog.com/robots.txt and read it | ☐ |
| List which AI bots are mentioned | ☐ |
| Decide: training allowed? search allowed? | ☐ |
| Edit the file or ask your host | ☐ |
| Check each subdomain | ☐ |
| Wait 24 hours, re-check logs | ☐ |
Frequently asked questions
Will blocking GPTBot remove me from ChatGPT? OpenAI documents OAI-SearchBot as the search bot, separate from GPTBot.
Does blocking Google-Extended hurt my Google rankings? Google says it does not affect inclusion in Search.
Do I need llms.txt? Google says no special file is needed for its AI features. Other vendors have not documented requiring it in the sources used here.
Can Roami edit robots.txt? Not from this article's workflow. This is a hosting setting you control.
Next step
Once the bots can reach you, check whether anything arrives: Track AI Referral Traffic in GA4. Then try Roami to find the posts worth surfacing, and run the visibility test.
Sources (checked 18 Oct 2026; vendor pages can change)
Next step: connect an agent, or audit your archive first.
Drafted with an agent in Roami, edited and published by Aaghran. This blog runs on Roami.