Skip to content
Geonos

Launching December 2026. Alexa for Shopping for Amazon sellers comes first. Get the launch email

How-to

robots.txt and AI bots: how to let ChatGPT and Google in

· Updated · 7 min read · By Dafne Gutierrez Herrera, founder of Geonos

Leer en español

The short answer

Allow OAI-SearchBot, Claude-SearchBot, PerplexityBot, Bingbot and Googlebot so you can appear in AI answers. Decide on training bots such as GPTBot, ClaudeBot and Google-Extended separately, and publish your sitemap.

Search bots and training bots are different settings

Most AI companies run more than one bot, and each one has its own job and its own line in robots.txt. OpenAI says its settings are independent: you can allow OAI-SearchBot to appear in ChatGPT search results and disallow GPTBot so your content is not used to train its models. The same split exists at Anthropic and at Google.

That gives you three groups to think about:

  • Search bots collect pages so an engine can show and link your site in answers. Blocking them keeps you out of those answers.
  • Training bots collect content for model training. Allowing or blocking them is a business decision, and it does not change whether you show up in search.
  • User-triggered fetchers open a page because a person asked a question. They are not crawlers.

The bots to know

BotOwnerWhat it doesWhat blocking it does
GooglebotGoogleCrawls for Google Search, which also feeds AI Overviews and AI ModeRemoves you from Search and from those AI features
Google-ExtendedGoogleA control token for using your content in Gemini training and groundingNo effect on Search, according to Google
BingbotMicrosoftCrawls for Bing, which Copilot relies onRemoves you from Bing and from Copilot answers
OAI-SearchBotOpenAIShows sites in ChatGPT searchYour site is not shown in ChatGPT search results
GPTBotOpenAICollects content for trainingSignals that your content should not be used for training
ChatGPT-UserOpenAIOpens a page when a person asks ChatGPTOpenAI says robots.txt rules may not apply to it
Claude-SearchBotAnthropicIndexes content to improve search resultsMay reduce your visibility in Claude search
Claude-UserAnthropicFetches a page when a person asks ClaudeAnthropic says site owners can control it with robots.txt
ClaudeBotAnthropicCollects content for trainingSignals that future content should be excluded from training
PerplexityBotPerplexitySurfaces and links sites in Perplexity searchYour site is not shown in Perplexity search
Perplexity-UserPerplexityFetches a page when a person asks PerplexityPerplexity lists it as a robots.txt control

Each company's description comes from its own bot page, last checked 12 November 2026. Companies change these pages, so check the source before you rely on a row.

What Google needs

Google says there are no additional requirements to appear in AI Overviews or AI Mode, and no special optimizations. A page needs to be indexed and eligible to appear with a snippet. So for Google the work is the normal SEO work: a page Googlebot can crawl, a sitemap, and no accidental noindex. Meeting this does not guarantee that Google crawls, indexes or shows a page.

Google-Extended is a separate token. It controls whether your content is used for some of Google's AI systems, such as Gemini. Google says it has no effect on Google Search.

What ChatGPT, Claude and Perplexity need

For ChatGPT search, OpenAI says to use OAI-SearchBot in robots.txt. A site that opts out of OAI-SearchBot is not shown in ChatGPT's search answers. Perplexity recommends allowing PerplexityBot and the IP ranges it publishes. Anthropic says disabling Claude-SearchBot may reduce your visibility in Claude's search.

Two things to remember about user-triggered fetchers:

  • OpenAI says ChatGPT-User acts on a person's request, so robots.txt rules may not apply to it, and it is not used to decide what appears in search.
  • Anthropic and Perplexity both describe their user bots as something site owners can control.

Allowing a bot does not make an engine name you. It makes you eligible. Whether you are cited depends on your content and on who else is saying the same things.

Check your own site in five steps

  1. Open the file. Go to yoursite.com/robots.txt. Look for Disallow: / under User-agent: * or under any bot in the table.
  2. Check each bot by name. A bot follows the group that names it. If a bot has its own group with Disallow: /, a general Allow above it does nothing for that bot.
  3. Open the sitemap. Go to yoursite.com/sitemap.xml. It should list the pages you want found, and robots.txt should point to it with a Sitemap: line. An empty sitemap gives engines no list to start from.
  4. Check the server, not only the file. robots.txt says what you allow. Your CDN, firewall or host can still refuse the bot. Compare what your site returns to a bot with what it returns to a normal browser. Bot protection settings are the usual cause when they differ.
  5. Wait, then re-check. OpenAI and Perplexity both say changes can take about 24 hours to reach their systems.

Two limits people hit without knowing

Google documents a size limit for each file, and both cut off silently:

  • robots.txt: Google reads the first 500 KiB. Rules after that point are ignored, so a long file can lose the rules at the bottom.
  • A sitemap file: up to 50 MB uncompressed and 50,000 URLs. Past that, split it into several files and list them in a sitemap index.

Most small sites are nowhere near either limit. If yours is a large catalog, check.

How we set up our own site

We follow this on getgeonos.com. The robots.txt names the search bots and the training bots and allows the public site. It keeps the API, the owner area, the demo and our design files out of crawling. A sitemap lists every public page, in English and in Spanish, and robots.txt points to it. We did not rely on the file alone: our free audit also compares what the server returns to each bot with what it returns to a browser.

Allowing the training bots is our choice. It may not be yours. If you sell content, or you have contracts about who can train on it, block those bots and keep the search bots open.

How to measure what arrives

  • ChatGPT: OpenAI adds utm_source=chatgpt.com to the links in its results, so you can filter those visits in Google Analytics 4.
  • Google: traffic from AI Overviews and AI Mode is counted in the Performance report in Search Console, under the Web search type, not as a separate line.
  • Everything else: keep the saved answer and the date, before and after each change, and compare.

Being named is not the same as being cited, and nobody can promise you either.

How Geonos helps

The free audit checks whether search engines and AI crawlers can reach your pages under your robots.txt, compares what your server sends each bot with what it sends a browser, checks the size limits above, and lists what to fix first. Geonos then tracks the questions your buyers ask in Google and in AI answer engines.

Sources

Frequently asked questions

Do I need to allow GPTBot to appear in ChatGPT?

No. OpenAI says its settings are independent. OAI-SearchBot controls whether your site appears in ChatGPT search, and GPTBot controls whether your content may be used for training.

Does blocking Google-Extended remove me from AI Overviews?

No. Google says Google-Extended has no effect on Google Search. AI Overviews and AI Mode use the normal search index, so they follow Googlebot.

How long do robots.txt changes take to work?

OpenAI and Perplexity both say about 24 hours for their systems to adjust. Google recrawls robots.txt on its own schedule.

Is robots.txt enough?

No. It says what you allow. A firewall, CDN or host setting can still refuse a bot that robots.txt allows. Check what the server actually returns to the bot.

Will allowing these bots get me cited in ChatGPT?

It makes you eligible, nothing more. Whether an engine names you depends on your content, your facts being consistent, and what other sources say about you. Measure it with the same prompts over time.

Keep reading

Get the launch email

Geonos is launching in December 2026, starting with Alexa for Shopping for Amazon sellers. Leave your email and we will write to you on launch day.