OpenAI Crawlers and robots.txt: A Practical GPTBot vs OAI-SearchBot Guide

Publishers keep mixing up three separate controls:

• OAI-SearchBot for ChatGPT search inclusion
• GPTBot for model-training access
• ChatGPT-User for user-triggered fetches

That confusion leads to bad robots.txt decisions. Teams block every OpenAI bot, then wonder why they never show up in ChatGPT answers. Or they allow everything because they want AI visibility, then realize they also approved training access they never meant to grant.

This guide gives you the practical setup.

If you want a fast baseline check before editing crawler rules, start with Fennec’s Robots.txt Tester, Bot Simulator, and full SEO Audit.

The current official split between OpenAI bots

OpenAI’s current crawler documentation separates the roles clearly:

• OAI-SearchBot is used to surface websites in ChatGPT search features
• GPTBot is used for content that may be used to train OpenAI foundation models
• ChatGPT-User is used for some user-initiated actions and is not the control point for ChatGPT search eligibility

That means your choice is not binary. You do not have to either allow all OpenAI crawling or block all OpenAI crawling.

The default decision most publishers actually want

For most editorial, SaaS, and documentation sites, the clean default is:

1. Allow OAI-SearchBot
2. Decide separately whether to allow GPTBot
3. Do not rely on ChatGPT-User for search visibility decisions

Why this setup is usually the least risky:

• It preserves eligibility for ChatGPT search answers
• It keeps your training decision separate from your discovery decision
• It makes your policy easier to explain internally to legal, editorial, and product teams

A simple decision matrix

Goal	OAI-SearchBot	GPTBot	Notes
Appear in ChatGPT search and allow training	Allow	Allow	Broadest OpenAI access
Appear in ChatGPT search but opt out of training	Allow	Block	Most common publisher setup
Block OpenAI search and training	Block	Block	Maximum restriction
Unsure and testing impact	Allow	Block first	Safer than blocking both

OpenAI’s docs also note that these settings are independent, and changes to search eligibility can take about 24 hours to adjust after a robots.txt update.

The robots.txt patterns that match each policy

1. Allow ChatGPT search, block training

This is the setup most publishers mean when they say “we want AI visibility, but not training use.”

User-agent: OAI-SearchBot
Allow: /

User-agent: GPTBot
Disallow: /

2. Allow both search and training

User-agent: OAI-SearchBot
Allow: /

User-agent: GPTBot
Allow: /

3. Block both search and training

User-agent: OAI-SearchBot
Disallow: /

User-agent: GPTBot
Disallow: /

4. Block only a sensitive section

If you want public articles available but want to keep private assets, staging sections, or thin parameter pages out of crawler paths:

User-agent: OAI-SearchBot
Disallow: /staging/
Disallow: /internal/

User-agent: GPTBot
Disallow: /

Use this only when the paths are truly not meant for discovery. If the content should stay eligible for search but not be indexed in traditional search engines, robots.txt is often the wrong tool.

robots.txt does not equal deindexing

Google’s guidance is still the cleanest way to avoid a common mistake here: robots.txt controls crawling, not guaranteed removal from search results.

That matters because teams often do this:

1. Block a folder in robots.txt
2. Add noindex inside those blocked pages
3. Assume the URLs will disappear everywhere

That does not work reliably. If a crawler cannot fetch the page, it cannot see the noindex rule. Google’s documentation explicitly says noindex must remain crawlable to be processed, whether via a meta robots tag or an X-Robots-Tag header.

So use the right control for the right job:

• Use robots.txt to manage crawl access
• Use meta robots or X-Robots-Tag to manage indexing
• Use authentication or access controls for truly private content

If you are auditing broader crawl/index conflicts, pair this with Fennec’s Canonical Checker, Sitemap Checker, and Technical SEO workflows.

The implementation checklist I recommend

1. Audit your current crawler layers

Check more than the root robots.txt file:

• CDN or WAF rules
• bot management products
• origin-level deny rules
• middleware that treats unknown bots differently

Cloudflare’s AI Crawl Control docs are useful here because they expose crawler-level allow/block states, bot categories, and robots.txt violation tracking. This is often where teams discover that robots.txt says “allow” while an upstream WAF still blocks the traffic.

2. Decide by content class, not by panic

Split URLs into groups:

• public articles and docs
• gated tools or account areas
• internal search or faceted junk URLs
• staging, preview, and QA environments

Do not make a sitewide block decision before you know which URL groups are actually worth exposing.

3. Keep your important pages easy to discover

Google’s AI features documentation still points to the basics:

• allow crawling
• keep internal linking strong
• make important content available as text
• avoid burying key facts behind client-side interactions

That applies to AI crawlers too. Letting OAI-SearchBot in will not help if your best pages are orphaned, thin, or hard to render. Use GSC Management and AI Overview workflows to connect crawler access with real discovery signals.

4. Verify the bot before changing policy

User-agent strings can be spoofed. If you are seeing strange crawl behavior, verify before you block:

• OpenAI publishes IP JSON files for OAI-SearchBot, GPTBot, and ChatGPT-User
• Cloudflare exposes verified detection IDs for major AI crawlers
• server logs should confirm whether the traffic is actually coming from the documented bot

This matters most when a security team sees “GPTBot” in logs and reacts with a broad block before validating the source.

5. Test the result after deployment

Run a short validation loop:

1. Fetch /robots.txt and confirm a 200 response
2. Check that the intended user-agent group is present
3. Use Bot Simulator to compare crawler behavior on key URLs
4. Confirm important pages remain internally linked and accessible
5. Watch logs for allowed versus denied requests by bot

6. Measure the outcome, not just the rule

OpenAI’s publisher FAQ says ChatGPT referrals include utm_source=chatgpt.com. That gives you a clean measurement path in analytics after you allow OAI-SearchBot.

Track at least:

• ChatGPT referral sessions
• landing pages receiving those sessions
• crawl hits from OAI-SearchBot
• changes in citation-ready content coverage

If you never connect crawler policy to measurement, you are just editing files without learning anything.

Common mistakes to avoid

Blocking OAI-SearchBot and expecting ChatGPT citations

OpenAI states that sites opted out of OAI-SearchBot will not be shown in ChatGPT search answers. If visibility is your goal, that block works against you.

Using robots.txt when the real need is noindex

If the page must stay out of traditional search, use page-level or header-level indexing controls instead of assuming crawl blocking will solve it.

Forgetting upstream security controls

Cloudflare, custom WAF rules, bot filters, and origin rate limits can override your crawler intent.

Treating ChatGPT-User like a normal crawler

OpenAI says ChatGPT-User is not used for automatic web crawling or search inclusion decisions. It belongs in a different conversation about user-triggered fetches, actions, and security review.

Recommended policy for most Fennec-style sites

If you publish public marketing pages, blog posts, guides, and product docs, the safest starting point is:

User-agent: OAI-SearchBot
Allow: /

User-agent: GPTBot
Disallow: /

Then:

• allow crawl access to the pages you want cited
• keep those pages indexable if you expect discovery
• verify CDN and WAF behavior
• measure ChatGPT referral traffic after rollout

That policy is simple, reversible, and aligned with how OpenAI currently separates search access from training access.

Sources

• OpenAI crawler documentation
• OpenAI Publishers and Developers FAQ
• Google Search Central: robots.txt introduction
• Google Search Central: block indexing with noindex
• Google Search Central: robots meta tag and X-Robots-Tag
• Google Search Central: AI features and your website
• Cloudflare AI Crawl Control bot reference
• Cloudflare AI Crawl Control: manage AI crawlers
• Cloudflare AI Crawl Control with WAF

Q&A

Can I appear in ChatGPT search without allowing GPTBot?

Yes. OpenAI documents that OAI-SearchBot controls ChatGPT search inclusion, while GPTBot controls whether content may be used for training. You can allow OAI-SearchBot and disallow GPTBot independently.

Will blocking GPTBot remove my pages from Google or other search engines?

No. GPTBot is OpenAI's training crawler, not Google's search crawler. Google indexing rules are separate, and Google recommends using noindex or access controls instead of robots.txt when the goal is removal from search results.

What should I do about ChatGPT-User?

Treat ChatGPT-User as user-triggered fetch traffic, not an automatic crawler for search inclusion. OpenAI says it is not used to determine whether content appears in ChatGPT search, so manage search eligibility with OAI-SearchBot first.