Free technical SEO tool
Robots.txt Checker
Fetch the live robots.txt file and check syntax, site-wide blocks, sitemap declarations, and access policies for search and AI crawlers.
Check a website
Enter a domain or any page URL. We always inspect /robots.txt at that site’s root.
The check runs through Fennec’s Cloudflare API and never changes the target website.
Fetching and analyzing…
Check result
This is a live request. CDN, WAF, and robots.txt controls are separate layers.
AI crawler access overview
Specific user-agent groups take priority over wildcard rules. Partial means only some paths are disallowed.
| Crawler | Purpose | Access | Rule source | Path rules |
|---|
Parsed rule groups
Discovered sitemaps
View raw robots.txt
What does this check cover?
One check answers whether the file is reachable, whether its rules parse, and what different crawler types are allowed to access.
⚡
Live fetch
See the final URL, HTTP status, response time, file size, and content type.
✓
Syntax diagnostics
Find missing user agents, invalid paths, bad sitemap URLs, and site-wide blocks.
AI
AI crawlers
Review search, training, and user-triggered AI crawler access separately.
</>
Raw evidence
Keep the fetched robots.txt visible so SEO and engineering teams can verify it.
How one check works
The checker sends a website URL through the edge API, safely retrieves robots.txt, and turns it into diagnostics, sitemap discovery, and crawler access results.
- Resolve the root-level /robots.txt automatically
- Limit redirects, response time, and file size
- Return rule diagnostics, sitemaps, and crawler access
Common robots.txt rules
Rules are case-sensitive and apply only to the same protocol, hostname, and port. Place the file at the site root.
Allow every crawler
An empty Disallow value means there are no path restrictions.
User-agent: *
Disallow:Block directories
Paths start with / and matching is case-sensitive.
User-agent: *
Disallow: /admin/
Disallow: /internal-search/Allow an exception
A more specific Allow can open a resource inside a blocked directory.
User-agent: *
Disallow: /private/
Allow: /private/public-guide/Declare a sitemap
Use a complete absolute URL. Multiple Sitemap lines are valid.
Sitemap: https://example.com/sitemap.xmlHow to manage AI crawlers
Do not treat every AI crawler as the same use case. Search discovery, model training, and user-triggered retrieval commonly use different user agents.
| User-agent | Provider | Purpose | Type |
|---|---|---|---|
| OAI-SearchBot | OpenAI | Discovery, summaries, and citations in ChatGPT search | Search |
| GPTBot | OpenAI | Web collection that may be used to improve and train models | Training |
| ChatGPT-User | OpenAI | Visits initiated when a ChatGPT user requests a page | User action |
| Claude-SearchBot | Anthropic | Discovery and relevance for Claude search results | Search |
| ClaudeBot | Anthropic | Web collection that may contribute to model training | Training |
| Claude-User | Anthropic | Web visits initiated by a Claude user | User action |
| PerplexityBot | Perplexity | Search indexing and linked results in Perplexity | Search |
| Google-Extended | Controls some Gemini and Vertex AI training or grounding uses; it does not affect Google Search inclusion | AI use |
Note: OpenAI says robots.txt rules may not apply to user-triggered ChatGPT-User visits, and Perplexity says Perplexity-User generally ignores robots.txt. The checker marks these visits as “Rules may not apply.”
Separate AI crawler policy by purpose
One robots.txt file can manage search discovery, model training, and user-triggered retrieval separately instead of applying one switch to every AI crawler.
- Keep search discovery and citation access
- Decline model-training crawlers independently
- Flag user-triggered visits where robots.txt may not apply
A common “allow search, block training” policy
This example preserves discovery in ChatGPT and Claude search while declining the OpenAI and Anthropic training crawlers. Review it against your business and licensing policy before publishing.
# Keep AI search discovery available
User-agent: OAI-SearchBot
Allow: /
User-agent: Claude-SearchBot
Allow: /
# Decline model-training crawlers
User-agent: GPTBot
Disallow: /
User-agent: ClaudeBot
Disallow: /
# Default policy for other compliant crawlers
User-agent: *
Content-Signal: search=yes, ai-input=yes, ai-train=no
Allow: /
Sitemap: https://example.com/sitemap.xml
Three controls for three different jobs
Control layers: robots.txt communicates crawl preferences, a CDN or WAF enforces network blocks, and authentication protects genuinely private resources.
- robots.txt: guidance for compliant crawlers
- CDN / WAF: enforced network blocking
- Authentication: protection for private resources
Robots.txt questions
Can robots.txt remove a page from Google?
Not reliably. robots.txt controls crawling, not guaranteed index removal. If a crawler cannot fetch a page, it also cannot read a noindex directive on that page.
What happens if robots.txt is missing?
It normally means there are no path-level crawl restrictions, although the server, CDN, WAF, authentication, and HTTP status can still block crawlers. A reachable baseline file with a Sitemap declaration is still useful.
What does Disallow: / mean?
It normally blocks the corresponding user agent from crawling the entire site. Under User-agent: *, it can stop every compliant crawler from accessing the site.
Can robots.txt protect private content?
No. The file is public, and non-compliant bots may ignore it. Protect sensitive content with authentication, authorization, and network access controls.
Why is an AI crawler blocked even when robots.txt allows it?
robots.txt is only one layer. Cloudflare, a WAF, bot management, rate limits, CAPTCHAs, geography rules, or the origin firewall may still return 403 or 429.
