S SCHEMA.BIZ

Last updated:

Robots.txt Checker

This robots.txt checker reads a site's file, tests any user agent and path, and shows which AI crawlers it blocks.

Our server fetches the site's /robots.txt as SchemaBizBot, then the file is checked here in your browser.

How to use the robots.txt checker

"Fetch robots.txt" takes any address on a site: our server fetches that site's /robots.txt as SchemaBizBot and sends the file back, and it's checked in your browser, with the status, redirects and size of the fetch. "Paste robots.txt" checks a file you haven't published yet, in your browser only. A link such as /seo/robots-txt-checker/?url=https://example.com/ runs the check when it opens, and "Copy a link to this check" makes one.

"Try an example robots.txt" loads a small shop's file and tests Googlebot on /search?q=trail+shoes. It's blocked by line 4, Disallow: /search, in the * group, while /search/help is allowed by the longer Allow rule on line 5. GPTBot and CCBot share a group on lines 8 and 9 that disallows everything, OAI-SearchBot has its own Allow: /, and the Crawl-delay on line 6 gets a note: Google ignores it.

What the results show

  • Test a user agent and path. Type a product token (the list suggests common ones) and a path or full URL. The answer is Allowed or Blocked, with the rule that decided it, its line, and whether the crawler's own group or the * group applied. The deciding line is highlighted in the file view.
  • Problems. Each line the parser couldn't use, with its number: unknown fields, rules before any User-agent, paths that don't start with /, misspellings Google accepts but other crawlers may not (useragent, dissallow), lines without a colon, relative sitemap addresses, fields Google ignores (crawl-delay, host, noindex), and a * group that disallows the whole site.
  • AI crawlers. The 28 tokens below, each tested on the path you tested (/ until you test another), with the rule or group that decided, and below it the AI training preference set by a __usage_ai-training group, if any.
  • Sitemaps the file lists, and the file itself with line numbers.

When /robots.txt doesn't answer 200, the status decides instead of the rules. A 4xx answer such as 404 means there's no file, and every crawler may fetch everything. A 5xx answer, or 429 for Google, means the file is unavailable: RFC 9309 tells crawlers to assume everything is disallowed, and Google stops crawling the site for 12 hours, then falls back to the last copy it read. An HTML page served at /robots.txt is an error: crawlers find no rules in it.

How crawlers read robots.txt

Robots.txt became a standard in 2022 as RFC 9309, and the checker follows it, with Google's documented behavior where Google says more.

  1. Pick the group. A crawler looks for groups whose User-agent line names its product token, ignoring case. Several such groups are merged. Only if there's none does it use the * group, and with neither, nothing is blocked. A version number in the line, as in Googlebot/2.1, is ignored: only the token counts.
  2. Find the matching rules. Each Allow and Disallow value is matched against the start of the path and query. * stands for any run of characters and a final $ for the end of the address. Paths are case-sensitive, and percent-encoded characters compare equal to the characters they stand for.
  3. The longest match wins. When an Allow and a Disallow match with the same length, the Allow wins. With no matching rule, the path is allowed. /robots.txt itself is always allowed.

Google reads the first 500 KiB of the file and ignores the rest, and the checker does the same, with an error saying which line the cut falls after. Google supports four fields: user-agent, allow, disallow and sitemap. Sitemap lines belong to no group, so they can go anywhere in the file.

AI crawlers and robots.txt

AI companies use separate tokens for separate jobs, so one file can let an AI search engine cite your pages while keeping model training out. The matrix groups them by purpose:

  • Model training: crawlers whose content may train models, such as GPTBot, ClaudeBot, meta-externalagent and Amazonbot. Google-Extended and Applebot-Extended are control tokens with no crawler of their own: blocking them tells Google and Apple not to use pages for their models, without changing Google Search or Applebot's crawl for search.
  • AI search: crawlers that index pages for answers with links, such as OAI-SearchBot, Claude-SearchBot and PerplexityBot. Blocking these can keep your pages out of those answers.
  • User-requested: fetchers that load a page because a person asked, such as ChatGPT-User, Claude-User and Perplexity-User. OpenAI, Perplexity, Google, Meta and Amazon say robots.txt may not apply to theirs; Anthropic says its bots honor it.

A different kind of line is on its way: User-agent: __usage_ai-training, from an IETF AIPREF draft. It names no crawler. A Disallow in its group says content on those paths may not be used to train AI models, whichever crawler fetched it, and it doesn't affect search. A * group doesn't apply to it, and a file without its group states no preference. Microsoft says Bing honors it from February 1, 2027, and Google says it intends to support it in early 2027, reading an opt-out there as a Google-Extended disallow when the file has no Google-Extended group. The checker shows it under the matrix.

Every token in the list was checked on its vendor's own page on 2026-11-06 (OpenAI, Anthropic, Perplexity, Google, Apple, Common Crawl, Meta, Amazon, Mistral AI, DuckDuckGo, You.com, Diffbot). Tokens that appear only on third-party lists, such as Bytespider, aren't in it. This file keeps training crawlers out and leaves everything else to the * group:

# Keep model training crawlers out, let AI search in
User-agent: GPTBot
User-agent: ClaudeBot
User-agent: Google-Extended
User-agent: Applebot-Extended
User-agent: CCBot
Disallow: /

User-agent: *
Disallow: /cart/

The AI readiness checker runs the same matrix for one page, along with its noindex and snippet controls and its structured data, and schema markup for AI covers what that markup tells those engines once they can read it.

Common robots.txt mistakes

  • Blocking a page to remove it from Google. A blocked page can't show its noindex, so it can stay indexed. Allow the crawl and use noindex.
  • A specific group that forgets the general rules. A Googlebot group replaces the * group for Googlebot; copy every rule it should still follow.
  • Blocking CSS and JavaScript. Google renders pages, and blocked scripts and stylesheets can change what it sees.
  • A staging file in production. User-agent: * with Disallow: / blocks every crawler without a group of its own, search engines included. The checker warns about it.
  • Wrong case or a missing slash. Disallow: admin matches nothing; Disallow: /Admin doesn't block /admin.
  • One file for every subdomain. Each host and protocol needs its own: https://shop.example.com/robots.txt doesn't cover https://example.com/.

Pages a sitemap lists shouldn't be blocked either: check the file's sitemaps with the sitemap checker, and the pages themselves with the meta tag checker.

Frequently asked questions

FAQ

Does robots.txt keep a page out of Google?

No. It stops crawling, not indexing: a blocked page can still be indexed from links to it, without its content. To keep a page out of results, let Google crawl it and use a noindex robots meta tag or X-Robots-Tag header. The meta tag checker reads both.

Is robots.txt case-sensitive?

Field names and user agent names aren't: Disallow, disallow and GPTBot, gptbot all match. Paths are: Disallow: /Private/ doesn't block /private/. The checker matches the same way.

Why is a crawler allowed when my file has a Disallow for everyone?

A crawler with its own group follows only that group. Google says groups for a specific user agent and the * group aren't combined, so a User-agent: GPTBot group with only Allow: / lets GPTBot in even when User-agent: * has Disallow: /.

Do AI crawlers obey robots.txt?

Their vendors say the training and search crawlers do. Several user-requested fetchers, such as ChatGPT-User, Perplexity-User and Google-Agent, act for a person, and their vendors say robots.txt may not apply to them. The matrix marks those.

Does the checker store my robots.txt?

A pasted file is checked in your browser and never leaves it. For a site address, our server fetches its /robots.txt as SchemaBizBot and sends the file back to your browser, where it's checked. Like every tool on the site, the checker records what kind of check ran: counts of groups, rules and problems, the problem codes and how many listed AI crawlers are blocked, never the file, a rule or what you typed into the tester. A fetched file's address is recorded without its query string. See the privacy policy.