Everything robots.txt controls, and the one thing it does not

What is robots.txt?

A site publishes robots.txt at the root of a host to tell crawlers which URL paths they may fetch, and that is the entire scope of the file: robots.txt asks a crawler not to fetch a page, does not make the page private, and does not reliably keep the URL out of search results. The rules come from RFC 9309, the Robots Exclusion Protocol, a Standards Track document published in September 2022 by Martijn Koster, Gary Illyes, Henner Zeller and Lizzi Sassman. RFC 9309 requires the rules to be "accessible in a file named '/robots.txt' (all lowercase) in the top-level path of the service", and Google's robots.txt specification adds that the rules "apply only to the host, protocol, and port number where the robots.txt file is hosted." Read the file as traffic control for crawlers that choose to obey it.

Last verified: 8 September 2026

Why does a page blocked in robots.txt still show up in Google?

Google lists the URL because a link pointed at it, and listing a URL takes no fetch. Google's robots.txt introduction, last updated 10 December 2025, states that a robots.txt file "is not a mechanism for keeping a web page out of Google," and Google's specification adds that Google "can't index the content of pages which are disallowed for crawling, but it may still index the URL and show it in search results without a snippet." A disallow rule stops the fetch and nothing else, so the result carries anchor text from the linking page in place of a description.

The fix runs opposite to instinct. A noindex robots meta tag works only when the crawler fetches the page and reads it, so a disallow line is exactly what stops noindex being seen. Delete the disallow, serve noindex, and let Google fetch the page to learn it should drop out. For content that must not be readable at all, use authentication: robots.txt is a public file listing your sensitive paths to anyone who opens it.

What does each robots.txt directive do?

RFC 9309 defines three fields, user-agent, allow and disallow, plus three special characters, and Google's specification adds the sitemap field.

Percent-encode a literal *, $ or # inside a path, as RFC 9309 recommends.

Crawl-delay sits outside the standard. Google's robots.txt specification, last updated 31 August 2026, lists the four fields Google reads and states that "other fields such as crawl-delay aren't supported." Support is per crawler: Anthropic's documentation publishes Crawl-delay: 1 under User-agent: ClaudeBot as an alternative to a full block. Write the line only for crawlers whose operator documents it.

Which rule wins when allow and disallow both match?

The longest matching path pattern wins, measured in octets, and an exact tie goes to allow.

winner = the allow or disallow rule whose path pattern contains the most octets; on an exact tie, allow

RFC 9309 puts it as "The most specific match found MUST be used. The most specific match is the match that has the most octets," and settles ties with "If an 'allow' rule and a 'disallow' rule are equivalent, then the 'allow' rule SHOULD be used." Google's specification reaches the same outcome: "In case of conflicting rules, including those with wildcards, Google uses the least restrictive rule."

Work it against https://example.com/blog/drafts/post-1:

19 beats 13 beats 6, so the third rule wins and the crawler fetches the page. Delete that line and 13 beats 6, so the disallow takes over. For a tie, Disallow: /p and Allow: /p both count 2 octets, so allow wins and /page stays crawlable. Line order changes nothing, so count characters instead of rearranging rules.

How do crawlers pick which user-agent group applies?

A crawler matches its product token against the user-agent lines case insensitively, obeys that one group, and falls back to the * group only when no group matches. RFC 9309 requires that "Crawlers MUST use case-insensitive matching to find the group that matches the product token" and that "If no matching group exists, crawlers MUST obey the group with a user-agent line with the '*' value". Two groups naming the same token get merged, which RFC 9309 states as rules that "MUST be combined into one group" and Google's specification describes as crawlers that "internally merge the groups".

The consequence catches people out. Give GPTBot its own group and GPTBot stops reading the * group, so every disallow written for everyone else stops binding it. Repeat the shared disallow lines inside each named group. Spelling each AI crawler token exactly matters for the same reason: a typo drops a crawler back into the * group.

How do you block AI crawlers in robots.txt?

Name each operator's token in its own group, and decide separately for training crawlers and for the search crawlers that cite and link back.

# https://example.com/robots.txt
 
User-agent: *
Disallow: /cart/
Disallow: /account/
Sitemap: https://example.com/sitemap.xml
 
# Model training
User-agent: GPTBot
Disallow: /
 
User-agent: ClaudeBot
Disallow: /
 
User-agent: Google-Extended
Disallow: /
 
# Answer engines that cite and link
User-agent: OAI-SearchBot
Disallow: /cart/
Disallow: /account/
 
User-agent: PerplexityBot
Disallow: /cart/
Disallow: /account/
 
User-agent: Claude-SearchBot
Disallow: /cart/
Disallow: /account/

The repeated /cart/ and /account/ lines are the group rule in action, not padding: each named crawler reads only its own group. Three caveats come from the operators. Google documents that "Google-Extended doesn't have a separate HTTP request user agent string" and works "in a control capacity," and that blocking it "does not impact a site's inclusion in Google Search nor is it used as a ranking signal in Google Search." OpenAI separates GPTBot, which crawls to improve its foundation models, from OAI-SearchBot, which surfaces sites in ChatGPT search, so blocking the GPTBot token leaves ChatGPT search access intact. Perplexity documents that Perplexity-User ignores robots.txt rules in most cases because a user requested the fetch, and OpenAI runs the same kind of on-demand fetcher under a third token, ChatGPT-User, separate from both GPTBot and OAI-SearchBot. The same per-operator split extends beyond this list: Amazonbot crawls to improve Amazon's products and may train Amazon's AI models unless its token is blocked, and Applebot-Extended can be disallowed on its own to stop Apple Intelligence training while leaving Siri, Spotlight and Safari search untouched.

Blocking training crawlers while keeping answer engines is a visibility decision, and it is measurable. RedReplier tracks the public threads naming a brand, which is what social media monitoring means in practice.

What happens when robots.txt is missing, huge, or returns an error?

The HTTP status decides, and the two error classes point in opposite directions. RFC 9309 states that on a 4xx "the crawler MAY access any resources on the server," and that on a 5xx "the crawler MUST assume complete disallow." A 404 opens the whole host and a 503 closes it, so an erroring origin during a deploy pauses crawling of everything. RFC 9309 also sets a parsing limit of "at least 500 kibibytes," which Google enforces by ignoring content past 500 KiB, caps reuse of a cached copy at 24 hours unless the file is unreachable, matching Google's own 24 hour cache, and asks crawlers to follow "at least five consecutive redirects" before treating the file as unavailable. Check that your CDN returns 200 for /robots.txt on every host and protocol you serve.

Is robots.txt the same as llms.txt?

No. robots.txt is a standard with an RFC behind it and documented support from every major crawler operator, while llms.txt is a proposed convention that none of the crawler documentation cited here mentions. The two answer different questions: robots.txt says which paths a crawler may fetch, and llms.txt proposes which content a model should read first.

Neither file is a legal control. A crawler that ignores robots.txt breaks a convention, and whether that crosses into unlawful collection turns on terms of service, access controls and jurisdiction. For guaranteed, permitted access to a platform's data, the choice between scraping and an API settles it.

Frequently asked questions

Does robots.txt keep a page out of search results?

No. Google's robots.txt introduction states directly that robots.txt "is not a mechanism for keeping a web page out of Google." A disallowed URL that other sites link to can appear in results as a bare URL with no snippet. Use a noindex meta tag on a crawlable page, or authentication, when exclusion is the goal.

Does the order of lines in robots.txt matter?

No. RFC 9309 resolves conflicts by the number of octets in the matching path pattern, so Disallow: /blog/drafts/ at 13 octets beats Allow: /blog/ at 6 octets wherever the two lines sit in the file. Order matters only for grouping, since every allow and disallow belongs to the user-agent line above it.

Does one robots.txt file cover subdomains and both protocols?

No. Google's specification states that the rules "apply only to the host, protocol, and port number where the robots.txt file is hosted." A file at https://example.com/robots.txt says nothing about https://shop.example.com/ or about the http version. Publish a separate file on every host you want to control.

Can I put noindex in robots.txt?

Google's robots.txt specification lists four fields Google reads, user-agent, allow, disallow and sitemap, and noindex is not among them. Serve noindex as a robots meta tag in the page head or as an X-Robots-Tag HTTP header. Both require the page to stay crawlable, so remove any disallow covering it.

Does robots.txt stop scrapers?

No. RFC 9309 describes rules a crawler chooses to obey, with no enforcement mechanism, so a scraper that ignores the file fetches the pages anyway. Perplexity says so openly about its own Perplexity-User fetcher, which ignores robots.txt rules in most cases because a user asked for the page. Rate limiting, authentication and WAF rules are the controls that enforce anything.

Does blocking AI crawlers hurt search visibility?

Blocking Google-Extended does not, on Google's own statement that it "does not impact a site's inclusion in Google Search nor is it used as a ranking signal in Google Search." Blocking answer-engine crawlers is a different trade: Anthropic states that blocking Claude-SearchBot may reduce visibility in Claude's search results, and OAI-SearchBot and PerplexityBot exist to surface and link sites. Block the training tokens and keep the search tokens when citations matter to you.

Who wrote RFC 9309?

Martijn Koster, Gary Illyes, Henner Zeller and Lizzi Sassman wrote RFC 9309, the Robots Exclusion Protocol. It became a Standards Track document in September 2022. Google's own robots.txt specification builds on the same rules and adds fields RFC 9309 leaves out, such as sitemap.

How big can a robots.txt file be before Google ignores it?

RFC 9309 sets a parsing limit of at least 500 kibibytes for a robots.txt file. Google enforces that ceiling by ignoring any content past the 500 KiB mark, so rules written beyond that point never apply. Keep the file short and put the logic in a handful of user-agent groups rather than one sprawling list.

Does blocking GPTBot also block ChatGPT search?

GPTBot and OAI-SearchBot are separate tokens under OpenAI's documentation. GPTBot crawls to improve OpenAI's foundation models, while OAI-SearchBot surfaces sites inside ChatGPT search. Disallow GPTBot on its own and ChatGPT search access stays intact, since each crawler reads only its own user-agent group.

How long does Google cache robots.txt before rechecking it?

Google caches a fetched robots.txt file for up to 24 hours before checking it again, unless the file becomes unreachable. RFC 9309 sets the same ceiling for generic crawlers. A change to a disallow line can take most of a day to take effect, so plan deploys of new blocking rules with that lag in mind.