Blocking OAI-SearchBot vs GPTBot Costs You Two Different Things

What is the difference between OAI-SearchBot vs GPTBot?

OpenAI runs OAI-SearchBot vs GPTBot as two crawlers with two unrelated jobs: OAI-SearchBot fetches pages so ChatGPT's search feature can surface them, and GPTBot fetches pages to train OpenAI's generative AI foundation models, per OpenAI's own bot documentation at developers.openai.com, checked 9 September 2026. Each one reads its own named block in robots.txt, and a rule written for one has no effect on the other. That separation is the whole point of the design: OpenAI states it "uses OAI-SearchBot and GPTBot robots.txt tags to enable webmasters to manage how their sites and content work with AI."

What are the exact user agent strings for OpenAI's crawlers?

OpenAI publishes the full user agent string for every agent it operates, and each one carries a self-describing URL you can check. Copy these character for character when you write a log filter, because a substring match on GPT alone will catch three different agents at once.

OAI-SearchBot also sends a second documented variant that inserts the token robots.txt before the URL: Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/131.0.0.0 Safari/537.36; compatible; OAI-SearchBot/1.4; robots.txt; +https://openai.com/searchbot. Write your log filter against the bare token OAI-SearchBot so both variants land in the same bucket.

Which OpenAI bot should you actually block?

Block the one whose output you do not want, and the cost column below is the part site owners skip. OpenAI ties each agent to a different product surface, so the three rules buy three different outcomes.

The full behaviour of that third agent, including why a robots.txt rule behaves differently for a user-triggered fetch, is covered in the dedicated ChatGPT-User page.

Why do most site owners block the wrong one?

The mistake is treating "block the AI crawlers" as one decision when OpenAI built it as two. GPTBot is the name people recognise from coverage of AI training data, so it goes into a blanket rule alongside OAI-SearchBot, and the blanket rule removes the site from the surface that sends real readers. Traffic from ChatGPT's search feature depends on OAI-SearchBot being allowed to read the page it cites; training input does not send anyone to your site at all. Decide the two questions separately, then write two blocks.

What robots.txt should you copy for each intent?

Pick the intent that matches your business and paste the matching block at the root of your domain. Each User-agent group stands alone, so a directive under one name never reaches another.

Stay out of model training, keep ChatGPT search traffic. This is the setting most publishers and SaaS sites want:

User-agent: GPTBot
Disallow: /

User-agent: OAI-SearchBot
Allow: /

Leave OpenAI entirely, including the search surface and user-requested fetches:

User-agent: GPTBot
Disallow: /

User-agent: OAI-SearchBot
Disallow: /

User-agent: ChatGPT-User
Disallow: /

Allow everything except the paths that should never reach a model or an answer:

User-agent: GPTBot
Disallow: /members/
Disallow: /checkout/
Disallow: /account/

User-agent: OAI-SearchBot
Disallow: /members/
Disallow: /checkout/
Disallow: /account/

The precedence rules that decide which line wins when Allow and Disallow both match a URL live in the full robots.txt reference.

Is robots.txt enforced, or just honoured?

Robots.txt is honoured by convention and enforced by nothing. RFC 9309, the Robots Exclusion Protocol standard published in September 2022, states plainly in its Security Considerations that "The Robots Exclusion Protocol is not a substitute for valid content security measures," and its opening section notes that these rules "are not a form of access authorization." A Disallow line is a request that a well-behaved crawler reads before fetching, and a badly behaved one can ignore without any technical consequence. If a page must not be read by anyone outside your login wall, put it behind authentication or a server-side IP rule, and treat the robots.txt entry as documentation of intent for the crawlers that do comply.

How do you verify a request really came from OpenAI?

Match the source IP address against the JSON files OpenAI publishes, because a user agent string is a header any client can type. OpenAI lists current ranges at openai.com/gptbot.json, openai.com/searchbot.json, openai.com/chatgpt-user.json and openai.com/adsbot.json, each file carrying a creationTime field and an array of CIDR prefixes. Checked on 10 September 2026, openai.com/searchbot.json listed 39 IPv4 prefixes with a creationTime of 2026-01-02, and openai.com/gptbot.json listed 21 prefixes with a creationTime of 2025-10-30. Refetch the file on a schedule instead of hardcoding the ranges, and treat any request carrying an OpenAI token from an address outside the current list as a spoof, not as a crawler ignoring your rules.

How long does a robots.txt change take to reach OpenAI?

OpenAI's documentation states that for search results "it can take ~24 hours from a site's robots.txt update for our systems to adjust." That delay applies to the search surface, so a site that flips OAI-SearchBot from disallowed to allowed should expect roughly a day before ChatGPT's search behaviour reflects it. Ship the change, note the timestamp, and hold off on a second edit until a full day has passed, because two changes inside the propagation window make the result impossible to read.

What should you track after you set the rules?

Track whether ChatGPT actually cites you, because a correct robots.txt is an input and a citation is the outcome. AI search brand monitoring watches where ChatGPT, Claude and Gemini reference a brand in their answers, which is the only direct read on whether allowing OAI-SearchBot bought you anything. The same training-versus-search split appears in other vendors' stacks: Applebot-Extended is Apple's training opt-out that leaves Siri, Spotlight and Safari search alone, and CCBot is Common Crawl's open archive that outside labs filter into training sets. Getting quoted at all is a separate discipline from getting crawled, which is what answer engine optimization covers.

Frequently Asked Questions

Does blocking GPTBot remove me from ChatGPT search?

No. OpenAI documents GPTBot as the crawler for training generative AI foundation models and OAI-SearchBot as the crawler that powers ChatGPT's search features. A Disallow under User-agent: GPTBot leaves OAI-SearchBot free to crawl, so ChatGPT search can still surface and cite your pages.

What happens if I block OAI-SearchBot?

OpenAI states that sites opted out of OAI-SearchBot "will not be shown in ChatGPT search answers, though can still appear as navigational links." You lose the citation and the click-through that comes with it. OpenAI recommends allowing OAI-SearchBot in robots.txt to help a site appear in search results.

Is there one directive that blocks all OpenAI bots at once?

No single token covers them. OpenAI documents GPTBot, OAI-SearchBot, ChatGPT-User and OAI-AdsBot as separate robots.txt user-agent tokens, and each needs its own named group. A wildcard User-agent: * group applies to compliant crawlers that find no group naming them specifically, so an agent with its own block reads that block instead.

Which OpenAI agent is not a crawler?

ChatGPT-User. OpenAI documents it as handling user-initiated actions in ChatGPT and Custom GPTs, meaning it fetches a page because a person asked ChatGPT to open that link. OpenAI notes that robots.txt rules may not apply to it the same way, since the request depends on a live user action.

What does OAI-AdsBot do?

OpenAI documents OAI-AdsBot as the agent that validates the safety of web pages submitted as ChatGPT ads, and states that "data collected by OAI-AdsBot is not used to train generative AI foundation models." It has its own robots.txt token and its own published IP list at openai.com/adsbot.json. It is separate from both the training crawl and the search crawl.

Can a crawler ignore my robots.txt file?

Yes. RFC 9309 describes the Robots Exclusion Protocol as a set of rules crawlers read, and its Security Considerations section states the protocol "is not a substitute for valid content security measures." Nothing in the protocol blocks a request at the network layer, so anything that must stay private needs authentication or a server-side block instead.

Should I hardcode OpenAI's published IP ranges instead of refetching them?

OpenAI's IP list files carry a creationTime field that changes whenever the range updates, so a hardcoded copy goes stale the next time OpenAI rotates its infrastructure. Refetch openai.com/gptbot.json, searchbot.json, chatgpt-user.json or adsbot.json on a schedule instead, and treat a request carrying an OpenAI user agent from an address outside the current list as a spoof rather than a crawler ignoring robots.txt.

Why shouldn't I make a second robots.txt change right after the first?

OpenAI's documentation puts the propagation time for search at roughly 24 hours after a robots.txt update. A second edit inside that window lands before the first one has finished taking effect, so whatever result you see afterward can't be traced to either change. Ship one edit, note the timestamp, and wait out the full day before touching the file again.

Does robots.txt give me any legal authority over crawlers?

RFC 9309 states in its opening section that robots.txt rules "are not a form of access authorization." A Disallow line asks a compliant crawler to skip a page; it carries no mechanism to stop one that doesn't comply. Content that actually needs protection belongs behind a login wall or a server-side IP rule, with the robots.txt entry serving only as a record of intent.

Can GPTBot and OAI-SearchBot rules live in the same robots.txt file?

They can, and the combination of Disallow for GPTBot and Allow for OAI-SearchBot is the exact setting OpenAI's own split makes possible for site owners who want out of training while keeping ChatGPT search traffic. Each User-agent group is read on its own, so the GPTBot block and the OAI-SearchBot block never interfere with each other in the same file.