Meta's AI Training Crawler meta-externalagent, and How to Block It

What is meta-externalagent?

Meta operates meta-externalagent as the crawler that collects public web pages for training foundation AI models, according to Meta's Meta Web Crawlers documentation, updated 21 May 2026 and checked 9 September 2026. Meta's exact wording is that the crawler "crawls the web for use cases such as training foundation AI models or improving products by indexing content directly." One robots.txt group controls it for an entire site, and Meta states that changes to robots.txt take up to 24 hours to take effect because its crawlers cache the file for that long.

What is the exact meta-externalagent user agent string?

Meta documents two forms of the string, and both begin with the same lowercase token:

meta-externalagent/1.1 (+/documentation/sharing/webmasters/web-crawlers)
meta-externalagent/1.1

Meta introduces both with the sentence "The specific UA string that you will see in your log files will be similar to one of the following", so the version suffix and the parenthesised path are not a stable contract. Match your log filters and firewall rules on the substring meta-externalagent, case insensitively, and they keep working when Meta bumps 1.1 to 1.2.

How does meta-externalagent differ from Meta's other crawlers?

Meta runs five documented crawlers, and each one has a separate robots.txt token, so blocking one leaves the other four untouched. Two of them tell you outright that they can ignore your file.

The pairing worth understanding is meta-externalagent against meta-externalfetcher. The first is a bulk crawler that walks the web on Meta's schedule to build training data. The second fires when a person asks Meta AI to look at a specific link, which puts it in the same category as OpenAI's user-triggered fetcher, ChatGPT-User, and Meta says that one can bypass your rules. FacebookExternalHit sits apart from both: block it and links to your site lose their preview card on Facebook, Instagram and Messenger.

How do you block meta-externalagent in robots.txt?

Add a User-agent group naming the token and disallow the whole site. Put this at the root of your domain, in the file served at /robots.txt:

User-agent: meta-externalagent
Disallow: /

Meta's own documentation shows the same group used the other way, allowing everything and carving out one directory:

User-agent: meta-externalagent
Allow: /                    # Allow everything
Disallow: /private/         # Disallow a specific directory

If your goal is to keep your pages out of AI training across vendors, this group is one line in a longer file that also names Apple's training opt-out token, Applebot-Extended and Common Crawl's CCBot. Path matching, wildcards and group precedence are covered in the full guide to robots.txt syntax, so keep the crawler groups here and the rule mechanics there.

Does blocking meta-externalagent actually stop Meta?

Blocking works only because the crawler chooses to read your file and obey it. Robots.txt is a convention, not an access control: your server still returns 200 to anyone who asks, and the standard has no enforcement, no authentication and no penalty. Meta demonstrates the limit in its own documentation by naming two crawlers, meta-externalfetcher and facebookexternalhit, that can fetch pages your file disallows.

Enforcement lives at the server. A deny rule on the user agent string, an IP block, a WAF rule or a paid access gate like Cloudflare's pay per crawl returns 403 instead of content, and that decision is yours, not the crawler's.

Can you verify that a request from meta-externalagent is really Meta?

Meta's crawler documentation gives no way to verify it. The page publishes no IP ranges, no autonomous system number and no forward-confirmed reverse DNS rule for meta-externalagent, which leaves the user agent string as the only identifier, and any client on the internet can send that header with one curl flag. The single contact Meta offers on that page is webmasters@meta.com.

Treat the string as a label, not proof. If you are enforcing a block instead of requesting one, write the rule against request behaviour you can observe, such as rate and path pattern from a single IP, and reserve user agent matching for the crawlers that identify themselves honestly.

Does capitalisation matter when you block meta-externalagent?

Capitalisation of the crawler name does not matter. RFC 9309 requires crawlers to match the product token in a User-agent line case-insensitively, so meta-externalagent, Meta-ExternalAgent and META-EXTERNALAGENT all select the same group. Spelling does matter: a missing hyphen or a stray character produces a token that matches no crawler, and the block fails without any error to tell you. Path values are the opposite case. RFC 9309 says Disallow and Allow paths should be matched case sensitively, so /Private/ and /private/ are two different rules.

Should you block meta-externalagent if you want AI visibility?

Blocking meta-externalagent removes your pages from Meta's AI training collection, and it does not remove you from Meta AI's answers, because Meta assigns citation and linking to a different crawler. Meta writes that allowing Meta-WebIndexer "helps us cite and link to your content in Meta AI's responses", so a site that wants citations without contributing training data disallows meta-externalagent and allows meta-webindexer.

That split is a decision, and the way to check it is measurement, not assumption. RedReplier's LLM brand monitoring for AI search tracks where ChatGPT, Claude and Gemini cite your brand in their answers, so you can watch what a robots.txt change does to citations instead of guessing.

Frequently Asked Questions

Is meta-externalagent the same as facebookexternalhit?

No. Meta documents them as separate crawlers with separate robots.txt tokens and separate jobs. Meta-ExternalAgent crawls the web to train foundation AI models and index content, while FacebookExternalHit fetches a link that someone shared on Facebook, Instagram or Messenger in order to build its title, description and thumbnail. Disallowing one has no effect on the other.

Does blocking meta-externalagent break Facebook link previews?

No. Link previews are produced by FacebookExternalHit, which reads the facebookexternalhit token in robots.txt. A group that disallows only meta-externalagent leaves preview crawling untouched. If you want to keep previews, do not add facebookexternalhit to your disallow list.

How long does a robots.txt change take to reach Meta's crawlers?

Meta's documentation asks you to allow up to 24 hours, because its crawlers cache the contents of robots.txt for up to that long. A block added this morning can still see crawler hits this afternoon without anything being broken. If requests continue past 24 hours, check that the file is served at the domain root and returns 200.

Can I block meta-externalagent with a meta robots tag instead?

Meta states that it prefers industry standard practices like robots.txt over non-standard formats like NoAI tags, and the only crawler control its documentation describes is robots.txt. A page level meta tag also requires the crawler to fetch and parse the page first, which defeats the purpose if your aim is to prevent the fetch. Use the robots.txt group.

What is the difference between meta-externalagent and meta-externalfetcher?

Meta-ExternalAgent crawls broadly on Meta's own schedule to build training and indexing data. Meta-ExternalFetcher fetches individual links at a user's request to support agentic AI features, and Meta states that this crawler may bypass robots.txt rules for that reason. A robots.txt disallow is therefore a reliable instruction to the first and an unreliable one to the second.

Where does Meta publish the official meta-externalagent documentation?

Meta publishes it at developers.facebook.com under Sharing, Webmasters, Meta Web Crawlers, at https://developers.facebook.com/documentation/sharing/webmasters/web-crawlers. That page lists every documented Meta crawler, the user agent strings, the robots.txt example and the webmasters@meta.com contact address. Check it before writing firewall rules, since Meta revises the crawler list and the version suffixes.

How current is Meta's documentation on meta-externalagent?

Meta updated its Meta Web Crawlers page on 21 May 2026, and this article's facts were checked against that page on 9 September 2026. Crawler names and version suffixes change over time, so a rule built from an old copy of the page can miss a new token. Check the live page at developers.facebook.com before you write a firewall rule.

Who do you contact if meta-externalagent misbehaves on your site?

Meta lists one contact for its crawler documentation, webmasters@meta.com. That same page publishes no IP list, no ASN and no verification method, so a report to that address is your only channel besides the robots.txt block itself. Use it for behavior your own server rules cannot control.

Is meta-externalfetcher the same kind of crawler as ChatGPT-User?

Meta-ExternalFetcher fires when someone asks Meta AI to look at one specific link, the same job OpenAI's ChatGPT-User does for ChatGPT. Both act on a single user request instead of crawling broadly on a schedule, which separates them from bulk crawlers like meta-externalagent. Meta states that meta-externalfetcher may bypass robots.txt, so blocking meta-externalagent has no effect on it.

Can I allow meta-externalagent on part of my site instead of blocking it everywhere?

Meta's own documentation shows a group that allows the whole site and excludes one directory, using Allow: / followed by Disallow: /private/. You can point that same pattern at any folder you want to keep out of training while leaving the rest open. It is the same robots.txt group, just with different Allow and Disallow lines instead of a blanket Disallow: /.