Model training · Meta
Meta-ExternalAgent
Meta-ExternalAgent is Meta's crawler for content that may be used in model training. It honours robots.txt.
- Operator
- Meta
- Purpose
- Model training
- robots.txt
- Honoured
- Last checked
Documented user agent
meta-externalagent/1.1 (+https://developers.facebook.com/docs/sharing/webmasters/crawler)Match on the token Meta-ExternalAgent rather than the whole string, since the version number inside the string can change. A user agent alone can be forged, so verify it as shown below.
robots.txt rules
To block it everywhere:
User-agent: Meta-ExternalAgent
Disallow: /To allow it explicitly, for example when a broader rule blocks other agents:
User-agent: Meta-ExternalAgent
Allow: /What blocking it changes
Meta says Meta-ExternalAgent is used for training foundation AI models or improving products by indexing content directly; disallowing it in robots.txt opts out of that crawl.
- Meta says robots.txt changes can take up to 24 hours to take effect.
- Meta documents two related agents: Meta-WebIndexer, which crawls to improve Meta AI search results, and Meta-ExternalFetcher, which fetches links at a user’s request and may bypass robots.txt.
How to verify a request is genuine
Meta’s crawler page does not publish IP ranges for this agent; it directs questions to Meta. Treat the user agent alone as unverified.
Sources
Meta's documentation for Meta-ExternalAgent, read on 15 Oct 2026. Operators change these pages without notice; check before relying on a detail.
Check how your site treats Meta-ExternalAgent
The AI crawler access check reads your robots.txt and fetches a URL as this agent, so you see what a CDN or firewall does as well.
Run the free check All AI crawlers