---
title: "Meta-ExternalAgent: What Meta’s Crawler Does on Your Site"
url: https://hostmy.blog/meta-externalagent-wordpress/
date: 2026-11-15
modified: 2026-11-15
lang: en
author: "Aditya Sharma"
description: "Meta runs five crawlers with different jobs, and two may bypass robots.txt. What each does, the documented user agents, and demo log lines."
categories:
  - "RankReady"
image: https://hostmy.blog/wp-content/uploads/2026/10/hmb-card-2811-1024x538.jpg
word_count: 1227
---

# Meta-ExternalAgent: What Meta’s Crawler Does on Your Site

Meta-ExternalAgent is the crawler Meta uses for training foundation AI models or improving its products by indexing content directly, and it is one of five Meta crawlers with different jobs. Blocking it does not block Meta's link-preview crawler or the fetcher that visits a page when a person asks, and two of those can bypass your robots.txt. Knowing which is which is the whole job.

This post sets out what each Meta crawler does, the user-agent strings Meta documents, which ones may ignore robots.txt, and how to recognise them in a log. It is built on Meta's own crawler documentation. The log lines shown are invented demonstration lines, explained below.

![Numbered outline of this post, Meta-ExternalAgent: What Meta](https://hostmy.blog/wp-content/uploads/2026/10/hmb-diagram-2811.png)The route through this post.

## A note on the demo lines

The log lines in this post are invented examples. They use Meta's documented user-agent strings, addresses from the documentation range 203.0.113.0/24 and made-up paths and sizes. They are not from our server and they show no real traffic, volume or behaviour. They show you what to look for in your own log.

## The Meta crawlers

Meta's [documentation on its web crawlers](https://developers.facebook.com/docs/sharing/webmasters/web-crawlers/) lists several, and the differences between them are the point.

| Crawler | What it is for | Robots.txt, per Meta |
| ------- | -------------- | -------------------- |
| Meta-ExternalAgent | Crawling for uses such as training foundation AI models, or improving products by indexing content directly | Documented as controllable with a robots.txt rule |
| Meta-WebIndexer | Improving the quality of Meta AI search results, which helps Meta AI cite and link to your content | Presented as controllable with robots.txt |
| Meta-ExternalFetcher | Fetching individual links at a user's request, including to help AI complete tasks on websites | Meta says this crawler may bypass robots.txt rules |
| FacebookExternalHit | Fetching a shared link to gather, cache and show its title, description and thumbnail | Meta says it might bypass robots.txt for security or integrity checks, such as malware scanning |
| Meta-ExternalAds | Improving advertising and other business products and services | Presented as controllable with robots.txt |

The page we read does not describe FacebookBot, and it does not list IP addresses or a verification method, so this post makes no claim about either. Meta's documentation index may cover them.

## The user-agent strings

Meta lists two forms for each crawler, one with a documentation link and one without. The token comes first, in lower case for the newer crawlers.

| Crawler | User-agent string, as documented |
| ------- | -------------------------------- |
| Meta-ExternalAgent | meta-externalagent/1.1 (+/documentation/sharing/webmasters/web-crawlers) and meta-externalagent/1.1 |
| Meta-WebIndexer | meta-webindexer/1.1 (+/documentation/sharing/webmasters/web-crawlers) and meta-webindexer/1.1 |
| Meta-ExternalFetcher | meta-externalfetcher/1.1 (+/documentation/sharing/webmasters/web-crawlers) and meta-externalfetcher/1.1 |
| FacebookExternalHit | facebookexternalhit/1.1 (+http://www.facebook.com/externalhit_uatext.php) and facebookexternalhit/1.1 |
| Meta-ExternalAds | meta-externalads/1.1 (+/documentation/sharing/webmasters/web-crawlers) and meta-externalads/1.1 |

Matching on the token, in a case-insensitive search, is safer than matching the whole string, because of the two forms. Here are four invented lines. The first two show Meta-ExternalAgent reading robots.txt and a post. The third is a link-preview fetch. The fourth is a user-triggered fetch of a page that your robots.txt disallows, which Meta says it may do.

`203.0.113.50 - - [22/Oct/2026:14:05:12 +0000] "GET /robots.txt HTTP/1.1" 200 612 "-" "meta-externalagent/1.1 (+/documentation/sharing/webmasters/web-crawlers)"
203.0.113.50 - - [22/Oct/2026:14:05:14 +0000] "GET /sample-post/ HTTP/1.1" 200 18452 "-" "meta-externalagent/1.1 (+/documentation/sharing/webmasters/web-crawlers)"
203.0.113.51 - - [22/Oct/2026:14:09:40 +0000] "GET /sample-post/ HTTP/1.1" 200 18452 "-" "facebookexternalhit/1.1 (+http://www.facebook.com/externalhit_uatext.php)"
203.0.113.52 - - [22/Oct/2026:14:11:02 +0000] "GET /members-only/ HTTP/1.1" 200 4100 "-" "meta-externalfetcher/1.1"`

The last line is the one worth reading carefully. In this invented example the path would be disallowed in robots.txt, and the fetcher was served anyway. Meta documents that behaviour for this crawler, so it is not a sign of a broken rule. It is a sign that robots.txt is not the right tool for it. If a page must not be fetched at all, the answer is access control on the page, not a robots rule.

## What blocking each one does

| If you block | You lose | You do not affect |
| ------------ | -------- | ----------------- |
| Meta-ExternalAgent | Nothing visible, since it is for training and indexing | Link previews and the user-triggered fetcher |
| Meta-WebIndexer | Meta says allowing it helps Meta AI cite and link to your content | Training and link previews |
| FacebookExternalHit | Rich link previews when your pages are shared, since this is what gathers the title and thumbnail | Training |
| Meta-ExternalFetcher | Nothing you can enforce, because it may bypass the rule | Everything else |

The FacebookExternalHit row is a consequence rather than a quotation. Meta describes the crawler as the one that gathers a shared link's title, description and thumbnail, so blocking it would remove those from shares. It is the mistake people make when they paste a list of Meta names into robots.txt without reading what each is for.

## The 24-hour delay

Meta says its crawlers may cache the contents of robots.txt for up to 24 hours, so a change can take that long to take effect. Do not conclude from an hour of continued visits that a new rule failed. This is the same caution that applies to other vendors, and [do AI crawlers respect robots.txt? A log-based test](/ai-crawlers-respect-robots-txt/) builds it into the method.

## Count them in your own log

`grep -ciE "meta-external|meta-webindexer|facebookexternalhit" /var/log/nginx/access.log
grep -iE "meta-externalagent" /var/log/nginx/access.log | awk '{print $7}' | sort | uniq -c | sort -rn | head
grep -iE "meta-externalfetcher" /var/log/nginx/access.log | awk '{print $7,$9}' | sort | uniq -c | sort -rn | head`

The first counts all the Meta names together. The second shows what Meta-ExternalAgent asked for. The third shows the paths and status codes for the user-triggered fetcher, which is where a request to a disallowed path would show up. The log path varies, as [reading your server log by hand](/read-server-log-ai-crawlers/) explains, and the field numbers assume the common log format.

## Deciding

A site that does not want its writing used for AI training can block Meta-ExternalAgent and leave the others alone. A site that wants Meta AI to be able to cite and link to its content, as Meta describes, leaves Meta-WebIndexer allowed. A site that wants rich previews when its pages are shared leaves FacebookExternalHit allowed. The fetcher is not yours to decide with robots.txt, and the other two can bypass it in the cases Meta describes. The steps for writing the rules are in [blocking or allowing 29 AI crawlers, and the robots.txt trap that exposes you](/block-ai-crawlers-robots-txt/) and [WordPress robots.txt: what to allow, block and test](/wordpress-robots-txt/). They are not repeated here.

## Verify before you count

Because the page we read gives no addresses, verifying a Meta request needs another source, and until you have one a count by name is a count of claims. [How to verify a real AI crawler visit: IP lists and reverse DNS](/verify-ai-crawler-visits/) shows the method for vendors that publish lists. The same reasoning applies here: do not act on a number you have not verified.

## What this post cannot tell you

It cannot tell you whether any Meta crawler visits your site, how often, or what it fetches. The demo lines are invented, and Meta's page states intent. Your log has the answer. We have not published our own findings, and we make no claim about them.

## A routine

`1. Count the Meta names in your log.
2. Split the count by crawler.
3. Check the fetcher's requests against your robots.txt.
4. Decide on training, search citation and previews separately.
5. After a change, wait a day before testing it.`

## Where RankReady fits

RankReady's crawler list includes Meta-ExternalAgent and Meta-ExternalFetcher, lets you allow or block each one at a time, and keeps a log of which AI crawler fetched which URL. The limits above still apply: a robots.txt rule cannot stop a fetcher that Meta says may bypass it. It runs alongside Yoast, Rank Math, AIOSEO and SEOPress, and the free [AI Readiness Checker](https://hostmy.blog/plugins/rankready/ai-readiness-checker/) runs 41 checks on your live site.

If you split your Meta crawler count by name tonight, which of the five would be asking?