Meta's New Crawling Terms: What Publishers With a Facebook Page Agree To on November 20

"You authorize us (or any entity acting on our behalf) to use automated software (such as web scrapers, bots, robots, spiders, crawlers, and user agents) and create test credentials to access, index, cache, analyze, collect, retrieve, or crawl the domains, web pages, or URLs … that are associated with your products or services." That is the opening of one bullet under "Licenses & Rights You Provide Meta" in Meta's preview of its new Commercial Terms, effective November 20, 2026.

Nobody signs it. Under the terms' update rule, you agree "by continuing any access or use of any Meta Company Products for business or commercial purposes after any notice of an update" (Meta). Virtually every news organization has at least one Meta business account, as Nieman Lab pointed out about itself, so the question in October was what that one bullet lets Meta do with a newsroom's website. Meta has edited the bullet once and moved its start date once. What's left is a contract-shaped permission sitting next to a robots.txt-shaped preference, and those are not the same kind of control.

This is not legal advice; it's a reading of public documents and reporting, with links to all of them.

The bullet, before and after October 17

The version Meta announced in early October went further than the text above. After the crawl permission, it said: "Meta may use any content from such domains, web pages or URLs to provide, develop, improve, research, train and maintain the integrity of, the Meta Company Products, services and features provided under these Commercial Terms" (PPC Land, quoting the UK-English text). The word that set off the reporting was "train", sitting in a contract that reaches far past advertisers: The Register lists managing a Page or a business Group, listing products, using a product catalog and using Ads Manager or Meta Business Suite among the uses the terms cover.

Two days after Nieman Lab's story, Meta removed that sentence. "We've heard you and want to clarify: the crawling terms we consolidated only applied to our ads and business products," the company said, adding that "it does not change how Meta uses web crawlers" (TechCrunch). Here is what that edit did and didn't touch:

The compare card is the summary. To see the edit word for word, switch between the two texts below; each part of the clause is labelled kept, removed, added or narrowed. Pick a part to see why it matters.

Four of the eight parts survive untouched, and they are the ones that decide what reaches Meta's servers: who may crawl ("any entity acting on our behalf"), with what (automated software and test credentials), where (anything "associated with your products or services") and how often ("periodically pull updates"). What changed is the stated reason. Lawyers Nieman Lab went back to weren't persuaded the reason matters much. Media lawyer Matt Topic said "'develop' is broad enough to include 'train'" and that "business product" doesn't appear to be a defined term; Cornell's James Grimmelmann said the purposes are "still broad enough to drive a truck through" (Search Engine Journal, quoting Nieman Lab).

Two details make the edit less clean than it sounds. First, sources disagree on when it reached every locale: SEJ found the old sentence still in the UK-English Self-Serve Ad Terms preview on October 19, and Relevant Audience found the revised wording on the same page on October 20. Second, advertisers had already granted a version of this. SEJ reports that the current Self-Serve Ad Terms (August 22, 2025) give Meta a right to crawl an advertiser's domains when it places an ad order, with uses that include "research and training" (Search Engine Journal). What's new in 2026 is the reach: the Commercial Terms cover businesses that never buy an ad.

Two doors into the same newsroom

Meta's defence leans on a control publishers already know. Its spokesperson told Nieman Lab that "content owners can still manage how their content is accessed by using robots.txt", and that the first version "wasn't clear" that robots.txt preferences are respected; the updates don't give Meta "new rights that supersede those preferences" (SEJ). Andy Stone said on X that the clause is about pulling accurate information on products for sale on third-party sites (SEJ).

That defence describes one route by which a page reaches Meta. The contract opens a second. Step through both:

Steps 1 and 2 are how AI crawling already works: a crawler asks, the site's robots.txt answers, and the answer is a request rather than a lock. Steps 3 and 4 are new. The permission doesn't come from your server at all; it comes from your social media manager logging into Business Suite on November 21. Step 5 is where publishers and Meta part ways.

What a robots.txt line can and can't do here

Start with what it can do, because it's real. Meta publishes its crawlers and their tokens (Meta for Developers):

• Meta-ExternalAgent "crawls the web for use cases such as training foundation AI models", and indexes content to improve products.
• Meta-ExternalAds "crawls the web for use cases such as improving advertising and other business-related products and services": the same purpose the revised clause now names.
• Meta-WebIndexer improves Meta AI search results, and allowing it helps Meta link to your content in answers.
• FacebookExternalHit fetches shared links for their title, description and image (the preview card), and "might bypass robots.txt when performing security or integrity checks".
• Meta-ExternalFetcher "fetches individual links at a user's request" and "may bypass robots.txt" for that reason.

Meta says changes take up to 24 hours to apply, because its crawlers may cache robots.txt for that long. Paste your own file below, or edit the sample, which disallows the training and ads crawlers and leaves everything else open:

Now what it can't do. Add User-agent: meta-externalfetcher with Disallow: / to the sample and the checker shows it blocked, with a warning: Meta says user-requested fetches may bypass robots.txt. More important, robots.txt speaks to user agents, and the contract doesn't name any. It covers "any entity acting on our behalf", and none of its text says a disallow line limits what you granted. SEJ notes it's unclear whether robots.txt settings limit what a business agrees to when it accepts the terms (SEJ), and University of Wisconsin law professor Benjamin Sobel told Nieman Lab that following robots.txt is voluntary and gives a site no enforceable legal right (SEJ). A December 2025 federal ruling reached a related conclusion from another direction: robots.txt is not a technological measure that controls access (our write-up).

So a robots.txt block is evidence of what you want and an instruction Meta's documented crawlers say they follow. Against a contract that says you authorized crawling, it's an argument rather than an answer. The European Publishers Council put it bluntly in its statement to the Commission: "Technical controls over crawler access cannot substitute for clear, enforceable contractual limits" (PPC Land). Meta's on-record answer is the opposite: the updates don't supersede robots.txt preferences. Until that sentence appears in the terms, it's a spokesperson's statement, not a clause.

Meta's account against the publishers'

Meta's position, in its own words across the week: the update consolidates terms rather than adding rights, "does not change how Meta uses web crawlers", applies "only" to ads and business products, and "is not about scraping news sites" (spokesperson Daniel Roberts to The Verge, via SEJ). SEJ also reports that Meta's Business Help Center described the update as moving the Ad Creative Generative AI Terms into the Self-Serve Ad Terms to "reduce overlap", without mentioning crawling or training (SEJ). On October 23 it told Reuters it needed more time to explain to partners what is and isn't changing, and moved the effective date to November 20 (PPC Land). Meta's preview page now shows "Effective Date: November 20, 2026" (Meta).

The publishers' position has three parts:

1. 01
   The terms are take-it-or-leave-it. The only way to refuse is to stop using Meta's business products; neither version offers a partial opt-out (PPC Land). Danielle Coffey of the News/Media Alliance called the terms "outrageous and probably illegal" and an abuse of market power (The Register). Santa Clara law professor Eric Goldman said users need a genuine option to reject amended terms (same source).
2. 02
   One deleted word isn't an exclusion. The European Publishers Council says "removing that one word does not establish an express exclusion of AI training, fine-tuning, grounding", that "the authorisation remains extensive", and that "little has changed in substance." It asked the Commission to look at the terms under the Digital Markets Act and competition law, and failing interim measures, to get Meta to postpone the provisions (PPC Land).
3. 03
   Collection can't be undone. The same statement warns about content once "collected and incorporated into datasets, models or competing services." In the UK, News Media Association chief executive Theo Bamber called the original a "shameless landgrab attempt from Meta to scrape anything and everything from news publisher and business websites" and asked ministers and regulators to intervene (InPublishing).

Our read: Meta's narrowing is real but small. The purpose clause got shorter; the permission to send automated software with test credentials to an undefined set of "associated" sites got no shorter at all. If Meta means what its spokesperson said about robots.txt, the cheapest fix is a sentence in the terms saying so. It hasn't added one.

Your list for the next four weeks

If your organization publishes anything and touches Meta's business tools, here's what to do before November 20:

1. 01
   List every Meta business asset you have. Pages, ad accounts, catalogs, business Groups, Business Suite logins and WhatsApp channels. Any of them can make you a business customer (The Register).
2. 02
   Read the preview yourself, in your locale. Both the Commercial Terms preview and the Self-Serve Ad Terms. The UK text lagged the US text by at least a day.
3. 03
   Set robots.txt on purpose. Decide on meta-externalagent, meta-externalads and meta-webindexer one by one; leave facebookexternalhit alone if you want link previews. Allow 24 hours.
4. 04
   Watch your logs for Meta's user agents on paywalled or logged-in paths. Meta declined to say whether test credentials would get past paywalls or paid content (SEJ).
5. 05
   Decide whether you accept. The only exit the terms offer is to stop business use. If you object, put it in writing and route it through your trade body; the EPC, the News/Media Alliance and the UK NMA are all pressing Meta and regulators now.
6. 06
   Check back before the 20th. The News/Media Alliance's deadline for Meta's answer is October 26, and the EPC wants a Commission response before November 20. The text may change again.

Sources

• Commercial Terms preview, effective November 20, 2026 — Meta
• Commercial Terms (current, effective March 6, 2024) — Meta
• Meta web crawlers — Meta for Developers
• Meta's new commercial terms will allow the scraping of news sites, legal experts say — Nieman Lab
• Meta grants itself right to scrape content from business customers for model training — The Register
• Meta's new terms expand crawling rights beyond advertisers — Search Engine Journal
• Meta removes language that raised scraping and AI training concerns — TechCrunch
• Meta removes training from crawl clause in commercial terms — Search Engine Journal
• Meta strikes "train" from its Commercial Terms crawling clause but keeps the right to crawl — Relevant Audience
• Meta's new terms let it crawl business websites to train AI from Oct 30 — PPC Land
• European publishers ask EU to halt Meta's crawling terms before Nov 20 — PPC Land
• NMA condemns Meta — InPublishing