Web Bot Auth, Explained: How Signed Requests Let a Site Verify Which AI Agent Is Calling

On 25 October 2026 we sent a plain GET to https://chatgpt.com/.well-known/http-message-signatures-directory. The answer was 243 bytes of JSON: one Ed25519 public key, a kid, a validity window, and a field reading "signature_agent":"https://chatgpt.com". The response headers carried a Content-Digest and its own Signature-Input with tag="http-message-signatures-directory", valid for exactly 24 hours. That small file is the public half of an AI agent's identity. Any site that receives a request claiming to come from it can fetch the file, check the request's signature, and know, with cryptographic certainty rather than a guess, that whoever holds the matching private key signed it.

That's the whole idea of Web Bot Auth, and it's a big change from how sites have identified bots for thirty years. The mechanism is also narrower than most write-ups suggest. Below, you'll build a signature base line by line, break a signed request in a simulator, and see where the line falls between what's specified, what's proposed and what one vendor has shipped. Our web crawler system design covers where a signing service sits in a crawler; this post is about the protocol itself.

Why a Name and an Address Prove Nothing

Today a well-behaved bot identifies itself two ways: a User-Agent string such as GPTBot/1.1, and a list of IP ranges it publishes so sites can allowlist it. OpenAI's crawler documentation still works this way, pointing site owners at JSON files of IP ranges for OAI-SearchBot, GPTBot and ChatGPT-User.

The draft's authors list why that's not enough (draft §2): a user agent "can be spoofed meaning anyone may attempt to act as that agent", and IPs on cloud platforms "have layers of ownership", so a published range binds weakly to the company actually renting the machines. Cloudflare's original Web Bot Auth post put it more bluntly: user agent headers alone "are easily spoofed and are therefore insufficient for reliable identification". At the IETF session where this work started, the BBC said 23% of requests to bbc.com are bots and that "few bots publish source network info, and often running on same cloud infra" (IETF 123 BoF minutes).

A third option, a shared API key per site, fails for a different reason: the draft notes it's "impractical to scale for any agent beyond select partnerships". A search crawler can't sign up with every website on the internet.

Keys on a Shelf, Signatures on Every Request

The design splits identity into two pieces. A key directory is a public file the operator hosts on their own origin. A signature travels on every request and points back at that directory. Step through it:

The subtle part is step 2. In the draft, the identifier is the URL, not the key (§4.1). A verifier ends up with a pair: the URL it fetched, and a key that URL provides. That's why rotation works without breaking anything. The operator adds the new key next to the old one, waits, then drops the old one, and every site that knew the URL keeps recognising it (§5.5.2). The kid of each key must equal its RFC 7638 thumbprint, so a verifier can check the directory's labels instead of trusting them.

What the draft requires on every Web Bot Auth signature (§5.2):

• Cover at least one of @authority or @target-uri, so the signature is bound to the site it was sent to.
• Cover the Signature-Agent member for this signature's label, so nobody can swap the directory pointer.
• Include created, expires, keyid and tag="web-bot-auth". Expiry is RECOMMENDED to be no more than 24 hours.
• Optionally narrow it with @method, @path, @target-uri or @query-param. The draft says agents SHOULD, and spells out why: a signature over @authority alone "verifies against any method, path, or body sent to that authority until it expires".

The Signature Base, Line by Line

RFC 9421 doesn't sign the raw request. Both sides build the same plain-ASCII string, the signature base, from the covered components in order, and the signature is computed over that string (RFC 9421 §2.5). Each line is a quoted component name, a colon, a space and the normalised value. The last line is always "@signature-params", which repeats the covered list and the parameters, so the metadata is signed too.

For a GET to shop.example/products/blue-kettle that covers method, authority, path and the agent pointer, the base is five lines:

Four details carry most of the protocol's behaviour:

1. 01
   Values are normalised first. @authority is the host in lowercase with the default port dropped, and @path excludes the query string (RFC 9421 §2.2.3 and §2.2.6). SHOP.example:443 and shop.example produce the same line.
2. 02
   Signature-Agent is a dictionary. In the working-group draft the header looks like Signature-Agent: sig1="https://agent.example", and the covered component names the member with ;key="sig1". The member's value appears in the base with its quotes. For the default directory type the value must be a bare origin; the verifier appends the well-known path itself (§5.5).
3. 03
   The label on Signature is not signed, but the agent key is. An intermediary may relabel sig1 to something else; it may not change the Signature-Agent member key a signature covers (§6.6.1).
4. 04
   The body isn't covered unless the agent sends and signs a Content-Digest. The draft deliberately doesn't require it, since "most automated traffic is GET" (§5.2).

Break It Yourself

Here's the same request with a verifier attached. Change the path or the host as a proxy might, point Signature-Agent at a different directory, move the verifier's clock past expires, or deliver the request twice. Then untick "covers @method and @path" and edit the path again.

Three things to take from the simulator:

• Untick the narrow coverage and the path becomes free. With only @authority and the agent pointer covered, which is the draft's minimum, /admin verifies as happily as /products/blue-kettle. That's conformant, and it's why the draft says a signature "has bearer-token semantics" until it expires (§6.9).
• Pointing Signature-Agent elsewhere fails at the key step, not the crypto step. Verifiers must look keys up by the (URL, key) pair (§5.4). If they indexed by keyid alone, a request could borrow one operator's key and claim another operator's URL.
• Replays are only caught if someone keeps a list. The draft defers replay protection to RFC 9421 §7.2.2 and calls nonce enforcement "deployment policy" (Appendix C.6). Cloudflare's Web Bot Auth docs say it does no nonce validation and recommends a short expires, "a minute is often sufficient". Untick "verifier records nonces" to see that world.

What the Site Does on Its Side

Verification follows the order RFC 9421 §3.2 sets out: parse the headers, check the parameters (tag, expiry), find the key, rebuild the base, verify. The draft adds its own rules at each step (§5.4), and the key fetch happens lazily, after the first request names a directory the verifier hasn't seen.

Here's the same exchange on the wire, including the directory fetch. Header names are exactly as the draft and RFC 9421 define them; everything in angle brackets is a placeholder, not a real value.

Two status codes come from the draft itself. A site that wants a signature from a client that didn't send one can ask with Accept-Signature, and the draft says the status SHOULD be 403 (§5.3). One that sees a signature or nonce used more often than it allows MAY answer 429 and ask for a fresh one.

What a Valid Signature Does Not Tell You

This is where most coverage gets ahead of the text. The draft is explicit about what a verified signature means (§4.1): a holder of a key that URL publishes signed the covered message. It "says nothing about who operates the Agent, whether the Agent is benign, or whether the request is authorized. Those are origin policy."

So a verified request gives a site a stable handle it can log, rate limit, allowlist or block, the way it handles an IP or a user agent today, except this one can't be forged. The draft's own use-case table (Appendix A) maps nearly everything to that handle: rate limiting per URL, access control, different content, auditing logs. Commerce sits on top of the same handle. Cloudflare's pay per crawl tells registered crawlers to send signature-agent, signature-input and signature on every request so it can quote prices with a 402 and a crawler-price header.

What it doesn't give you:

• Intent. Whether the request is for search, training or a user's question isn't part of the signature. The working group's charter lists a vocabulary of bot "intents" as out of scope.
• The human. Authenticating the end user is out of scope in both the charter and the draft (§4.6). A signed ChatGPT agent request tells you it came from OpenAI's agent infrastructure, not which user asked for it.
• Anything from an unsigned request. "A verifier that sees none has learned nothing about the sender: not that it is automated, not that it is human, not that it is evading anything" (§6.11).
• Fast revocation. The draft defines none. A compromised key keeps verifying until each verifier's cached directory expires (§6.3).

Standard, Draft, or One Vendor's Build

Four different things get called "Web Bot Auth", and they're at different stages:

The third row matters if you're implementing this. The working-group draft says signers "MUST send the dictionary form" of Signature-Agent (§5.2.1) and verifiers MAY accept the old string form. Cloudflare's docs list, among the reasons verification fails, a message that "uses the dictionary form from later drafts, such as sig2="https://signature-agent.test"", and they list the key component parameter as unsupported. So today a signer that follows the WG draft to the letter fails at Cloudflare, and one that satisfies Cloudflare sends the form the WG draft has moved away from. Expect that gap to close, but check both before you ship.

How it got here:

The charter's own milestones had the standards-track specifications going to the IESG by 30 April 2026. The protocol draft was only adopted in September, so the schedule has already slipped.

The Arguments the Working Group Hasn't Settled

The open questions are on the record, mostly in the founding session's minutes and the draft's own considerations sections. We'll stick to those.

Centralisation. Cloudflare's own background talk at the BoF set the goal: "Cryptographic trust should not add a new centralized choke point", for authorization or business decisions (minutes). The protocol's answer is that any operator can stand up a directory on its own domain with no registration. Practice is lagging the principle: Cloudflare's verifier still asks bots to submit a form before their signatures earn verified status. The charter puts "tracking or assigning reputation to particular bots" out of scope, and the draft declines to define how trust is accrued or shared between sites (§4.6). That leaves room for the big verifiers' lists to become the de facto registry.

Who can afford to check. In the same session Brian Campbell noted that tools to tell users from bots are "only available to large sites" and asked the group to "be honest about the resulting incentive". Vercel said smaller companies may lack the resources to maintain bot verification. The charter commits a best-practice document to "address impacts on the openness of the web".

Discrimination against unsigned traffic. The draft is careful here: absence of a signature "is not evidence about the party that did not send it", and the document "neither requires an origin to treat unsigned requests differently nor gives it grounds to" (§6.11). Cloudflare's verified bots post, meanwhile, says that as signed verification rolls out "the number of unauthenticated web crawlers should diminish". Both can be true; the protocol won't force a site's hand, and the market might.

Privacy and humans. Signing is a choice to be identified: "If an agent wishes not to identify itself, this is not the right choice of protocol for it" (§7.1). The draft warns against keys tied to a specific person, because that "exposes personally identifiable information and makes the key usable for user tracking" (§7.2). Delegation, meaning a signature that says an agent acts for a user, is out of scope and "expected to be specified separately" (Appendix D.1).

Scrapers, Remote Browsers and the User's Own Browser

If you run a server-side crawler or agent, the trade is simple to state. Signing makes you recognisable everywhere, under one name you control, through any IP or cloud. Sites can then allow, meter or charge you by that name, and they can also block it with one rule. Spoofing someone else's user agent gets less useful at every site that verifies, because the impostor can't produce the signature. Practically: one key per agent and purpose (§6.5), a signing service that keeps keys off fetcher disks, expires measured in seconds or minutes rather than the 24-hour ceiling, @method and @path covered, and a directory that stays reachable without bot protection in front of it (Appendix C.2).

Remote browsers sit in between. Cloudflare counts Browserbase and Anchor Browser as signed agents, and the draft's Appendix D.2 shows a remote browser adding its own signature over an agent's. That records that the browser forwarded the agent's request. It doesn't record that the agent authorised it.

In-browser extraction is a different case. CrawlPilot runs inside the user's own Chrome, on their session and their connection, and the page loads because the person opened it. The working group drew its boundary right there: the charter leaves end-user authentication out of scope, and at the BoF the chair said an "agent on laptop is explicitly out of initial charter", while Vercel described browser-based agents as "similar to today" (minutes). A per-user signing key would be exactly the human-correlated identifier the draft's privacy section warns against. So Web Bot Auth doesn't ask anything new of an extension like ours, and it doesn't excuse anything either: the site's existing bot detection, its terms and robots.txt apply as before. Our anti-bot detection deep dive covers that side.

Keep This

Sources

• Backman, Richer and Sporny, RFC 9421: HTTP Message Signatures, RFC Editor, February 2024
• Meunier and Major, draft-ietf-webbotauth-httpsig-protocol-00: HTTP Message Signatures for automated traffic, IETF, 1 September 2026, and its datatracker history
• draft-meunier-web-bot-auth-architecture, the individual draft (first posted 15 April 2025)
• IETF, Web Bot Auth working group charter and milestones
• IETF 123, Web Bot Auth BoF minutes, Madrid, 21 July 2025
• Cloudflare Docs, Web Bot Auth (updated 8 October 2026)
• Cloudflare Blog, Forget IPs: using cryptography to verify bot and agent traffic, 15 May 2025
• Cloudflare Blog, Message Signatures are now part of our Verified Bots Program, 1 July 2025
• Cloudflare Blog, Introducing pay per crawl, 1 July 2025
• Cloudflare Blog, The age of agents: cryptographically recognizing agent traffic, 28 August 2025
• OpenAI, Overview of OpenAI crawlers
• OpenAI's key directory, https://chatgpt.com/.well-known/http-message-signatures-directory, fetched by us with curl on 25 October 2026 (headers and body as described above)