The bot identity

To the sites your agents read, an enrolled edge is CommonMeasureBot: one product name for every organisation using Common Measure, and one key per enrolled edge. Publishers are told to trust a signature, never the name.

What a publisher sees

An enrolled edge signs every request it makes to a source — the page its agent asked for, and the robots.txt, licence and manifest it reads to rule on that page — with its own Ed25519 key, as an RFC 9421 HTTP message signature under the Web Bot Auth profile. The request carries Signature, Signature-Input and Signature-Agent; the last names the origin serving the key directory below, which a verifier reaches by appending /.well-known/http-message-signatures-directory to it, and the signature’s keyid is the key id assigned at enrolment and shown under Edge keys on the API keys page. A publisher resolves the key id in the directory and verifies the signature. An edge that has not enrolled sends the product token with no signature and is not this identity.

A key id names one edge. It never names your organisation or a person, and the hub does not disclose which organisation a key id belongs to without that organisation’s agreement. The key id is the agent identifier on the telemetry wire, so an owner can match the requests it served to the report it receives.

The identity documents

The hub serves the public documents a verifier reads, from the enrolled keys that are not revoked:

• /.well-known/http-message-signatures-directory — the Web Bot Auth key directory. Each entry’s kid is a key id. The response carries one member of Signature-Input and Signature for each listed key, made by that key’s edge over the directory’s authority and tagged http-message-signatures-directory.
• /.well-known/signature-agent-card.json — the signature agent card: the product token, how the bot identifies itself, where its keys are, the contact for complaints and, depending on the hub’s release floor, the source’s terms (below).
• /.well-known/content-telemetry.json — the Content Telemetry manifest, carrying the enrolled keys and the hub’s receiver endpoint.
• /bot — the page for publishers: what the bot is, how to address it in robots.txt, where its keys are, and how to raise a complaint about a key id.

The documents name absolute URLs on the origin the hub is configured to serve them on. Revoking an edge key removes it from the directory and the manifest at once. The directory declares a cache age of one hour; a publisher that cached it keeps the revoked key for at most that long.

The source’s terms on the card

Under web_bot_auth, the card declares rfc9309-compliance: ["User-Agent", "Allow", "Disallow", "Crawl-delay"].

The hub’s release floor is the lowest edge release whose keys its directory lists. When that floor is at or above the edge release that keeps the source’s terms in every policy mode, the card also declares source-terms. Below it, the card omits source-terms, because the directory then still lists keys of edges that do not yet keep all of those terms in every policy mode. When declared, the object is:

"source-terms": {
  "binding": "every-policy-mode",
  "content-signals": ["Content-Signal", "Content-Usage"],
  "licence": "rsl-1.0",
  "reporting": ["content-telemetry"]
}

• binding: the operator’s policy mode sets none of these terms aside.
• content-signals: the statements the edge reads for AI input. One that disallows ai-input refuses the fetch.
• licence: the edge reads an RSL 1.0 licence named in robots.txt or in a Link header before it uses the page. A licence that cannot be read refuses the fetch.
• reporting: the reporting demand types the edge can meet. A demand of one of these types is met through the enrolled hub or the licence’s endpoint, or the fetch is refused. A demand of any other type refuses the fetch.

The card does not yet declare payment. From the edge release that keeps the source’s terms in every policy mode, the edge refuses, in every mode, a licence’s payment and licence-server terms it cannot meet, with two exceptions it fetches without payment: the payment and licence-server terms of a licence with no usage <permits>, and a payment whose type it does not recognise.

Addressing the bot in robots.txt

Use the product token CommonMeasureBot. The edge reads User-Agent, Allow, Disallow and Crawl-delay from groups naming that token. If no group names it, the * groups apply. Records from every matching group are merged. The longest matching path decides, Allow wins a tie, and the longest Crawl-delay applies.

No operator setting lets an edge fetch a page these rules disallow, or follow a redirect to one. The same rules govern the requests the edge makes on its own account. It requests /.well-known/content-telemetry.json, and a licence that only a page’s Link header names, only where robots.txt at that URL’s origin allows it for CommonMeasureBot. It follows a redirect from either only where robots.txt at the target’s origin allows the target, and only in the target host’s Crawl-delay turn; where that turn does not fit the fetch, it does not follow the redirect. A licence that a License: line in your robots.txt names at the origin of the page being fetched is requested without that check, because your file could have refused it; a redirect from it is still checked. A licence your License: line names at any other origin, including the one your robots.txt redirects to (from http to https, say), is checked against robots.txt at that origin; where that file refuses it, the edge treats your page as having no readable licence and does not fetch it.

The edge asks for the manifest at the page’s own host. Where that answers 404, it asks once more, at the registrable domain (example.com for news.example.com), after reading that domain’s robots.txt, and asks no other host. That second request never goes to a public suffix such as co.uk or pages.dev: for a page on x.pages.dev or bbc.co.uk the manifest is asked only at the page’s own host. A public suffix host is asked for its manifest only when the page is on that host, or when a redirect you serve leads there, and then only where its own robots.txt allows it.

Each edge keeps Crawl-delay between its own requests to a host; separate edges are paced separately, and each enrolled edge signs with its own key id. A delay over 60 seconds is kept at 60 seconds. robots.txt is exempt from Crawl-delay, but observes back-off.

Each edge caches a robots.txt answer for up to 24 hours, or less where Cache-Control: max-age says so. Once it expires, the next request that needs the rules asks for the file again. After a failure it asks again no sooner than five minutes later. It can ask sooner where the failure was its own: the agent’s call ran out of time before the 5-second limit, or the file redirected to another host the edge is backing off from. Only the first 512 KiB is parsed; a larger file is cut at the last complete line within that limit. A body over 32 MiB, including after decompression, is unreachable.

A 4xx other than 429, including 401, 403 and 404, means there are no rules: every path is allowed. A 429, a 5xx or any status outside 2xx and 4xx, a redirect the edge does not follow, too many redirects, an unreadable response body or a network failure makes robots.txt unreachable. An unreachable robots.txt disallows everything, unless the edge holds an earlier answer from the host. The last answer then applies, however old: the last file, or a 4xx that allowed everything. The edge waits up to 5 seconds for robots.txt; no complete response by then counts as a network failure.

Back-off after a failure

A 429 or 503 with a usable Retry-After in seconds or HTTP-date is honoured, capped at one hour. Other 5xx responses, and a 429 or 503 without a usable header, start a 10-second wait. Each further failure after a wait has ended doubles the next wait, up to 15 minutes. A failure during a wait does not double it, but can extend it. Any other response, such as a 2xx, a redirect or a 404, resets the count but does not end a wait already under way.

The edge does not retry a failed request. Back-off is kept per host and per edge, like Crawl-delay; where both apply, a request waits for whichever ends later. Pages, redirect hops, robots.txt, licence and manifest probes all observe back-off. If the wait does not fit the next request’s remaining time budget, that request is refused.

Why a key is listed, and for how long

A verifier such as Cloudflare uses a key from the directory only when the directory response carries a signature made by that key. Your edge’s private key never leaves your machine, so the edge makes that signature itself: a short statement, signed with its key, that it agrees to be listed at the hub’s origin for a stated period. It covers the origin and nothing else, so the edge can sign it in advance and the hub can serve it beside whatever else the directory lists. The hub checks each signature against the key it enrolled and adds no signature of its own.

The edge signs at enrolment and again whenever it reaches the hub with a signature more than a day old: when it connects, delivers evidence, or starts. Each signature lasts seven days. The hub lists a key while its signature has at least two hours left, so a copy of the directory cached for its full hour still verifies. An edge that stops running leaves the directory up to seven days, less two hours, after it last reached the hub, and returns once it runs and reaches the hub again; verifiers pick it up when they next fetch the directory.

The Content Telemetry manifest does not depend on these signatures. It keeps every enrolled key that is not revoked, so a report can still be matched to a key that has left the directory.

The hub also lists a key only while its edge runs a supported release, currently 0.4.1 or later: each edge reports its release when it renews its signature, and an edge below the minimum, or one that reports none, leaves the directory. Key directory under Edge keys on the API keys page gives the reason for each machine that is not listed.

Revocation does not wait for a signature to run out: a revoked key leaves the directory on the next response. If the hub’s origin changes, every key leaves the directory until its edge signs for the new origin.

What is not published

Your organisation’s name, the mapping from key id to organisation, the ingest keys, and anything the edge delivers. The documents carry public keys and key ids only.