Skip to content
Common Measure
Edge documentation

Recognise and verify CommonMeasureBot

CommonMeasureBot is the name every Common Measure edge presents when it fetches a page for an agent. It fetches what an agent asks for and does not crawl. The name is shared; each enrolled edge signs its requests with its own Ed25519 key under Web Bot Auth (HTTP Message Signatures, RFC 9421). Trust a request by its signature, never by its User-Agent. The formats are in docs/contracts/bot-identity.md.

1. Address it in robots.txt

Section titled “1. Address it in robots.txt”

The product token is CommonMeasureBot. The edge obeys User-agent, Allow and Disallow, in every policy mode, and honours Crawl-delay. Where no group names it, the * group applies.

How a path is matched against your rules:

  • Percent-encodings are compared in one spelling, on the URL and on your rule alike: an encoded unreserved character is decoded and hex case does not matter. Disallow: /private/ refuses /%70rivate/staff.
  • Your rule is read as you wrote it. Its slashes are never merged, its %2F is never read as /, its %3B is never read as ; and its ; is never stripped, so Disallow: // refuses only a page whose path, or a reading of it, starts with a doubled slash.
  • The page’s path is read as requested and as every path four operations reach from it, in any order and including only some of them: %2F or %5C read as / and %3B read as ;, each run of / read as one, . and .. segments resolved, and each segment’s ; path parameters removed. Servers differ in which they apply and in what order: nginx serves /x//..%2Fprivate/staff as /private/staff, Tomcat serves /private;x=1/staff and /x/..;/private/staff as /private/staff, nginx in front of Tomcat serves /private%3Bx/staff as /private/staff when proxy_pass names a URI, and others serve //private/staff or /private%2Fstaff as /private/staff. An encoded ; (%3B, either case) is read as ;, as a proxy that decodes the path before a Java server does, and then stripped. A Disallow that any reading reaches refuses the page. An Allow reached by one reading does not lift that Disallow: under Disallow: / and Allow: /news/, //news/1 is refused. Readings are capped at 64, and a crafted path can reach the cap; a page past it is refused without comparing them.
  • A literal * or $ in a URL is matched by %2A or %24 in your rule (RFC 9309 §2.2.3): Disallow: /file-%2A.html refuses /file-*.html. A final $ anchors the end of the path: Disallow: /*.pdf$ refuses /a.pdf/b.pdf.
  • Content-Usage path rules and the url of an RSL <content> entry are matched the same way. Where the readings of a page fall under different <content> entries, the entry that covers the page as requested governs; where none does, the entry the fewest operations reach. So an entry covers these spellings only where no other entry of the licence covers the URL as requested, and the requested spelling can select a less restrictive entry: beside a catch-all / entry, //premium/x is governed by / and not by a /premium/ entry, and under a permissive /news/free/ entry, /news/free/..%2Fpaid/x is governed by /news/free/ and not by /news/.

Some spellings are not matched. A query your rule writes with a raw reserved character (Disallow: /go?u=https://x) does not match the URL’s encoded ?u=https%3A%2F%2Fx. An empty query is kept, so /news? is not under Disallow: /news$. A change of letter case is not read: a server on a case-insensitive file system or IIS serves /NEWS/1 as /news/1, and Disallow: /news/ does not cover it.

User-agent: CommonMeasureBot
Disallow: /private/

Served from a loopback site, that group refused a fetch of /private/staff before the page was requested; the site logged only the robots.txt request. The agent was told:

refused before the crossing: http://127.0.0.1:8405/robots.txt disallows CommonMeasureBot at http://127.0.0.1:8405/private/staff: the `User-agent: CommonMeasureBot` group, which names CommonMeasureBot; rule `Disallow: /private/`, a literal path prefix. A `Disallow` binds in every policy mode: refused before the request.

A robots.txt that answers 429 or 5xx, or cannot be reached, is treated as RFC 9309 §2.3.1 says: the last copy the edge holds governs, and with none held the edge refuses the page as a complete disallow.

The same rules govern the requests the edge makes on its own account. It requests /.well-known/content-telemetry.json, and a licence that only a page’s Link header names, only where robots.txt at that URL’s origin allows it for CommonMeasureBot. It follows a redirect from either only where robots.txt at the target’s origin allows the target, and only in the target host’s Crawl-delay turn; where that turn does not fit the fetch, it does not follow the redirect. A licence that a License: line in your robots.txt names at the origin of the page being fetched is requested without that check, because your file could have refused it; a redirect from it is still checked. A licence your License: line names at any other origin, including the one your robots.txt redirects to (from http to https, say), is checked against robots.txt at that origin; where that file refuses it, the edge treats your page as having no readable licence and does not fetch it.

The edge asks for the manifest at the page’s own host. Where that answers 404, it asks once more, at the registrable domain (example.com for news.example.com), after reading that domain’s robots.txt, and asks no other host. That second request never goes to a public suffix such as co.uk or pages.dev: for a page on x.pages.dev or bbc.co.uk the manifest is asked only at the page’s own host. A public suffix host is asked for its manifest only when the page is on that host, or when a redirect you serve leads there, and then only where its own robots.txt allows it.

2. What a signed request carries

Section titled “2. What a signed request carries”
Header Value
User-Agent CommonMeasureBot/<version> (+<bot page>)
Signature-Agent the origin of the hub the edge enrolled with, as a quoted string
Signature-Input sig1=("@authority" "signature-agent");created=…;expires=…;keyid="<key id>";alg="ed25519";nonce="…";tag="web-bot-auth"
Signature sig1=:<base64>:

Where the request carries Content-Telemetry-ID, that header is covered too. The page fetch, each redirect hop and the robots.txt, licence and Content Telemetry manifest probes are signed; a signature lasts 300 seconds. For edges enrolled with Common Measure Hub, Signature-Agent is "https://hub.commonmeasure.ai", whose key directory, agent card and bot page are at:

  • https://hub.commonmeasure.ai/.well-known/http-message-signatures-directory
  • https://hub.commonmeasure.ai/.well-known/signature-agent-card.json
  • https://hub.commonmeasure.ai/bot

An edge that is not enrolled, or has learnt its key was revoked, sends User-Agent: CommonMeasureBot/<version> and no signature. Such a request cannot be told from anyone else using the name.

3. Verify a signature

Section titled “3. Verify a signature”
  1. Check that Signature-Agent names an origin you trust.
  2. Fetch the key directory from that origin and find the key whose kid is the keyid in Signature-Input. No key means unknown or revoked.
  3. Rebuild the signature base: one line per covered component, then the parameters. @authority is the host the request was sent to, lower case, with the port only where it is not the default.
  4. Verify the Ed25519 signature over the base with the key’s x.

This script does those steps for one request, using OpenSSL 3 for the signature check. Save the request’s headers as JSON (Host, which is :authority in HTTP/2, Signature-Agent, Signature-Input, Signature, and any other header the signature covers) and pass the origin you trust:

#!/usr/bin/env python3
"""Verify one CommonMeasureBot request.
Usage: verify-bot.py <headers.json> <trusted origin>
headers.json holds the request's Host, Signature-Agent, Signature-Input and
Signature headers, and any other header the signature covers. The trusted
origin is the key directory's origin you accept. Needs OpenSSL 3."""
import base64, json, re, subprocess, sys, tempfile, time, urllib.request


h = {k.lower(): v for k, v in json.load(open(sys.argv[1])).items()}
_, params = h["signature-input"].split("=", 1)
components = re.findall(r'"([^"]+)"', params[: params.index(")") + 1])
keyid = re.search(r'keyid="([^"]+)"', params).group(1)
created = int(re.search(r"created=(\d+)", params).group(1))
expires = int(re.search(r"expires=(\d+)", params).group(1))
assert 'tag="web-bot-auth"' in params and 'alg="ed25519"' in params, "not a Web Bot Auth signature"
assert created - 60 <= time.time() <= expires, "outside the signature's validity window"


origin = h["signature-agent"].strip('"')
assert origin == sys.argv[2], f"signed for {origin}, not the origin you trust"
url = origin + "/.well-known/http-message-signatures-directory"
keys = json.load(urllib.request.urlopen(url))["keys"]
key = next((k for k in keys if k["kid"] == keyid), None)
assert key, f"key {keyid} is not listed at {url}: unknown or revoked"


values = {"@authority": h["host"].lower()}
lines = [f'"{c}": {values[c] if c in values else h[c]}' for c in components]
lines.append(f'"@signature-params": {params}')
base = "\n".join(lines).encode()


sig = base64.b64decode(h["signature"].split("=", 1)[1].strip(":"))
x = base64.urlsafe_b64decode(key["x"] + "==")
der = bytes.fromhex("302a300506032b6570032100") + x  # Ed25519 SubjectPublicKeyInfo
with tempfile.TemporaryDirectory() as d:
    for name, data in (("pub.der", der), ("base", base), ("sig", sig)):
        open(f"{d}/{name}", "wb").write(data)
    ok = subprocess.run(["openssl", "pkeyutl", "-verify", "-pubin", "-keyform", "DER",
                         "-inkey", f"{d}/pub.der", "-rawin", "-in", f"{d}/base",
                         "-sigfile", f"{d}/sig"], capture_output=True).returncode == 0
print(("verified" if ok else "SIGNATURE DOES NOT VERIFY") + f": key {keyid} at {origin}")
sys.exit(0 if ok else 1)
Terminal window
python3 verify-bot.py headers.json https://hub.commonmeasure.ai

The script was run against requests a signing edge made to a loopback site, with the key directory served from a second loopback origin, not against a live edge enrolled with the hub. A request as sent verified; the same request with another Host did not; with the key removed from the directory the script reported it unlisted; and against the wrong trusted origin it refused before fetching anything:

verified: key 759-Mmz_OXqWF8SmrNfiTnjwLCn9bw38Si84ne8rBsU at http://localhost:8499
SIGNATURE DOES NOT VERIFY: key 759-Mmz_OXqWF8SmrNfiTnjwLCn9bw38Si84ne8rBsU at http://localhost:8499
AssertionError: key 759-Mmz_OXqWF8SmrNfiTnjwLCn9bw38Si84ne8rBsU is not listed at http://localhost:8499/.well-known/http-message-signatures-directory: unknown or revoked
AssertionError: signed for http://localhost:8499, not the origin you trust

The directory is cacheable for an hour (Cache-Control: public, max-age=3600); fetch it once per hour rather than per request. The directory response also carries each listed key’s own signature over the directory’s authority (binding0, binding1, …), which a verifier may check as well (contract §The directory proof).

4. What a key id is

Section titled “4. What a key id is”

A key id names one enrolled edge. It is the RFC 7638 thumbprint of the edge’s public key: SHA-256 over {"crv":"Ed25519","kty":"OKP","x":"<x>"} with no spaces, base64url without padding. You can recompute it from the directory:

Terminal window
python3 -c 'import base64,hashlib,sys; print(base64.urlsafe_b64encode(hashlib.sha256(("{\"crv\":\"Ed25519\",\"kty\":\"OKP\",\"x\":\"%s\"}" % sys.argv[1]).encode()).digest()).rstrip(b"=").decode())' 11qYAYKxCrfVS_7TyWQHOg7hcvPapiMlrwIaaPcHURo
kPrK_qmxVWaYVA9wwBF6Iuo3vVzz7TxHCTwXBygrS4k

That is the RFC 8037 example key; the key listed in Common Measure Hub’s directory on 23 September 2026 matched its kid the same way. The key id is also the agent_id of the usage reports that edge’s activity produces (Receive usage reports about your content).

5. Revocation

Section titled “5. Revocation”

A key leaves the directory when an organisation owner revokes it at the hub, when the edge runs commonmeasure disconnect, when the member it was enrolled for leaves the organisation, or when the organisation closes. A key also drops out when the edge’s directory proof lapses. A verifier that cached the directory can accept a revoked key for at most its cache age, one hour. The edge learns of an owner’s revocation at its next relay run and then stops signing; until then its requests fail verification once directory copies expire. The hub’s own 401 on a standing check or directory-proof upload also records revocation and stops signing, including after member removal or organisation closure; a 401 written by something in front of the hub, such as an ingress page, does not. The edge retains the hub’s refusal text. A session already open refuses each signature while the edge’s record says revoked or cannot be read, and its fetches fail rather than go unsigned (bot identity §Revocation).

6. Raise a complaint about a key

Section titled “6. Raise a complaint about a key”

Write to the contact in the hub’s agent card (contacts), which the bot page also gives. For Common Measure Hub it is mailto:hello@commonmeasure.ai (read from the card on 23 September 2026). Quote the key id and the requests concerned: time, URL, User-Agent and Signature-Input. To stop one edge at once, refuse requests signed with its key id at your server; a robots.txt group applies to every edge.