An atlas of agent assurance

When an AI agent does something that matters, several different systems will later make a statement about it. An identity system says who the agent was. A policy engine says the action was allowed. A sandbox says what the agent could reach. A trace says what the agent reported doing. A signed receipt says which bytes were issued. An evaluation says how the agent’s model behaved on tests. A regulation says what should have been true.

Each of those statements is useful, and each answers a different question for a different party. This atlas lays them side by side. Its data file counts 7 families, 45 named projects and standards, and 18 mechanisms, with the public artifacts that address the one cell no family fills.

Every project or standard named in the family sections below is a standard, law or program of a recognized public, standards or industry body; an independent evaluation organization; or an organization-owned project measured at 5,000 GitHub stars or more. Every figure carries its read date. Each list is a representative sample of its family. Smaller projects that do the same work are counted by the mechanisms they implement rather than named.

The map in one table

The families map onto the three jobs that Three Jobs, Not One separates. Families 1 and 2 carry the policy job: the decision on what may be attempted. Family 3 does the containment job: enforcing a boundary the agent’s code does not control. Families 4 and 5 are where systems usually attempt the evidence job. The table shows why neither finishes it: the party a trace describes writes the trace, and a signature fixes bytes without saying what they describe happened. Evaluation and governance are complementary roles. The evaluators and regulators who work in them scrutinize the other families. Neither role can stand in for any of them.

Who writes the record

One question separates these mechanisms: who writes the record each one produces. The table uses the witness-scope terms of the agent evidence vocabulary: self is the observed party, or the system that runs it; peer is a party alongside it with its own stake, such as a gateway operator or an evaluator paid by the developer it evaluates; external is a party outside the observed system’s control. The writer given for each mechanism is this atlas’s reading of a typical deployment, not a property of the software.

The “writer” column shows one pattern. Every mechanism that records the agent’s actions in deployment comes from the agent’s own side or from a peer with a stake in the outcome. Every mechanism written from the outside records something other than those actions: an identity, a launch state, a log inclusion, a timestamp, a test result, or an assessment of what the developer let the evaluator see.

1. Identity and delegated authority

Question

Who or what is acting, for whom, under which delegated authority?

Establishes

That a request carried a credential bound to an identity a trusted issuer vouched for, and, through delegation or token exchange, the principal on whose behalf it was issued and the scope it was granted.

Cannot establish

That the holder is the agent it claims to be once a key or token leaves its workload; what the agent did with the authority; or whether a particular action matched what the principal intended. An identity makes an actor accountable only when some other record says what the actor did.

Seam to family 2 (policy, mediation and admission)

The policy engine decides on the identity it is shown. That identity usually names a workload, not a task. When a user delegates to an agent, which calls a tool, which calls a sub-agent, the token at the enforcement point often carries the agent’s own identity and a broad scope. The human principal, the narrowing at each hop and the purpose of the call are lost or merely asserted. The resulting decision allows “this workload may call this tool”. The two families leave open whether the call was within what the user delegated for this task. No record answers that question for the user who delegated.

2. Policy, mediation and admission

Question

Was the action permitted, mediated or refused under a stated policy, and may a relying party accept its evidence?

Establishes

That a request which reached an enforcement point was evaluated against a stated policy, and allowed or denied.

Cannot establish

Anything about requests that never passed through that point, anything about what happened after an allow, or that the policy was right. The decider writes the decision log, so the log reports the decision and nothing about the effect.

Seam to family 3 (containment)

The policy engine evaluates the action as declared at the gateway. Containment governs what the process can actually reach. Whatever the agent does without crossing the gateway, such as code it runs inside an allowed tool, a direct socket, or a file a later step reads, is invisible to the policy engine. The decision and the effect happen in different places at different times, and nothing binds one to the other, so a decision log cannot tell anyone what an allowed action did.

3. Containment

Question

Which resources and channels could the workload reach?

Establishes

That execution was restricted: the system calls, files, network destinations and memory it could touch were bounded, which bounds the damage a misbehaving agent can do.

Cannot establish

What happened inside the allowed boundary. A sandbox limits what a process can reach and keeps no record of what it did. It cannot show by itself that it was in place for a particular run; that needs attestation by a party outside it. An escape it did not detect looks the same as no escape.

Seam to family 4 (observation and tracing)

The contained process is also the process that reports on itself. Traces come from instrumentation inside the sandboxed workload, so a denied connection, a killed process, or a tool that did something other than what it logged produces no span. The trace still looks complete. Containment and observation sit at different vantage points, and the operator running both has to reconcile two partial views by hand.

4. Observation and tracing

Question

What was seen while the workload ran, and from which vantage?

Establishes

What the instrumentation emitted: correlated spans, timings and attributes, as reported.

Cannot establish

That the events happened independently of the party emitting them; that nothing is missing, since tracing has no concept of a missing span and sampling drops spans by design; or that the record is intact, since spans are unsigned and can change anywhere in the collection pipeline.

Seam to family 5 (records, receipts and transparency)

Signing a trace fixes its bytes and changes nothing about whether the trace is true or complete. Whatever the emitter omitted before signing remains omitted. The signed version now carries more authority than the unsigned one did.

5. Records, receipts and transparency

Question

How are observations encoded, signed, committed, ordered, retained and disclosed?

Establishes

Integrity and origin of bytes: the holder of a key signed these bytes and no others. A transparency log adds that an entry was included at a point in the log’s history and that the history was not rewritten, which prevents equivocation for anyone who checks.

Cannot establish

That what the bytes describe occurred; that the set of records is complete; that their content is true; or that the signer observed anything at all.

Seam to family 6 (conformance and evaluation)

A receipt proves which bytes were issued. An evaluation measures how a system behaved. Deployments rarely bind the two: an evaluation result seldom names, by digest, the weights, scaffold, tools, configuration and grader it ran against. A production receipt seldom points back to the evaluation that licensed the deployment. Where no digest binds them, the deployer’s statement that “this deployed agent is the one that was evaluated” is all a regulator or a customer receives.

6. Conformance and evaluation

Question

What did testing establish about behavior or reader agreement?

Establishes

Measured behavior of a specific system version, under a specific harness and task set, at a specific time.

Cannot establish

Behavior in deployment; that the evaluated system is the deployed one, unless the two are bound by digest; the absence of capabilities it did not test; or the soundness of its own result, unless someone else can recompute it from preserved inputs.

Seam to family 7 (governance, requirements and vocabulary)

Governance texts name outcomes such as record-keeping, testing, logging and third-party assessment, without fixing an evidence format, a required vantage, or a way for a regulator to re-check a result. The European harmonised standards that would give a presumption of conformity are, on CEN-CENELEC’s own page, still under development. Operating conditions for evaluator independence have been written down, AEF-1 among them; no comparable text says how a third party re-checks the evidence an evaluator produces, so the evaluator’s client relies on the evaluator’s report of that evidence.

7. Governance, requirements and vocabulary

Question

What is required, reported or asserted, and in which stable terms?

Establishes

Obligations, risk categories, process requirements and routes to conformity.

Cannot establish

That any obligation was met. Several instruments are voluntary, and several rest on standards not yet published. None specifies evidence that a distrusting party could re-check.

Seam back to family 1 (identity and delegated authority)

Accountability obligations end in a person or an organization. The record that would show which principal authorized which agent action is not something any of these instruments requires in a checkable form. The chain from “this obligation applies to the deployer” back to “this delegation authorized this call” is closed by documentation alone, which leaves the regulator reading the deployer’s own account.

The open cell

Across the “cannot establish” column, one question is never answered: did this action happen as recorded, seen from a vantage the agent could not reach, and is anything missing? Each family supplies part of an answer: identity names who acted, policy records the decision, containment bounds the reach, tracing carries the actor’s own account, receipts fix the bytes, and evaluation measures a tested version, while governance requires that an answer exist. No family includes a record of the action made from beyond the agent’s reach.

Signing more records does not close that cell; the seam between tracing and receipts shows why, since a signature fixes a self-report’s bytes without changing who wrote it. Where the cell can be closed at all, closing it takes records made from a vantage the observed party does not control, bound to what they observed, with an explicit statement of what they do not cover and a rule for what a missing record means. The rest of this site works in that cell. The version ledger lists each piece with its status.

Where the public evidence artifacts sit

My vocabulary, conformance corpus and admission rules address that cell. Two Rust crates supply the byte handling of family 5 that they rest on. The last column follows each README’s statement of what the repository does not do.

The conformance corpora

The historical baseline at release v0.13.0 of agent-evidence-vectors carries 817 vectors in 13 corpora. Each corpus tests one specification or one contract; several test specifications written by others, so that anyone can check independent implementations of those specifications against the same cases.

Standards work in review, and public listings

These are my contributions to other bodies’ work, listed where the repository meets the naming rule above: a pull request to each of two projects and an Internet-Draft. No body had merged or adopted any of them when I read their states. Each carries the state I read.

Checking this page

• A “cannot establish” line is the limit of its family. The projects listed under a family are measured against that line alone.
• Star counts measure attention, and they drift. I read each status from the body’s own page or repository. The re-derive tool in the site’s repository re-reads each count, and the claim ledger gives the time of each read.
• The atlas data is published as data/atlas.toml in the same repository, one node per project, mechanism and artifact with its link and measurement, so any source that emits the same shape can regenerate the page.