Experiments: evidence, effects and bytes

The atlas’s open cell asks whether a record of an agent action comes from a vantage the agent cannot control, whether its evidence binds to the actual effect, and what was missed. A signature and an offline verifier answer questions about retained bytes. They do not answer who could alter or omit the underlying observations. These experiments keep those questions separate.

Historical report meaning and diagnostic rule controls

The worked report example retains a historical report replay and two separate diagnostic controls. Inspect its Lab record or check the retained accounting. These are existing author-operated measurements; no new numbered study or live customer workload is introduced.

E1. PEER pilot: the snapshot gap

The local agent-evidence-observer prototype mediates one file replacement, commits to the before-state, signs a bounded result, and checks a hash-chain head under a second key. It does not isolate an agent. Its own record says witnessScope: PEER and unmediatedEffects: not-established. The two keys are controlled by one local operator.

I ran the exact pilot script against the package source identified by SHA-256 273e4985cb4de671145bcd55ef85545be213e8520f766ab7692e9d05edae60d1. The recorded machine result contains three scripted intervals. At that read, the package source was a local unpublished prototype. The observer now has public source, used at a different pin in E6; that does not supply or identify the exact older source bytes for this pilot. Its original result remains a local measurement without an independent reproduction.

The population for this pilot is three scripted intervals: one brokered write plus its retry, one durable direct write, and one transient direct write. The broker’s declared coverage population is calls with valid request IDs. The direct effects are outside that denominator. There is no sample of live agents, tool calls, reads, network effects, file modes, concurrent races, hostile host operators, or independent witnesses. The only tested observation point is the sequential file-tree snapshot at a broker call or seal. The test deliberately controls both sides; its contradiction is a counterexample to a claim of complete coverage, not a measured detection rate in deployment.

To re-run after obtaining the matching observer source, create its Python 3.12+ environment with the dependencies in its pyproject.toml, then use:

/path/to/observer/.venv/bin/python experiments/observer-vantage/run.py \
  --observer-root /path/to/agent-evidence-observer > /tmp/observer-pilot.json
diff -u experiments/observer-vantage/recorded.json /tmp/observer-pilot.json

The script hashes the package metadata and every src/probity_observer/*.py file before importing the package. A different digest is a different implementation, even if the three outcomes match. It prints no ephemeral signing keys or timestamps. The claim ledger records this run and its limits as E-01 through E-03.

E2. Below-agent observation: registered design, no result

Hypothesis. Under a pinned launch configuration, a host broker unavailable to the agent records every accepted or denied file replacement through one declared channel. A direct attempt to alter the watched tree, observer process, key, or journal fails. If any channel loses its record, the result becomes incomplete instead of an absence claim.

Unit and denominator. One launch interval is the unit. Before each interval, pin the agent image and policy digests, authority, allowed path scope, observer build/configuration, initial tree root, nonce, and observer key. Report counts for attempted, accepted, denied, replayed, dropped, and unresolved requests. Record the paths and privileges reachable by the agent as part of the denominator. Do not convert “no event seen” into “no effect” when reachability or liveness is unknown.

Interventions and controls. Run the same workload under (a) the current PEER prototype as a negative control and (b) a host-side isolation boundary whose configuration is published. Exercise a normal write, a false self-report, direct workspace write, key read, process signal, alternate channel, path traversal or symlink, lost response retry, broker death, and a missing event. Retain failed and null runs. The agent must not choose the observer’s key, journal, commitment, or before-root. A separate operator reproduces one live interval and pins the key independently.

Primary outcomes. For each attempted effect, report its raw trace, broker decision, resulting durable bytes, record, refusal or gap, and verification result. The threshold to claim a bounded below-agent file-write observation is a demonstrated inability of the tested agent to address the observer and watched tree except through the broker, no missing record for the tested channel, and a signed prior commitment bound to that interval. A single escaped effect or missing record blocks that claim. This does not establish all filesystem effects, all agent actions, or host-operator honesty.

This registered design has no separately reproduced isolated-agent result or measured overhead here. Upstream now runs an author-operated Linux boundary probe, whose CI provenance is recorded in E6. That artifact does not satisfy this design’s separate-operator requirement. The observer acceptance criteria remain a target; no conformant Observed Effect statement is established by the retained pilot or E6.

E3. Separately witnessed history: registered design, no result

Hypothesis. A witness run by a separate principal, with a pinned key and retained highest head, detects deletion, reorder, replay, divergent heads, and an absent committed interval when shown inclusion and consistency evidence. A second key held by the same operator cannot establish that property.

Unit and denominator. One committed interval and its checkpoint pair, plus each attempted history mutation. Publish the witness operator boundary, retention policy, head pinning method, hash/signature profile, and every accepted or refused fork attempt. Test honest append, truncation, two conflicting suffixes, withheld interval, witness unavailability, and key substitution. An offline consumer with an independently pinned head must reproduce refusals. Report mutation count, detected count, undecidable count, and time/space overhead; never recode an unavailable witness as success.

There are zero separately operated witness runs. The local PEER pilot’s second key is a protocol test only. The witness experiment will remain unclaimed until a separately operated history and a second reproducer exist.

E4. APS/PriorSeal E2: decision evidence and exact-call fixtures

The immutable run record checks APS decision evidence against PriorSeal’s exact-call examples. Its checker imports no APS or PriorSeal implementation. It uses draft-pidlisnyi-aps-03, the fixture documentation, pinned public test keys, and a fixed reference time of 2026-09-19T10:05:00.000Z. Implementation independence does not establish independent custody of an action observation.

The fetch script pins 31 files from APS 948f99b85343bef2c6fa677c8543965caacfc087 and PriorSeal d749d2691c3e6be139de4020e7b27cdafca2c428. RESULTS.json contains 96 rows, one per claim per input: 86 pass, four fail, and six not-exercised. These are fixture outcomes, not a deployment success rate. The four failures are the denied APS dispatch, the expired APS reference-time check, and the over-limit pair’s call-value and per-action-cap checks.

The report’s 22 recomputable values agree, and three quoted fixture values agree. Report assertions about PriorSeal receipt validity remain unexercised. SPEC-GAPS.md records the missing written PriorSeal signing profile, hash constructions recovered by trial, and the checker’s mapping from APS spend units to the PriorSeal asset. No result evaluated by this checker differs from the producer’s stated outcome.

NEGATIVES.json records five variants and two controls:

The surviving rehashed variant is not a whole-case acceptance: the APS cross-check still refuses it. It demonstrates the precise limit of this checker at an unexercised principal-signature claim. The fixtures do not establish chain execution, live authorization currency, live revocation, decision-level single use, cumulative spend, independently operated custody, or production usage.

To reproduce, check out the immutable vectors revision above and run the commands in its README. The checker pins agent-evidence-vectors 0.15.0 and pycryptodome 3.23.0; mutation generation additionally pins cryptography 44.0.2. The corpus package supplies byte and Ed25519 primitives, not APS or PriorSeal policy decisions. The ledger records the outputs and ceilings as E-05 through E-07.

E5. AGT TRACE transcript bytes: a source-pinned reproduction

The runner calls the real AuditEntry, MerkleAuditChain, and both AGT session mappers at AGT 60931ce. Before import, it verifies each retrieved source against its exact Git tree blob. Its byte oracle is rfc8785.py 655cbe0. IDs, times and contexts are fixed. The recorded result includes all six inputs, digests, source SHA-256 values, Python 3.12.14, and Pydantic 2.13.5.

The sink escapes non-ASCII characters while the model emits UTF-8 characters. Both sort keys by Unicode code point and use Python’s number spelling; RFC 8785 requires UTF-16 code-unit ordering and ECMAScript number serialization. A JSON-value match therefore does not establish a transcript-byte match. These results concern the pinned AGT implementation’s transcript hash, not the current TRACE library’s separate reverse adapter or an entire protocol-conformance claim.

uv run --no-project --python 3.12.14 --with pydantic==2.13.5 \
  python experiments/trace-transcript/run.py > /tmp/trace-transcript.json
diff -u experiments/trace-transcript/recorded.json /tmp/trace-transcript.json

The runner also accepts --source-dir for an offline rerun from cached source blobs; it checks the same immutable pins. This experiment signs no record and executes no live agent. It establishes byte disagreements on the selected inputs, with no measured claim about effect capture, coverage, custody, or production failures. The ledger records its provenance and results as E-08 and E-09.

E6. Observer declaration, admission and replay

The runner executes the unchanged public observer demo at 8562c25. It checks 12 Git blob pins, privately copies the verified sources, and only then imports them. The declaration fixes interval admission-demo-1, scope /work, and operation write-file. The consumer records those expectations and a signed witness head before the broker begins. The broker performs one durable file replacement, seals its history, and the consumer persists its admission before returning success.

The retained bundle is the original admission-run artifact from observer CI run 36808961399, downloaded and hash-checked. Its archive SHA-256 is bcc22cd27b1f9404e961f7329770620994a76348d085b7b711ab586986493892. The provenance record supplies a digest for each of the eight retained evidence files and records the omission of two empty lock files, so the evidence survives the original artifact’s expiry. The PR run reports head cc7191e344aabcd26baec8cbac922050ad1420b2; its logs show checkout of synthetic merge e8040a8032ea197ea7a185a832808c5fb9ee9355. The tested tree matches the pinned observer revision. That original archive records the tested PR tree. A later native main run 36825509807 checked out exact revision 8562c25fb7ec97596ea9d0297c495340298914b6 and passed the same 148 tests, admission demo and eight Linux boundary-probe checks. The provenance record includes this additional run and its distinct artifact digests. None was independently operated.

The atlas re-admits those signed bytes in a fresh scratch consumer store and requires its decision and durable state to equal the retained records. Its deterministic result also records three controls:

Every refusal must leave consumer state unchanged. Fresh stores for the latter controls ensure replay protection cannot mask a missing authority or signature check. A newly executed demo generates new keys, nonces and signatures; the comparison checks complete semantic outcomes and source digests, while the original bundle has its own exact file manifest.

To reproduce on Linux:

git clone https://github.com/probityai/agent-evidence-observer observer-source
git -C observer-source checkout --detach 8562c25fb7ec97596ea9d0297c495340298914b6
uv run --no-project --python 3.12.14 --with cryptography==46.0.7 \
  python experiments/observer-admission/run.py --observer-root observer-source \
  --bundle experiments/observer-admission/retained \
  --manifest experiments/observer-admission/provenance.json \
  --expect experiments/observer-admission/recorded.json
uv run --no-project --python 3.12.14 --with cryptography==46.0.7 \
  python experiments/observer-admission/run.py --observer-root observer-source \
  --output /tmp/observer-admission-new \
  --expect experiments/observer-admission/recorded.json

The output directory must be empty. After the exact source checkout and runtime are available, neither command fetches sources or contacts a witness. The atlas CI job runs both commands and the source-pin/refusal attack tests on every push and pull request, retaining each fresh bundle.

This is an author-produced same-operator fixture, with witnessScope: PEER and evidence_vantage: artifact. Its public keys and head are fixture expectations, not externally acquired trust. It establishes the tested declaration-to-admission ordering, signed-byte checks and local at-most-once admission. Its coverage remains broker-write-calls-with-valid-request-id, with unmediatedEffects: not-established. It does not establish independent custody, live-agent behavior, wall-clock freshness, complete effect capture, production isolation, cross-host consensus, or exactly-once downstream effects. Deleting or rolling back consumer state can erase replay protection. The ledger records these outcomes and ceilings as E-10 through E-13.