A green test suite can contain negative conclusions

A payments team wants an agent to issue refunds within an approved limit. Its tests cover a permitted refund, a refused refund, an altered approval and a repeated request. Each test can pass. That does not mean each refund succeeded. A test also passes when the checker correctly refuses the altered approval.

The distinction matters after an incident. A valid record can describe a failed action. An invalid record cannot establish whether the action succeeded. Probity keeps the test outcome, the record’s validity and the recorded result separate, with the inputs and reader needed to repeat the check.

This note demonstrates those meanings with published conformance fixtures. The refund workflow is an example of the customer problem. The measurements below come from synthetic artifacts.

Read the retained Lab record or check its retained accounting. The maintained checker reads the original outputs; it does not rerun captured reader code.

Reproduce the historical report

We replayed all 272 cases in agent-evidence-vectors==0.12.0 on CPython 3.13.7 and 3.14.3. Both runs reproduced the complete historical report and harness bytes. All 272 test outcomes passed. The report contained 31 passing, 239 failing and two inconclusive checks.

Thirty valid artifacts described a failing or degraded result. The other 209 negative checks came from invalid artifacts. The test suite passed because each case produced its expected answer. The two indeterminate cases retained the historical unsupported_input cause.

The original report, original harness output and complete case table retain each conclusion. The report SHA-256 is 37288d8dfc01f56cb5ef553232258dc596928bc962e2309e0c7dc09ae588baf2. The harness SHA-256 is ee349e66697a2f4f95dd8223fd261810f99f5f2807e03bde03c9a9f57359529d.

The source is commit 2f3aee40a454df6de0f571d119f057d7dbb8dd67. Its 61 accepting, 209 rejecting and two indeterminate fixtures have corpus digest 8b035678def9e5ac00ba761b8c640e4412c57163134afb9d2c1a90f49d573a52. The measured installation used the published wheel with SHA-256 cb346a41f715029f8e68542150560b08a71ac23eb7660eaa411aea4fd2ec1f59. All 1,191 package files matched that wheel and their source Git blobs. The replay configuration records the exact package, environment and command:

agent-evidence-vectors --emit-w3c-report report.json

Run it from an empty directory with that installed package. The experiment used the reference reader, removed AEE_EXTERNAL_VERIFIER and AEE_SUBSTRATE_KEYS, and exposed the expected answers. It ran no model inference or new agent task.

Check which rules change the result

A separate experiment used the 232 fixtures in the published 0.13.0 report corpus. We copied its reader twice and disabled one predicate in each copy. The fixture documents, resolution stores and all other checks stayed fixed. Each actual reader ran over every fixture.

The first change accepted a witness whose state was inconclusive. The second accepted a control that named another checker or constraint set. All 116 accepting fixtures and every other rejecting fixture stayed unchanged.

The rule configuration, complete rule results and source record identify the exact readers and modifications. These controls test the predicates themselves. They do not remove rejection labels after the reader runs.

What a customer can use

A release test answers whether the system produced the expected decision for the selected case. An incident report answers what the retained evidence supports. Keeping those answers separate prevents a dashboard from turning a correct refusal into a successful agent action.

The retained files let another reader check the same inputs, rules and results. This experiment used the same reference implementation on two runtimes. A second-party replay also reported the historical file digests. Repeated output supports reproducibility; it does not establish an independently written implementation.

The September 30 draft provides the reporting context. It remains draft correspondence. These fixture results establish no live refund, customer deployment or independent custody. A deployed workflow needs its own permitted action, refusal, retry and fault results, with records checked at the actual effect boundary.