Making the CAVA benchmark claim computable.

Verification claims should survive contact with someone who is not trying to help you.

That happened to CAVA this week.

An independent researcher included the CAVA arXiv paper in a public replication survey of papers that publish quantitative figures about their own verification components. The first determination was fair: `NOT RELEASED`.

Not because the reported benchmark figures were shown to be wrong. The issue was simpler and more uncomfortable. The paper referenced internal paths for the benchmark harness, and a reader outside OSuite could not reach the code needed to recompute the reported numbers.

That is exactly the kind of criticism a verification system should take seriously.

What was missing

The CAVA paper reported a deterministic benchmark profile around action canonicalization, wrapper-bypass detection, and false-positive control.

The claim that triggered the replication check was specific:

The problem was not the shape of the benchmark. The problem was reachability.

The artifact manifest pointed to internal repository paths. That meant an outside reader could not clone a public repo, run the harness, and recompute the claim. For a paper about verification and replayable evidence, that gap was not cosmetic. It cut directly against the principle we argue for in the product.

If OSuite says agent governance should be replayable, then our own research artifacts should be replayable too.

What we changed

We published a public reproducibility mirror inside the CAVA repository:

OndCo/CAVA reproducibility artifact

The artifact includes:

• —the executable benchmark harness;
• —the full expected benchmark snapshot;
• —the publication-safe result snapshot;
• —a manifest explaining what is included and excluded;
• —a verifier that recomputes the benchmark and compares the result to the shipped snapshots.

The reproduction path is intentionally boring:

git clone https://github.com/OndCo/CAVA.git
cd CAVA
npm run verify:paper

The verifier recomputes the benchmark result and reports:

{
  "ok": true,
  "benchmark_id": "cava-paper-benchmark-v1",
  "seed_scenarios": 96,
  "executable_variants": 384,
  "wrapper_bypass_catch_rate": 1,
  "false_positive_control": 1
}

That does not mean every part of OSuite is public. It means the specific benchmark claim in the paper is now publicly reachable and recomputable without access to the private OSuite product repository.

What the independent recheck did

The researcher did not simply accept the verifier's output.

He changed `wrapper_bypass_catch_rate` in the expected snapshot from `1` to `0.5` and reran the verifier. It failed. That matters because it shows the verifier is not just printing the expected number back to the reader.

He then inserted an unconditional failure into the CAVA canonicalization path and reran the artifact. The run aborted. That matters because it shows the shipped CAVA code is actually on the benchmark path.

After those checks, the public survey record was updated:

Determination as of 2026-08-06: COMPUTABLE, and recomputed rather than merely judged so.

The correction trail is public:

Replication survey reply record

What this does not claim

This is not a product certification.

It is not an audit of every OSuite deployment path. It does not publish customer-specific adapter rules, enterprise policy thresholds, managed evidence graph internals, tenant data, or private signing and KMS configuration. Those remain outside the open artifact for normal commercial and security reasons.

The claim is narrower and cleaner:

That is the line we want to keep.

It would be easy to overstate this. We are deliberately not doing that. The useful trust signal is not that someone praised CAVA. The useful signal is that a public criticism identified a reachability gap, OSuite closed it, and the correction was independently recomputed rather than taken on trust.

Why it matters for agent governance

Agent governance will get noisy.

There will be dashboards, audit trails, policy badges, blockchain receipts, trust registries, connector directories, and plenty of claims that something is safe because a platform says it is safe.

That is not enough.

The better question is whether a claim can be replayed later by a party that was not part of the original decision. In product terms, that is why OSuite keeps pushing toward action fingerprints, approval-bound receipts, bounded leases, runtime exposure graphs, external verifier checkpoints, and proof bundles.

This CAVA correction is small, but it is aligned with that same philosophy.

The benchmark result is less interesting than the process:

• —a third party asked whether the claim was reachable;
• —the first answer was no;
• —the artifact was published;
• —the verifier was stress-checked;
• —the determination became `COMPUTABLE`;
• —the correction trail remained public.

That is closer to how serious AI-agent governance should work.

Not perfect claims. Checkable ones.

Links

• —CAVA arXiv paper
• —CAVA public reproducibility artifact
• —Public replication survey reply record
• —CAVA open-source repository