OSuite Runtime Boundary Benchmark tests whether an agent action is still inside the boundary that was approved, across 18 runtime surfaces, 38 risk families, 6 locales, and 10 obfuscation styles.
Most AI agent security discussions still look at the wrong object.
They look at the prompt, the transcript, the model response, the tool name, or the runtime label. Those are useful signals. They are not the control point.
The control point is the moment an intended action becomes a side effect.
That is the reason we built the OSuite Runtime Boundary Benchmark. It tests a concrete enterprise question: when an AI agent is about to act, can the system tell whether the action is still inside the boundary that was approved?
The benchmark contains 6,000 synthetic agent action records.
It covers 18 runtime surfaces, 38 risk families, 6 locales, and 10 obfuscation styles. The corpus includes 1,000 safe baselines and 5,000 risky records across families such as credential material access, public egress, model-code execution, package lifecycle execution, CI/CD token pivoting, workflow sink risk, stale verifier proofs, payment actions, governance bypass headers, and tenant boundary crossing.
This is not a benchmark about whether a model can write a good answer. It is a benchmark about whether a governance layer can classify a runtime action before that action becomes operational consequence.
Download the full benchmark whitepaper
The runtime-label baseline knew where the action came from. It could see labels like shell, browser, workflow, MCP, or code agent.
That was not enough.
| Classifier | Exact match | Risky protection | Safe allow |
|---|---|---|---|
| Runtime label | 16.7% | 0.0% | 100.0% |
| Operation only | 87.4% | 84.9% | 100.0% |
| Effect + destination | 69.9% | 63.9% | 100.0% |
| Resource + effect + destination | 82.6% | 81.9% | 100.0% |
| Reference boundary runner | 100.0% | 100.0% | 100.0% |
The point is not that every production system will score this cleanly. The point is that runtime labels by themselves are structurally weak. They can identify a lane, but they cannot decide whether a credential read became a public export, whether a draft became a payment action, whether a safe URL became a signed URL leak, or whether an approval still covers the action that actually ran.
A log can tell you that something happened.
That is not the same as proving the action was inside the approved boundary before it happened.
For agent systems, the evidence has to bind more than a timestamp and a tool call. It has to bind the proposed action, the affected resource, the destination boundary, the policy checked, the approval state, the lease condition, the execution result, and the replay path.
Otherwise the enterprise is not governing the action. It is reconstructing a story after the action already happened.
OSuite is built around the action boundary rather than the transcript boundary.
| OSuite layer | Role in the benchmark problem |
|---|---|
| CAVA | Turns heterogeneous runtime events into canonical action objects with stable fingerprints. |
| Policy + Decision Score | Routes the action as allow, observe, review, dual approval, or block based on consequence. |
| BAF | Binds approval to a bounded action lease rather than a reusable permission. |
| AREG | Reconstructs the runtime security graph, blast radius, and evidence chain. |
| Proof bundle | Preserves replayable evidence so the decision can be checked later. |
The benchmark is designed to stress the seam between "the agent said it was doing X" and "the runtime action would actually do Y."
That seam is where real incidents start.
Enterprise buyers do not need another dashboard that says an agent was active.
They need to know whether the agent was about to cross a boundary that the organization did not approve.
That is a harder bar than logging. It is also a more useful bar for buyers. A CISO does not want a perfect-looking transcript after a leaked secret, unauthorized payment, public message, or destructive workflow update. They want controls that fire before the side effect forms.
The OSuite Runtime Boundary Benchmark is our attempt to make that bar measurable.
It is not a claim that OSuite prevents every possible agent incident. It is a public test harness for a narrower and more important question:
Can the governance layer recognize the action boundary before the action runs?
The full whitepaper includes corpus construction, baseline failure cards, representative case pages, an incident-inspired chain, reproducibility notes, scoring formulas, and appendices with the full family index.
Download the full OSuite Runtime Boundary Benchmark whitepaper
The public benchmark project is also available on GitHub: OndCo/Agent-Action-Boundary-Benchmark.
Request enterprise access and send your first governed decision today.
