Enterprises are wiring agents to real credentials. Nothing measures both sides of the risk.
Autonomous agents are being connected to CI systems, directories, cloud IAM and data vaults. Two questions decide whether that is safe: how deep an agent can drive an attack, and whether a model can catch the same attack in the telemetry it leaves behind. Existing public benchmarks do not jointly measure offensive depth and defensive detection on the same enterprise estate.
Cybench, the best-known public CTF benchmark, is a good ruler but can't be the whole measurement, which is why we brought it inside promptfoo as a reference point and then built past it:
A promptfoo-native cyber evaluation layer. Three deliverables.
Everything is defined, executed and reviewed through promptfoo itself. No bespoke harness. We call it a "promptfoo-native cyber evaluation layer" until offense and defense are consolidated into one plugin.
-
1
Cybench, running through promptfoo.
We extended promptfoo (using NIST CAISI's cyber-evals harness, built on UK AISI's Inspect framework) to run the public Cybench security tests directly from promptfoo and review them in the promptfoo UI, like any other eval.
-
2
Our seven-hop Enterprise Identity-to-Cloud Takeover benchmark.
An authored attack chain of seven capability-distinct hops, from a foothold on an edge network to a company's most protected data. It measures how far a model can drive the attack, and exactly where its reasoning stops.
-
3
A defensive mirror, using telemetry from the same simulated estate.
The same attack seen from the blue team: can a model author a detection that catches the intrusion in the telemetry the attack produces, without raising false alarms on normal activity?
The environment is a high-fidelity simulated enterprise identity and cloud-IAM estate, an HTTP/JSON/HMAC simulation, not literal Active Directory or AWS.
Two measured ladders. Three models, ten attempts each.
Both evals were run against the same three models at the released difficulty under contamination-resistant scoring. Offense produces a capability ladder whose death-point moves outward as models get stronger; defense produces a detection ladder scored only when the rule-pack raises zero false alarms. Separate evals on one estate. Read each ladder on its own.
Read side by side, not against each other: offense measures whether a model can execute the attack, defense whether it can author the detection, orthogonal capabilities on the same estate. Fuller statistical detail, the hop-by-hop chain, and robustness testing are in the appendix.
Rigor is the product.
The point of the instrument is a number we can defend. These are the controls that make the ladders above mean what they say:
- →Fresh instance, fresh secrets. Every offensive run is a freshly generated instance with fresh secrets.
- →Out-of-band verification. Memorized or replayed answers do not score.
- →Per-hop evidence. Each run shows where the model's reasoning stops, not just a pass/fail bit.
- →Harness failures excluded and replaced. An infrastructure error is re-run, not counted as a win or a loss.
- →Defense requires zero benign false alarms. A rule-pack scores only if it stays silent on normal activity.
- →Defense grounding checked against nine estate-generated native capture flows.
- →N = 10, wide intervals stated honestly. An early small-sample read looked like "about two in three" and collapsed once we ran ten; we corrected it in public and standardized on ten.
Cyber evaluation becomes one more thing the platform measures.
We didn't build a bespoke harness: offense chains, defensive twins and the Cybench cross-check are all defined, executed and reviewed through promptfoo itself. That was a deliberate bet on the platform.
One hub for every evaluation
Enterprises can run Cybench and authored offense and defense evals directly from promptfoo, and review every result in the same interface they use for model quality. No separate security-eval exercise, no bespoke tooling. That makes promptfoo the single hub through which all of an enterprise's evaluations run: high-stakes, recurring, and sticky by nature.
- →A new, defensible category. Frontier cyber-capability is a board-level, regulator-adjacent concern; owning its measurement pulls promptfoo up-market.
- →It showcases the platform. Every run lands in promptfoo view: the task-by-model grid, full transcripts on drill-down, named depth scores.
- →Recurring by construction. Contamination-resistant means it's re-run on every new model, not solved once and retired.
A technical review of the flagship offense-and-defense pair.
We would value a technical review of the flagship offense-and-defense pair: whether it measures the capabilities that matter most, where it should be deepened, and which additional evaluation areas should come next. We propose a focused methodology and results walkthrough, followed by agreement on the next two benchmark tracks if the approach holds up under review.
Whether the flagship offense + defense pair measures the capabilities that matter most, and where it should be deepened.
A focused walkthrough of the chain, its calibration and the defensive twin, in promptfoo.
If the approach holds up under review, we align on the next two benchmark tracks to build.
Supporting detail.
The hop-by-hop chain, the design axes, statistical detail, robustness testing, and the full roadmap.
A · The chain, hop by hop
The agent starts on an edge network with no handed foothold, recons its way in, then chains seven capability-distinct hops to recover a protected tenant secret. Every gate secret is derived from a per-run nonce, so nothing transfers between runs, and every hop is scored.
B · Designed to stay meaningful: four axes
A multi-stage attack across observability, CI, directory, federation, cloud IAM, KMS and tenant data: the estate agents actually operate in, not a puzzle box. It measures blast radius in the deployed world.
Every run is a freshly generated instance with fresh secrets, scored out of band; the defense corpus is de-oracled. That defeats secret-replay and memorizing an instance's literal hostnames, roles, and tags, so the score reflects solving this instance, not recalling a past one.
Offense scores where a model's reasoning runs out; defense scores when it catches the attack. A capability horizon and a detection latency, not a single pass/fail bit.
The attack runs generate the exact telemetry the defense eval is graded on. The attacker produces the defender's ground truth: real model-driven attacks, not imagined ones. Every hardening of the attack deepens the defensive test for free.
C · Statistical detail & robustness
Intervals and separation
Clopper-Pearson 95% intervals throughout. Each ladder's top-to-bottom gap is significant: offense GLM-5.3 4/10 vs DeepSeek 0/10, Fisher one-sided p = 0.043; defense GLM-5.3 5/10 vs DeepSeek 0/10, p = 0.016 (both < 0.05). Adjacent rungs are directional only at N = 10, not all-rungs-separated. GLM-5.2 runs were executed externally.
Defense grounding & graceful degradation
A passing CI conformance test over nine estate-generated native captures (both escalation families × successful / blocked / abandoned / benign, plus an authoritative insider) at zero benign false alarms and full malicious recall, with provenance and MFA-assurance boundaries grounded independently (the offense→defense adapter synthesizes no security facts). Earliest-landmark credit catches the smuggle at tag-landing (hop 4), before privilege escalation or key release; recall is arrival-order-invariant and degrades gracefully under telemetry loss (≈1.0 → 0.79 at 8% event loss → 0.67 at 22%). A full grounded calibration corpus is the stated next milestone.
D · Roadmap: ten evaluation areas
The built pair already exercises five of the ten areas that span how enterprise agents are wired in; the other five are scoped. Breadth comes after review, deepening on evidence rather than front-loading a factory.
Credential discovery and misuse
Finding and abusing leaked or rotated credentials to gain a foothold.
Identity federation
Smuggling claims across a signature boundary to impersonate an entitled principal.
Cloud IAM
Role assumption, PassRole and trust-policy abuse to escalate privilege.
Data exfiltration
Reaching a protected data store and extracting the tenant secret past a honeypot.
Persistence and lateral movement
Establishing durable access, then pivoting across systems and tenants.
Prompt injection and malicious artifacts
Hijacking an agent through poisoned content, tools or build artifacts.
Tool authorization boundaries
Whether an agent stays inside the tools and scopes it was granted.
Secret handling
Whether an agent stores, logs and transmits secrets safely.
Destructive actions
Whether an agent avoids irreversible or out-of-scope operations.
Detection and incident-response behavior
Whether a model spots the intrusion and responds without false alarms, the flagship's defensive twin.