Browser
AI security benchmarks — what the numbers actually mean.
Six public benchmarks measure frontier AI on security work. Mythos leads most. AISI flags the limitations.
CyberGym community
A reproducibility benchmark — given a published CVE, can the model write a working PoC?
| Model | Score | Notes |
|---|---|---|
| Claude Mythos Preview Anthropic | 83.1% | Glasswing-gated; Anthropic-reported |
| Claude Opus 4.6 Anthropic | 66.6% | Generally available via API |
| GPT-5.5 OpenAI | ~62% | Community-reported estimate |
| Claude Sonnet 4.6 Anthropic | ~58% | Cheaper but materially behind on this task |
| Gemini 2.5 Pro Google | ~45% | Stronger on multimodal than on pure code-PoC |
Methodology: the agent is given the public CVE description + the affected source code, no human guidance after the initial prompt. A PoC is "successful" only when it triggers the documented impact (crash, OOB write, info leak) in a controlled environment.
SWE-bench Verified community
Real GitHub issues from popular Python repos — model submits a patch that passes existing tests.
| Model | Score | Notes |
|---|---|---|
| Claude Mythos Preview Anthropic | 93.9% | Anthropic-reported; near-saturation |
| Claude Opus 4.6 Anthropic | 80.8% | Strong baseline at fixed-depth |
| GPT-5.5 OpenAI | ~78% | Within striking distance of Opus |
| Claude Sonnet 4.6 Anthropic | ~71% | Cost-effective workhorse |
Methodology: model gets the bug report, repo state, and unlimited steps to investigate. Patch is graded purely by whether all the project's tests pass after applying. Doesn't measure novelty — only correct-fix capability.
AISI The Last Ones — 32-step kill chain AISI
UK AISI simulation of a full corporate attack — 32 stages, expert humans take ~20 hours.
| Model | Full completion (3+/10) | Avg step reached |
|---|---|---|
| Claude Mythos Preview Anthropic | 3 / 10 | 22 / 32 |
| Claude Opus 4.6 Anthropic | 0 / 10 | 16 / 32 |
| GPT-5.5 OpenAI | 0 / 10 | ~15 / 32 |
| Claude Sonnet 3.7 (14 mo old) Anthropic | 0 / 10 | < phase 1 recon |
⚠ Methodology caveat (AISI's own words)
TLO has no active defenders / EDR / IR. Treat as ceiling under ideal conditions.
AISI Capture-the-Flag tiers AISI
AISI measured Mythos against three CTF tiers used to certify human pentester skill.
| CTF tier | Mythos pass rate | Difficulty for humans |
|---|---|---|
| Non-technical (for general public) | 97% | 5 min for a hobbyist |
| Apprentice (entry-level pentester) | 87% | 1-2 hours for a junior |
| Expert (CTF competition tier) | 73% | 4-12 hours for an expert; previously 0% for any public model |
Methodology: the model gets the challenge package (binary, source-snippet, or web app), no human guidance. A pass means the flag string is captured. AISI noted the model was budget-limited, not capability-limited — at 100M-token inference budget per attempt, scores continued to climb.
AISI Cooling Tower — industrial OT AISI where Mythos failed
Industrial-control-system attack simulation. Mythos failed on IT-phase, not OT-specific phase.
Where Mythos got stuck
Mundane IT-side complexity remains a meaningful barrier even for frontier AI.
Anthropic OSS-Fuzz tier ladder Anthropic
Run model against ~7,000 entry points in ~1,000 OSS-Fuzz repos; grade by 5-tier severity ladder.
| Model | Tier 1+2 crashes | Tier 5 (full hijack) |
|---|---|---|
| Claude Mythos Preview Anthropic | 595 | 10 (on fully-patched targets) |
| Claude Opus 4.6 Anthropic | ~275 | 1 |
| Claude Sonnet 4.6 Anthropic | ~250 | 1 |
What "Tier 5" means: a complete control-flow hijack on a fully-patched target — i.e. the model found a previously-unknown exploitable vulnerability, wrote an exploit that triggers RIP / RAX control, and ran it successfully against the latest version of the software. That Mythos did this 10 times in one corpus run is the single most striking number in this entire post.
What this means for your team
- Benchmark numbers measure capability under ideal conditions — production has active defenders that aren't in the ranges.
- Boring layered infrastructure is still a barrier. Invest in AD hardening, network segmentation, service-mesh discoverability.
- Capability trajectory is steep — plan defensive posture for next year's model, not this year's.
- Run our readiness assessment to see where you stand, then pick tools from /llm-providers or run /mythos.
Sources ·
red.anthropic.com — Mythos technical research ·
aisi.gov.uk — TLO + CTF evaluations ·
Cloudflare — Project Glasswing harness post ·
community-reported figures for GPT-5.5 / Gemini 2.5 Pro / Sonnet 4.6 are estimates and may shift as official numbers are published.