bots-bench: Benchmarking AI Models and Agents for SOC

Primary Benchmark View

CyBT-CTF first, BOTSv3 as the public cross-check

CyBT-CTF is the blinded cyber investigation read and is the default comparison here. BOTSv3 remains useful as a public, exposure-prone benchmark, so it is available as an opt-in cross-check instead of the headline result.

BOTSv3 Frontier Leaderboard

Public BOTSv3 agent + model leaderboard

This section and the detailed sections below are the public BOTSv3 report surface. Each entry pairs an AI agent (Claude Code, OpenAI Codex, opencode, Louie) with a model. Use the CyBT-CTF table above for the blinded default comparison.

Notes on the leaderboard

Read status flags before ranking. Some public BOTSv3 rows show answer-only or traceability signals that can inflate a headline score. When a row is flagged, use the adjusted score shown in the table rather than treating the raw rank as a clean win.
Effort settings are compared explicitly. A model can behave very differently at low, medium, and high effort. The leaderboard keeps each effort visible so strong low-effort runs do not hide weaker higher-effort results, and the Effort filter can collapse to the best available setting when that is the question.

BOTSv3 Follow-Up

Model coverage by ATT&CK tactic

Each polygon uses one representative config: the best-scoring run among the latest tested version of each model family.

BOTSv3 model progress over time

How each family scales by release date

One point per model line — strongest effort tier kept. Lines connect chronologically within each model family so successive generations are visible.

BOTSv3 Efficiency

Public BOTSv3 efficiency tradeoff explorer

BOTSv3 only. Switch among cost-vs-speed, cost-vs-solve-rate, and cost-vs-combined-score modes. Bubble size encodes first-pass solved-count so cost can stay on the axes.

Benchmark Program

What this benchmark is, and what it refuses to fake.

Systems Under Test

Agent harnesses and models on the board

BOTSv3 Provider Upgrades

What changed across model generations on BOTSv3

Summarizing version-step changes from publishable BOTSv3 runs in this snapshot.

BOTSv3 Benchmark Atlas

The public BOTSv3 corpus underneath these scoreboards

This is the public BOTSv3 corpus view, not CyBT-CTF. The point is representative investigation work, not toy prompts: 100+ overlapping log and alert providers, incident families grounded in the BOTSv3 IR corpus, and question-level hardness.

Track coverage

ATT&CK coverage

Chart Wall

Question hardness

Each dot is a benchmark question. Left means fewer configs solve it. Up means it burns more time or more attempts. Bigger dots mean more repeated attempts, and color stays tied to track.

Track Incident Search

Claude 4.6 Controversy

Anthropic Apr-23 postmortem: did we catch a regression?

Anthropic published an April 23 postmortem describing three product/harness-level regressions affecting recent Claude runs. We reran the same 17 previously-contaminated IDs under a matched post-window envelope to isolate model effects from window confounds.

Subset 17-ID control · with tools

Opus 4.6 high: 14/17
Opus 4.7 high: 5/17
Paired delta: +9 for 4.6 on same IDs, same envelope, post-window.

Matched envelope: MCP, parallel=2, timeout=360s, no session persistence, high effort.

Subset 17-ID · no tools

Opus 4.6: 6/17 (high), 7/17 (default)
Opus 4.7: 1/17 (both effort lanes)
Only q2_sq201_id_cl solves for 4.7 — general-knowledge-answerable.

Interpretation

Window-confound alone does not explain the 4.6 > 4.7 gap.
Residual gap likely includes model-behavior and tool-use differences.
Host-level config changes around Apr 17 remain a credible, unquantified confound for non-isolated runs.

Evidence: Anthropic postmortem · findings-09 · short report.

BOTSv3 Benchmark Hygiene

Could the model already know the BOTSv3 answers?

This BOTSv3 check asks whether a model can answer from prior exposure or memory before using the investigation tools.

No-tools priors versus scored runs

We keep the same score axis as the main benchmark so you can see how much of the corpus a model reaches before any tool use.

Can we trace answers back to evidence?

Loading literal-answer coverage and traceability statuses...

BOTSv3 Harness Versions

Same model across Claude Code MCP run dates on BOTSv3

Comparing earliest vs latest BOTSv3 MCP runs for the same model (date-based proxy for harness evolution).

BOTSv3 Prompt Effects

Cross-validation and planning loops on BOTSv3

Looking for clean BOTSv3 cross-validation and OODA / OSCAR-style planning-loop pairs...

BOTSv3 Latest Runs

Pass-rate snapshot cards for published BOTSv3 executions

Looking for local run artifacts…

BOTSv3 Retry Lab

What happens when a failed BOTSv3 config gets more tries?

These rows show configs where the evaluator allows extra retry passes after a miss. That can improve score, but it also blows out time and token budgets.

BOTSv3 Config Microscope

Drill into one BOTSv3 configuration, one question at a time

The map and microscope share one selection. Pick a config here, or click a point in the solve-versus-time map, and inspect its question-level pattern instead of only looking at rollups.

Config Map

Solve rate versus time

Human-readable config labels, not raw run IDs. Pick the score dimension and the time dimension you care about, then click a point to sync the microscope below.

X metric Y metric

Configuration Question metric

BOTSv3 Supporting Boards

Secondary cuts of the same snapshot

Additional ways to slice the same public BOTSv3 snapshot.

BOTSv3 Benchmark Rules

How the BOTSv3 runs work

BOTSv3 Experiment Surface

What we are varying

The dimensions we vary across BOTSv3 runs. CyBT-CTF is tracked separately in the comparison and deep-dive pages.

Benchmarking AI models and agents for SOC and IR investigations.