SShogo Bench

Agent Index v1.0 · 2026-11-20

Methodology

The Shogo Agent Index answers one question: how well does this model do agentic work? Not trivia, not chat style, not competition math on its own. It is a weighted average of 17 published benchmarks where the model has to act: edit a repo, run commands, use tools over many steps, operate a computer or produce a professional deliverable.

1. Benchmarks and weights

Five pillars, weighted by how much of real agent work they represent. Inside each pillar, benchmarks with stronger task realism, harder ceilings and more independent results get more weight.

PillarBenchmarkWeightModels
Agentic codingSWE-bench Verified8%32
Terminal-Bench 2.08%44
DeepSWE6%26
FrontierSWE5%25
CursorBench4%22
GSO-Bench4%30
Long-horizon autonomyMETR Time Horizons10%35
Vending-Bench 26%68
PostTrainBench4%21
Knowledge-work agentsAPEX-Agents8%62
GDPval7%11
Remote Labor Index5%18
DeepResearch Bench5%24
Computer useOSWorld 2.06%9
OSWorld4%9
Security & context learningCybench5%21
CL-bench5%19

Benchmarks we show but leave out of the index (reasoning, math, knowledge, saturated classics) are on the leaderboard. Weights only change with a version bump and a changelog entry.

2. Normalisation

Each result is mapped to 0–100 so benchmarks can be averaged: the random-guess baseline (or starting value) becomes 0 and the ceiling becomes 100. Vending-Bench 2 reports dollars, so it is log-scaled from the $500 starting balance to $20,000. Where a leaderboard has several agents or harnesses for the same model, we take the best reported result and link to it, the same convention the leaderboards themselves use.

3. Filling gaps without guessing

No model has been run on every agentic benchmark; many new ones have results on only a handful. Averaging only what exists would reward models that skipped hard benchmarks. Instead, for each benchmark we fit a logistic curve linking its results to Epoch AI’s Capabilities Index (ECI), a general capability score built from dozens of benchmarks. A missing result is estimated from the model’s ECI on that curve. When a model has no ECI, we infer its position from its own measured results on the same curves.

Estimated components are shown as estimates everywhere (lighter bars, “est.” labels), and they widen the model’s range.

4. Evidence labels

5. Ranges

Every score has a 95% range. Measured results carry a small fixed uncertainty (±3 points before weighting). Estimated results carry the residual error of their fit; because all estimates for a model come from the same capability score, their errors are added together rather than assumed to cancel. Two models whose ranges overlap should be read as tied.

6. Agent-effective price

The usual “blended price” assumes 3 input tokens per output token. Agents are far more input-heavy: every turn resends the system prompt, tool definitions and history, and most of that is served from the prompt cache. The agent-effective price assumes 20 input tokens per output token, with 85% of input served from cache:

price = (20/21) × (0.85 × cache_read + 0.15 × input) + (1/21) × output

Models without a cache discount are charged full input price for cached tokens. Prices are list prices from the lab (via models.dev), or OpenRouter where the lab doesn’t publish one; every API host’s own price is on the model page.

7. Data and updates

Results come from the Epoch AI Benchmarking Hub (CC BY 4.0), which runs some benchmarks itself and collects the rest from the benchmark authors’ leaderboards. Every result links to its source. Model specs and prices come from models.dev; per-host prices and uptime from OpenRouter. A pipeline pulls all sources, validates them (units, ranges, a source URL on every score, no orphan records) and rebuilds the site. See sources.

8. Known limits

9. Independence

Shogo is an agent platform that runs models from many labs; it does not sell a model of its own. No lab pays for placement or reviews scores before they’re published. If you spot an error, the data and code are open: tell us and it goes in the changelog.