Benchmarks
What each benchmark measures, who leads it, and its weight in the Agent Index. Benchmarks with weight 0 are shown for context but don’t move the index.
Agentic coding
500 real GitHub issues from Python repos, human-validated. The agent edits the repo until the hidden tests pass.
Hard tasks done entirely in a terminal: compiling, debugging, data wrangling, sysadmin. Best reported agent per model.
Long, multi-file software-engineering tasks; pass@1 with mean cost and agent steps per task.
Frontier-difficulty engineering projects scored on implementation, performance and research quality.
Real coding-agent sessions from Cursor, graded on whether the change does what the developer asked.
Software optimisation: make a real codebase measurably faster without breaking it.
Long-horizon
How long a task (in human expert time) the model completes with 50% reliability. Scored on METR’s task-suite average.
Run a simulated vending business for a year starting with $500. Score is the final bank balance (log-scaled).
The agent post-trains a small model end to end on a GPU budget; scored on the resulting model.
Agentic play across text and grid games (NetHack, Crafter, BabyAI).
Knowledge work
Long tasks from investment banking, consulting and corporate law, done with real office tools.
Deliverables from 44 occupations judged blind against expert human work. Score is win rate.
Real paid freelance projects; share completed to a standard a client would accept.
PhD-level research reports with citations, graded for quality and citation accuracy.
A simulated software company: browse, code, message coworkers.
Computer use
Security & context
Professional capture-the-flag challenges solved without hints.
Learn new rules, procedures and domain knowledge from context, then apply them.
Exploit development across tiers of difficulty.
Coding
225 hard Exercism exercises in six languages, edited through Aider.
Reimplement a program from its behaviour alone.
Frontier-difficulty programming problems.
Head-to-head web-app builds voted on by people (Elo).
Research-grade scientific coding problems.
Unusual ML tasks solved by writing and running PyTorch code.
Speed up numerical algorithms; score is speedup.
Reasoning
Graduate-level science questions written to be Google-proof.
2,500 expert-written questions at the edge of human knowledge.
Abstract visual puzzles that are easy for people and hard for models.
The original ARC abstraction and reasoning corpus.
Trick questions about space, time and social reasoning.
Find the best move in chess puzzles, no engine.
Unpublished research-level physics problems.
Math
Hundreds of unpublished, expert-written math problems.
Research-level math problems that take experts days.
Competition math at AIME level.
The hardest tier of the MATH competition dataset.
Write rigorous mathematical proofs.
Knowledge
Short factual questions; measures hallucination.
Language-model capability assessment across domains.
Decision-theory questions.