DeepSWE
Long, multi-file software-engineering tasks; pass@1 with mean cost and agent steps per task.
Agent Index weight
6%
in the Agentic coding pillar
Models tested
26
from the benchmark’s leaderboard and reports
Results
| # | Model | Score | Normalised | Harness | Cost/task | Date |
|---|---|---|---|---|---|---|
| 1 | GPT-6 Astra Supported OpenAI | 74.1% | mini-swe-agent | $4.43 | — | |
| 2 | Gemini 3.8 Flash Supported Google | 73.8% | mini-swe-agent | $2.36 | — | |
| 3 | Claude Opus 5 Supported Anthropic | 73.6% | mini-swe-agent | $11.84 | — | |
| 4 | GPT-5.6 Sol Supported OpenAI | 72.7% | mini-swe-agent | $8.39 | — | |
| 5 | Claude Fable 5 Supported Anthropic | 69.9% | mini-swe-agent | $13.41 | — | |
| 6 | GPT-5.6 Terra Estimated OpenAI | 69.6% | mini-swe-agent | $4.95 | — | |
| 7 | GLM-5.3 Supported Z.ai (Zhipu) | 69.0% | mini-swe-agent | $3.99 | — | |
| 8 | Kimi K3 Supported Moonshot AI | 68.5% | mini-swe-agent | $4.65 | — | |
| 9 | Grok 4.6 Supported xAI | 67.5% | mini-swe-agent | $3.45 | — | |
| 10 | GPT-5.6 Luna Estimated OpenAI | 67.2% | mini-swe-agent | $3.03 | — | |
| 11 | GPT-5.5 Supported OpenAI | 67.0% | mini-swe-agent | $7.23 | — | |
| 12 | Gemini 3.7 Flash Estimated Google | 65.5% | mini-swe-agent | $2.03 | — | |
| 13 | GLM-5.3-Flash Supported Z.ai (Zhipu) | 63.4% | mini-swe-agent | $0.48 | — | |
| 14 | Claude Opus 4.8 Supported Anthropic | 59.0% | mini-swe-agent | $13.22 | — | |
| 15 | Qwen3.8 Max Preview Estimated Alibaba (Qwen) | 57.5% | mini-swe-agent | $3.73 | — | |
| 16 | Muse Spark 1.2 Estimated Meta | 54.9% | mini-swe-agent | $3.70 | — | |
| 17 | Claude Sonnet 5 Supported Anthropic | 53.8% | mini-swe-agent | $26.40 | — | |
| 18 | Grok 4.5 Estimated xAI | 53.8% | mini-swe-agent | $2.42 | — | |
| 19 | Muse Spark 1.1 Estimated Meta | 53.3% | mini-swe-agent | $2.36 | — | |
| 20 | GPT-5.4 Supported OpenAI | 51.8% | mini-swe-agent | $5.65 | — | |
| 21 | Gemini 3.6 Flash Estimated Google | 46.7% | mini-swe-agent | $4.42 | — | |
| 22 | GLM-5.2 Supported Z.ai (Zhipu) | 43.8% | mini-swe-agent | $3.92 | — | |
| 23 | Gemini 3.5 Flash Supported Google | 37.4% | mini-swe-agent | $7.34 | — | |
| 24 | Kimi K2.7 Code Estimated Moonshot AI | 30.5% | mini-swe-agent | $2.82 | — | |
| 25 | Claude Sonnet 4.6 Supported Anthropic | 29.9% | mini-swe-agent | $5.52 | — | |
| 26 | Gemini 3.1 Pro Preview Supported Google | 11.7% | mini-swe-agent | $2.14 | — |
For models without a DeepSWE result, the index estimates one from Epoch’s Capabilities Index using a logistic fit on 26 models (residual SD 0.57 on the logit scale). Estimates are labelled and widen the model’s range.