SShogo Bench

DeepSWE

Long, multi-file software-engineering tasks; pass@1 with mean cost and agent steps per task.

Agent Index weight
6%
in the Agentic coding pillar
Models tested
26
from the benchmark’s leaderboard and reports
Normalised for the index
0 → 0, 1 → 100
Official site

Results

#ModelScoreNormalisedHarnessCost/taskDate
1GPT-6 Astra Supported
OpenAI
74.1%mini-swe-agent$4.43—
2Gemini 3.8 Flash Supported
Google
73.8%mini-swe-agent$2.36—
3Claude Opus 5 Supported
Anthropic
73.6%mini-swe-agent$11.84—
4GPT-5.6 Sol Supported
OpenAI
72.7%mini-swe-agent$8.39—
5Claude Fable 5 Supported
Anthropic
69.9%mini-swe-agent$13.41—
6GPT-5.6 Terra Estimated
OpenAI
69.6%mini-swe-agent$4.95—
7GLM-5.3 Supported
Z.ai (Zhipu)
69.0%mini-swe-agent$3.99—
8Kimi K3 Supported
Moonshot AI
68.5%mini-swe-agent$4.65—
9Grok 4.6 Supported
xAI
67.5%mini-swe-agent$3.45—
10GPT-5.6 Luna Estimated
OpenAI
67.2%mini-swe-agent$3.03—
11GPT-5.5 Supported
OpenAI
67.0%mini-swe-agent$7.23—
12Gemini 3.7 Flash Estimated
Google
65.5%mini-swe-agent$2.03—
13GLM-5.3-Flash Supported
Z.ai (Zhipu)
63.4%mini-swe-agent$0.48—
14Claude Opus 4.8 Supported
Anthropic
59.0%mini-swe-agent$13.22—
15Qwen3.8 Max Preview Estimated
Alibaba (Qwen)
57.5%mini-swe-agent$3.73—
16Muse Spark 1.2 Estimated
Meta
54.9%mini-swe-agent$3.70—
17Claude Sonnet 5 Supported
Anthropic
53.8%mini-swe-agent$26.40—
18Grok 4.5 Estimated
xAI
53.8%mini-swe-agent$2.42—
19Muse Spark 1.1 Estimated
Meta
53.3%mini-swe-agent$2.36—
20GPT-5.4 Supported
OpenAI
51.8%mini-swe-agent$5.65—
21Gemini 3.6 Flash Estimated
Google
46.7%mini-swe-agent$4.42—
22GLM-5.2 Supported
Z.ai (Zhipu)
43.8%mini-swe-agent$3.92—
23Gemini 3.5 Flash Supported
Google
37.4%mini-swe-agent$7.34—
24Kimi K2.7 Code Estimated
Moonshot AI
30.5%mini-swe-agent$2.82—
25Claude Sonnet 4.6 Supported
Anthropic
29.9%mini-swe-agent$5.52—
26Gemini 3.1 Pro Preview Supported
Google
11.7%mini-swe-agent$2.14—

For models without a DeepSWE result, the index estimates one from Epoch’s Capabilities Index using a logistic fit on 26 models (residual SD 0.57 on the logit scale). Estimates are labelled and widen the model’s range.