SShogo Bench

SWE-bench Verified

500 real GitHub issues from Python repos, human-validated. The agent edits the repo until the hidden tests pass.

Agent Index weight
8%
in the Agentic coding pillar
Models tested
32
run by Epoch AI on a shared harness
Normalised for the index
0 → 0, 1 → 100
Official site

Results

#ModelScoreNormalisedHarnessCost/taskDate
1Claude Opus 4.7 Supported
Anthropic
83.5%source—Apr 20, 2026
2GPT-5.5 Supported
OpenAI
80.6%source—Apr 24, 2026
3Gemini 3.5 Flash Supported
Google
79.3%source—Jun 1, 2026
4Claude Opus 4.6 Supported
Anthropic
78.7%source—Feb 18, 2026
5GLM-5.2 Supported
Z.ai (Zhipu)
78.7%source—Jun 25, 2026
6DeepSeek V4 Pro Estimated
DeepSeek
77.6%source—Jun 18, 2026
7Qwen3.7 Max
Alibaba (Qwen)
77.3%source—Jun 18, 2026
8GPT-5.4 Supported
OpenAI
76.9%source—Mar 6, 2026
9Claude Opus 4.5 (latest) Supported
Anthropic
76.7%source—Feb 5, 2026
10Kimi K2.6 Estimated
Moonshot AI
76.7%source—May 8, 2026
11Qwen3.6 Max Preview Estimated
Alibaba (Qwen)
76.7%source—May 28, 2026
12Gemini 3.1 Pro Preview Supported
Google
75.6%source—Feb 24, 2026
13Gemini 3 Flash Preview Supported
Google
75.4%source—Feb 18, 2026
14Claude Sonnet 4.6 Supported
Anthropic
75.2%source—Feb 21, 2026
15GPT-5.3 Codex Supported
OpenAI
74.8%source—Feb 25, 2026
16GLM-5.1 Estimated
Z.ai (Zhipu)
74.2%source—May 15, 2026
17GPT-5.2 Supported
OpenAI
73.8%source—Feb 12, 2026
18Kimi K2.5 Supported
Moonshot AI
73.8%source—Feb 17, 2026
19GPT-5 Supported
OpenAI
73.6%source—Feb 6, 2026
20Claude Opus 4.1 (latest) Supported
Anthropic
73.3%source—Feb 11, 2026
21Gemini 3 Pro Preview Supported
Google
72.9%source—Feb 13, 2026
22GLM-5 Supported
Z.ai (Zhipu)
72.1%source—Feb 15, 2026
23Claude Sonnet 4.5 (latest) Supported
Anthropic
71.3%source—Feb 5, 2026
24Claude-Opus-4 Supported
Anthropic
70.7%source—Feb 6, 2026
25GPT-5.1 Supported
OpenAI
68.0%source—Feb 18, 2026
26GPT-5 Mini Estimated
OpenAI
64.7%source—Feb 1, 2026
27o3 Supported
OpenAI
62.3%source—Feb 12, 2026
28Claude 3.7 Sonnet Supported
Anthropic
61.0%source—Feb 4, 2026
29Qwen3.6 Plus Estimated
Alibaba (Qwen)
57.9%source—May 14, 2026
30Gemini 2.5 Pro Supported
Google
57.6%source—Feb 13, 2026
31GPT-4.1
OpenAI
48.5%source—Feb 8, 2026
32GPT-4o (2024-08-06) Supported
OpenAI
31.0%source—Feb 11, 2026

For models without a SWE-bench Verified result, the index estimates one from Epoch’s Capabilities Index using a logistic fit on 31 models (residual SD 0.22 on the logit scale). Estimates are labelled and widen the model’s range.