SShogo Bench

OTIS Mock AIME

Competition math at AIME level.

Agent Index weight
0%
shown for context only
Models tested
202
run by Epoch AI on a shared harness
Normalised for the index
0 → 0, 1 → 100
Official site

Results

#ModelScoreNormalisedHarnessCost/taskDate
1Claude Fable 5 Supported
Anthropic
100.0%source—Aug 6, 2026
2Claude Fable 5.1 Supported
Anthropic
100.0%source—Sep 1, 2026
3Claude Opus 5.5 Supported
Anthropic
100.0%source—Sep 22, 2026
4Claude Sonnet 5.5 Estimated
Anthropic
100.0%source—Sep 29, 2026
5Gemini 4 Argon Supported
Google
100.0%source—Oct 13, 2026
6Gemini 4 Carbon Supported
Google
100.0%source—Nov 2, 2026
7Gemini 4 Hydrogen Estimated
Google
100.0%source—Nov 13, 2026
8GPT-5.5 Supported
OpenAI
100.0%source—Apr 24, 2026
9GPT-5.5 Pro
OpenAI
100.0%source—Apr 24, 2026
10GPT-5.6 Sol Supported
OpenAI
100.0%source—Jul 9, 2026
11GPT-6.1 Sol Estimated
OpenAI
100.0%source—Sep 29, 2026
12GPT-6.1 Terra Estimated
OpenAI
100.0%source—Oct 26, 2026
13GPT-6.2 Astra
OpenAI
100.0%source—Oct 27, 2026
14GPT-6 Astra Supported
OpenAI
100.0%source—Aug 30, 2026
15GPT-6 Sol Estimated
OpenAI
100.0%source—Sep 22, 2026
16Kimi K3.1 Pro
Moonshot AI
100.0%source—Nov 6, 2026
17Qwen3.8 Max 0902
Alibaba (Qwen)
100.0%source—Sep 2, 2026
18Claude Fable 5.2 Supported
Anthropic
99.7%source—Nov 6, 2026
19GPT-5.6 Terra Estimated
OpenAI
99.7%source—Jul 9, 2026
20DeepSeek V4.1 Flash 1017 Estimated
DeepSeek
99.4%source—Oct 19, 2026
21Mistral Large 4
Mistral
99.4%source—Nov 2, 2026
22Muse Wonder Estimated
Meta
99.4%source—Oct 12, 2026
23Qwen3.8 Max Preview Estimated
Alibaba (Qwen)
99.4%source—Aug 4, 2026
24Doubao Seed Evolving (Nov 2026)
ByteDance Seed
99.2%source—Nov 2, 2026
25Grok 4.6 Supported
xAI
99.2%source—Aug 14, 2026
26Muse Spark 1.3 Estimated
Meta
99.2%source—Sep 17, 2026
27Claude Haiku 5.5 Estimated
Anthropic
98.9%source—Oct 7, 2026
28Claude Opus 5 Supported
Anthropic
98.9%source—Jul 24, 2026
29Gemini 3.8 Flash Supported
Google
98.9%source—Sep 2, 2026
30GPT-6 Luna
OpenAI
98.9%source—Sep 22, 2026
31DeepSeek V4 Pro 0813
DeepSeek
98.6%source—Aug 18, 2026
32Claude Opus 4.8 Supported
Anthropic
98.3%source—Jun 7, 2026
33DeepSeek V4.1 Flash Estimated
DeepSeek
98.3%source—Oct 8, 2026
34GPT-5.6 Luna Estimated
OpenAI
98.3%source—Jul 9, 2026
35Grok 4.7 Estimated
xAI
98.1%source—Sep 22, 2026
36Muse Spark Open
Meta
98.1%source—Oct 15, 2026
37Claude Opus 4.7 Supported
Anthropic
97.8%source—Apr 17, 2026
38GPT-5.4 Supported
OpenAI
97.8%source—Jul 15, 2026
39Grok 4.5 Estimated
xAI
97.8%source—Jul 8, 2026
40Gemini 3.7 Flash Estimated
Google
97.2%source—Aug 14, 2026
41Kimi K3 Supported
Moonshot AI
97.2%source—Jul 16, 2026
42DeepSeek V4 Pro Estimated
DeepSeek
96.7%source—Jun 17, 2026
43Doubao Seed 2.1 Turbo
ByteDance Seed
96.1%source—Oct 30, 2026
44GPT-5.2 Supported
OpenAI
96.1%source—Dec 11, 2025
45Kimi K2.6 Estimated
Moonshot AI
96.1%source—May 2, 2026
46Gemini 3.1 Pro Preview Supported
Google
95.6%source—Feb 20, 2026
47Doubao Seed 2.1 Pro
ByteDance Seed
95.6%source—Oct 30, 2026
48Gemini 3.5 Flash Supported
Google
95.6%source—May 25, 2026
49Gemini 3 Flash Preview Supported
Google
95.6%source—Aug 6, 2026
50Kimi K2.7 Code Estimated
Moonshot AI
95.6%source—Aug 7, 2026
51Qwen3.7 Max
Alibaba (Qwen)
95.6%source—Aug 7, 2026
52Claude Sonnet 5 Supported
Anthropic
94.7%source—Jul 1, 2026
53Claude Opus 4.6 Supported
Anthropic
94.4%source—Feb 6, 2026
54DeepSeek V4 Flash 0731
DeepSeek
94.4%source—Aug 2, 2026
55Gemini 3.6 Flash Estimated
Google
94.2%source—Aug 2, 2026
56Doubao-Seed-2.0 Pro
ByteDance Seed
93.9%source—Oct 30, 2026
57GLM-5.3-Flash Supported
Z.ai (Zhipu)
93.9%source—Aug 26, 2026
58GLM-5.1 Estimated
Z.ai (Zhipu)
93.3%source—Aug 10, 2026
59Grok 4.3 Beta
xAI
93.3%source—Jun 17, 2026
60Qwen3.6 Plus Estimated
Alibaba (Qwen)
93.3%source—Aug 7, 2026
61Qwen3.7 Plus
Alibaba (Qwen)
93.3%source—Aug 7, 2026
62Grok 4.20 (Non-Reasoning) Estimated
xAI
92.2%source—Jul 13, 2026
63Kimi K2.5 Supported
Moonshot AI
92.2%source—Feb 2, 2026
64Gemini 3 Pro Preview Supported
Google
91.4%source—Nov 19, 2025
65GPT-5 Supported
OpenAI
91.4%source—Oct 29, 2025
66Doubao Seed 2.0 Lite (Apr 2026)
ByteDance Seed
91.1%source—Oct 30, 2026
67GLM-5.3 Supported
Z.ai (Zhipu)
91.1%source—Aug 24, 2026
68Qwen3.6 27B
Alibaba (Qwen)
91.1%source—Aug 7, 2026
69Qwen3.6 Max Preview Estimated
Alibaba (Qwen)
91.1%source—Aug 7, 2026
70Inkling Small
thinkingmachines
90.0%source—Aug 14, 2026
71Muse Spark
Meta
88.9%source—Apr 8, 2026
72GPT-5.4 mini
OpenAI
88.9%source—Aug 7, 2026
73GPT OSS 120B Supported
OpenAI
88.9%source—Dec 11, 2025
74Inkling Estimated
thinkingmachines
88.9%source—Aug 5, 2026
75Qwen3.5 397B-A17B
Alibaba (Qwen)
88.9%source—Aug 7, 2026
76GPT-5.1 Supported
OpenAI
88.6%source—Nov 13, 2025
77DeepSeek V3.2 Supported
DeepSeek
87.8%source—Dec 16, 2025
78GPT-5.4 nano
OpenAI
87.8%source—Apr 14, 2026
79GPT-5 Mini Estimated
OpenAI
86.7%source—Oct 30, 2025
80Nemotron 3 Ultra
NVIDIA
86.7%source—Aug 10, 2026
81Qwen3 235B A22B Thinking 2507
Alibaba (Qwen)
86.7%source—Dec 10, 2025
82Qwen3.5 Plus Estimated
Alibaba (Qwen)
86.7%source—Aug 7, 2026
83Qwen3.6 35B-A3B
Alibaba (Qwen)
86.7%source—Aug 7, 2026
84Qwen3.7 Flash
Alibaba (Qwen)
86.7%source—Aug 7, 2026
85GLM-5.2 Supported
Z.ai (Zhipu)
86.4%source—Jun 25, 2026
86Claude Opus 4.5 (latest) Supported
Anthropic
86.1%source—Nov 24, 2025
87Claude Sonnet 4.6 Supported
Anthropic
85.8%source—Feb 20, 2026
88Gemini 2.5 Pro Supported
Google
84.7%source—Nov 16, 2025
89o3 Supported
OpenAI
84.4%source—Aug 7, 2026
90Qwen3.5 Flash
Alibaba (Qwen)
84.4%source—Aug 7, 2026
91Qwen3.6 Flash
Alibaba (Qwen)
84.4%source—Aug 7, 2026
92Grok 4 Supported
xAI
84.0%source——
93GLM-4.7 Estimated
Z.ai (Zhipu)
83.3%source—Jan 29, 2026
94Kimi K2 Thinking Estimated
Moonshot AI
83.1%source—Nov 13, 2025
95Gemma 4 26B A4B
Google
82.2%source—Aug 28, 2026
96o4-mini Estimated
OpenAI
81.7%source—Apr 16, 2025
97GPT-5 Nano
OpenAI
81.1%source—Oct 31, 2025
98Gemini 3.1 Flash Lite
Google
80.0%source—Aug 6, 2026
99GLM-5 Supported
Z.ai (Zhipu)
80.0%source—Feb 12, 2026
100Claude Sonnet 4.5 (latest) Supported
Anthropic
77.8%source—Oct 28, 2025
101Grok-3 mini
xAI
77.8%source—Apr 10, 2025
102o3-mini Estimated
OpenAI
76.9%source—Feb 27, 2025
103Gemma 4 31B IT
Google
73.3%source—Aug 6, 2026
104o1 Estimated
OpenAI
73.3%source—Feb 27, 2025
105Qwen3 Max Estimated
Alibaba (Qwen)
73.3%source—Oct 6, 2025
106Gemini 2.5 Flash Estimated
Google
73.1%source—Apr 22, 2025
107Claude Sonnet 4 Supported
Anthropic
71.1%source—May 22, 2025
108Gemini 3.5 Flash Lite
Google
71.1%source—Aug 6, 2026
109MiniMax-M3 Estimated
MiniMax
71.1%source—Aug 10, 2026
110Qwen3-30B-A3B-Thinking (Jul 2025)
Alibaba (Qwen)
70.3%source—Aug 30, 2026
111Qwen3.5 35B-A3B
Alibaba (Qwen)
70.0%source—Aug 30, 2026
112Doubao-Seed 1.6
ByteDance Seed
69.4%source—Oct 30, 2026
113Claude Opus 4.1 (latest) Supported
Anthropic
68.9%source—Aug 5, 2025
114GPT Chat Latest
OpenAI
68.1%source—Aug 2, 2026
115seed-oss-36b-instruct
Unknown
67.5%source—Aug 28, 2026
116Qwen3 32B
Alibaba (Qwen)
66.9%source—Aug 30, 2026
117Claude Haiku 4.5 Estimated
Anthropic
66.7%source—Oct 22, 2025
118DeepSeek-R1 Estimated
DeepSeek
66.4%source—May 29, 2025
119Qwen3 14B
Alibaba (Qwen)
66.4%source—Aug 28, 2026
120GPT OSS 20B
OpenAI
65.3%source—Aug 27, 2026
121Claude-Opus-4 Supported
Anthropic
64.4%source—May 28, 2025
122Qwen3 30B A3B
Alibaba (Qwen)
62.8%source—Aug 28, 2026
123Qwen3 30B A3B Instruct 2507
Alibaba (Qwen)
62.2%source—Aug 30, 2026
124Qwen3.5 9B
Alibaba (Qwen)
61.7%source—Aug 27, 2026
125Qwq 32B
Alibaba (Qwen)
59.2%source—Aug 28, 2026
126GLM-4.7-Flash
Z.ai (Zhipu)
58.3%source—Aug 30, 2026
127Claude 3.7 Sonnet Supported
Anthropic
57.8%source—Mar 13, 2025
128Gemini 2.0 Flash Thinking (Jan 2025)
Google
57.8%source—Mar 7, 2025
129Qwen3 8B
Alibaba (Qwen)
56.1%source—Aug 27, 2026
130Qwen3.5 4B
Alibaba (Qwen)
55.8%source—Aug 28, 2026
131DeepSeek-R1-Distill-Qwen-32B
DeepSeek
55.6%source—Aug 28, 2026
132Grok 3
xAI
55.6%source—Apr 10, 2025
133qwen3-4b-instruct-2507
Unknown
52.2%source—Aug 28, 2026
134DeepSeek-R1-Distill-Llama-70B
DeepSeek
51.4%source—Mar 7, 2025
135DeepSeek-R1-Distill-Qwen-14B
DeepSeek
50.6%source—Aug 28, 2026
136o1-mini
OpenAI
46.9%source—Mar 6, 2025
137GPT-4.1 mini
OpenAI
44.7%source—Apr 14, 2025
138DeepSeek R1 0528 Qwen3 8B
DeepSeek
43.9%source—Aug 28, 2026
139GPT-4.1
OpenAI
38.3%source—Apr 14, 2025
140DeepSeek V3 0324
DeepSeek
37.8%source—Apr 1, 2025
141GPT-4.5
OpenAI
37.8%source—Feb 28, 2025
142Mistral Medium 3
Mistral
32.2%source—May 7, 2025
143Gemini-2.0-Flash
Google
31.1%source—Feb 25, 2025
144Mistral Small 3.2
Mistral
30.3%source—Aug 30, 2026
145Magistral Small 1.0
Mistral
30.0%source—Aug 6, 2026
146GPT-4.1 nano
OpenAI
28.9%source—Apr 14, 2025
147Magistral Small 1.2
Mistral
28.1%source—Aug 30, 2026
148Gemini 1.5 Pro (Feb 2024)
Google
23.1%source—Feb 25, 2025
149Gemma 3 27B
Google
22.5%source—Aug 27, 2026
150DeepSeek-R1-Distill-Qwen-1.5B
DeepSeek
21.4%source—Aug 28, 2026
151Llama 4 Maverick
Meta
20.6%source—Apr 8, 2025
152Qwen Plus
Alibaba (Qwen)
17.8%source—Apr 7, 2025
153Gemma 3 12B
Google
16.7%source—Aug 28, 2026
154Gemini 1.5 Flash (Sep 2024)
Google
16.3%source—Feb 25, 2025
155Qwen2.5-Max
Alibaba (Qwen)
16.1%source—Apr 1, 2025
156DeepSeek-V3
DeepSeek
15.8%source—Feb 25, 2025
157Phi 4
Microsoft
13.8%source—Feb 25, 2025
158Grok-2 (Dec 2024)
xAI
11.5%source—Feb 25, 2025
159Llama 3.1-405B
Meta
9.7%source—Feb 25, 2025
160Claude-Sonnet-3.5 Estimated
Anthropic
8.5%source—Feb 25, 2025
161Mistral Large 2 (Nov 2024)
Mistral
8.5%source—Feb 25, 2025
162Qwen2.5-72B Estimated
Alibaba (Qwen)
8.1%source—Feb 25, 2025
163Qwen3-1.7B
Alibaba (Qwen)
8.1%source—Aug 28, 2026
164Llama 4 Scout
Meta
7.8%source—Apr 8, 2025
165Gemma 3 4B
Google
7.5%source—Aug 28, 2026
166Qwen2.5-32B
Alibaba (Qwen)
7.4%source—Feb 25, 2025
167GPT-4o mini
OpenAI
6.9%source—Jul 30, 2025
168GPT-4 Turbo
OpenAI
6.7%source—Feb 27, 2025
169Mistral Small 3
Mistral
6.7%source—Aug 30, 2026
170GPT-4o (2024-08-06) Supported
OpenAI
6.4%source—Feb 25, 2025
171Qwen Turbo
Alibaba (Qwen)
6.1%source—Apr 7, 2025
172Mistral Small 3.1
Mistral
5.8%source—Mar 18, 2025
173Llama 3.3 70B
Meta
5.1%source—Feb 25, 2025
174Claude 3 Opus Estimated
Anthropic
4.7%source—Feb 25, 2025
175Gemini 1.5 Flash 8B
Google
4.6%source—Feb 25, 2025
176Tulu 3 (Tülu 3) 70B
Allen Institute for AI
4.4%source—Mar 7, 2025
177Claude 3.5 Haiku
Anthropic
4.3%source—Feb 25, 2025
178Llama 3-70B
Meta
4.3%source—Feb 25, 2025
179Llama 3.1-70B
Meta
3.6%source—Feb 25, 2025
180granite-4.0-micro
Unknown
2.8%source—Aug 30, 2026
181Llama 3.2 90B
Meta
2.6%source—Feb 25, 2025
182Claude 2
Anthropic
2.5%source—Mar 7, 2025
183Claude 3 Sonnet
Anthropic
2.5%source—Feb 25, 2025
184Hermes 2 Theta Llama-3 70B
Nous Research
2.5%source—Apr 1, 2025
185Qwen2.5-7B
Alibaba (Qwen)
2.5%source—Aug 30, 2026
186GPT-3.5-turbo
OpenAI
2.2%source—Aug 7, 2026
187Claude 2.1
Anthropic
1.9%source—Mar 7, 2025
188Llama 3-8B
Meta
1.9%source—Aug 30, 2026
189Mistral Large
Mistral
1.9%source—Feb 25, 2025
190Claude-Haiku-3
Anthropic
1.8%source—Feb 25, 2025
191Llama 3.1-8B
Meta
1.7%source—Aug 27, 2026
192Gemma 2 27B
Google
1.4%source—Mar 7, 2025
193Gemini 1.0 Pro
Google
1.1%source—Feb 25, 2025
194Gemma 3 1B
Google
1.1%source—Aug 28, 2026
195GPT-4
OpenAI
1.1%source—Oct 23, 2025
196DeepSeek LLM 67B
DeepSeek
0.8%source—Aug 28, 2026
197granite-4.0-1b
Unknown
0.8%source—Aug 30, 2026
198Gemma 2 9B
Google
0.6%source—Mar 7, 2025
199Llama 3.2 1B Instruct
Meta
0.6%source—Aug 30, 2026
200granite-4.0-350m
Unknown
0.3%source—Aug 28, 2026
201Mistral 7B v0.3
Mistral
0.3%source—Aug 30, 2026
202Llama 2-70B
Meta
0.0%source—Feb 25, 2025