WeirdML v3
Unusual ML tasks solved by writing and running PyTorch code.
Agent Index weight
0%
shown for context only
Models tested
27
from the benchmark’s leaderboard and reports
Results
| # | Model | Score | Normalised | Harness | Cost/task | Date |
|---|---|---|---|---|---|---|
| 1 | Claude Fable 5.2 Supported Anthropic | 43.1% | claude_code | — | — | |
| 2 | GPT-6 Astra Supported OpenAI | 42.2% | codex_cli | — | — | |
| 3 | GPT-6.1 Sol Estimated OpenAI | 38.3% | codex_cli | — | — | |
| 4 | Claude Opus 5.5 Supported Anthropic | 31.2% | claude_code | — | — | |
| 5 | GPT-6.1 Terra Estimated OpenAI | 28.1% | codex_cli | — | — | |
| 6 | Claude Fable 5.1 Supported Anthropic | 26.0% | claude_code | — | — | |
| 7 | Gemini 4 Argon Supported Google | 20.3% | gemini_cli | — | — | |
| 8 | Claude Sonnet 5.5 Estimated Anthropic | 19.9% | claude_code | — | — | |
| 9 | GPT-6 Sol Estimated OpenAI | 19.7% | codex_cli | — | — | |
| 10 | Claude Opus 5 Supported Anthropic | 19.0% | claude_code | — | — | |
| 11 | Claude Fable 5 Supported Anthropic | 15.6% | claude_code | — | — | |
| 12 | Muse Wonder Estimated Meta | 15.1% | codex_cli | — | — | |
| 13 | GPT-5.6 Sol Supported OpenAI | 15.0% | codex_cli | — | — | |
| 14 | Claude Haiku 5.5 Estimated Anthropic | 11.7% | claude_code | — | — | |
| 15 | Kimi K3.1 Pro Moonshot AI | 9.9% | kimi_code | — | — | |
| 16 | DeepSeek V4.1 Flash 1017 Estimated DeepSeek | 9.1% | codex_cli | — | — | |
| 17 | Grok 4.7 Estimated xAI | 8.5% | codex_cli | — | — | |
| 18 | GPT-6 Luna OpenAI | 7.9% | codex_cli | — | — | |
| 19 | Kimi K3 Supported Moonshot AI | 7.5% | kimi_code | — | — | |
| 20 | Gemini 3.8 Flash Supported Google | 7.4% | gemini_cli | — | — | |
| 21 | Muse Spark 1.3 Estimated Meta | 6.7% | codex_cli | — | — | |
| 22 | Claude Opus 4.5 (latest) Supported Anthropic | 6.5% | claude_code | — | — | |
| 23 | DeepSeek V4.1 Flash Estimated DeepSeek | 5.9% | opencode | — | — | |
| 24 | GPT-5.6 Luna Estimated OpenAI | 5.2% | codex_cli | — | — | |
| 25 | GPT-5.2 Supported OpenAI | 5.0% | codex_cli | — | — | |
| 26 | Mistral Large 4 Mistral | 4.2% | opencode | — | — | |
| 27 | Inkling Estimated thinkingmachines | 3.9% | opencode | — | — |