Aider Polyglot
225 hard Exercism exercises in six languages, edited through Aider.
Agent Index weight
0%
shown for context only
Models tested
45
from the benchmark’s leaderboard and reports
Results
| # | Model | Score | Normalised | Harness | Cost/task | Date |
|---|---|---|---|---|---|---|
| 1 | GPT-5 Supported OpenAI | 88.0% | source | — | — | |
| 2 | o3-pro OpenAI | 84.9% | source | — | — | |
| 3 | Gemini 2.5 Pro Supported Google | 83.1% | source | — | — | |
| 4 | o3 Supported OpenAI | 81.3% | source | — | — | |
| 5 | Grok 4 Supported xAI | 79.6% | source | — | — | |
| 6 | DeepSeek V3.2 Supported DeepSeek | 74.2% | source | — | — | |
| 7 | Claude-Opus-4 Supported Anthropic | 72.0% | source | — | — | |
| 8 | o4-mini Estimated OpenAI | 72.0% | source | — | — | |
| 9 | DeepSeek-R1 Estimated DeepSeek | 71.4% | source | — | — | |
| 10 | Claude 3.7 Sonnet Supported Anthropic | 64.9% | source | — | — | |
| 11 | o1 Estimated OpenAI | 61.7% | source | — | — | |
| 12 | Claude Sonnet 4 Supported Anthropic | 61.3% | source | — | — | |
| 13 | o3-mini Estimated OpenAI | 60.4% | source | — | — | |
| 14 | Qwen3 235B A22B Alibaba (Qwen) | 59.6% | source | — | — | |
| 15 | Qwen3 235B A22B Instruct 2507 Alibaba (Qwen) | 59.6% | source | — | — | |
| 16 | Kimi K2 0905 Estimated Moonshot AI | 59.1% | source | — | — | |
| 17 | DeepSeek V3 0324 DeepSeek | 55.1% | source | — | — | |
| 18 | Gemini 2.5 Flash Estimated Google | 55.1% | source | — | — | |
| 19 | Grok 3 xAI | 53.3% | source | — | — | |
| 20 | GPT-4.1 OpenAI | 52.4% | source | — | — | |
| 21 | Claude-Sonnet-3.5 Estimated Anthropic | 51.6% | source | — | — | |
| 22 | Grok-3 mini xAI | 49.3% | source | — | — | |
| 23 | DeepSeek-V3 DeepSeek | 48.4% | source | — | — | |
| 24 | GPT-4o (2024-08-06) Supported OpenAI | 45.3% | source | — | — | |
| 25 | GPT-4.5 OpenAI | 44.9% | source | — | — | |
| 26 | GPT OSS 120B Supported OpenAI | 41.8% | source | — | — | |
| 27 | Qwen3 32B Alibaba (Qwen) | 40.0% | source | — | — | |
| 28 | Gemini-2.0-Flash Google | 38.2% | source | — | — | |
| 29 | Gemini 2.0 Pro Google | 35.6% | source | — | — | |
| 30 | o1-mini OpenAI | 32.9% | source | — | — | |
| 31 | GPT-4.1 mini OpenAI | 32.4% | source | — | — | |
| 32 | Claude 3.5 Haiku Anthropic | 28.0% | source | — | — | |
| 33 | Qwen2.5-Max Alibaba (Qwen) | 21.8% | source | — | — | |
| 34 | Qwq 32B Alibaba (Qwen) | 20.9% | source | — | — | |
| 35 | Gemini 2.0 Flash Thinking (Jan 2025) Google | 18.2% | source | — | — | |
| 36 | DeepSeek-V2.5 (Sep 2024) DeepSeek | 17.8% | source | — | — | |
| 37 | Qwen2.5 Coder 32B Instruct Alibaba (Qwen) | 16.4% | source | — | — | |
| 38 | Llama 4 Maverick Meta | 15.6% | source | — | — | |
| 39 | Yi-Lightning 01.AI | 12.9% | source | — | — | |
| 40 | Cohere Command A Cohere | 12.0% | source | — | — | |
| 41 | Codestral 25.08 Mistral | 11.1% | source | — | — | |
| 42 | openhands-lm-32b-v0.1 Unknown | 10.2% | source | — | — | |
| 43 | GPT-4.1 nano OpenAI | 8.9% | source | — | — | |
| 44 | Gemma 3 27B Google | 4.9% | source | — | — | |
| 45 | GPT-4o mini OpenAI | 3.6% | source | — | — |