Commissioned, Curated and Published by Russ. Researched and written with AI.
The headline narrative around open-weight AI models is that they have caught up with frontier cloud models. The reality is more specific than that – and more useful for making deployment decisions.
The gap has narrowed significantly on some dimensions, closed almost entirely on others, and remains real on a distinct set of tasks. Understanding which is which is the actual engineering question.
The benchmark picture
| Model | Type | SWE-bench | GPQA Diamond | MMLU | MATH-500 |
|---|---|---|---|---|---|
| GPT-5.4 | Cloud | – | 83.9% | 87–88 | – |
| Claude Opus 4.6 | Cloud | 80.8% | 87.4% | – | – |
| Claude Sonnet 4.6 | Cloud | 79.6% | 74.1%* | – | – |
| Llama 4 Maverick | Open-weight | – | 69.8% | 85.5 | – |
| Llama 4 Scout | Open-weight | – | 69.8% | 80.5† | – |
| Qwen 3 72B | Open-weight | – | – | 83.1 | – |
| DeepSeek R1 | Open-weight | – | – | – | 97.3% |
* No standard academic benchmarks (MMLU-Pro, AIME) have been independently published for Claude Sonnet 4.6. GPQA Diamond figure from nxcode.io.
† MMLU Pro score, not standard MMLU.
Sources: GPT-5.4 GPQA Diamond from mindstudio.ai head-to-head (March 2026); MMLU from historical GPT-4o data (GPT-5.4 standard MMLU not separately published). Claude Sonnet 4.6 and Opus 4.6 SWE-bench from nxcode.io, digitalapplied.com, morphllm.com; Opus 4.6 GPQA Diamond from mindstudio.ai. Llama 4 from allaboutai.com and getbind.co – Maverick’s claims come largely from Meta’s own LMarena benchmarks. Qwen 3 72B from SitePoint. DeepSeek R1 MATH-500 from blog.getbind.co.
The gap on GPQA Diamond – graduate-level science and engineering questions, one of the hardest benchmarks – tells an interesting story: Claude Opus 4.6 (87.4%) actually outperforms GPT-5.4 (83.9%), while open-weight Llama 4 (69.8%) is a meaningful step behind. On SWE-bench (real coding tasks on GitHub repositories), Sonnet 4.6 (79.6%) and Opus 4.6 (80.8%) lead, with open-weight models not yet publishing comparable numbers on this benchmark.
Where open-weight has genuinely closed the gap
Graduate-level reasoning (GPQA Diamond). Llama 4 at 69.8% sits about 4-5 points behind Claude Sonnet 4.6 (74.1%), and further behind Claude Opus 4.6 (87.4%) and GPT-5.4 (83.9%). The gap is real but narrower than it was – and on this specific benchmark, Opus 4.6 actually leads GPT-5.4. This is the hardest category in the table – graduate-level science and engineering questions – and open-weight is competitive.
Mathematics. DeepSeek R1’s MATH-500 score of 97.3% is frontier-level. For tasks involving mathematical reasoning or quantitative analysis, a locally-run DeepSeek R1 is not a meaningful step down from cloud.
Multimodal reasoning. Llama 4 Maverick pushes into the low-70s on MMMU (multimodal academic reasoning), edging past GPT-4o. If your use case involves image and text together, Maverick is worth testing seriously.
General knowledge. The MMLU gap of 3 to 5 points translates to real differences at the edges, but for the middle 80% of general knowledge queries, users in practice cannot distinguish the outputs.
Where frontier still wins clearly
First-attempt code correctness. A comparison across 500 real pull requests found Claude Sonnet getting code right on the first attempt more often than DeepSeek R1, which generates longer reasoning chains but does not convert them into higher first-pass accuracy. Claude Sonnet 4.6’s SWE-bench Verified score of 79.6% – real GitHub coding tasks, not synthetic exercises – versus the open-weight field where comparable numbers aren’t yet published reflects a gap that shows up in practice.
Ambiguous and vague prompts. Claude handles under-specified inputs better. When the prompt is weak or the task is not fully defined, frontier models are more likely to produce a sensible interpretation. Open-weight models are more likely to take a literal wrong turn. This matters for agentic workflows where inputs are not always clean.
Long context. DeepSeek R1 has a 64K token context window. GPT-4o and Claude support much larger contexts, which matters for large codebase analysis, long document processing, or multi-step agent tasks.
Consistency under distribution shift. Frontier models tend to degrade more gracefully on unusual, poorly-formatted, or edge-case inputs. Open-weight models can fail harder when inputs fall outside their fine-tuning distribution.
The benchmark vs reality problem
Llama 4 is the useful case study. Meta’s benchmark numbers show it beating GPT-4o and DeepSeek V3 on reasoning and coding. Independent evaluators found a different story.
Rootly benchmarked Llama 4 specifically against coding-centric models and found it underperforms relative to its headline numbers. The divergence is not unusual – models can be fine-tuned to perform well on benchmark categories without those gains generalising to the real task distribution. HumanEval, in particular, is a narrow benchmark that does not fully capture real-world coding reliability.
This does not mean Llama 4 is bad. It means benchmark scores are a starting point, not a conclusion. The right question is whether the model performs on your specific task, with your data, in your infrastructure. That requires testing, not table-reading.
The practical 80/20
For engineering teams evaluating local inference, the honest answer is roughly an 80/20 split.
For 80% of real engineering tasks – code generation from clear specifications, structured extraction, classification, summarisation, Q&A over documentation, test generation – the gap between Qwen 3 72B or DeepSeek R1 and a frontier model is small enough to test locally before committing to API spend. On many of these tasks, the gap will be negligible in practice.
For the remaining 20% – tasks requiring frontier-level reasoning on ambiguous inputs, first-attempt coding reliability where rework is expensive, complex multi-step reasoning where errors compound, and tasks requiring very long context windows – frontier models are materially better and the gap matters.
The 20% is real. But it is smaller than most teams assume before they test. Default assumptions about frontier necessity are often based on the hardest tasks, applied to the full workload.
For the hardware options for running these models locally, see the hardware comparison. For the broader architectural argument, see the local AI moment piece.