Module 01 · 11 min

Benchmarks: Which Ones Matter and Which Ones Are Marketing

⚡This topic changes fast (models, prices, tools). The idea and the judgment don't expire, but double-check current names and prices. Last reviewed: July 2026.

Every time a new model drops, the announcement is full of charts: 'state of the art across 12 benchmarks,' '95% on such-and-such test.' It's easy to get dizzy and just pick the biggest number. This lesson will teach you how to read those numbers with a critical eye — because while many are legit, plenty are pure marketing.

What a Benchmark Is (and Why They Exist)

A benchmark is a standardized test for models: a set of tasks with known answers where you measure what percentage each model gets right. They help us compare models somewhat fairly. The problem is that once a test becomes famous, it starts getting corrupted.

Which benchmark matters for YOUR task?

Interactivo

What to measure

SWE-bench / real code

solving real repo issues, not toy puzzles

Tip: Ignore the 'general' score: look at who fixes real bugs. The coding specialist usually wins, not the most famous one.

A rule that never expires: define your task first, pick the model by how it performs ON THAT task, not by its fame.

A model's 'general' score is marketing. What actually matters is how it performs on YOUR specific task. Pick what you're going to do and notice what you should really be looking at — and why the 'best on average' doesn't always win at your thing.

The One That Matters Most for Building: SWE-bench

SWE-bench gives the model REAL GitHub issues and measures whether it can write the patch to fix them. That feels a lot closer to actual building than a multiple-choice test. It comes in two main flavors: SWE-bench Verified (Python tasks, more widely known) and SWE-bench Pro (harder, multiple languages and repos, resistant to cheating).

⚠️ The Data Contamination Problem

When a benchmark becomes super popular, its answers end up in the training data for new models. At that point, the model isn't SOLVING it — it's REMEMBERING it. By 2026, SWE-bench Verified became so contaminated that even OpenAI stopped reporting it, and scores above ~80% don't say much anymore. A sky-high score might just mean 'it memorized the test,' not 'it's better at building.'

The Other Trap: Self-Reported Numbers

Many leaderboards list scores submitted by the model maker itself, not an independent third party. On one famous leaderboard in 2026, out of 100 models listed, only ONE was independently verified; the other 99 were self-reported by the companies. That doesn't mean they're always lying, but you should take any self-awarded score with a grain of salt.

Benchmarks That Are Usually Just Marketing

Saturated general-knowledge tests (like MMLU): almost all top models are hitting the ceiling here, so a 0.5% difference makes zero difference to you.
'State of the art on ALMOST all benchmarks': take a close look at which ones they left out; companies cherry-pick the charts that make them look good.
Any round, spectacular number that doesn't state who measured it or under what conditions.

Worked Example: Two Models, Which One to Trust

Model A boasts a '95% on SWE-bench Verified.' Model B reports a '60% on SWE-bench Pro.' Beginners pick Model A because it has the bigger number. But Verified is contaminated and saturated above 80%, whereas Pro is harder and offers real differentiation between 23% and 69%. A 60% on Pro tells you MORE than a 95% on Verified. Looking at the number without knowing the test is a trap.

💡 The Only Test That Doesn't Lie: Your Own

The best benchmark is your own project. Take a real task from what you're building, run it through two or three models, and compare the results yourself. Five minutes of this will tell you more about which model actually works for you than any chart in an ad.

Key Takeaways

SWE-bench (especially Pro) is the benchmark that comes closest to real-world building.
Watch out for data contamination: a super high score might mean recall, not capability.
Be skeptical of self-reported scores that lack independent verification.
Saturated tests and 'state of the art on almost everything' are marketing red flags.
The ultimate test is running your own task through the models you're considering.

Don't pick a model based on the number in the ad. Pick it based on how it handles YOUR task. That's the only benchmark that actually pays the bills.

← Previous

The 2026 Model Landscape: Which One to Build With

How to Structure Prompts That Actually Work

Comentarios

Inicia sesión (arriba) para leer y participar en la conversación de la comunidad.