Journal
LLM Benchmarks Explained: MMLU, HumanEval and How to Read Them
MMLU and HumanEval are the two most quoted LLM benchmarks in the world. Here is what each actually measures, why scores are not comparable across vendors, and how to avoid the most common benchmark trap.
Every few months a new large language model ships with a press release full of benchmark numbers that look straight out of a video game: “MMLU 90.3%, HumanEval 96.2%, +11 points over the previous generation.” These numbers get quoted everywhere, and almost nowhere are they properly explained.
This article breaks down the two most cited benchmarks — MMLU and HumanEval — what each one actually measures, why the numbers are not directly comparable across vendors, and how to read a benchmark table without fooling yourself. You can follow the published scores for leading models in our LLM Knowledge & Coding Benchmarks dataset.
What MMLU actually measures
MMLU (Massive Multitask Language Understanding) is a multiple-choice knowledge test. It consists of about 15,000 questions spanning 57 subjects — from abstract algebra and astronomy to law, medicine and moral scenarios. Each question has four answer options, and the benchmark reports the model’s accuracy (the percentage answered correctly).
Think of MMLU as a breadth-of-knowledge exam with options to guess from. A random baseline scores ~25% (four options); a strong machine learning system of a decade ago scored ~30%; the first frontier models crossed 80–90% in 2023–2024, and the top models now hover in the low 90s — close to the practical ceiling of the benchmark as it was originally labeled.
Key properties to remember:
- It is knowledge, not reasoning. MMLU rewards memorized facts and, increasingly, pattern-matching against the internet’s store of model-answer content.
- It is multiple choice. The answer is always present; the task is selection, not generation.
- It saturates. Once good models hit the mid-90s, the benchmark stops distinguishing between them — which is why the community moved to harder forks like MMLU-Pro.
What HumanEval actually measures
HumanEval is a code-generation benchmark built by OpenAI: 164 hand-written Python programming problems — function signatures with docstrings describing the expected behavior. The model must generate a complete function, and the solution is verified by executing it against hidden unit tests.
The reported metric is pass@1: the probability that a single generated solution passes all tests. In practice, because generation has randomness, it is estimated over multiple samples.
HumanEval catches things multiple choice can’t:
- Fluency matters. The model must produce syntactically valid, runnable code.
- Semantics matter. The code must do the right thing, verified by execution, not by a grader’s judgment.
- Under-specification matters. Real problems have edge cases (empty input, unbounded integers) the docstring only implies.
The flip side: with only 164 problems, HumanEval is a small, known dataset — it leaks into training corpora, and every vendor tunes against it. Scores in the 90s from different companies tell you less than the same score from different models five years ago. As the dataset aged, the field moved to more robust and harder suites such as HumanEval+ (with added test cases), LiveCodeBench (held-out, time-stamped problems) and SWE-bench (real GitHub issues).
MMLU vs HumanEval: the real difference
| Dimension | MMLU | HumanEval |
|---|---|---|
| Modality | Knowledge & language understanding | Code generation |
| Format | Multiple choice (57 subjects, ~15k items) | Function synthesis (164 problems) |
| Scoring | % accuracy vs labeled answers | pass@1: % passing hidden unit tests |
| Random baseline | ~25% (4 options) | ~0% |
| Frontier range (2025–26) | ~85–93% | ~85–97% |
| Main weakness | Saturation, memorization | Small size, train/test contamination |
The two benchmarks measure orthogonal capabilities: one is “how much do you know,” the other is “can you produce working software.” A model can be excellent at MMLU and mediocre at HumanEval, or vice versa. This is precisely why our LLM leaderboard shows both columns side by side instead of a single aggregate score.
The benchmark trap: why cross-vendor scores are not comparable
This is the part almost nobody reads in the footnotes, and it matters:
- Self-reported vs independently measured. Most published scores come from the vendor’s own evaluation pipeline. Prompt wording, sampling temperature, the number of attempts and the exact item set all shift results by a few points — sometimes more.
- Contamination. Frontier models train on the open web, and every popular benchmark is on it. Massive-but-public MMLU leaks; small HumanEval leaks worse.
- Version drift. “MMLU” is not a single number: there is 5-shot, 0-shot, in-context vs fine-tuned, Pro forks, and each produces a different value.
- Ceiling effects. At 93% vs 91%, the difference is closer to noise than to “3% smarter.”
The safe way to read any table (including ours — we report scores exactly as published, without re-testing): treat vendor-reported numbers as upper bounds of capability under favorable conditions, prefer independent evals (ARTEL, HELM, LMArena) when you need to compare, and always demand the evaluation setup alongside the score.
How LLM benchmarks fit the AxioStats matrix
Benchmarks live in the Tech & AI × Time-Series & Hard Data cell of the matrix — published, structured, vendor-reported facts with explicit methodology. To explore:
- LLM Knowledge & Coding Benchmarks — MMLU & HumanEval leaderboard
- Framework Weekly Downloads — who developers actually use
- Wikipedia Pageviews: Artificial Intelligence — public interest in AI
More reading on the blog:
FAQ
Which benchmark is better: MMLU or HumanEval? Neither — they measure different things. If you care about general knowledge breadth, MMLU; if you care about producing working code, HumanEval (or better, its harder successors like LiveCodeBench or SWE-bench).
Why is HumanEval so small? It was designed in 2021 as a minimal, hand-curated probe (164 problems). Its virtue — human-written, unambiguous — is also its flaw: it cannot be expanded cheaply and it leaks into training data.
What score should a good coding model get on HumanEval? Reaching 90%+ is table stakes for a frontier coding model in 2026. The discriminating benchmarks are the harder, dynamically-updated ones: HumanEval+ (extra tests), LiveCodeBench (temporally held out) and SWE-bench (real-world issue resolution).