Benchmark

Definition

A benchmark is a curated collection of test cases with defined evaluation metrics and methodology, enabling standardized comparison of model or system performance. Benchmarks serve two purposes: external benchmarks (MMLU, HumanEval, HELM, SWE-bench) enable cross-model comparison on publicly agreed-upon tasks; internal benchmarks measure performance on a specific organization's actual use cases. Both are essential: external benchmarks indicate general capability; internal benchmarks predict production performance.

Engineering Context

External benchmarks (MMLU, HumanEval, HELM) measure general model capabilities but often don't predict task-specific performance. Build internal benchmarks for your specific use case. A strong internal benchmark: 200+ examples covering the full input distribution, automated scoring (avoid manual evaluation for anything run repeatedly), and historical baselines to detect regression. Beware of benchmark contamination: models trained on data that includes benchmark test sets will score artificially high. For model selection, always run candidate models against your internal benchmark rather than relying solely on published numbers. Track benchmark scores over time as a leading indicator of system health.

Related Terms