Epoch Capabilities Index

Epoch Capabilities Index: the explanation

ECI links benchmarks through models that have been evaluated on more than one of them. These overlapping results allow a statistical method to estimate relative test difficulty and place models on a common capability scale. It is therefore not simply an average of published percentages. Performance on a difficult test can provide different information from performance on an easier one. The [official ECI overview](https://epoch.ai/data/eci-documentation) explains how overlapping benchmark observations are combined to support comparisons between models with different evaluation records.

The index helps track capabilities when available evaluations differ or older tests become less informative. Recent models might solve almost every task on an established benchmark while still struggling with a newer one. A shared scale attempts to connect those observations rather than treating each test as an isolated result. For you, the benefit is a broad reference point before examining detailed evidence. That reference describes capabilities represented in the available evaluation data, not every characteristic that might make an assistant useful.

ECI values use a conventional statistical scale. They are neither universal success percentages nor direct measurements of workplace productivity. Estimates can change when additional models or benchmark results enter the calculation. The index also inherits limitations from its underlying evaluations, including differences in testing conditions and possible contamination. A specialized model may suit your field particularly well without receiving the highest general score. You should therefore examine relevant task results and practical operating requirements before turning the index into a selection decision.

A concrete example

You compare models for an assistant that will summarize documents and answer questions. A higher ECI gives one model an initial advantage as a broad capability signal. You then inspect language support and test representative documents from your workflow. Another model may ultimately suit your response-time or budget constraints better. The index helps organize your comparison, but it does not predict the complete experience of using the finished assistant.

Why it matters

ECI can make comparisons easier when models have been tested on different benchmark collections. Keep the date of your comparison because estimates may be revised as more data becomes available. When choosing an agent, also examine its tools, permissions and actual task execution. The index alone does not describe privacy arrangements, service pricing, response consistency or interface quality. It is one piece of evidence within a broader practical assessment.

Common misconceptions

• An ECI value is not the percentage of your tasks that a model will solve, because it places benchmark observations on a shared statistical scale rather than measuring success across a universal collection of everyday requests.
• Combining benchmarks does not remove their underlying limitations, because incomplete observations, different evaluation procedures or prior exposure to test material can still affect the evidence used to estimate and compare the capabilities of different models.

Related terms: measuring and comparing

benchmark

Benchmark is a structured test with defined rules for comparing system performance.

SWE-bench

SWE-bench is a benchmark for resolving software issues drawn from real projects.

GPQA Diamond

GPQA Diamond is a selected subset of GPQA containing difficult multiple-choice science questions.

FrontierMath

FrontierMath is an advanced mathematics benchmark developed by Epoch AI with mathematical specialists.

OSWorld

OSWorld is a family of benchmarks and environments for evaluating agents that perform computer tasks.

Terminal-Bench

Terminal-Bench is a collection of tests for measuring how well an AI agent completes tasks through a terminal, an interface controlled with text commands.

SimpleQA

SimpleQA is a benchmark for checking how accurately a language model answers short factual questions.

AIME (benchmark)

AIME (benchmark) refers to using problems from the American Invitational Mathematics Examination to evaluate language models.

Where this site covers it

The term appears on 9 pages of this site. A few of them:

• LLM benchmarks: how to read model rankings Article
• Claude Fable 5: pricing, context and benchmarks Model
• Claude Fable 5.1: pricing, context and benchmarks Model
• Claude Haiku 4.5: pricing, context and benchmarks Model
• Claude Haiku 5.5: pricing, context and benchmarks Model
• Claude Opus 4.1: pricing, context and benchmarks Model
• Claude Opus 4.5: pricing, context and benchmarks Model
• Claude Opus 4.6: pricing, context and benchmarks Model