Skip to main content
METR's AI Graph Update: Abstract network of interconnected nodes and lines, symbolizing complex data evaluations.

Editorial illustration for METR's AI Graph Update Adds Hard-to-Score New Evaluations

METR's AI Graph Update Adds Hard Evaluations

• 2 min read

METR dropped an update this week to the chart that's been bouncing around AI Twitter for months, the one showing how long a task an AI model can complete without human help, doubling roughly every four months. The nonprofit's original time horizon plot became a shorthand for measuring progress toward autonomous AI agents, but it also got flattened into takes that stripped out most of the nuance in how it was built. The new version tries to correct course by adding tasks that take human testers far longer to complete than anything in the first batch, plus a set of more open-ended problems that don't have a clean pass-fail answer.

That last part matters. METR built its original rankings on tasks with a clear right answer, the kind a script can grade automatically. Coding problems work well for this.

Real-world judgment calls don't. The organization's own researchers acknowledge this gap directly, framing the update around a question that's been nagging at AI labs and forecasters alike: does progress on checkable, verifiable tasks actually transfer to the messier work where nobody can simply check the answer.

A new report from the AI evaluation nonprofit METR finds that some AI capabilities are continuing to improve along an exponential trajectory—but it also indicates that models aren’t quite as good at tasks that require judgment as they are at coding.

Why this matters

METR's time-horizon chart has become the de facto scoreboard for "how close are we," and that's always been a little dangerous given how narrow the underlying tasks are. This update is an admission of that limit: coding benchmarks are easy to score and easy to game with more compute, but judgment-heavy work doesn't compress into a clean curve. For developers and founders leaning on these numbers to time product bets or fundraising narratives, the lesson is to stop treating one exponential line as a proxy for general capability.

Sayash Kapoor's point about evaluations needing to get "more creative" is really an admission that the field's measurement tools are lagging the systems they're meant to measure. If METR, one of the more careful shops in this space, is struggling to score and interpret its own new tasks, anyone citing time-horizon growth as proof of imminent AGI should slow down. We'd rather see messier, harder-to-graph benchmarks that actually probe judgment than another smooth line that flatters coding agents while saying nothing about whether they can be trusted to decide anything.

Common Questions Answered

What is METR's time horizon chart and why has it become important in AI development?

METR's time horizon chart tracks how long a task an AI model can complete without human help, with the duration doubling roughly every four months. This chart has become the de facto scoreboard for measuring progress toward autonomous AI agents and is widely used by developers and founders to time product bets and fundraising narratives.

Why did METR update its original AI Graph with hard-to-score new evaluations?

The original chart had become oversimplified and stripped of nuance as it bounced around AI Twitter, leading to misleading interpretations of AI progress. METR added harder-to-score evaluations to correct course and provide a more accurate representation of AI capabilities, particularly for tasks requiring human judgment.

What key difference did METR's report find between AI performance on coding tasks versus judgment-heavy work?

METR's report indicates that while some AI capabilities continue to improve along an exponential trajectory, models aren't quite as good at tasks that require judgment as they are at coding. Coding benchmarks are easy to score and easy to game with more compute, but judgment-heavy work doesn't compress into a clean exponential curve.

What limitation does the updated METR chart address about using narrow task benchmarks for AI progress?

The update acknowledges that the original chart's narrow underlying tasks created a dangerous limitation for measuring true AI progress. The new version demonstrates that relying solely on easily-scorable metrics like coding benchmarks can distort understanding of AI capabilities and lead to inaccurate product development timelines.

LIVE19:50METR's AI Graph Update Adds Hard-to-Score New Evaluations