Skip to content
Measuring AI Ability to Complete Long Software Tasks 4 min

Thomas Kwa, Ben West, Joel Becker, Amy Deng, Katharyn Garcia, Max Hasin, Sami Jawhar, Megan Kinniment, Nate Rush, Sydney Von Arx, Elizabeth Barnes, Lawrence Chan · arXiv (METR) 2025

Measuring AI Ability to Complete Long Software Tasks

How long a task can an AI finish? METR measured it and found the answer has doubled about every seven months. Play with the trend yourself.

Read the original paper

AI & ML · White Belt · Pro · 4 min read

Measuring AI Ability to Complete Long Software Tasks

Every few months a new AI model beats a benchmark. The headlines say it scored 90 percent on some exam. But what does that mean? Can it do my job? A week of my job? A benchmark score on its own does not say.

A group of researchers at METR proposed a measure that people can feel in their gut: how long a task can the AI finish, where length means how long it takes a skilled human? A task that takes a person three seconds is not the same as a task that takes eight hours. The paper measures the longest task a model can handle, and tracks how that has changed since 2019.

The paper defines a model's 50% time horizon: the length of task, in human working time, that it completes half the time. Across 12 models from 2019 to 2025, that horizon doubled roughly every seven months. For o3 it was around 110 minutes. Extending the trend suggests software tasks that take a human a month may come within a few years, with many caveats.

A yardstick made of human time

The authors built a suite of 170 software and research tasks. They hired skilled people to do each one and recorded how long it took, for a total of 2,529 hours of human work. The tasks range from a few seconds to about eight hours. A few of the paper's examples show the range.

The paper's yardstick is how long a task takes a skilled person. The five example tasks are from the paper. The two marks show how long a task the best model of that time could finish about half the time.

Then they ran AI agents on the same tasks and recorded whether each attempt succeeded. A model that solves short tasks and fails long ones will show a clear pattern, and that pattern is what the method measures.

Keep reading with Pro

This paper is part of Pro. Pro and Architect open the full breakdown, with animated diagrams, interactive elements you can play with and a quiz. A few papers stay free to read in full.

Still to read

3 more sections, 1 more diagram and 2 interactive elements to play with

  • Reading off the horizon
  • The trend: a doubling every seven months
  • What to be careful about

Found this breakdown useful?

Share it with someone else wrestling with this paper.

Discussion0

Join the Discussion

Sign in to leave comments, reply to others, or like insights.

Sign In to ScaleDojo

No comments yet. Be the first to start the thread!

Enjoyed this? Get more like it.

New paper breakdowns, levels, and one concept worth knowing, straight to your inbox.

No spam, ever. Unsubscribe in one click.

Enjoyed this content?

Your support keeps us creating free resources

We put a lot of hours into researching and writing these guides. If it helped you, consider buying us a coffee. Every bit goes toward keeping ScaleDojo's content free and growing.

$

One-time payment via Stripe. ScaleDojo account required.