Every few months a new AI model beats a benchmark. The headlines say it scored 90 percent on some exam. But what does that mean? Can it do my job? A week of my job? A benchmark score on its own does not say.
A group of researchers at METR proposed a measure that people can feel in their gut: how long a task can the AI finish, where length means how long it takes a skilled human? A task that takes a person three seconds is not the same as a task that takes eight hours. The paper measures the longest task a model can handle, and tracks how that has changed since 2019.
The paper defines a model's 50% time horizon: the length of task, in human working time, that it completes half the time. Across 12 models from 2019 to 2025, that horizon doubled roughly every seven months. For o3 it was around 110 minutes. Extending the trend suggests software tasks that take a human a month may come within a few years, with many caveats.
A yardstick made of human time
The authors built a suite of 170 software and research tasks. They hired skilled people to do each one and recorded how long it took, for a total of 2,529 hours of human work. The tasks range from a few seconds to about eight hours. A few of the paper's examples show the range.
Then they ran AI agents on the same tasks and recorded whether each attempt succeeded. A model that solves short tasks and fails long ones will show a clear pattern, and that pattern is what the method measures.
