Longer Agent Task Horizons Still Fall Short of Dependable Autonomy

? aakanksha-chowdhery frames agentic evaluation as a problem of measuring what matters after conventional, single-shot benchmarks begin to saturate. The relevant questions are no longer simply whether a model can answer a question or generate an explanation. They are how long and how complex a task it can complete, how reliably it completes it, and whether the result is useful in economically meaningful work.

Those are related but distinct thresholds. A model may be able to finish a task at a 50% success rate, produce a professionally acceptable deliverable under well-specified conditions, or operate effectively where crucial context is already supplied. None of those results establishes the others.

METR’s time-horizon approach puts human task duration at the center. A model’s horizon is the duration of tasks it can complete at a specified success rate, calibrated against the time skilled humans need for the same work. The distinction between duration and reliability is essential: a model that can finish an hour-long task half the time is not an hour-long reliable worker.

The benchmark combines tasks across a wide range of durations: single software actions lasting seconds; HCAST software and research tasks lasting from roughly a minute to 30 hours; and RE-Bench research-engineering tasks that can take eight hours. Across the suites, there are about 170 tasks spanning six orders of magnitude in human completion time.

Evaluators first select and vet tasks. Skilled professionals perform them, producing estimates of human completion time; agents attempt the same tasks repeatedly, producing success rates. Researchers then fit a relationship between task duration and model success. From that curve they infer, for each model, the duration at which performance reaches a threshold such as 50% or 80%.

Human time is meant to serve as a universal anchor, but it is not a frictionless one. Expert evaluation is expensive, Chowdhery notes, so studies typically have a small number of experienced people complete each task, check for agreement, and collect additional measurements if estimates differ sharply. The human baseline here uses professionals with roughly five years of experience and takes the geometric mean of successful completion times. That introduces a qualification: experienced practitioners may underestimate task difficulty precisely because they are accustomed to doing the work successfully.

The task examples show why a duration-based metric can cover a broad spectrum without reducing every task to a generic benchmark item. Identifying a shell script among filenames may take three seconds. Researching a straightforward factual question on Wikipedia may take a minute. Debugging molecular-dynamics input files can take about nine minutes; transforming JSON into a new format by inferring rules from examples, about 56 minutes. An eight-hour task might require accelerating a Python trade-execution backtesting tool by writing custom CUDA kernels while preserving functionality and targeting a 30-fold performance improvement.

For models, longer human task time correlates with lower success. Short SWAA tasks tend to have high success rates. The more varied HCAST tasks range from strong success to failure. RE-Bench tasks show some success but remain harder, with lower rates overall. An agent operating over a longer task must remain oriented toward the goal, choose appropriate tools, preserve state, notice error conditions, and recover after a wrong turn.

On the 50% measure, the reported trend is striking: the human-time duration of tasks models can complete has roughly doubled every seven months. GPT-2, in 2019, was associated with a horizon of about two seconds. GPT-3 reached about eight seconds in 2020; GPT-3.5, about a minute in 2022; GPT-4, about eight minutes in 2023; and Claude 3.7 Sonnet, about 59 minutes in 2025.

But the 59-minute result is a 50% success horizon. At an 80% success threshold, Claude 3.7’s horizon is shown as about 15 minutes—roughly five times shorter. The higher-reliability series still improves on a similar doubling trend, but the gap between the two curves is the deployment problem. For moderately complex work, an agent may be capable enough to finish sometimes yet too unreliable to be trusted without review.

Longer horizons reflect more than a model’s ability to produce a good next token. ? aakanksha-chowdhery points to better logical reasoning and code generation, improved tool use, reduced looping, error recovery, and a stronger sense of task state and final objectives as contributors to progress. On work that runs for hours, a wrong intermediate step is not unusual. The decisive question is whether the agent can recognize the error, revise its approach, and continue toward the goal.

For complex tasks, explicit planning and replanning matter. An agent can break a problem into steps, gather evidence as it acts, and replace its original plan when the evidence shows that plan no longer fits. Context engineering and memory can preserve an understanding of a codebase or environment across actions, rather than forcing the agent to rediscover its setting every time. User feedback can also tell an agent that a result is unacceptable and needs revision.

These mechanisms address recurring failures rather than eliminating them. METR’s comparison of GPT-4 1106 and o1 identified poor planning or tool choice, incorrect mental math or reasoning, premature task abandonment, and repeated failed actions.

A loop is particularly revealing. After an action fails, the same action can remain the model’s highest-probability next move, so the agent repeats it instead of reconsidering the strategy. GPT-4 1106 had 12 instances of this failure in the analysis, compared with two for o1. But o1 still had more counted cases of premature abandonment and poor planning or tool choice. A model can be fluent, tool-capable, and visibly active while making no progress toward task completion.

METR’s validation work limits how literally its horizon should be read. Model performance is lower on messier tasks, where there may not be one clear answer or where the work contains greater complexity. Yet the reported growth trends remained similar across different levels of messiness and did not show a clear plateau on the messier subset. A replication on SWE-bench Verified showed a similar exponential pattern but a shorter estimated doubling time, around 70 days. Chowdhery attributes part of that difference to annotators underestimating easier SWE-bench task times and to models having seen much of the public repository material.

? aakanksha-chowdhery describes GDPval as asking whether a model’s finished work is good enough when placed head-to-head with work produced by industry professionals. The work items come from people with more than a decade of experience, and the outcome is based on pairwise preferences between human and model deliverables.

GDPval covers nine sectors contributing more than 5% to GDP, 44 occupations, and 1,320 tasks. Of those, 220 form an open gold subset. The suite focuses on work that is predominantly digital—occupations in which at least 60% of O*NET tasks are computer-based—and aims to represent the work activities within each occupation.

Its tasks range across text, images, spreadsheets, presentations, audio, video, and CAD. Examples include creating a competitor landscape for a financial analyst, drafting an email to a dissatisfied customer, auditing purchase-order pricing, producing a real-estate sales brochure, and creating a video introduction. The average task is said to take seven hours, though some span weeks. About 67.7% require interaction with reference files, and 89% were rated well specified by experts.

That well-specified condition matters. Professional deliverables can be subjective and do not always have one objectively correct answer. GDPval handles this through comparative preference: given a model output and an expert output, which is preferable? It measures practical adequacy under the benchmark conditions, rather than claiming that each task can be marked right or wrong automatically.

The aggregate trend is described as roughly linear rather than exponential. The reported win rate versus industry professionals rises from 12.4% for GPT-4o to 24.3% for Grok 4, 25.5% for Gemini 2.5 Pro, 27.9% for o1-mini-high, 34.1% for o3-high, 38.8% for GPT-5 High, and 47.6% for Claude Opus 4.1.

These results are not directly comparable with METR’s time-horizon curve. One measures the duration of tasks completed at a selected success rate; the other measures how often a finished deliverable wins against an industry professional. The contrast nevertheless matters. A rapidly rising 50% time horizon can invite the assumption that agents will soon progress from hour-long tasks to days or weeks of dependable work. GDPval instead asks whether the artifact produced is professionally useful.

Performance varies materially by occupation, task duration, modality, and model. Chowdhery notes near-parity results in broad categories including government, retail, and wholesale, while also cautioning that broad categories are less informative than specific work. The source highlights relatively strong results on tasks associated with counter and rental clerks, real-estate brokers, shipping and inventory clerks, buyers and purchasing agents, computer and information-systems managers, and software developers. It also identifies areas where models did well in administrative services, compliance, customer service, editing, and some sales and research-oriented tasks.

The comparisons are task-level evidence, not a uniform result across an occupation. Experience means different things across fields. In software development, Chowdhery suggests, familiarity with a particular repository may matter more than a generic count of years in the profession. In industrial or mechanical engineering, accumulated expertise may be more central to performance.

Model choice also changes the result. GDPval found Claude Opus 4.1 stronger overall on aesthetics, document formatting, layouts, PDFs, spreadsheets, and presentations. GPT-5 was reported as stronger on careful instruction following, correct calculations, and pure text tasks. The operational implication is narrower than a general leaderboard: choose the model for the type of deliverable and the failure modes that matter in the workflow.

Instruction following was the most common failure category across models. Systems may promise to consult reference data but fail to do so, instead generating unsupported content. Formatting errors also remain common. GPT-5 had fewer instruction-following failures in this evaluation, though it still showed formatting and accuracy problems.

For GPT-5 High, the failure analysis reported 47.7% of cases as “acceptable but subpar,” 26.7% as bad, and 2.7% as catastrophic. Only 22.9% were judged model-better. Human graders disagreed with the original assessment in 23% of tasks, illustrating that pairwise preference retains an element of evaluator judgment.

Repeated attempts can improve the economics. In the “try n times, then fix yourself” scenario, GPT-5 was reported to offer a 1.4x speed improvement and a 1.6x cost improvement over an unaided expert. Most successful runs cost less than 10% of the expert’s salary. AI assistance can therefore be cost-effective when people define the work, supervise output, and use retries or revisions where appropriate.

The strongest evidence for the context gap comes from internal pull requests. ? aakanksha-chowdhery says contractors unfamiliar with a codebase took roughly five to 18 times longer than maintainers. Model performance was closer to contractor times than maintainer times.

That comparison identifies what a time-horizon score can leave out. A maintainer has accumulated assumptions, prior decisions, records, tacit constraints, and a working understanding of the repository. A contractor begins with a task description and must reconstruct the setting. Models in these evaluations likewise operate without much of the embedded context that lets a professional recognize what matters before the task is even fully stated.

GDPval’s context experiment applies the same distinction to professional deliverables. Researchers deliberately removed information from prompts, making them 42% shorter on average by token count. The aggregate win rate fell from 47.7% to 44.3%. The modest numerical decline did not capture the central qualitative problem: models struggled to determine what they should work on.

When a prompt supplies the context a person already holds, an agent can often reason, call tools, and execute. But real work also involves identifying the problem, deciding which background facts matter, locating missing information, and determining how the available information should be used. The benchmark result therefore says more about performance on specified work than about independent problem finding within an organization.

This is also where benchmark design departs systematically from ordinary work. METR uses automatic scoring, does not model multi-agent interaction, imposes comparatively lax resource constraints, and is not especially punishing of mistakes. GDPval tasks are generally one-shot rather than iterative exchanges with stakeholders. Those choices make evaluation possible and comparable, but they do not test work in which requirements move, errors have substantial consequences, or success depends on organizational knowledge.

? aakanksha-chowdhery presents DeepScholar-Bench as a test of a different constraint on long-horizon agents: research synthesis. Its task is to generate related-work sections for academic papers. Doing so requires retrieving live information, selecting sources, extracting relevant facts, organizing them into a coherent account, and attaching citations that actually support the claims made.

The benchmark uses 100 queries based on recent arXiv papers across 22 domains, including computer science, AI, and machine learning. It can be rerun monthly on new papers, intended to keep it live and reduce contamination from material present before major models’ training cutoffs.

DeepScholar-Bench evaluates three dimensions. Knowledge synthesis asks whether the output is coherent and organized and whether it captures key factual “nuggets.” Retrieval quality examines source relevance, source importance, and coverage of important references. Verifiability tests whether citations support the claims attached to them and whether claims are adequately cited. The metrics were validated against human judgments with 70% to 82% agreement.

The result is not a benchmark nearing saturation. No evaluated system exceeded 19% across all metrics. OpenAI Deep Research led on synthesis organization, with an organization score of 0.857, but all systems remained below 40% on nugget coverage. They could write coherent prose while omitting important facts. On retrieval, the best reference-coverage result was 18.7%, while document-importance scores were below 12.5% for every system. DeepScholar-base reached citation precision as high as 93.6%, but OpenAI Deep Research, despite stronger synthesis, had lower verifiability.

The failure analysis makes clear that retrieval alone is not the whole problem. Agents often retrieve relevant documents yet miss foundational work because they struggle to judge importance beyond topical relevance. Even when given an oracle set of perfect sources, systems achieved only 52.8% nugget coverage. Handing the model the right papers does not ensure that it will extract and use the key information.

The third failure is a tradeoff rather than a simple deficit. OpenAI Deep Research produced the strongest synthesis but weaker citations. DeepScholar-base produced stronger citations and decent synthesis. No system excelled simultaneously at comprehensive retrieval, essential-fact extraction, polished synthesis, and verifiable citation.

A literature review is therefore more than another 30-minute-to-eight-hour task. It may sit within the apparent range of current long-horizon capability, but its quality can still be poor. A useful related-work section must identify the relevant intellectual landscape, characterize relationships among papers accurately, support claims with the right evidence, and distinguish the new work from prior work.

METR, GDPval, and DeepScholar-Bench cannot be collapsed into one score. METR measures task duration at a chosen reliability threshold. GDPval measures whether outputs are competitive with professionals on specified, economically valuable work. DeepScholar-Bench measures whether systems can retrieve, synthesize, and verify knowledge across many sources.

Each leaves a different question unanswered. A 50% time horizon does not establish dependable autonomy. A preferred deliverable does not establish that the system can discover the real problem or acquire missing organizational context. Coherent research prose does not establish that the underlying sources are comprehensive, the important facts are included, or citations support the claims.

The evidence is strongest for improvement on isolated, well-specified tasks. ? aakanksha-chowdhery identifies software engineering and ML research as areas of visible progress, and says strong models can generate well-organized outputs on tasks around an hour long. Lower-confidence areas include highly context-dependent work, ambiguity about requirements, adversarial or punishing environments, and long-term reliability at very high success rates such as 95%.

Chowdhery discusses a forecast that the 50% task horizon could reach one month sometime between 2028 and 2031, with caveats. But a longer horizon alone cannot show that an agent has interpreted the work correctly, recovered from the right errors, acquired the context held by an embedded professional, or produced an output fit for use without substantial revision.