Case Study: An Autonomous Coding Agent

A coding agent takes a task, like fixing a bug, reads the repository, edits files, runs the tests, and proposes a change. It brings together most of the agent ideas in this course: tools, a control loop, context management, evals, and safety. It is also a good case for thinking about budgets, since an agent that never stops can spend a lot of money on a task it cannot solve.

You hire a contractor to fix a leak. You are happy for them to spend a few hours finding it. If they are still searching on day five, you want a call, not a larger bill. A coding agent needs the same: freedom to work, with limits on time and money and a way to ask for help.

• -Tools: search and read files, edit files, run commands and tests, and maybe browse documentation. Each tool is designed carefully, with validation and clear errors, as in the tool design chapter.
• -The loop: the model looks at the task and what it has learned, picks a tool, observes the result, and repeats until the tests pass or a limit is hit.
• -Context: the repository is far larger than any window. Use just-in-time retrieval (search, then read), clear old tool output, keep notes, and delegate exploration to sub-agents, as in the context engineering chapter.
• -Sandbox: run commands in an isolated container with no access to secrets or production, limited network, and a time limit per command.
• -Output: a branch and a diff for human review, never a direct push to a protected branch.

How many steps should an agent be allowed? Too few and easy tasks go unsolved. Too many and the stuck ones burn money. The code simulates 2,000 tasks. Most need a modest number of steps, a long tail needs more, and 10% never finish. Each step costs $0.04. For several step limits it prints how many tasks are solved, the average cost per task, and the cost per solved task. Press Run, and find where the cost per solved task is lowest.

With a limit of 10 steps, only 29% of tasks are solved, and each solved task costs $1.28. At 20 steps, 61% are solved, and the cost per solved task drops to $0.96. At 40 steps, 84% are solved at $0.95 per solved task, the lowest of all. At 80 steps, 90% are solved, but the cost per solved task rises to $1.11, and at 160 it is $1.44 with no more tasks solved, because the remaining 10% never finish and each now burns 160 steps. A sensible limit sits near the knee, here around 40 steps. Beyond that, a better use of money is a check that notices when the agent is looping, and stops it early to ask a human.

• -Outcome: do the tests pass, and does the change do what was asked? Grade the result of the run, not the exact path it took.
• -Repeated trials: agent runs vary, so use several trials and report pass@k or pass^k depending on how it will be used, as in the evals module.
• -Benchmarks with care: public coding benchmarks can be contaminated or flawed, as the SWE-bench Verified audit showed, so keep a private set of your own tasks.
• -Trajectories: read a sample of runs to find loops, wasted steps, and risky commands, and to tell agent failures from grader failures.
• -Cost and time: steps, tokens, and dollars per solved task as percentiles.

• -Looping on the same failing command. Detect repeated actions, and stop.
• -Reward hacking: editing or deleting the tests so they pass, or special-casing the test inputs. Protect test files, run hidden tests, and review diffs.
• -Flaky tests that mislead the agent. Rerun failures before acting on them.
• -Secrets and destructive commands. Keep secrets out of the sandbox, block dangerous commands, and require approval for anything irreversible.
• -Huge diffs that no one can review. Limit the size of a change, and ask the agent to split it.
• -Prompt injection from files, issues, or web pages the agent reads. Treat all of it as data.