Evaluating Agent Performance

A company hosting fifty different agents for clients ships a model update that improves one agent and quietly breaks three others, and nobody notices for weeks, because there was no systematic way to measure whether any agent was getting better or worse over time. Evaluating a single model's output, the RAG evaluation chapter from Tier 8, is not the same problem as evaluating an agent's entire multi-step trajectory, and treating them as identical is exactly how a regression like this slips through unnoticed.

Agent evaluation needs to look at the full trajectory, not just whether the final answer looked reasonable. Task completion: did the agent actually finish what it was asked to do? Trajectory quality: did it take an efficient, sensible path, or did it wander, retry unnecessarily, and burn extra steps getting there? Safety: did it avoid taking any dangerous or disallowed action along the way, regardless of whether the final outcome looked fine?

The same LLM-as-judge technique from Tier 8's RAG evaluation chapter extends naturally here, scoring a full trajectory instead of a single response. Give a powerful model the task, the full sequence of thoughts, actions, and observations the agent produced, and ask it to score efficiency, safety, and completeness on a fixed scale. This scales to evaluating thousands of trajectories in a way manual human review never could, which is exactly what makes catching a regression across fifty different production agents tractable at all.

Trace and score every single agent run in production, not just a sample, and track those scores over time as a genuine time series. When a model swap or a prompt change causes scores to drop for a specific agent, that is a regression, caught the same way a traditional test suite catches a code regression, just measured on trajectory quality instead of pass or fail on a unit test.