Evals define what success means for an agent

Without evals, debugging is reactive (see: tracing and evaluation serve different jobs): wait for complaints, reproduce manually, fix, hope nothing else regressed. Evals force you to specify what success looks like and turn failures into test cases. Start with 20 to 50 tasks drawn from real failures. Define unambiguous success criteria where two experts would independently reach the same verdict.

The most common mistake is waiting too long to start. Teams delay building evals because they think they need hundreds of tasks. In reality, 20 to 50 tasks drawn from real failures is enough to catch regressions and guide improvements. Each user-reported bug becomes a test case. Each edge case the agent mishandles becomes a task with a reference solution. Over time, the eval suite becomes the most valuable artifact in the project: a precise, executable definition of what ‘good’ means.

But if the agent’s output is hard to turn into a checkable artifact in the first place, that is a product problem before it is an eval problem (see: a hard-to-eval product is hard for users to verify).