The eval set is thirty frozen customers

Agent benchmarks are synthetic: tidy repos, planted bugs, environments no customer's machine has ever resembled. Then the release meets a real tangle of half-migrated configs and fails in ways the eval suite never rehearsed.

Your test data can be entire computers. From customers who opted in, keep scrubbed snapshots of their actual environments: the weird ones, the ones that broke things. Before each release, clone all thirty, run the new agent against every frozen world in parallel, and diff outcomes against the last version. Between releases the corpus sits idle as copy-on-write clones, costing effectively nothing on the flat-rate box.

A regression now fails against the mess that would have found it in production.