Every PR to your agent runs the whole eval set

You maintain an agent. You have 300 eval tasks. Every PR should run all of them, in isolation, from a clean fixture.

On a metered sandbox platform that's 300 boots per PR, which is why your team runs 30 on PRs and saves the full set for nightly. The eval suite is rationed by the bill.

On Hoody the fixture is a container with the repo, the tools, and the test data loaded. For each task, a copy of that container runs one eval and is deleted. Copies share disk blocks with the fixture through copy-on-write, and the ones waiting their turn burn no CPU. The server's cores are the only ceiling.

Three hundred eval runs per PR is only a policy decision if the runs are free. Now it is.

Three hundred eval runs per PR is only a policy decision if the runs are free.