The eval you watch instead of score

Two prompt variants, same task, and the benchmark disagrees with your gut. Run them side by side: two containers, each agent driving its own desktop. Each display gets an ffmpeg capture into its own pipe path. One HTML page, two iframes, both ?video URLs.

You watch variant A sail through the checkout flow while variant B reopens the same dropdown four times. The scores said they were within two points of each other. The video says otherwise, immediately and unarguably.

Evals compress behavior into numbers, which is what makes them scale and what makes them lie. Sometimes you need the behavior itself, at native framerate, in parallel.

The eval harness is two video players on one page.