Aller au contenu
#0559
AgentSnapshotsContainers+1

Quarantine the flaky test while you sleep

CI went red at 5:50pm on a change nobody can explain. Someone reruns it, it goes green, everyone goes home, and the test stays to ruin another Friday.

At 11pm a cron entry starts an agent with one job: find the flaky one. It restores the 5:50pm snapshot of the CI container so the environment matches the failure, runs the suite thirty times, and tallies per test. Three tests fail two to six times each. It finds the shared cause: an assertion that assumes the clock hasn't crossed midnight: patches it, reruns thirty times clean, and opens a PR titled quarantine: time-dependent assertions in billing_test.

A human reruns twice and gives up. Thirty reruns on a container you already pay for cost nothing but the night.

The flaky test gets found by the only engineer with thirty reruns of patience.