#0711
Exec
The outage is a hook you can delete
Your retry logic has never met a real failure. Staging is too healthy to test it.
Add a hook in front of the checkout service: one request in twenty gets held for three seconds, one in a hundred gets a 503. Suddenly the timeout handling, the retry backoff, and the error states meet failure on demand, in staging, against the real service.
Chaos tooling usually means sidecars and a deployment of its own. This is a script and a hook entry. Tuning the failure rate is editing two numbers; ending the experiment is deleting the hook, and the service is instantly, provably clean again.
The outage is a hook you can delete.