Skip to content
#0715
ExecNotifications

Two hundred alerts in, one page out

2:07 AM. A shared dependency dies, and forty services fire the same alert into hoody-notifications, one POST each, over and over.

A hook on the notifications service fingerprints each payload. The first of a kind forwards to your phone. Duplicates inside the window short-circuit into a counter: accepted, counted, never delivered. When the window rolls over, the next arrival flushes one summary instead: same alert, 214 sources, still firing.

The services didn't change; they still shout freely. The senders don't coordinate; the middle does. On-call wakes to one page carrying a number that says how bad it is, not a phone that's just a buzzing brick.

The pager storm arrives as one page with a count.