A WebSocket server that won't stay down

Your real-time service is a Node.js WebSocket server holding three thousand connections. It crashes maybe once a week: usually a malformed frame from a misbehaving client. When it crashes, three thousand reconnects pile up and the next minute is rough.

You configure it as a daemon program with autorestart: "true". The crash happens at 2:14 AM. supervisord notices the process exited, spawns a new one within milliseconds, and the new process binds the same port and starts accepting. Reconnections start hitting a healthy server before most clients have finished their backoff jitter.

You see the crash in the morning by GETing the logs URL: stderr shows the exception, stdout shows the recovery, status shows current uptime since 2:14:01 AM. You file the bug. Production never noticed.