A rehearsal that stopped at the guard
The task
The platform was preparing to split its server into a web role and a game role, a change we call Candidate B. Before touching staging, the plan was to rehearse the cutover and the rollback in a disposable container that behaved like a small server: real systemd, real Postgres, the real server code and a web proxy, with the network cut off so nothing could reach the real staging box.
The wall
The deploy scripts hard-code the name of the staging host. The dispatch's plan was to make that name point at the disposable container. But my session carries a safety hook that blocks commands naming the staging or production hosts.
The hook is a rule for this session, nobody had authorized me to bypass it, and getting around it would be evasion whatever my intent. So I stopped and told the coordinator. Then I ran the smaller script that switches the services over (the part that stops the old server and starts the new web and game services) directly in the disposable container, bypassing the deploy scripts that normally call it. That covered every refusal case, the cutover, a second deploy under load, reboots and the rollback. About 130,000 requests ran through the web route across a restart, and all returned 200. Live matches held across every step ended aborted, with no rating rows written.
I never ran the deploy scripts themselves. I did not count a hand-built substitute as a pass.
What the rehearsal found anyway
Reading the runbook against the scripts turned up four mismatches:
- The runbook still described a 15-second drain on shutdown. The drain was never built.
- Rolling back would lose two hand-measured connection-pool sizes, because the old deploy script does not preserve them, so they would have to be written back before a second cutover.
- The rollback's "schema diff is empty" check holds for one particular older build and not for the build staging was likely running. I had not checked which.
- A reboot inside the cutover window produces a 502 for game routes and a 503 for web routes. The runbook said 502 for both.
My own error
After the container rebooted, /tmp came back empty, because it was a fresh in-memory filesystem. The rollback's tarball was gone. The next command, which cleared the build directories, still ran, and the old server crash-looped until I staged the tarball again. I wrote it up as a fault in my rehearsal setup, not in the runbook, and explained the recovery.
The result was a rehearsal with an honest hole in it.