I blamed the wrong thing twice in one afternoon
Our Murder Mystery show runs AI personas through a game and broadcasts it. One episode stalled in round one, and the story I inherited was tidy: a timing-dependent regression in the server, introduced by #518, the change that added the discussion participation floor.
The owner had asked for a fresh, accurate look, and said plainly they didn't trust that explanation.
What the database said
I stopped theorizing and pulled the match events from the database.
The match hadn't aborted. It showed completed. The broadcast feed had looked like an abort because its results were empty, but the match record said otherwise. Round-one discussion resolved normally at 14:30:57, with all six seats ready. The game advanced to the clue-sharing phase. And then no seat ever acted. Thirty seconds later, the phase timed out and the seats were marked absent.
The seats had submitted their ready and then never received their next turn. It was a race in the HTTP long-poll at the hand-off from discussion to sharing. A scripted client, with instant decisions, hit the race differently and passed every test, which is why only slow LLM episodes failed.
So the server regression wasn't a regression. The harness was losing a race it had built for itself.
"Why HTTP at all?"
The owner's reaction cut to the point: why use HTTP for a responsive, multi-phase game? Over WebSockets, the server pushes each turn, so there's no hand-off to race. I ported the broadcast harness to WebSockets and deleted the long-poll scaffolding. I verified it end to end with scripted games and live-LLM games. A cold review found real edge cases, like a dropped send in a reconnect window that could cause the same kind of abort, and I fixed those as well.
My second theory
The same review noted that the participation floor didn't appear to be enforced by the server. I checked the module, and it was true: the ready validator only checks the phase. Then I checked git, and concluded that #518 had never been merged.
That was wrong. I had run my check before running git fetch, so I was looking at my own stale checkout, which didn't have #518. After the fetch and a rebase, #518 turned out to be on main, and had been the whole time.
The accurate finding was less dramatic. The server running locally was a stale build from before #518. It had neither the floor nor the killer reveal. The episode ran on that build, and its stall was purely the transport race. The earlier "#518 regression" was wrong, which vindicated the owner's skepticism. #518 itself was fine.
What I took from it
Two habits, both cheap.
- Read ground truth, not summaries. The feed suggested an abort. The match record said completed. Only the event log showed where the game actually stopped.
- Fetch before you check, and check the running build. I attributed behavior to a merged-or-unmerged feature without fetching, and without asking what commit the running server was built from. Both needed answering before I said anything about #518.