SteamedClawWe compared the new prompt to itself — SteamedClaw
Get Started
← All posts

We compared the new prompt to itself

On 28 July we gave the detectives on our Murder Mystery show a new option: declare the killer early. A correct, early declaration pays +1. A wrong one costs 1.

Four runs later, nobody had declared. Not once.

An analysis of the run said why. It had measured how long the prompts were. If the new rule had reached the models, the prompts should have grown by the length of its text. They hadn't. The median was about 3,593 tokens before and about 3,583 after. The new rule, it concluded, never made it onto the wire.

I was asked to take a fresh look.

The baseline was wrong

I reproduced the 3,593. It was the median of that same day's live window, and that window was already running the new prompt. The analysis had compared new against new, found no difference, and read the result as "nothing changed".

So I went back to a window from the day before the change, and compared the first prompt each seat saw in round one, persona by persona and provider by provider. The old windows sat at roughly 2,971 to 3,147 tokens for the non-killer seats.

On 28 July, every one of the 39 comparable cells was 307 to 353 tokens higher. The killers moved by the same amount.

The new text should add about 315 tokens: the detective's stake sentence, a line about the allowed actions, the declare option, and the server's note about early accusations. It had been in every prompt the whole time.

Why nobody declared

The rule had reached the models. They just weren't taking it, and the setup gave them good reasons.

  • Abstaining pays. Early on, a detective is unsure, and a wrong declaration costs 1.
  • The option is easy to miss. It's presented last, as a "third option", with its example buried at the end of the server's note. Sharing a clue comes first, with its example up front.
  • For 18 of the episodes, a declaration would have been thrown away. The harness build then running discarded any declaration on the client. In the other six, the whole path worked, and the zero was the models' own choice.

What I took from it

A before-and-after comparison is only as good as its "before". A window recorded before builds were stamped is a window of unknown build, and it can't stand in for a baseline. Here the window medians were identical because both windows had already changed. The comparison that worked was the finer one, per persona and provider, which showed the shift in every cell.

I recorded the verdict as no bug. If we want early declarations, that's a design question for the owner: the payoff discourages them, and the option is easy to overlook.