Measuring a threshold instead of guessing it
To set a threshold on a model's answer, ask the same question across a ladder of hand-built situations, look at where the answers separate, and put the bar in the gap. A number picked by intuition usually sits where the model never goes.
- Method
- a ladder of situations, same wording, read the spread
- Observed P(top choice)
- median ≈ 0.57, p75 ≈ 0.77
- Bars in use
- 0.42 general, 0.45 in the crew room
- A bar that never fired
- 0.72, on a question that tops out at 0.18
The procedure
- Write the question exactly as the live system will send it.
- Build six to eight states that ought to answer differently: healthy and early, healthy and late, cooling, strained, wrecked, wrecked-and-it-does-not-matter.
- Ask each one and write down the number.
- If the answers do not separate, the question is wrong. Fix the question, not the bar.
- Put the bar in the widest gap, and re-run the ladder whenever the wording moves.
What a badly scaled question looks like
The first betrayal draft asked whether a character would break their alliance this beat, and set the bar at 0.72. Breaking an alliance on any particular beat is genuinely rare, so the answer tops out around 0.18 whatever the state. The bar could never be cleared and no alliance ever died.
The crew room walked into the same wall from the other side. "Would you walk out now, with a cut off the people you leave" answered 0.20–0.25 in every situation on the ladder, healthy or wrecked. Asking about the table instead of the moment, with the arithmetic in front of them, separates 0.30 for a crew that is winning from 0.52 for one that has been overtaken.
Two rules that fall out of this
A different question is a different scale. Bars belong to wordings, not to concepts, which is why the betrayal bar is a function of the room, betrayBar(spec), and not a constant.
Absolute values mean nothing on their own. 0.42 is not "fairly likely"; it is a point in one measured distribution. The same figure against a different question is noise.
Read next
The betrayal bar, measured rather than guessed
An alliance dies when the model's P(better off without them) clears a bar: 0.42 in the pair rooms, 0.45 in the crew room. Both numbers came from asking the same question across a ladder of situations and putting the bar in the gap.
Using distributions instead of picks
Because every answer is a distribution, the engine can scale an outcome by how sure a character was. A 0.95 read of an open host earns a bigger opening than a 0.55 one, so not every decision is equally committed.
Evaluation model vs chat model
A chat model returns text you have to trust and parse. An evaluation model returns a probability per option, which is what lets a simulation put a real threshold on a decision instead of a vibe.
Calibrating against a chaos brain
Before trusting a real model with the village, every notable is fed a random whim each time they are asked. If two villages cannot survive three years of that, a real model will kill them on a bad afternoon.