What an evaluation model is
An evaluation model takes one shared state and a set of typed questions, and answers all of them in a single request. Each answer comes back as a full probability distribution instead of a sentence.
- Shape
- state in, N typed answers out, one request
- Answer types
- choice, score, boolean
- What you get back
- a distribution per question, not a token stream
- Model here
- typesafe-ai/jev, via the Vercel AI Gateway
A chat model is asked to continue text. An evaluation model is asked to judge: you give it a situation once, hand it a list of questions with declared answer shapes, and it returns a scored answer to every one of them together.
What matters is not the API ergonomics. It is that the output is a measurement. A choice question comes back as mass spread over the options, so you can see that a character chose to approach the host at 0.71 with 0.25 still sitting on holding court. That is an almost-did you can put on screen, and a number you can put a threshold on.
The three question types
Every question in Parlour is one of those three. Nothing in the system asks for free text, which is why nothing in the system has to parse any.
| Type | You declare | You get back |
|---|---|---|
| choice | a set of labelled options | a probability for each option |
| score | ordered levels, lowest to highest | a level, with the mass around it |
| boolean | a proposition | P(true) |
Why that shape suits a simulation
A simulated person is not writing an inner monologue. They are making a handful of gut-checks about a room: what do I do, who is it about, how badly do I want it, how dangerous is that one, am I better off without this alliance. Those are questions with answer sets, asked of the same state, and asking them together is both cheaper and more coherent than nine separate calls that cannot see each other.
The engine then uses the whole distribution instead of flattening it (see distributions, not picks), and every threshold in the rules is measured against what the model actually returns.
Read next
typesafe-ai/jev, and what Parlour uses it for
Jev is the evaluation model Parlour runs on. It answers nine typed questions per character per beat in one call, is billed on input tokens only, and costs roughly three cents for a five-minute party.
Evaluation model vs chat model
A chat model returns text you have to trust and parse. An evaluation model returns a probability per option, which is what lets a simulation put a real threshold on a decision instead of a vibe.
Typed questions: choice, score, boolean
Parlour asks three shapes of question: a choice over options, a score on ordered levels, and a boolean. The option sets are built per character per beat from the rules, so the model can only pick something legal.
Using distributions instead of picks
Because every answer is a distribution, the engine can scale an outcome by how sure a character was. A 0.95 read of an open host earns a bigger opening than a 0.55 one, so not every decision is equally committed.