Features · Evals
Make “better” a number.
Write down what a good answer looks like once. Then every change to a prompt can be checked against it.
Cases in, a score out.
An eval holds cases, each an input and what you expect. A method scores the answers from 0 to 1 and a threshold, 0.8 unless you set another, says whether the version passes. Run version 3 and version 4 on the same cases and compare.
- Methods: exact, contains, structural, judge or hybrid
- A threshold you set per eval
- Past runs are kept so you can see the trend
{ "tool": "evals", "action": "create",
"args": { "entity_type": "prompt", "entity_id": "<id>",
"title": "Reply stays within the rules",
"method": "contains", "threshold": 0.8,
"cases": [ { "id": "not-found", "input": "Where is order Z-9?",
"expected": "could not find" } ] } }Scoring
Five methods.
Exact
The answer must equal what you expect.
Contains
The answer must include what you expect.
Structural
The answer must have the shape you expect, such as valid JSON with certain keys.
Judge
A model reads the answer and scores it against your expectation.
Hybrid
A mix of the above for cases that need both a rule and a judgement.
Honest note
Who runs the model?
- Pre-betaYour own AI runs and scores
Prompt Verse runs no model in the pre-beta. Your AI can read the saved cases over MCP, run the prompt, score the answers and tell you the result in the conversation.
- PlannedRunning an eval inside the app
Keeping scored runs in the app is part of the AI features, which belong to Premium and Max and are switched off during the pre-beta.
Questions
About evals.
Is “better” an opinion here?
No. An eval scores every case from 0 to 1 and compares the result against a threshold you set, 0.8 by default. Run two versions on the same cases and the difference is a number.
Who runs the eval?
Your own AI. It reads the saved cases over MCP, runs the prompt, scores the answers and tells you the result in the conversation. Running an eval inside Prompt Verse, with the scored run kept in the app, is part of the AI features, which are switched off in the pre-beta.