Most prompt reviews are arguments about wording. They end when someone senior gets tired, not when anyone is proven right. There is a better place to have the argument.
Prompts are not specs
Teams argue about prompt wording because the prompt is the only written description of what the product should do. That makes every edit risky, and it makes every model upgrade a rewrite.
Write the cases instead
A versioned test set of real inputs, each with a grader, says what good looks like. Ours for support triage has three parts:
- 2,400 cases sampled from production and redacted
- A rubric graded by a model, reviewed by the support leads
- Hard assertions for anything touching refunds
Once the set is reviewed like code, the prompt becomes an implementation detail.
Make it a gate
Run the set on every pull request and fail the build below the threshold. Two lines in the workflow file:
- uses: lattis/evals-action@v1
with: { suite: support-triage, threshold: 0.90 }What changes for the team
Disagreements become pull requests to the test set, not to the prompt. New cases arrive with every bug report, and the bar only moves up.
“If you cannot score it, you cannot ship it.”
