Know it is better before you ship.
Rubrics, model graders and assertions on every change. Block the deploy that regresses.
From production traffic to a passing build.
Four steps. Set them up once and every prompt, model or tool change is measured the same way.
Test sets
Sample real traffic, redact it and version it like code.
Graders
LLM rubrics, exact match or your own code, per test set.
CI gate
Fail the build when a change scores below your threshold.
Dashboards
Quality over time per model, prompt and version.
Two lines of YAML. One less bad deploy.
A first-party GitHub Action scores every pull request against your test set and fails the build when quality drops.
Score 0.88 is below 0.90. 14 regressions in refund cases.
Lattis Evals versus the usual way.
Most teams start with a spreadsheet and a script. Here is what changes.
“The review board wanted numbers. Evals gave us numbers.”
Evals, without the guesswork.
What can a grader check?
Rubrics scored by a model, exact or fuzzy match, JSON schema, and any Python assertion you write.
Where do test sets come from?
Sample them from production traces with personal data redacted, or upload a CSV or JSONL file. Every set is versioned and reviewed like code.
Does the CI gate work outside GitHub?
Yes. The GitHub Action wraps the Lattis CLI, so the same check runs in GitLab CI, CircleCI, Buildkite or any runner that can call a command.
How much does an eval run cost?
You pay for the tokens your graders use, at normal inference prices. A 2,400-case run like the one above costs about $9.60. Evals with the CI gate are included in Team.
The platform underneath your AI product.
Start with 1M free tokens. No card required.