GitHub Actions
Gate every pull request on eval scores.
Deployment
Official
v1.2.0
Install
~/infra
$
lattis ci add github-actions --suite support-triage
✓ workflow written .github/workflows/evals.yml
✓ suite linked support-triage
● first check passed 0.93
Lattis · production
ci.evals.runs
Eval runs on pull requests
46 / day
ci.evals.score
Median score per run
0.92
ci.evals.blocked
Pull requests held below threshold
3
Overview
Run your eval set on every pull request and fail the check when a change scores below your threshold. The action posts the score, the cases that regressed and a link to the full report as a comment.
What the check reports
ci.evals.score
gauge
Score for the changed prompt or model
ci.evals.delta
gauge
Change against the main branch
ci.evals.regressions
count
Cases that scored lower than main
ci.evals.cost
gauge
Grader spend for the run, in USD
ci.evals.report
link
Full report, posted as a pull request comment
Setup
01
Add your API key
Save LATTIS_API_KEY in the repository secrets.
02
Add the action
Run the install command or paste the two lines of YAML into your workflow.
03
Require the check
Mark Lattis evals as required in branch protection.
Run your first agent in five minutes.
1M free tokens a month. No card, no sales call, production keys in two minutes.
$ npm install @lattis/sdk
$ export LATTIS_API_KEY=lt_••••••••
$ npx lattis run hello-agent
✓ plan · ✓ search_docs · ✓ answer
● run completed · 3 steps · 0.9 s