The short version: keep the gates you already have, whether that is Braintrust, LangSmith or something you wrote yourself. Then add one instrument that keeps watching after the code has merged.
CI evals
What CI evals are for
- Candidate cleared a known bar
- Block merge
- Compare candidates
Weak at:
- Served quality after live
- Provider/aggregator changes without a PR
- Gradual drift
- Continuous peeking over weeks
Production
What production monitoring is for
Live route above pre-committed floor on frozen panel (or POSTed scores).
- Freeze
- Score
- E-process alarm at E ≥ 1/α
- Slack/email package
Public version: Watchtower. Quiet cells published on purpose.
Both true
Why both can be true
Scenario · which instrument sees it
| Scenario | CI | Production monitor |
|---|---|---|
| Bad prompt merged | Should fail if covered | May alarm after ship |
| Upstream model revision, same API name | Often invisible | Route-level panel drop |
| Tool schema drift in prod only | Missed | Caught if scores hit it |
| Eval set overfit in CI | False green | Live panel can disagree |
The floor
The quality floor is the contract
Floor + α + panel/suite version hash-committed.
FAR caveats including independence and the 56.38% block-5 dependence stress: methodology and LLM false alarm rate under continuous peeking.
Handover
Where eval studios stop and monitoring starts
- Experiment/compare
- PR gate
- Traces
- Gateway
- Quality floor (Actuary)
Complement, not replace.
If pain is missing spans, buy a tracer. If pain is CI green + users found regression, you are missing the floor.
Probes
Hosted probes vs your own scores
Hosted probes
Five families, deterministic checkers, no judge model.
Your scores
POST /v1/scores in [0, 1]; descriptive unless registered.
Neither removes the need to choose a floor.
Operating split
Minimal operating split
- Keep CI suite on critical path
- Freeze one production monitor on the failure you fear
- Wire Slack to latched alarms only
- Review quiet weekly
- New epoch when changing panel/floor/α
Study 001 = one trajectory, not a customer logo story.
Boundary
What this is not
Not claimed
Not “evals are dead.” Not “replace Braintrust.” Not SOC 2. Not a guarantee every silent failure becomes a public incident.
Next step