Actuary

Learn

Reference notes on LLM quality monitoring

Seven notes, each one written against the public record and linking back to it. What a quality monitor does that a tracer cannot. What false alarms really cost you once you start checking every day. Why a green CI gate and a worse production system are not a contradiction. How an agent can write to your CRM, be told it succeeded, and still be wrong — and how it picks the almost-right record in the first place. And two real incidents, read field by field.

The shelf

  1. Monitoring vs observability

    LLM quality monitoring is not tracing

    Note 01

    LLM quality monitoring is not tracing. Keep Langfuse/Braintrust; add a frozen floor, α, and anytime-valid alarms when production quality drops.

    Reads with how it works

  2. Note 02

    LLM alert fatigue is often unbudgeted peeking. FAR needs design-time α, independence caveats, and anytime-valid evidence—not a tighter threshold.

    Reads with methodology

  3. Note 03

    CI evals can be green while production LLM quality drops. Add a frozen quality floor and anytime-valid monitoring beside Braintrust — not instead of it.

    Reads with how it works

  4. Note 04

    When an agent mutates CRM/ERP and the API returns 200, quality only shows up when a human notices. Keep tracers — add a quality floor after ship.

    Reads with methodology

  5. Note 05

    Agent picks a near-right CRM/ERP contact or account and writes on it. Landscape note on near-miss match vs traces, CI, and a frozen quality floor.

    Reads with methodology

  6. Note 06

    Claude Haiku 4.5 code-reasoning incident opened 2026-08-05 UTC: effect size 0.700, E=20.829, aggregator route — facts from Actuary’s public register.

    Reads with the incident register

  7. Note 07

    Gemini 3.5 Flash Lite code-reasoning incident opened 2026-08-05 UTC: effect size 0.633, E=32.365, aggregator route — facts from Actuary’s public register.

    Reads with the incident register

Primary sources

The notes cite three surfaces. Those come first.

Every claim in these notes traces back to one of the sources below. If a note and the record ever disagree, believe the record — and please tell us, so we can fix the note.

How it works

The loop: freeze the design, score one panel, alarm, issue the certificate.

Read the loop

Methodology

The guarantee, the multiplicity gate, and the regime where this detector is badly wrong.

Where it fails

Watchtower

The live record: model × family cells, quiet spells included, incidents with permanent ids.

Open the record