math-001 · PUBLISHED SAMPLE
A supply room has 8 sealed cases with 6 filters per case and 3 loose filters. Every filter costs $4. What is the total cost in dollars? Return only the numeric answer.
How it works
A day in the life of a monitor.
Floor, panel and error budget are frozen before any data. Then every day is the same: one panel of checks, one score, evidence that compounds. This is the whole loop, exactly as it runs.
01 Day one
Tell us the task; the first monitor is set up with you.
STEP 1
Name the task
The thing your feature must keep doing — extract the fields, keep the format, get the number right.
STEP 2
Freeze the design
Panel, floor, α — written down before the first observation. Changing any of them opens a new epoch, on the record.
STEP 3
Watch the record
One page per stream: every panel, the running interval, the whole lineage. Nothing is backfilled, ever.
Two monitors are free, forever. Onboarding is by hand this month — and we say so.
02 What runs
Five task families, checked by code.
Hosted checks probe five task families — structured output, extraction, instruction following, code reasoning, maths. Each is a registered pool of 150 items with known answers; deterministic checkers score every response in [0, 1]. No judge model anywhere.
An epoch freezes a five-member panel from the pool, committed by hash before its first observation. Panel members stay unpublished so nobody can tune against them; the samples below are from the published set.
instruction-following-001 · PUBLISHED SAMPLE
Write one line about garden sensors. Start exactly with BRIEF:,
end exactly with :END, include moisture, use exactly
9 whitespace-separated words, and never use the word urgent.
03 The check
One check, end to end.
Budget is reserved before a single provider call — a denied reservation makes no calls at all. Every dispatched member returns scored; a terminal failure scores zero and stays on the record. Only a complete panel advances the evidence. Read it hourly or yearly — the guarantee already paid for every look.
The public record is registered to run one panel per stream per day from clock start; hosted checks for your tenant run on the cadence your design declares. The premise, and where it fails: the guarantee boundary.
04 Your scores
Already scoring your feature? Send the number.
Declare a suite — a built-in checker and its parameters, never code we execute — then one JSON call per observation. An eval harness, a human-review queue, any metric you trust: if it lands in [0, 1], it can be watched.
PUT /v1/suites/checkout · application/yaml
checker: math parameters: absolute_tolerance: 0.01 relative_tolerance: 0.001
POST /v1/scores
{"suite":"support-agent","suite_version":3,
"model":"gpt-5-mini","score":0.86,
"ts":"2026-07-29T09:14:07Z"}
Descriptive, and it says so
Streams built from sent scores keep full history, the API and the record — and are labelled descriptive, because their rules were not frozen in advance.
META IS METADATA ONLY — NEVER YOUR PROMPTS OR OUTPUTS
Registered
Freeze the design with us — panel, floor, α, reviewed — and the same machinery carries the published false-alarm bound and ships signed certificates.
05 The alarm
An alarm is a package, not a ping.
Evidence crosses 1/α and latches. One alert in Slack and email — stream, log e-value, onset estimate, incident link. The incident keeps the full trajectory; registered designs add a certificate anyone can replay offline. Quiet is published too: no alarm is a finding, not a reassurance.
REGISTERED STUDY 001 · 2026-07-24 · ALARM AT PANEL 12 · REGISTERED BOUND ≤ 19 BLOCKS · ONE REAL TRAJECTORY, NOT A RATE
The one detection on the record so far, made on purpose to prove the alarm path. Read study 001 →
06 The dials
The dials are yours. The freeze is the point.
| Printed field | Typed value |
|---|---|
| Suite | five built-in checker types · parameters, not code · versioned — every score names the version it was made under |
| Checker parameters | math tolerances · extraction case rule · constraint lists · code normalisation · a JSON Schema you provide |
| Panel | five members drawn from the registered pool · panel_hash in the design |
| Floor · α | declared per stream, before data · default α 0.05 |
| Cadence | declared in the design's schedule |
| Alerts | Slack + email · test-fire any time |
| Stream control | pause / resume · settings via the API |
The public record starts with three open-weight routes — Llama 3.3 70B, Qwen 2.5 72B, Mistral Small 3.2 24B — pinned as model · provider · quant. Your route goes on watch by request. Request your model →
07 Start
Two monitors free, forever.
Send one score today, or tell us the task and get back a configured monitor — panel, floor, α, in writing.