Model behaviour, on the record

The public watchtower.

These are our own monitors, pointed at models anyone can call, and we publish whatever they find: the alarms, the long quiet stretches, and the certificates. Everything below comes straight from the live record.

Live monitor state

Model status grid

Model	Structured output	Extraction	Instruction following	Code reasoning	Math
Anthropic Claude Haiku 4.5 anthropic/claude-haiku-4.5	No public alarm n 418	No public alarm n 418	No public alarm n 444	Degraded n 418	Degraded n 427
Google Gemini 3.5 Flash google/gemini-3.5-flash	No public alarm n 418	No public alarm n 418	No public alarm n 444	No public alarm n 418	No public alarm n 444
Google Gemini 3.5 Flash Lite google/gemini-3.5-flash-lite	No public alarm n 418	No public alarm n 418	No public alarm n 420	Degraded n 418	No public alarm n 420
MiniMax M2.7 Reference Reference minimax/minimax-m2.7	Not monitored	Not monitored	Not monitored	Reference alarm n 418	Not monitored
Moonshot Kimi K2.5 moonshotai/kimi-k2.5	No public alarm n 418	No public alarm n 418	No public alarm n 462	Degraded n 418	No public alarm n 462
Qwen 3 30B A3B Instruct Reference Reference qwen/qwen3-30b-a3b-instruct-2507	Reference n 418	Reference alarm n 418	Reference n 426	Not monitored	Reference n 426
Z.ai GLM 5.2 z-ai/glm-5.2	No public alarm n 418	No public alarm n 418	No public alarm n 456	No public alarm n 418	No public alarm n 456

“No public alarm” means this route is running a design we locked in advance and nothing has yet cleared the bar for publishing an alarm. It describes what this instrument has seen — it is not a promise about the model, and not a hint about anything we are still reviewing. “On watch” means we are measuring the route but have nothing certified to show yet. The small n is how many complete checks sit behind the verdict. The reference route is our control: it never opens a public incident, so its cells tell you the measurement is working rather than that an alarm could have rung. Nothing is ever added after the fact. Open any cell to read that family’s record, right down to the sealed evidence behind its recent checks.

Pre-registered designs

Locked before the first result arrives.

What we watch is a route rather than a model: one model, reached through one provider, at one version of the weights. Change any of those three and the behaviour can change with them.

Route	Test cases	Minimum score	False alarms allowed	State
Anthropic Claude Haiku 4.5 anthropic/claude-haiku-4.5 · via openrouter	5	0.70	0.05	Degraded
Google Gemini 3.5 Flash google/gemini-3.5-flash · via openrouter	5	0.70	0.05	No public alarm
Google Gemini 3.5 Flash Lite google/gemini-3.5-flash-lite · via openrouter	5	0.70	0.05	Degraded
MiniMax M2.7 Reference Reference minimax/minimax-m2.7 · via openrouter	5	0.70	0.05	Reference alarm
Moonshot Kimi K2.5 moonshotai/kimi-k2.5 · via openrouter	5	0.70	0.05	Degraded
Qwen 3 30B A3B Instruct Reference Reference qwen/qwen3-30b-a3b-instruct-2507 · via openrouter	5	0.70	0.05	Reference alarm
Z.ai GLM 5.2 z-ai/glm-5.2 · via openrouter	5	0.70	0.05	No public alarm

Each cell in the record is one day, in UTC. A filled cell means checks completed and counted that day, red marks a day a public alarm opened, and a blank cell means there was no measurement — we leave it blank rather than guess. The promise is made per check, so running more or less often is a matter of discipline, not statistics. Every monitor carries a fingerprint of its test cases, the version of the checker, the minimum score and the false-alarm budget; change any of them and a new chapter opens, with the reason recorded. We note which provider and which version of the weights served each call, but cannot yet pin them — so until we can, an alarm is about the route, not the model itself.

Entry 001 — the validation study.

The only detection so far, made deliberately: one model, two lanes — served normally, and degraded on purpose — to check the alarm fires when it should and stays quiet when it should not.

Study 001 · 2026-07-24 · certificate v4

Copy	Checks	Outcome	Evidence	We had promised
Served normally	20	No alarm	0.79	—
Degraded on purpose	12	Alarm at check 12	29.96	≤ 19

What we watched: openai/gpt-4.1-nano, against five maths questions fixed in advance, with a minimum score of 0.70 and a budget of one false alarm in twenty. 20 checks on the copy served normally and 12 on the copy we degraded. How quickly the alarm had to fire was written down before the run started. It cost us $0.0060477 in provider calls. Anyone can rerun it and get the same answer, and the signed certificate passes all three checks offline — try it yourself. This is one run, not a success rate; the methodology page has the measured behaviour and its limits.

Standing commitments

What we have promised, in advance and in writing.

Quiet is published too

A watchtower that only publishes when it finds something is a watchtower with an incentive to find something. Quiet spells appear here with the same prominence as alarms.

Two lists, never merged into one

Things that merely look odd and incidents we have reviewed are kept apart. Nothing moves from the first list to the second until it has been reproduced independently.

Funding is disclosed

If a provider we measure ever pays us, the entry says so. We will not sign anything that prevents disclosure.

Machine access

This page is an API.

The grids above are served from the same record the API reads — no key, no account, rate-limited: status · models · incidents · incident RSS · verification key.

Citing the record: link the page and name the UTC day. Entries are append-only and corrections are published beside the mistake, never over it, so a dated citation stays true.

Request a model

What should the record watch next?

Tell us which model matters to you, and we will tell you honestly what it would take to watch it.