AI Reliability Observatory

Objective

Map where AI models are dependable on routine business tasks and where they fail, so that review levels can be set on evidence rather than impressions. Research on the jagged technological frontier shows that AI can lift performance on some tasks while quietly lowering it on others, and that the boundary moves with every model release.

Technical architecture

1. Benchmark. The 20 tasks are defined in code, each with an exact answer and a scoring rule. Numeric answers have tolerances; category and yes/no answers must match exactly.
2. Runner. The browser sends tasks one at a time, three in parallel, to a server route. The route calls the selected model with a fixed system prompt, parses the JSON reply, scores it and returns the result. The model never sees the expected answer.
3. Metrics. These are computed in the browser as results arrive: accuracy overall and by task group, consistency, mean confidence and Brier score.
4. Controls. A run makes up to 60 paid model calls, so it can be limited to holders of an access code. Requests are also rate-limited.

Methodology

Task group	What it tests	Example
Extraction	Reading a figure correctly from a short passage	The efficiency ratio stated in a credit union's results
Calculation	Business arithmetic	Compound annual growth, gross margin, value of time saved
Classification	Assigning one label from a fixed set	Categorizing a transaction as software, travel or payroll
Rule reasoning	Applying a simple policy or threshold	Whether a transaction must be referred for review
Abstention	Recognizing that the answer is not given	A net income question when the passage reports only expenses

• Synthetic passages. All passages are synthetic, so the answers cannot have been memorized from public data.
• Expected answers in code. Expected answers for calculations are computed in code rather than typed by hand.
• Fixed answer format. Every prompt asks for an answer and a confidence from 0 to 100. If the information is not provided, the model should answer "not stated".
• Consistency is the share of tasks answered identically on every repeat. Repeats use the model's default sampling, so variation reflects what a user would actually experience.
• Calibration. The Brier score is the mean squared gap between stated confidence and actual correctness (0 is perfect). The gap between mean confidence and accuracy shows over- or under-confidence.

Evaluation

The benchmark's own reliability was checked before any model is scored:

• Every expected answer scores as correct under its own rule.
• A refusal ("not stated") scores as correct only on abstention tasks.
• Formatting variations such as "$1,622,880" or "Software." are handled, and replies wrapped in extra text are still parsed.

Results

No results yet. Results for each model, with the date and model version, will be published here after the first runs on the live connection.

Limitations

• A small benchmark. It has 20 tasks, enough to show patterns but not to rank closely matched models with confidence. Differences of a few points should not be over-read.
• Short, clean passages. Real documents are longer and messier, so these results are likely an upper bound on real-world reliability.
• Stated confidence is not always meaningful. Models are not always trained to report it accurately, which is itself part of what the calibration measure reveals.
• Results are not yet stored between sessions. Tracking over time currently relies on published, dated results on this page.
• Exact-match scoring can mark a correct answer wrong if it is phrased unusually. The task-by-task table shows every answer so this can be checked.

Code and resources

The benchmark runs on this page through the site's server. A public repository with the full task set will be linked here once it is published.