Calibration error measures how far off your confidence estimates typically are. It is the square root of your Brier score: for each answer, take the confidence you stated that you were right (as a probability p) and the outcome o (1 if right, 0 if not), average (p − o)² across your graded answers, then take the square root. It sits on a 0–1 scale; always saying 50% scores 0.5.

Until 2026-11-17 MindFrame used the average gap, mean |p − o|. That measure could be gamed: on anything you were likely to get right, answering 100% every time gave the lowest expected gap. The Brier score is a proper scoring rule, so the best strategy is to state the confidence you actually hold.

Example: say "80% confident" on 10 questions and get 6 right. Each right answer contributes (0.8 − 1)² = 0.04 and each wrong one (0.8 − 0)² = 0.64, so your Brier score is (6 × 0.04 + 4 × 0.64) ÷ 10 = 0.28 and your calibration error is √0.28 ≈ 0.53. Saying 60% instead would have scored 0.24 and 0.49.

Lower calibration error is always better. Zero means you were certain and right every time. Because nobody is certain about everything, a well-calibrated thinker on hard questions still has a calibration error well above zero; what matters is that it falls as your confidence starts to match your results.

MindFrame shows your calibration error alongside a reliability diagram, which plots your stated confidence against actual accuracy, making it visually clear in which direction and how severely you are miscalibrated.