NLL lower is better
—Penalizes low probability on the correct class.
AN INTERACTIVE TUTORIAL
Temperature Scaling & Calibration
From model scores to calibrated probabilities: explore confidence, compare calibration metrics, and find the temperature that minimizes NLL.
Educational simulation, not research results. Artificial scores and labels, not patient records.
About these examples
0.513
Brier lower is better
—Squared error across all three probabilities.
ECE lower is better
—Confidence versus accuracy, grouped into bins.
Curves use the selected examples at every temperature. The filled dot marks the current T; the dashed line marks T = 1. The green ring marks the grid NLL minimum. Each plot has its own vertical scale. ECE can jump when examples cross bin boundaries.
How is the temperature selected?
- Keep the scores and reference labels fixed.
- For each candidate T, calculate softmax probabilities and average negative log-likelihood.
- Choose the smallest NLL on the grid, not the highest accuracy or the smallest ECE.
| State | T | NLL | Brier | Selected ECE |
|---|
The grid is an educational approximation, not a global optimization guarantee. The study uses SciPy minimize_scalar(method="bounded") over T = 0.05–10.0 on calibration logits and labels, with NLL probabilities clipped at 10−12. Classifier weights remain fixed. The reported T = 1.480195 is not the optimum of this simulation.
One prediction, three probabilities
Examples are selected by confidence at T = 1 to illustrate contrasting outcomes, not to represent the whole set. The class order stays fixed for every positive T.
Reliability by confidence bin
Each dot is a nonempty bin. The diagonal represents matching confidence and accuracy; vertical lines show the gaps. Counts and boundaries are listed below.
ECE calculation by bin
| Confidence bin | Count | Mean confidence | Accuracy | Weighted gap |
|---|
Equal-width bins share probability interval widths. Adaptive bins share approximately equal sample counts. Both use maximum predicted probability and the same weighted absolute-gap formula here. Neither binning choice guarantees a smaller error or a better estimate.
What each equation measures
Temperature-scaled softmax
The logits z are the model's scores. Dividing every score by the same positive temperature changes their separation, not their ordering. Higher T softens the distribution; lower T sharpens it.
Negative log-likelihood
Only the probability assigned to the reference class enters each example's loss. A probability of 0.9 gives a loss of 0.105; 0.01 gives 4.605. Confident wrong predictions receive a large penalty. NLL is the objective used to fit temperature in the study.
Multiclass Brier score
The reference is one-hot: 1 for the correct class and 0 for the others. This version sums squared errors over classes and averages over examples; it is not divided by the number of classes. Its range is 0 to 2.
Expected calibration error
Group predictions by their maximum probability. In each bin, compare the average confidence with the fraction correct, then weight that gap by the bin's sample share. ECE depends on binning and can hide within-bin differences. Minimizing NLL does not necessarily minimize ECE or Brier.
Simulation and study are different
The default 600 examples provide more populated bins than the original 12-example illustration, but no sample count guarantees a reliable calibration estimate. Small bins remain noisy; binning, data distribution and evaluation independence still matter.
Data construction and source methods
The simulated sets use a fixed-seed linear congruential generator (seed 20261024). Each of three logits is drawn uniformly from [0, 5); the reference label is sampled from softmax(z / 1.8). Thus, overconfidence at T = 1 is built into this teaching example. A finite-sample NLL minimum need not equal 1.8. The 60 examples form a prefix of the 600; the 12 hand-crafted examples are a different set. Increasing the sample size in this interface is not an experiment establishing an improvement.
The released study notebook computes both 15-bin equal-width ECE and top-label adaptive ECE. It fits T on the calibration partition, then fixes T and the routing threshold for evaluation. The tutorial uses its own scores, grid search and bin controls; it does not reproduce the paper's calibration estimates.
Study calibration implementation · Simulation and metric source
Study calibration results · Guo et al. (2017) · Nixon et al. (2019)