Assessing calibration
The preceding methods differ not only in cost but in whether they produce calibrated uncertainty, which raises the question of how to verify calibration in the first place. Reporting uncertainty without verifying its calibration is akin to reporting accuracy without a test set. We recommend a minimal set of diagnostics that should accompany maps with per-pixel uncertainty:
Coverage and reliability
Do the \(1\sigma\) ($$68%) prediction intervals contain 68% of held-out observations, and the \(2\sigma\) ($$95%) intervals contain 95%? Coverage above the nominal level indicates underconfident (conservative) intervals and coverage below indicates overconfidence. Plot observed coverage against nominal coverage for a range of probability levels (e.g., 10% through 90%). A well-calibrated model lies on the diagonal. This generalizes the coverage check to all levels simultaneously and is the regression analog of the classification reliability diagram [1].
Standardized residuals
For regression, compute \(z = (y_\text{true} - y_\text{pred}) / \sigma_\text{pred}\). If the predicted uncertainties are well-calibrated, \(z\) should follow \(\mathcal{N}(0, 1)\). The Z-score mean (ideal: 0.0) detects systematic bias; the Z-score standard deviation (ideal: 1.0) detects miscalibrated interval width. Values above 1 indicate overconfidence; below 1 indicate underconfidence.
Scoring rules
The interval score (Winkler score; [2]) simultaneously penalizes miscoverage and interval width, rewarding intervals that are both sharp and calibrated. The continuous ranked probability score (CRPS) evaluates the full predictive distribution. These are more informative than coverage alone, which can be achieved by making intervals arbitrarily wide.
Additional diagnostics
Probability integral transform (PIT) histograms and Q-Q plots provide visual checks of distributional calibration. The uncertainty–error correlation assesses whether predicted uncertainty is informative about where the model makes large errors, within the domain where errors can be measured.
The diagnostics above require reference observations, and so can only be computed where reference observations exist. This is an inconvenient constraint, because epistemic uncertainty is most consequential where the inference domain departs from the training and reference domain, and for a global map that departure is the norm rather than the exception. It is therefore worth thinking about what can and cannot be established. Three regimes may be distinguished. In the first, reference data are available at sufficient density that calibration can be verified directly, stratified by biome or land cover, and reported as such. In the second, no reference data exist in the target region, but the model’s behavior under extrapolation can be tested by proxy. We may hold out an entire region or biome from training (not merely pixels within it) and check whether predicted uncertainty rises there and whether it remains calibrated against that region’s held-out observations.
This is the only direct test of epistemic uncertainty under the conditions that matter, and it is distinct from spatial cross-validation at the pixel or block scale, which tests interpolation rather than extrapolation. Weaker checks, such as confirming that predicted uncertainty increases with feature-space distance from the training data [3], establish that the uncertainty is responsive but not that it is calibrated, and should be labeled accordingly. In the third regime, neither direct verification nor a credible proxy is available, and the map documentation may state that the uncertainty layer is unverified there rather than implying that the band carries the same meaning everywhere. A map’s uncertainty product could therefore come with its own map that delineates where calibration was verified, where it was tested under proxy extrapolation, and where it is inferred only.
When calibration diagnostics reveal problems, the response depends on the severity and pattern of the failure. Mild, smooth miscalibration where the reliability diagram shows a consistent curve away from the diagonal can often be corrected post hoc. Isotonic regression applied to the predicted CDF values, or Platt scaling for parametric models, can remap predicted intervals to achieve empirical coverage without retraining [1]. These are cheap fixes and should be attempted first. If miscalibration is more structured, for instance, systematic overconfidence in one biome but reasonable calibration elsewhere, post hoc recalibration on a stratified held out set can help, but the stratification should reflect the structure of the miscalibration, which requires enough validation data in each stratum to fit the recalibration reliably. When miscalibration is severe or regionally patterned in ways that post hoc correction cannot resolve, the model itself may need revisiting: retraining with a proper scoring rule as the loss function (e.g., CRPS for distributional regression, or Gaussian NLL rather than MSE for heteroskedastic models) can produce better-calibrated uncertainty at source.
Switching to a method that provides calibration by construction such as conformal prediction as a post hoc wrapper is another option, with the caveat that the resulting guarantees are marginal rather than conditional. At a minimum, if calibration cannot be achieved, this should be documented explicitly by reporting the empirical coverage alongside the nominal coverage, stratified by region or land cover class, so that downstream users can judge for themselves whether the uncertainty layer is trustworthy for their application.
Classification diagnostics
The categorical analogs of the above are the reliability diagram (observed frequency of correctness binned by predicted confidence), expected calibration error (ECE) as its scalar summary, and the Brier score as a proper scoring rule that, like the interval score, rewards both calibration and sharpness. There are, however, two caveats specific to maps, which is the context we care about. First, ECE and the standard reliability diagram assess only the top-label probability. For area estimation, where all class probabilities are summed over pixels, class-wise reliability diagrams or a class-wise ECE are the relevant diagnostics [4]. Second, per-class diagnostics on rare classes (change, in particular) are dominated by the few reference observations available, and should be reported with their sample sizes.
Qualifying a model before inference
The recalibration and retraining remedies above presuppose that miscalibration can be, and is, discovered while retraining is still an option. For global maps, training is typically cheap relative to inference over the full domain, so the cost of discovering a regionally patterned failure late is not the retraining itself, but rather the inference run that must now be repeated. Due to the compute use differential, calibration assessment should therefore be treated as a sanity check that precedes wall-to-wall inference. To this end, we suggest a qualification step with a few example components. (i) calibration diagnostics described above, computed on spatially and environmentally blocked holdouts, reported per stratum, since pooled diagnostics can hide regionally patterned miscalibration. (ii) feature space coverage analysis over the full inference domain, such as the area of applicability or a dissimilarity index [3], which requires predictor data only and can be computed before prediction is made. This identifies where the model will be extrapolating and therefore where the proxy-extrapolation test above should be targeted. (iii) end-to-end inference on a small number of tiles chosen to span ecologically distinct regions, including regions that the coverage analysis flags as poorly supported, evaluated against whatever reference data exist there. This should also surface tiling, compositing, and edge artifacts (see Map production and distribution) that do not appear in pixel-level cross-validation.
Of course, this does not guarantee calibration where reference data are absent, since such regions are unverifiable by modeling alone. However, testing models before inference makes sure that failures that could, in principle, have been detected are detected before expensive inference compute.