Validation beyond probability sampling
The preceding sections describe the recommended framework for map validation based on probability sampling and design-based accuracy assessment. In practice, however, this framework is not always feasible. Large-area mapping projects may lack purpose-designed probability samples, existing reference data may have been collected for different objectives, or certain products (particularly continuous-field variables) may require specialized validation strategies. In these situations, validation typically relies on one or more of three approaches: alternative inferential frameworks, existing reference datasets, and product-specific validation protocols. Although these approaches provide valuable evidence of map quality, they generally do not provide the same statistical guarantees as design-based accuracy assessment.
Alternative inferential framework. One alternative is to replace the design-based framework with model-based inference. Whereas design-based inference derives its validity from the randomness of the sampling process, model-based inference relies on a statistical model that describes how map errors relate to predictors such as landscape characteristics or data availability. This enables the use of non-probability reference data, such as opportunistic field plots or spatially sparse inventories, by extrapolating accuracy estimates to unsampled locations through the model [1]. The trade-off is that validity depends entirely on whether the assumed model is correct, which may be difficult to verify. Consequently, model-based inference has seen only limited adoption in operational map accuracy assessment [2].
Existing reference datasets. In practice, validation more commonly relies on existing reference datasets than on model-based inference. These include, for example, national forest inventories (NFIs), agricultural parcel registers, and airborne LiDAR campaigns. Whether such datasets can validate a map depends, among others, on its sampling design and the population it covers, its reference period and its independence from the training data. When the design is probabilistic and its population and definitions align with the product (for example an NFI used to assess a national forest map with a compatible forest definition), design-based estimation remains possible. However, in practice, NFIs are used to assess products with different thematic definitions, spatial extents, or target classes, such as global land cover products. In these situations, the original sampling design may no longer correspond to the validation objective, and the resulting assessment should be interpreted with appropriate caution. The results describe agreement at the reference locations rather than accuracy over the map, that is, verification rather than validation.
Product-specific validation protocols. Some geospatial products require dedicated validation protocols that extend beyond the general recommendations described above. Biomass mapping provides a representative example. The CEOS WGCV Land Product Validation subgroup has developed dedicated protocols that address the particular challenges of validating continuous-field products, including the use of plot-level field measurements, allometric uncertainty, and spatial scaling from plot to pixel [3]. These protocols recommend combining field inventory data with airborne LiDAR transects to bridge the scale gap between ground plots and satellite-derived estimates. Even under such protocols, however, spatial coverage remains fundamentally limited, and the resulting accuracy characterization applies only to the sampled regions.
Regardless of the validation strategy adopted, the principles of transparency and rigorous reporting remain unchanged. Reference data sources and their limitations should be clearly documented, the distinction between verification and validation should be made explicit, and results should be interpreted as indicative rather than as unbiased estimates of map accuracy.
The preceding sections have traced the full arc of the EO mapping pipeline, from the processing of satellite observations through to a validated, disseminated product. At each stage, we have highlighted choices that are easy to get wrong and hard to diagnose after the fact — from the silent differences between data providers, through the spatial structure of training splits, to the gap between algorithm-derived uncertainty and genuine accuracy assessment. We conclude by summarizing the key recommendations that emerge from this analysis, examining the structural and institutional challenges that the community must confront as ML-based mapping matures from a research activity into an operational capability, and acknowledging the limitations of the present work.