Representative of What? When a New Sample Falls Outside Your Model's Domain

A cocoa sourcing programme builds a soil organic carbon baseline for its first origin. The team collects samples across the supply shed, fits a model that predicts carbon stock from soil and landscape covariates, and validates it with cross-validation. The R² is 0.78. Good enough to report. A year later the programme expands to a second origin in a different country, and reuses the same model to estimate the new baseline, because rebuilding it would cost a field campaign the budget does not have. The verifier asks one question: what is your evidence that the second origin falls within the range of conditions the model was calibrated on? The team points to the 0.78. But that number describes how well the model predicts inside the first origin's conditions. It says nothing about the second. The room goes quiet.

This is the representativeness problem that monitoring teams meet most often and name least clearly. It is not really about the sample. It is about the relationship between three things: where you measured, where you now want a number, and the slice of the world the model has actually seen.

Two questions hiding in one word

Representativeness gets used for two distinct claims, and conflating them is where trouble starts. The first claim is about a sample and a population: that the sample reflects the area you want to describe, so an average computed from it is a fair estimate of the true average. This is the domain of sampling design, and the sense we work through in How Large Should Your Soil Carbon Sampling Campaign Be? and Signal versus Noise.

The second claim is about a new point and a model. Most monitoring today does not stop at a sample average. It fits a model, an allometric equation, a soil-carbon map, a spectral calibration, a process model such as RothC, or a default emission factor, and applies it to locations where nothing was measured. A prediction at a new location is only as trustworthy as the model's experience of conditions like that location. Outside the range of conditions in the training data, the model is extrapolating, and its error is neither bounded nor knowable from the fit statistics.

These map onto the two classical modes of inference. Design-based inference gets its validity from how the sample was selected, and answers the sample-to-population question. Model-based inference gets its validity from the model being correct where it is applied, and answers the new-point-to-model question 78. A sample can be a textbook probability sample of its own population and still land a prediction well outside a borrowed model's experience. The two questions are independent, and the second is the one that silences the room.

The applicability domain, defined

The concept that governs the second question already has a name, borrowed from a field that learned the lesson early. In chemistry, quantitative structure-activity models predict a molecule's properties from its structure, and regulators noticed these models made confident, wrong predictions for molecules unlike anything in the training set. The response was to define an applicability domain: the region of input space where a model produces reliable predictions, with an explicit rule for deciding whether a new case falls inside it 12.

Environmental monitoring uses the same idea, usually without the name. Every allometric equation is fit over a range of tree diameters 11, and applying it beyond that range is extrapolation, which is why the practical rule is to plot your inventory's diameter distribution against the equation's fitted range, as we describe in How to Calculate the Amount of Carbon in a Tree? A diameter range is an applicability domain in one dimension. A modern soil model depends on many covariates at once, the SCORPAN factors that digital soil mapping formalises 9, and the training data occupy a cloud in that high-dimensional space. The applicability domain is the region of that space the cloud actually covers, and the whole question becomes geometric: does the new point sit inside the cloud, or off in a corner the model never visited?

Does your calibration set cover the new region?

Everything so far treats one point at a time, which is the right question when you are about to report a number for one farm. It is the wrong question when a programme asks whether a model built in one origin can be reused across a whole second origin. That is a question about two distributions, and it has a direct test that takes an afternoon.

Label every calibration sample 0 and every location in the target area 1. Throw away the soil carbon values and keep only the covariates. Now train a classifier to tell the two labels apart, and cross-validate it. If the classifier cannot do better than chance, the two sets are statistically indistinguishable on the variables the model uses, and your calibration data are a plausible sample of the target's conditions. If it separates them easily, you have covariate shift, and now you have a number for it rather than an argument about it. This is the classifier two-sample test, and the theory that licenses reading a distance between domains off a classifier's accuracy is older than its use here 2223.

The classifier gives you three things a headline R² cannot. Its accuracy, read as the area under the ROC curve, is the summary number. Its variable importances say which covariate drives the mismatch, which is what you would actually act on. And its fitted probabilities give the density ratio between the two populations, the quantity that formalises covariate shift 24 and that turns up again when we come to uncertainty.

The second line is the one to put in front of a client. Reweighting a calibration set toward a new target population is standard practice 25, but weights concentrate the work in a shrinking handful of samples, and Kish's effective sample size measures how few are left 26. A 200-sample calibration set that reweights to an effective 14 for the new region has answered the question. That number is harder to argue with than any goodness-of-fit statistic, because it is a count of the data actually doing the work.

Move the sliders below and watch the three numbers move together. Two behaviours are worth provoking deliberately. Push the target area away from the calibration set and the classifier separates them, the effective sample size collapses, and the share of the target inside the domain falls; by the time the area under the curve reaches 0.94, seventy calibration samples are worth an effective fifteen. Then set the shift back to zero and widen the target instead. The classifier drops back to 0.50, apparently declaring the two sets identical, while well under half the target still sits inside the domain.

That second case is the honest limitation of the method, and it matters. A linear classifier separates on the mean, so it is blind to a target that shares the calibration set's centre and simply spans more ground, which is exactly what a supply shed does when it expands into more variable terrain. Use a flexible learner rather than a logistic regression, and never read the classifier on its own: report it alongside the share of the target that falls inside the domain. One number detects a shift in location, the other detects a shift in spread, and a programme can suffer either.

For a monitoring plan, that share is the sentence worth writing down: not “the model is applicable,” but “62% of the new supply shed falls inside the calibration domain, the remainder is concentrated in the high-rainfall north, and here is what we did about it.”

How to tell whether a new point is in-domain

Domain membership is measurable, and the methods form a ladder rather than a menu. Each rung fixes a specific blindness in the one below it, so the useful question is not which method is best but which blindness you can afford.

One variable at a time. For each covariate, does the new site fall between the minimum and the maximum of the training data? This catches the crudest extrapolation in five minutes. The published version of the check, the multivariate environmental similarity surface, scores a location from 0 to 100 by how deep inside the reference distribution its most extreme variable sits, and goes negative once any variable leaves the reference range, with the magnitude giving the departure as a percentage of that variable's range 17. A score of 100 means the point sits at the median of the reference data on every variable.

Its weakness is structural: it takes the minimum across variables, so it is only barely multivariate. A point can be comfortably inside the range of every variable and still be an impossible combination the model never saw, the rainfall of a wet site with the clay content of a dry one. No amount of per-variable checking finds that.

Novel combinations. The extrapolation detection method splits novelty into the two kinds explicitly 16. Its first index sums each variable's departure beyond the reference range, expressed as a fraction of that range, and is zero whenever every variable is in range. Its second index divides a point's Mahalanobis distance from the reference centroid by the largest such distance among the reference data themselves. Values above 1 mean the point is farther from the centre of the training cloud than any training point ever was: a novel combination, invisible to any range check.

Distance to the training cloud. Underneath both sits the Mahalanobis distance, which measures distance in units of the training data's own spread and correlation, so a step along a direction the training data barely explored counts for more than a step along one they covered well. Its reference distribution is chi-squared with as many degrees of freedom as covariates, which gives a threshold rather than a feeling: flag anything beyond the 95th percentile. For a linear model the same idea appears as leverage, the diagonal of the hat matrix, with the long-standing warning line at three times the mean leverage, and a plot of leverage against standardised residual is the diagnostic that regulators in the field that named the applicability domain actually use 1210.

Two caveats decide whether this works in practice. The covariance matrix has to be estimated robustly, because a handful of unusual training points inflate it and the inflated matrix then declares everything to be in-domain. And it inverts badly when covariates are collinear or numerous, which soil covariates reliably are, so work in the space of retained principal components or use a shrinkage estimator. Where variables are mixed continuous and categorical, Gower distance and simple nearest-neighbour distances are the robust non-parametric fallback.

The area of applicability, for maps. When the model produces a wall-to-wall map, the question is spatial: which pixels can the model be applied to? The area of applicability method answers exactly this 3. It standardises the covariates, weights each by its importance in the fitted model, measures the distance from each pixel to its nearest training point in that weighted space, and divides by the mean of all pairwise distances among the training data. Pixels beyond a threshold derived from the training data's own dissimilarities are outside the area of applicability, and the model's reported error does not apply there. A regional map can look complete and authoritative while large parts of it are, by this measure, out of domain 4.

That threshold is worth stating precisely, because it is widely misquoted. It is the outlier-removed maximum of the training points' own dissimilarity indices. The familiar upper-whisker expression, the 75th percentile plus 1.5 times the interquartile range, is the rule for deciding which values count as outliers, not the threshold itself. The preprint that circulates freely specifies a different rule again, the .95 quantile, which was changed during peer review; the published version is the one to implement.

How much support, not just how near. The dissimilarity index looks only at the single nearest training point, so it cannot tell a location propped up by one lonely sample from one backed by two hundred. Counting how many training points fall within the threshold distance of a location closes that gap, and the count correlates with how well the model actually performs there 15. It is a post-hoc measure that costs nothing extra once the distances are computed, and it is the difference between “inside the domain” and “inside the domain, and well supported.”

Spectral calibrations have their own version. If soil carbon is predicted from mid-infrared spectra rather than covariates, principal component analysis of the calibration set defines the space it knows, and two statistics are needed as a pair rather than singly. Hotelling's T² measures how far a new spectrum sits from the centre within that space; the Q-residual measures how far it sits outside it, in directions the calibration never modelled. A sample can pass either alone and still be one the calibration was never trained to read.

The validation trap: why a good R² can lie

There is a specific way this goes wrong that flatters the practitioner. When a model is validated by ordinary random cross-validation, held-out points are scattered among the training points. If the data are spatially autocorrelated, and soil and vegetation data almost always are, then every held-out point has a near neighbour in the training set. The model is being tested almost entirely inside its domain, on easy points. Spatially structured validation, which holds out whole blocks of space, repeatedly reveals that models with excellent random cross-validation scores predict poorly when they have to reach 56. A regional map can be unbiased on average across its extent and still be wrong at every individual project inside it.

The honest response is not simply to swap random for spatial cross-validation and move on, because the right validation design is itself debated: spatial blocking can be pessimistic if it holds out regions the model would never be asked to predict, and the most defensible option, where the budget allows, is a separate validation sample drawn by probability sampling from the target area, which estimates map accuracy without leaning on the model at all 78. The point that survives the debate is simpler: an accuracy number is only meaningful together with a statement of where, in covariate and geographic space, it was measured, and whether the place you now want a prediction resembles that.

There is now a sharper answer than choosing sides. Rather than blocking space arbitrarily, match the validation to the prediction: hold out points so that the distribution of distances from test point to training data resembles the distribution of distances from the places you will actually predict to the training data 13. The cross-validation then measures the reaching the model will really be asked to do, neither flattered by near neighbours nor penalised for regions nobody will ask about. The original formulation left one out at a time and did not scale; the k-fold version brings a four thousand point clustered dataset down from days to about a minute 14, which matters, because the usual reason teams fall back on random cross-validation is that the honest alternative was computationally out of reach. Clustered samples need care of their own 39.

What “inflate the uncertainty” should actually mean

Advice to widen the error bars for borderline points is easy to give and hard to act on. Widen it by how much? There is a rigorous answer, and its limitations are as instructive as its guarantees.

Split conformal prediction builds an interval from held-out residuals rather than from any assumption about the shape of the errors 18. Fit the model on part of the data, compute absolute residuals on a held-out calibration set, and take a specific order statistic of those residuals as the interval half-width. The resulting interval covers the truth at least 1 − α of the time, in finite samples, with no distributional assumption whatsoever.

The second line is not a technicality. Below that many calibration points the method returns an infinitely wide interval, which is its honest way of saying the data cannot support the claim being asked of them. A method that refuses to answer when it cannot is worth more to a monitoring programme than one that always produces a number.

A constant half-width across a whole project is rarely what you want, and the fix is standard: score conformity against fitted quantiles rather than a fitted mean, and the interval adapts to how uncertain the model is at each location while keeping the coverage guarantee 20. Sites in well-sampled conditions get tight intervals, awkward ones get wide intervals, and nobody has to choose the inflation factor by hand.

Here is the catch, and it is the whole subject of this article. That guarantee holds only if the calibration points and the new point are exchangeable, which is precisely what moving to a new region breaks. Conformal prediction is not an escape from the domain problem. Applied naively across a domain boundary, it produces intervals that look rigorous and are not.

What rescues it is the quantity we already built. If you know the density ratio between target and calibration populations, you can weight each calibration residual by it and recover the guarantee under covariate shift 19. The mechanism is elegant: the new point contributes its own weight to the quantile, so as it moves into territory the calibration data barely cover, more of the probability mass sits at infinity and the interval widens on its own, becoming unbounded once that weight exceeds α. The classifier from the coverage section is what supplies the ratio, which is why the two sections belong together. One honest caveat: the guarantee is exact only when the ratio is known, and in practice it is estimated, so treat the coverage as approximate and degrading with the quality of that estimate.

A harder limit is worth knowing before anyone promises it. Coverage conditional on the exact covariate values of a site cannot be achieved non-trivially for continuous covariates: any procedure claiming it must return infinitely wide intervals 21. The workable compromise is to guarantee coverage within strata you declare in advance, by region, soil texture or land use, each with its own calibration set. This is the right shape for MRV, and it carries an operational consequence: a thinly sampled region gets a coarse guarantee or none, which is information rather than a defect.

A decision rule you can defend

Put together, domain testing is a short, repeatable workflow that fits on one page of a monitoring plan. Define the calibration domain (the multivariate distribution of covariates in the training data). Locate each new point relative to it: the per-variable range check first, then a distance metric. Classify it as in-domain, borderline, or out-of-domain against thresholds fixed in advance. Then act on the class: in-domain points get the prediction as is; borderline points get it with an explicitly inflated uncertainty and a flag; out-of-domain points do not get a borrowed number at all, they trigger local reference sampling and re-estimation.

The value of writing this down is that it converts an argument (“we think the new region is similar enough”) into evidence (“here is where each new point falls, and here is the rule we applied”). That is the difference between a claim a verifier accepts and one a verifier probes.

A worked example: one model, two regions

A soil-carbon model is calibrated on full-sun and light-shade farms in one sourcing region. Sourcing expands to a second region of established shaded agroforestry. Five of the model's covariates, checked against the new region's representative site:

Three of five covariates fall outside the calibrated range, and the two that matter most for carbon, rainfall and shade, are among them. A Mahalanobis distance or an area-of-applicability index would put this site well beyond the training cloud. The verdict is unambiguous: the model is out of domain here, and applying it would report a number with no support. This is the same finding, in miniature, that independent evaluations reach at continental scale, where soil-carbon relationships calibrated on temperate row-crop systems do not transfer to shaded perennial systems without local re-estimation. We take up model choice and local recalibration in How Do You Model Soil Carbon Change Over Time?

When the answer is “out”: extending the domain cheaply

“Trigger local sampling” is where most treatments of this subject stop, and it is the point at which a programme manager needs the most help. The useful news is that extending a domain is far cheaper than building one, and the numbers are surprisingly small.

Choose the new points to cover the space, not to mimic the population. Conditioned Latin hypercube sampling, the default in digital soil mapping, selects sites so the sample reproduces the marginal distributions of the covariates and their correlations 29. Coverage sampling instead clusters the target area in covariate space and samples near each cluster centre, spanning the joint space rather than matching its margins 30. For calibrating a model, spanning is usually what you want, and one comparison recommends spreading the sample through the feature space of the most important covariates 31.

Resist anyone selling a clean winner between the two. The evidence is genuinely mixed: coverage sampling wins on average in some comparisons, the Latin hypercube wins at small sample sizes in others, and sample size matters more than the choice of method in most of them. Comparisons based on a single realisation of each design are close to uninformative, because the distributions of outcomes overlap heavily.

You often do not need to rebuild the model at all. Soil spectroscopy learned this first, where a large library that fails locally can be repaired by adding a handful of local samples to it, a practice called spiking. The numbers across the literature are consistent and small: around fifteen local samples cut prediction bias sharply 32, twelve took regional model performance from an R² between 0.07 and 0.36 to an R² between 0.69 and 0.86 33, and twelve to twenty local samples combined with a large library matched site-specific calibrations built from up to three hundred 34. Ten to thirty well-chosen samples is the working range.

The same economy shows up in allometry. Using published equations as priors rather than starting fresh, six trees can estimate parameters as well as forty to sixty in the classical approach 35. This is partial pooling doing the work: the borrowed model supplies the shape, the local data pull it toward local truth, and the between-site variance component is itself a measure of how transferable the relationship was in the first place.

If you can only afford to test, test for bias. A small local sample that cannot support a refit can still answer whether the borrowed model is systematically wrong here. That is a two-sample comparison, and it uses the same power formula as any change-detection design, which we work through in How Large Should Your Soil Carbon Sampling Campaign Be? Substituting the bias you would care about for the minimum detectable difference gives the number of local samples needed to catch it.

One counter-intuitive finding deserves a place in any transfer decision. When pedotransfer functions were moved to a new region, those built on only sand and clay outperformed versions using eleven covariates 38. More predictors mean a higher-dimensional space, a sparser training cloud, and more ways for a new site to fall outside it. Parsimony buys domain coverage, which is a better reason to prefer it than elegance. The cost of getting this wrong is well documented: pantropical equations overestimated biomass by about 40% at one central African site 36, and equations built from small samples carry an average site-level bias near +70%, ranging from −4% to +193% 37.

Resampling, and the domain that moves

There is a second, quieter way representativeness fails, and it shows up precisely when a programme does everything else right. Monitoring is built on resampling: revisit the same network over time, compare, and report the change. The unstated assumption is that the network stays representative of the system it monitors. Systems move. Management changes, a shade programme matures, a drought reshapes the soil moisture regime, land use shifts at the edges of the supply shed. A model calibrated once against the original system can slide out of domain relative to the system that now exists, without anyone re-running a check. Statisticians call this covariate shift or concept drift; in the field it looks like a baseline that quietly stops describing the present.

Drift is testable with the instrument already introduced. Run the domain classifier between monitoring cycles instead of between regions: label cycle-one covariates 0 and cycle-two covariates 1, and see whether a classifier can tell the years apart. If it can, the system has moved relative to the model, and the variable importances name what moved. It costs nothing beyond covariates you already hold, and it turns a warning into a scheduled check.

Why verifiers are starting to ask

The claim that environmental monitoring uses this idea without naming it needs one correction, and it is a useful one. Verra's VM0042 names it. Its definitions section defines a project domain as the set of conditions, including crop type, soil texture and climate, in which model application has been validated 40. The applicability domain is not a borrowed abstraction from chemistry here; it is a defined term in the methodology a large share of soil-carbon projects are written against.

Its companion module turns the definition into an auditable test. Every climate zone or nationally defined agricultural region the project declares must appear in the validation dataset; the three soil textural classes expected to dominate the project area must be included; and the data must span 15 percentage points of clay content. The module states the purpose plainly, which is to confirm the model has not been hyper-calibrated to one combination of conditions and then applied elsewhere. Two limits are worth stating accurately: the requirement attaches to measure-and-model quantification rather than measurement-only crediting, and it sets no minimum number of sites. The discipline is coverage, not sample size.

Every other framework is softer, and it is worth not overstating them. The IPCC treats this as good practice at national scale, advising that Tier 3 models be calibrated and tested against measurements reflecting the variability in climate, soil and land use over which they will be applied. The EU carbon removals regulation, despite frequent claims to the contrary, imposes no model validation requirement at all: its representativeness duty attaches to the baseline, and the substance is deferred to delegated acts still to come. Anyone citing it for this purpose has not read it.

This is no longer only a methodological nicety. The accounting frameworks that govern carbon claims are converging on the demand that uncertainty be quantified and disclosed, and a prediction applied outside a model's domain is uncertainty that has been hidden rather than quantified. The GHG Protocol Land Sector and Removals Guidance, effective 1 January 2027, asks companies to calibrate model-based and remote-sensing approaches with empirical data specific to the land and the management under analysis, and where removals are being claimed it hardens that from a recommendation into a requirement, with resampling at least every five years. The IPCC 2019 Refinement pushes toward higher-tier, locally-appropriate factors over borrowed defaults precisely because defaults are out-of-domain for many of the places they get applied 12. Verifiers are beginning to ask for spatially-structured or independent validation rather than a headline R², and for evidence that a reused model fits the site it is being reused on. The teams that will pass these questions are the ones who tested domain membership before they were asked, and can show the work.

Key takeaways