What Does a High R² Actually Tell You? Model Evaluation Metrics from Soil Carbon to Tree Biomass

The validation table in a project document usually looks reassuring. Sixty-four sites, an R² of 0.86, a model efficiency of 0.84, a root mean square error of 12 t C/ha. Each number is correct and each is standard. A reader who stops there has still learned less than the table suggests. It does not say whether the model can detect the change the project claims. It may not say whether the model leans high or low, or whether it beats anything simpler.

The gap is not a matter of hidden incompetence. Evaluation metrics answer narrow questions, and the questions differ from one metric to the next in ways their names do not reveal. Hydrology, weather forecasting, chemometrics, forestry and soil science each built their own toolkit, often for good local reasons, and a carbon project inherits all of them at once. One review of hydrological practice covered more than sixty error metrics and gathered them into a single open-source library 1.

This piece complements two others on this site. The piece on calibrating and validating a model covers how to split the data and which basic statistics to report. The piece on representativeness covers where a validated model may be applied. Neither looks closely at the metrics themselves: what each one measures, what it cannot see, and how the same model can receive opposite verdicts from two of them. That is the subject here, with examples drawn from soil carbon stocks and tree biomass.

A high R² can hide an error larger than the change being claimed

Start with the number that appears in nearly every validation table, and with an ambiguity that is rarely acknowledged. R² is not one statistic. In a 1985 note, Kvålseth compared the several formulas in use under the same symbol. Except for linear models with an intercept, they are not generally equivalent 2.

Two of them dominate model evaluation. The first is the squared correlation between predictions and observations. The second compares the squared errors around the one-to-one line with the spread of the observations around their own mean, which is the model efficiency discussed later in this piece. The first ignores bias entirely. The second does not. A table that reports R² without saying which one it means has left the reader to guess.

Both versions share one property, and for a carbon project it is the one that matters most. They measure a share, not a size. Each divides an error quantity by the total variance of the observations. The verdict therefore depends as much on how spread out the validation data is as on how wrong the model is. A large absolute error is a small share of a large total.

An illustration in soil carbon terms makes the scale concrete. Take validation sites whose stocks run from under 10 to about 150 t C/ha around a mean of 80, with a standard deviation of 30 t C/ha. An unbiased model with an R² of 0.84 on the one-to-one line then has a root mean square error of 12 t C/ha. The error variance is the remaining 16 percent of 900, or 144. Suppose a practice adds 0.3 to 0.5 t C/ha per year, an illustrative rate in the range discussed in the removal-rate benchmark. Over a decade it accumulates 3 to 5 t C/ha. The model's typical error is two to four times the change the project exists to measure, and the headline statistic calls it a good model.

The arithmetic runs in reverse just as easily, and it is the most useful thing to do with a reported R². Multiply the standard deviation of the validation observations by the square root of one minus the score, and the result is the error in units. If a paper does not report that standard deviation, its R² cannot be translated into anything a carbon claim can use.

There is a second reason to distrust a correlation-based summary, older and more visual. In 1973 Anscombe published four small datasets that share the same means, the same variances, the same fitted regression line and the same correlation coefficient 3. Plotted, they look nothing alike. One is a clean linear scatter and one a smooth curve. One is a straight line with a single outlier, and one a vertical stack of points with one point far to the side. His argument was that a statistical summary describes the data only when the data looks like what the summary assumes, and that the plot should come first.

Biomass data resembles the fourth of those datasets more often than anyone would like. Destructive sampling is expensive, so a calibration or validation set for an allometry often contains many small trees and very few large ones. One large tree, many times the mass of the others, can account for most of the variance in the sample. A model that predicts that tree reasonably well receives a high R² from it, whatever it does on the rest. Legates and McCabe made the general point for hydrological models: correlation-based measures are oversensitive to extreme values and insensitive to additive and proportional differences between model and observations 4.

Most metrics are three numbers in different combinations

The list of metrics in use is long enough to feel arbitrary. It runs from mean error, mean absolute error and root mean square error to model efficiency and the Kling-Gupta efficiency. It continues with Willmott's index of agreement, Lin's concordance correlation coefficient and the RPD of the spectroscopists, each with normalized and modified variants. Much of that variety disappears once the errors are described by three properties. The first is how far the mean of the predictions lies from the mean of the observations. The second is how the spread of the predictions compares with the spread of the observations. The third is how closely the two vary together.

Mean squared error splits exactly into those three parts. Kobayashi and Salam wrote the partition out in 2000 for comparing simulated with measured values. It has a squared bias term and a term for the difference between the two standard deviations. The third term measures lack of correlation, weighted by the product of the standard deviations 5. Gauch and colleagues proposed a variant that separates the slope of the relationship from the scatter around it. They argued that its three components are “distinct and additive” and “relate transparently to regression parameters” 6. Both answer the question a single RMSE leaves open: is the error a constant offset, a stretched or compressed range, or noise?

The same three numbers rebuild model efficiency. Murphy showed in 1988 how a skill score based on mean squared error decomposes when it is measured against the mean of the observations. It equals the squared correlation minus a penalty for conditional bias and a penalty for unconditional bias 7. Read the other way round, the squared correlation is model efficiency with both bias penalties removed. That is the exact algebraic reason r² cannot see a model that runs ten percent high: the terms that would have caught it are absent from its definition.

Gupta and colleagues used a version of this decomposition in 2009 to diagnose a habit of hydrological models calibrated on efficiency 8. Because the variability term interacts with the correlation term, the best-scoring model tends to reproduce less variability than the observations show. Their response was an alternative criterion, now known as the Kling-Gupta efficiency (KGE). It is one minus the distance from the ideal point in a space whose axes are the correlation, the ratio of standard deviations and the ratio of means 8. Its interpretation has a catch that a later section returns to.

Other indices draw on the same ingredients. Lin's concordance correlation coefficient multiplies the correlation by a factor that falls as the means and the spreads drift apart. It therefore rewards agreement with the one-to-one line, not merely with some line 9. Taylor's diagram places the correlation, the two standard deviations and the error left after removing the mean offset on a single plot. The law of cosines links all four 10. Willmott's 1981 index of agreement divides the squared error by a potential error: the largest error possible, given how far predictions and observations lie from the observed mean 11. Its 2012 refinement uses absolute errors and runs from minus one to one. Over its positive range it is a simple rescaling of the absolute-error efficiency of Legates and McCabe 12. None of them measures anything new. Each picks its own weighting of offset, spread and correlation, and inherits the blind spots of that weighting.

The regression of observed on predicted values offers a fourth view of the same structure. Its slope reports the stretch, and its intercept reports the offset once the slope is one. The calibration piece explains which way round to fit it, and why the choice of axes is not cosmetic13. A broad review of how environmental models are characterized reached the same practical conclusion. Use several complementary metrics, and look at the data graphically before trusting any of them 14.

The tool below makes the weighting visible. It generates a validation set of soil carbon stocks and lets you distort the model one way at a time. The options are a constant offset, a stretched or compressed range, random scatter, a single extreme site, and a wider or narrower validation range. The panel recomputes twelve metrics and splits the squared error into its three sources. Watch which numbers move and which stay still.

The same error scores better on a wider validation range

Any metric that divides by the variance of the observations inherits a property that has nothing to do with the model. Keep the error fixed and widen the range of the validation data, and R², model efficiency and the concordance coefficient all improve. Narrow the range and they fall, below zero for efficiency, while the error in units has not moved.

In the illustration above, the 12 t C/ha model scores an efficiency of 0.84 across sites with a standard deviation of 30 t C/ha. Within one region, where stocks vary with a standard deviation of 12 t C/ha, the same error gives an efficiency of zero. Within one farm, where the standard deviation might be 8 t C/ha, it gives minus 1.25. Nothing about the model differs between the three statements. Only the population it is scored against changes.

The consequence for carbon projects is direct. A model validated across a continental gradient of soils and climates reports skill over the full width of that gradient. The project then applies it within a narrow band of sites that differ mainly by management, which is exactly where the denominator shrinks. The validation statistic is honest about the dataset it was computed on, and silent about the application.

Soil spectroscopy built this property into its standard vocabulary. The ratio of performance to deviation, RPD, divides the standard deviation of the reference values by the standard error of prediction. A widely cited scheme from 2001 sorts soil properties into three categories by RPD. Above 2.0 they are predicted accurately. Between 1.4 and 2.0 the authors judged that prediction could be improved, and below 1.4 it may not be reliable 15. RPD and R² on the one-to-one line compare the same two quantities, the prediction error and the spread of the observations. For an unbiased model, RPD equals one over the square root of one minus that R², so it shares the same dependence on the spread of the samples. Bellon-Maurel and colleagues argued that the standard deviation also describes the spread of skewed soil properties poorly. They proposed the interquartile range in its place, as RPIQ 16.

In the illustration, the same 12 t C/ha model has an RPD of 2.5 across the wide gradient, comfortably in the top class. Within the region it has an RPD of 1.0, in the bottom one. The spectrometer and the model are unchanged. The workbench also shows why the interquartile range helps: with a narrow range, adding one organic-rich site more than doubles the RPD and leaves the RPIQ where it was.

The remedy is not to abandon these metrics but to report what they depend on. A validation table should state the standard deviation, or better the range and the interquartile range, of the validation observations beside every skill score. A reader can then translate the score into units, and judge whether the validation population resembles the population the model will be applied to. That second question belongs to the piece on representativeness.

A skill score is a comparison, so name the rival

Model efficiency was proposed by Nash and Sutcliffe in 1970 for river flow forecasting, as the proportion of the initial variance accounted for by the model17. The initial variance is the spread of the observations around their own mean. Efficiency therefore compares the model with one particular rival: a forecaster who knows the mean of the validation data and predicts it everywhere. Zero means the model does no better than that rival, and a negative value means it does worse.

Whether that rival is a sensible yardstick depends on the problem. Schaefli and Gupta made the point for high mountain catchments fed by snow and ice melt, where flow follows a strong seasonal cycle. There, the mean flow of each calendar day already reaches efficiencies of 0.72 to 0.84, so a high score says little about whether the model adds anything beyond the calendar. They proposed scoring the model against an explicit benchmark suited to the catchment, and reporting this benchmark efficiency alongside the Nash value 18. Legates and McCabe had already endorsed replacing the observed mean in such formulas with a more demanding baseline, such as persistence or a seasonal mean 4.

The Kling-Gupta efficiency, increasingly used in place of model efficiency, changes the rival without announcing it. Knoben and colleagues showed that a model which predicts the observed mean everywhere scores a KGE of about minus 0.41, not zero 19. A KGE of zero is therefore already better than the mean, and KGE values cannot be read on the scale learned from model efficiency. A study that reports a KGE of 0.3, and a reader who files it beside an efficiency of 0.3, are talking about two different things.

A sharper version of the lesson came from land surface modeling. The PLUMBER experiment compared physically based land surface models with simple statistical models driven by the same meteorology at flux tower sites. For sensible heat flux, a linear regression on incoming shortwave radiation, fitted only at the other sites, outperformed all 13 land surface models on the standard statistics 20. The physical models were not useless. But the comparison showed how much of their apparent skill a far simpler rival could already supply, which a score against the mean would not have shown.

Carbon accounting has natural rivals that are rarely used this way. For a soil carbon model, one is the IPCC Tier 1 approach, with default reference stocks and stock change factors, which a Tier 3 model is meant to improve on. Others are a regression of measured change on a few site covariates, or the assumption of no change at all. A standard crop modeling text frames this comparison as a skill score against a naive simple predictor, and treats the observed mean as one choice among several 21. A Tier 3 model that cannot beat the Tier 1 factors on the project's own validation data has not shown what its higher tier promises. Its efficiency against the mean does not change that.

Squared, absolute or relative: decide which misses matter

Once the rival is named, the error itself still has to be summarized, and the three common choices weight misses differently. Root mean square error squares each error before averaging, so a few large misses dominate it. Mean absolute error weights every unit of error equally. Relative errors divide by something, and the choice of divisor turns out to matter more than the choice between the first two.

The choice between RMSE and MAE has a literature of its own. Willmott and Matsuura argued in 2005 that MAE is the natural measure of average error. RMSE, they argued, also varies with the variability of the error magnitudes and with the square root of the number of errors 22. Chai and Draxler replied that RMSE is the better choice when errors are normally distributed, and that avoiding it is not justified23. Hodson's 2022 review offers the useful resolution. RMSE is optimal for normal errors and MAE for errors following a Laplace distribution, so the choice should follow the shape of the errors, not habit 24.

Whichever is chosen, the gap between the two is informative. RMSE can never be smaller than MAE, and the gap widens as a few errors become much larger than the rest 22. For normally distributed errors RMSE is about 1.25 times MAE, so a ratio well above that points to a tail of large misses. In biomass validation that usually means the largest trees. In soil validation it usually means a handful of organic-rich or stony sites that the model handles badly.

Relative errors are where reporting conventions break down. A normalized RMSE divides the RMSE by something to make it unitless. Four divisors are in common use: the mean, the range, the standard deviation and the interquartile range of the observations. In the soil illustration, the RMSE is 12 t C/ha, the mean 80, the range about 145, the standard deviation 30 and the interquartile range about 40. The same model then reports a normalized error of 15, 8, 40 or 30 percent. Two studies compared by normalized RMSE without checking the divisor are not being compared on one scale.

Percentage errors computed point by point fail in a more specific way. The mean absolute percentage error divides each error by its own observation. It is therefore infinite or undefined when an observation is zero, and unstable when observations come close to it 25. Soil carbon change is the textbook instance. Measured change at individual plots straddles zero, with some plots losing carbon over the monitoring period while others gain. A percentage error on those plots mostly measures the arbitrary distance of each observation from zero.

The working rule is to keep errors in the units of the quantity wherever a decision depends on them. Normalize only when comparing across quantities, and state the divisor every time. Hyndman and Koehler's own alternative divides each error by the in-sample mean absolute error of a naive forecast 25. That scaling has the merit of bringing the rival back into the metric.

On a log scale, almost every allometry looks excellent

Allometric equations for tree biomass are usually fitted on logarithms. Regressing the log of biomass on the log of diameter, and often of height and wood density, linearizes a power law and stabilizes a variance that grows with tree size. It is a sound choice for fitting. It becomes a trap when the statistics of the log-scale fit are read as statistics of biomass.

Logarithms compress exactly the variation that matters. A harvest sample can span three orders of magnitude, from saplings of a few kilograms to trees of several tonnes. A log-scale fit across that range can explain nearly all of the log-scale variance while individual trees still miss by a quarter or more. The residual standard error on the log scale is a multiplicative error: a value of 0.36 means a typical tree lies within a factor of about 1.43 of its prediction. The pantropical model with height of Chave and colleagues, fitted to 4,004 destructively harvested trees, reports a residual standard error of 0.357 on the log scale26.

Returning to the original scale adds a second problem. The exponential of a predicted log value estimates the median biomass for that diameter, not the mean. The residual spread is symmetric on the log scale but skewed on the original one, so the median falls below the mean. Baskerville showed in 1972 that ignoring this underestimates biomass systematically, and gave the correction. He estimated the bias at roughly 10 to 20 percent and found it in most, if not all, earlier uses 27. Sprugel's 1983 note restated the usual correction factor, the exponential of half the squared residual standard error, and argued that it should always be applied. He put the typical bias at 10 percent or less, and warned that the residual standard error must come from natural logarithms 28.

With a residual standard error of 0.357, the factor is about 1.066, so an uncorrected equation underestimates mean biomass by about six percent. Six percent is a modest error for one tree. It is not modest as a systematic bias in a project total, because bias does not average out across trees the way random error does.

A common error in model selection comes from the same source. Comparing a log-log model with an untransformed or weighted nonlinear model by R² or by Akaike's information criterion is not valid. The two statistics are computed on different response variables. An R² of 0.97 on log biomass and one of 0.89 on biomass are not two scores on one scale. Furnival proposed an index in 1961 for exactly this comparison, which expresses the residual error of a transformed model in the units of the original variable 29. The FAO and CIRAD manual on allometric equations recommends it for comparing models with different transformations of the response 30. More broadly, Sileshi's review of biomass modeling practice lists the uncritical use of model selection criteria among the most common mistakes. It finds that misuse of R² and AIC has led to the selection of implausible models 31.

The scale of aggregation raises the same question one level up. A project claims carbon for plots and strata, not for single trees, and errors that are random across trees partly cancel in a plot total while bias does not. A model chosen for its tree-level error may not be the one that predicts plot totals best, a point developed in the piece on calculating tree carbon. Evaluation belongs at the scale at which the number is used.

The tool below fits a log-log allometry to a simulated harvest and reports the same fit in both currencies. It shows R² on the log scale and on biomass, the error for individual trees, and the bias in the stand total with and without the correction factor. At the default settings the fit explains 97 percent of the variance of log biomass and 85 percent of the variance of biomass itself. The median tree misses by about a quarter, and the five largest of fifty trees supply nearly two thirds of the biomass variance.

A model can score well on stocks and still miss the change

Soil carbon models are often validated on stocks: measured carbon per hectare at a group of sites, against the model's simulated value at the same sites. The stock is what the field measurement produces, so this is natural. It is also generous to the model in a specific way. Stocks vary enormously between sites, mostly for reasons of texture, climate and land-use history that are known before the model is run. Much of a model's apparent skill on stocks comes from that between-site variation.

The extreme version is a process model initialized on each site's measured starting stock. Its simulated stock after ten years equals the measured starting value plus a simulated change, so it reproduces most of the between-site variation automatically. Its efficiency on stocks will be high whether or not the simulated change bears any relation to reality. What the project needs from it, and what credits are issued against, is the change.

Scored on change, the same model faces a much harder test. The variance of observed change between plots is small, so the denominator of every skill score shrinks. The observed change is also a difference between two noisy measurements, so its noise is large relative to its spread. With plausible values, a model can show an efficiency above 0.95 on stocks and a negative efficiency on change. The tool below reproduces that contrast.

The long-term experiments used to test soil carbon models are mostly evaluated over time, as trajectories within each experiment. In the nine-model comparison of Smith and colleagues, the statistics compared simulated with measured values through time, for each treatment at each experiment 32. That is closer to a test of change than a cross-site scatter of stocks, although only one of the datasets showed a clear trend in its measurements. When the same authors pooled all sites, they warned that the variation between sites far exceeded the change within them. A strong correlation would then appear even if a model were given only the initial carbon values 32.

The practical rule for a carbon project follows. Ask for validation on change, or on differences between treatments at the same site, in addition to validation on stocks. If only stocks are reported, ask how the model was initialized, because a model started from measured stocks has been handed most of its stock-level skill.

An error smaller than the measurement noise deserves suspicion

Every metric above compares the model with observations as if the observations were true. They are not. A measured soil carbon stock combines sampling error, laboratory error and bulk density error, and an observed change between two campaigns combines two such stocks. The piece on detection limits works through how large that error is. For evaluation, the consequence is that part of what a metric calls model error is measurement error, and no model can remove it.

Measurement error therefore imposes an upper limit on attainable skill. In the stocks-and-change tool, a perfect model, with no error at all in the simulated change, still scores an efficiency of only about 0.1 on change at the default settings. The measurement noise in the observed changes is nearly as large as their real spread. A reported efficiency on change well above that limit should prompt a question about how the observations were produced.

Soil modelers have long had a statistical way to account for this. Where an experiment has replicates, the variation between replicates estimates the measurement error, and a lack-of-fit test asks whether the model's error is significantly larger than that. The nine-model comparison used this test alongside its error and efficiency statistics, but only its two Tamworth datasets had the individual replicate values it needs. There, every model that simulated the full record had an error significantly larger than the measurement error 32. Harmel and Smith proposed a more general adjustment for hydrological and water quality models. Each deviation between model and observation is reduced according to the uncertainty of that observation before the goodness-of-fit statistic is computed 33.

The reverse reading matters just as much. A validation error clearly smaller than the known measurement error is not a sign of an exceptional model. It usually means the validation data was not independent of the calibration data, or that the observations were themselves partly produced by a model. The calibration piece explains how that leakage happens.

Decide the tolerance before computing the metric

Sooner or later a reader wants to know whether a score is good enough, and the literature offers thresholds. A widely cited scheme comes from watershed modeling. Moriasi and colleagues proposed in 2007 that a monthly streamflow simulation be judged satisfactory when model efficiency exceeds 0.50. The ratio of RMSE to the observed standard deviation must also be at most 0.70, and percent bias within 25 percent 34. An update in 2015 added statistics, daily and annual time steps and field-scale models, and tightened the streamflow bias limit to 15 percent. Its authors based the new criteria mainly on four hydrologic models and advised using them only with caution for other models 35.

Those thresholds were built for watershed models, with their own data, time steps and decisions. Nothing in them transfers automatically to soil carbon or to biomass. As the earlier sections showed, the same efficiency of 0.5 can describe a model fit for a regional inventory and a model that cannot detect a decade of practice change. A threshold that ignores the validation range and the size of the effect cannot tell the two apart.

Forestry offers a better habit. Robinson and Froese proposed in 2004 to validate models with equivalence tests, which reverse the burden of proof of an ordinary significance test 36. The null hypothesis is that the model is unacceptable: its predictions differ from the observations by more than a stated tolerance. The data must reject that hypothesis before the model is accepted. A small or noisy validation set then fails to validate the model, where an ordinary test would fail to reject it and leave an impression of agreement. Robinson, Duursma and Marshall extended the approach to the intercept and slope of the regression of observed on predicted values 37.

Medicine reached the same position from a different direction. Comparing two methods of clinical measurement, Bland and Altman argued in 1986 that correlation is the wrong statistic, since two methods can correlate strongly and still disagree. They proposed limits of agreement instead: the mean difference plus or minus two standard deviations of the differences 38. Whether disagreement of that size is acceptable is a question about the purpose, which only someone who knows the purpose can answer. Bland and Altman added that the acceptable difference should ideally be defined in advance 38. The same logic applies when a spectroscopic carbon measurement replaces dry combustion, or a terrestrial laser scan replaces a harvest.

What the equivalence test and the limits of agreement share is an order of operations: the tolerance comes first, from the decision, and the statistics second. For a soil carbon project the tolerance can be expressed in the units that matter. It might be an error in modeled change that is small next to the change being credited, or a bias small enough for the uncertainty deduction to absorb. Stating that number before the validation is run is the single most effective protection against a metric chosen after the fact to flatter the model.

When a model returns a range, score the range

Every metric so far scores a single predicted value. Many models now return a distribution, as an ensemble, a Bayesian posterior or a prediction interval, and scoring only its center discards what the model claims about its own uncertainty. The calibration piece covers the first check, whether intervals contain the truth at their stated rate. A complementary family of scores evaluates the whole distribution at once.

The continuous ranked probability score is a standard choice. It compares the predicted cumulative distribution with the observed value, and rewards distributions that are as narrow as possible while remaining calibrated. When the prediction is a single value, it reduces to the absolute error 39. It belongs to the class of proper scoring rules reviewed by Gneiting and Raftery, under which a forecaster obtains the best expected score by reporting what they actually believe39. Those two properties make it the natural successor to MAE for probabilistic carbon estimates, and it is expressed in the same units.

What a complete score sheet reports

No single metric survives all of the conditions above, and none needs to. A short combination, reported consistently, covers the three properties of the errors, the dependence on the validation population and the scale of the decision. The list below is a minimum, and more is welcome.

None of this asks for exotic statistics. It asks that the numbers in a validation table be chosen for the decision they inform and reported in the units that decision uses. They should then be read with an understanding of what each one cannot see. A model that passes that reading has been evaluated. A model described only by a high R² has been summarized.

Punti chiave