The ABC of model evaluation: a visual method for a clearer assessment of model accuracy

Abstract

Models that predict species’ distributions or ecological niches, based on inferred relationships between species occurrence and environmental variables, need to be evaluated for their quality or predictive capacity. However, commonly used model evaluation measures, such as the area under the receiver operating characteristic curve (AUC), correct classification rates (including sensitivity and specificity), Cohen’s kappa, and the true skill statistic, are known to be strongly affected by the species’ prevalence or extent of occurrence in the study area: widespread species yield apparently poorer models than species with restricted distributions. Nevertheless, models for widespread species may actually be quite accurate and correlate with independent measures such as species abundance. The problem lies in that these evaluation measures assess only the discrimination capacity of models: they start by converting continuous model predictions into an overly simple binary output (or a series of possible binary outputs, in the case of the AUC), and then evaluate the performance of these binary outputs instead of what the models really predict. We argue that, when the modelling aim does not require binary predictions, discrimination measures should be replaced (or at least complemented) with calibration or reliability measures applied directly to the continuous predictions. We propose a new model evaluation method, the Area Between the Curves (ABC), which can be applied to presence probability models and provides, besides various quantitative metrics, a visually explicit plot showing how much, in which direction and where in the explanatory and predictor space each model departs from observed distribution patterns. It also includes several methods to divide the predicted probability values into bins that allow meaningful comparisons between observations and predictions. We illustrate this procedure on a set of generalised linear models built for European tree distributions (http://www.euforgen.org/distribution_maps.html) gridded on UTM 50 × 50 km cells, including restricted to widespread species (prevalences ranging from 0.4% to 73%). We show that, unlike common evaluation measures based on discrimination ability, the ABC is not conditioned by species prevalence and, together with its associated plot, gives a clearer and more informative picture of the accuracy and weaknesses of model predictions. We also illustrate the advantages of the ABC over previously existing (albeit underused) measures of model reliability, such as the Hosmer-Lemeshow goodness-of-fit test and Miller’s (Cox’s) calibration statistics.