PT - JOURNAL ARTICLE AU - Owen L. Petchey AU - Mikael Pontarp AU - Thomas M. Massie AU - Sonia Kéfi AU - Arpat Ozgul AU - Maja Weilenmann AU - Gian Marco Palamara AU - Florian Altermatt AU - Blake Matthews AU - Jonathan M. Levine AU - Dylan Z. Childs AU - Brian J. McGill AU - Michael E. Schaepman AU - Bernhard Schmid AU - Piet Spaak AU - Andrew P. Beckerman AU - Frank Pennekamp AU - Ian S. Pearse TI - The Ecological Forecast Horizon, and examples of its uses and determinants AID - 10.1101/013441 DP - 2015 Jan 01 TA - bioRxiv PG - 013441 4099 - http://biorxiv.org/content/early/2015/01/11/013441.short 4100 - http://biorxiv.org/content/early/2015/01/11/013441.full AB - Forecasts of how ecological systems respond to environmental change are increasingly important. Sufficiently inaccurate forecasts will be of little use, however. For example, weather forecasts are for about one week into the future; after that they are too unreliable to be useful (i.e., the forecast horizon is about one week). There is a general absence of knowledge about how far into the future (or other dimensions, e.g., space, temperature, phylogenetic distance) useful ecological forecasts can be made, in part due to lack of appreciation of the value of ecological forecast horizons. The ecological forecast horizon is the distance into the future (or other dimension) for which useful forecasts can be made. Five case studies illustrate the influence of various sources of uncertainty (e.g., parameter uncertainty, environmental and demographic stochasticity, evolution), level of ecological organisation (e.g., population or community), organismal properties (e.g., body size or number of trophic links) on temporal, spatial and phylogenetic forecast horizons. We propose that the ecological forecast horizon is a flexible and powerful tool for researching and communicating ecological predictability, and for motivating and guiding agenda setting for ecological forecasting research and development.PredictionTwo types of predictions can be distinguished.Explanatory predictions are formulations of what should be expected if the general hypotheses of a theory or model are correct. Their aim is to assist with testing a model or a theory. They can be rejected or not, which validates or not the hypothesis (Popper 2002). Anticipatory predictions are formulations of a possible future assuming that the current interactions and processes will hold in the future (i.e. that the hypothesis of the models/theories are validated and can be extended in to the future). Their aim is to give a statement about what the future will be.ProjectionA statement about the future based on extrapolating models to domains for which there are no data (Coreau et al. 2009).ForecastThe best projection of the future from a model or expert.ScenariosAlternative futures based on consistent sets of assumptions, interactions and driving forces (Bennett et al. 2003); provide a set of plausible pathways to the futures, rather than predicting what the future will actually be (Coreau et al. 2009).AccuracyThe difference between observed and predicted value. High accuracy implies good prediction and low accuracy poor prediction. Accuracy is an important component of forecast proficiency (see below).PrecisionThe amount of uncertainty in predictions. Precise predictions will have low uncertainty (i.e., be closely grouped around the mean prediction). Imprecise predictions will have high uncertainty. Unlike accuracy, very high precision may indicate a poor predictive model that might result, for example, from failing to include a stochastic process. Low precision is also a sign of a poor predictive model. Hence, best is if a predictive model produces a prediction that has the same uncertainty as the real system being modelled.UncertaintyRegan et al. (2002) give two classes of uncertainty: epistemic and linguistic. Epistemic uncertainty is lack of knowledge in the state of a system, for example in parameter values, processes operating, representation of processes, system components, and inherent randomness (also see Clark et al. 2001). See Gregr & Chan (Gregr & Chan 2014) for discussion of the relationship between modelling assumptions and uncertainties.Intrinsic and realised predictabilityBeckage et al. (2011) recognise two types of predictability: the intrinsic predictability of a system, and the realised predictability achieved by a particular model of the system. The intrinsic predictability of a system is the predictability of the best possible model of that system, i.e., it is the greatest achievable predictability. Low realised predictability and high intrinsic predictability implies problems with the predictive model, such as uncertainty in parameter values. High predictability requires an intrinsically predictable system, and low uncertainty about the processes governing the system. A fully deterministic system has perfect intrinsic predictability, since perfect knowledge of parameters and initial conditions results in perfect predictions. A fully deterministic system may, however, be computationally irreducible.Forecast proficiencyA measure of how useful is a forecast, usually some function of accuracy and or precision. We first thought to use instead the term forecast skill, which comes from meteorology and there usually refers to a specific measure of accuracy, mean square error, and has already been used in environmental science to assess forecasts of marine net primary production (Seferian et al. 2014). Forecast skill is, however, often used to mean one measure, mean square error, and we do not wish to be so specific. We propose that in ecology, the term forecast proficiency be general, such that any measure of accuracy or match in precision can be a measure of forecast proficiency. Thus, a model with high accuracy and appropriate precision will have high forecast proficiency. Very high precision or very low precision may both be inappropriate and contribute to lower forecast proficiency.Measures of forecast proficiency for continuous variables include mean error or bias=E(ϵi)=1/n Σ ϵi, which gives a measure of whether predictions are consistently wrong in one direction. Mean squared error is given by MSE=E(ϵi2)=1/n Σ ϵi2. Taking the square root gives root mean squared error, and is in the units of the original variable. Another common measure is variance explained, R2=1-SSE/SST=1-Σϵi2/Σ yi2=1-MSE/VAR(yi). A relative of RMSE that is robust to outliers is Mean Absolute Error MAE=1/n Σ|ϵi|. The correlation between predicted and observed, , is sometimes used but is a weaker assessment since predictions that are biased or not falling on a 1-to-1 line in a predicted vs. observed plot can still have a perfect correlation of one (r=1). MSE has the useful property of combining accuracy and precision.For binary variables, the choices are less obvious. The observed values can be coded as zero and one and the predicted values kept as a probability between 0–1 and the Pearson correlation can be calculated. This is called the point-biserial correlation and is an easily understood metric but the values will be lower than correlation of continuous variables. Alternatively, a confusion matrix can be calculated. A confusion matrix is a 2x2 table giving counts of true and false positives and true and false negatives. One entry, the true positives, is a measure of accuracy, but a number of other values can be calculated from the confusion matrix that correct for an uneven ratio of positives to negatives. One commonly used metric in scenarios requiring thresholding is the AUC or Area Under the Curve (the curve being a receiver operator curve; though see the caveats in the main text).Forecast horizonThe distance in time, space, or environmental parameters at which forecast proficiency falls below the forecast proficiency threshold. Forecast horizon is closely related to concepts such as mean and maximal forecast time (e.g., Salvino et al. 1995).Forecast proficiency thresholdThe value of forecast proficiency above which forecasts are useful, and below which forecasts are not useful.Retrodiction / postdiction / hindcastingEach relates to the practice of testing the predictions of models / theories against observations already in existence at the time when the predictions were made. While care is required to understand how the existing observation might have influenced the predictions, prediction horizons can be calculated, and provide an indication about prediction into the future.