Statistical estimation methods in ecology are prone to overfitting, yet our field has long relied on within‐sample goodness‐of‐fit metrics that inherently favour overly complex models. More recently, cross‐validation using held‐out data has become common practice, but this approach does not necessarily align with the forecasting demands typical of applied ecological decision making. When the goal is to inform planning—making decisions today to achieve desired outcomes tomorrow—models must be evaluated in the same way they will be used: to forecast. We argue that appropriate – and the specific best – model complexity can only be identified when the validation strategy mirrors the temporal structure of the forecasting problem. Using caribou GPS collar movement data, we demonstrate that Unseen Future Validation and Testing—validation on data that occur strictly temporally after all training data—identifies "best" models that generalize better and avoid the overfitting that both within‐sample fit and conventional cross‐validation fail to detect. Our results illustrate that predictive, temporally aligned validation consistently identifies a different set of best movement models than other model scoring procedures: we believe this is doing a better job at correcting for overfitting. We conclude that ecological models intended to support real‐world planning and policy should adopt a predictive validation framework as standard practice to ensure that model selection reflects true forecasting performance and leads to better‐informed decisions for ecological systems.

Keywords: Predictive validation, Predictive Ecology, Caribou, Movement, Neural Nets