Information Theory for Model Diagnostics: Structural Error is Indicated by Trade-Off Between Functional and Predictive Performance

Benjamin L. Ruddell, Darren T. Drewry, Grey S. Nearing

Research output: Contribution to journalArticlepeer-review

17 Scopus citations

Abstract

Because of the possibility of getting the right answers for the wrong reasons, the predictive performance of a complex systems model is not by itself a reliable indicator of hypothesis quality for the purposes of scientific learning about processes. The predictive performance of a structurally adequate model should be an emergent property of its functional performance. In this context, any Pareto trade-off between measures of predictive performance versus functional performance indicates process-level error in the model; this trade-off, if it exists, indicates that the model's predictions are right for the wrong functional reasons. This paper demonstrates a novel concept based on information theory that is capable of attributing observed errors to specific processes. To demonstrate that the concept and method hold true for models and observations of real systems, we employ a minimal single-parameter-variation sensitivity analysis using a sophisticated ecohydrology model, MLCan, for a well-monitored field site (Bondville IL Ameriflux Soybean). We identify both functional and predictive error in MLCan, and also evidence of the hypothesized tradeoffs between the two. This trade-off indicates structural error within MLCan. For example, the sensible heat flux process can be calibrated to achieve good predictive performance at the cost of poor functional performance. In contrast, we find little structural error for processes driven by solar radiation, which appear “right for the right reasons.” This method could be applied broadly to pinpoint process error and structural error in a wide range of system models, beyond the ecohydrological scope demonstrated here.

Original languageEnglish (US)
Pages (from-to)6534-6554
Number of pages21
JournalWater Resources Research
Volume55
Issue number8
DOIs
StatePublished - Aug 1 2019

Keywords

  • benchmarking
  • complex systems
  • hypothesis testing
  • information theory
  • modeling
  • Pareto optimality

ASJC Scopus subject areas

  • Water Science and Technology

Fingerprint

Dive into the research topics of 'Information Theory for Model Diagnostics: Structural Error is Indicated by Trade-Off Between Functional and Predictive Performance'. Together they form a unique fingerprint.

Cite this