TY - JOUR
T1 - Sensitivity of inferred climate model skill to evaluation decisions
T2 - A case study using CMIP5 evapotranspiration
AU - Schwalm, Christopher R.
AU - Huntinzger, Deborah N.
AU - Michalak, Anna M.
AU - Fisher, Joshua B.
AU - Kimball, John S.
AU - Mueller, Brigitte
AU - Zhang, Ke
AU - Zhang, Yongqiang
PY - 2013/4
Y1 - 2013/4
N2 - Confrontation of climate models with observationally-based reference datasets is widespread and integral to model development. These comparisons yield skill metrics quantifying the mismatch between simulated and reference values and also involve analyst choices, or meta-parameters, in structuring the analysis. Here, we systematically vary five such meta-parameters (reference dataset, spatial resolution, regridding approach, land mask, and time period) in evaluating evapotranspiration (ET) from eight CMIP5 models in a factorial design that yields 68 700 intercomparisons. The results show that while model-data comparisons can provide some feedback on overall model performance, model ranks are ambiguous and inferred model skill and rank are highly sensitive to the choice of meta-parameters for all models. This suggests that model skill and rank are best represented probabilistically rather than as scalar values. For this case study, the choice of reference dataset is found to have a dominant influence on inferred model skill, even larger than the choice of model itself. This is primarily due to large differences between reference datasets, indicating that further work in developing a community-accepted standard ET reference dataset is crucial in order to decrease ambiguity in model skill.
AB - Confrontation of climate models with observationally-based reference datasets is widespread and integral to model development. These comparisons yield skill metrics quantifying the mismatch between simulated and reference values and also involve analyst choices, or meta-parameters, in structuring the analysis. Here, we systematically vary five such meta-parameters (reference dataset, spatial resolution, regridding approach, land mask, and time period) in evaluating evapotranspiration (ET) from eight CMIP5 models in a factorial design that yields 68 700 intercomparisons. The results show that while model-data comparisons can provide some feedback on overall model performance, model ranks are ambiguous and inferred model skill and rank are highly sensitive to the choice of meta-parameters for all models. This suggests that model skill and rank are best represented probabilistically rather than as scalar values. For this case study, the choice of reference dataset is found to have a dominant influence on inferred model skill, even larger than the choice of model itself. This is primarily due to large differences between reference datasets, indicating that further work in developing a community-accepted standard ET reference dataset is crucial in order to decrease ambiguity in model skill.
KW - CMIP5
KW - climate models
KW - evapotranspiration
KW - model validation
UR - http://www.scopus.com/inward/record.url?scp=84880882512&partnerID=8YFLogxK
UR - http://www.scopus.com/inward/citedby.url?scp=84880882512&partnerID=8YFLogxK
U2 - 10.1088/1748-9326/8/2/024028
DO - 10.1088/1748-9326/8/2/024028
M3 - Article
AN - SCOPUS:84880882512
SN - 1748-9326
VL - 8
JO - Environmental Research Letters
JF - Environmental Research Letters
IS - 2
M1 - 024028
ER -