OptIC project: An intercomparison of optimization techniques for parameter estimation in terrestrial biogeochemical models

We describe results of a project known as OptIC (Optimisation InterComparison) for comparison of parameter estimation methods in terrestrial biogeochemical models. A highly simplified test model was used to generate pseudo-data to which noise with different characteristics was added. Participants in the OptIC project were asked to estimate the model parameters used to generate this data, and to predict model variables into the future. Ten participants contributed results using one of the following methods: Levenberg-Marquardt, adjoint, Kalman filter, Markov chain Monte Carlo and genetic algorithm. Methods differed in how they locate the minimum (gradient-descent or global search), how observations are processed (all at once sequentially), or the number of iterations used, or assumptions about the statistics (some methods assume Gaussian probability density functions; others do not). We found the different methods equally successful at estimating the parameters in our application. The biggest variation in parameter estimates arose from the choice of cost function, not the choice of optimization method. Relatively poor results were obtained when the model-data mismatch in the cost function included weights that were instantaneously dependent on noisy observations. This was the case even when the magnitude of residuals varied with the magnitude of observations. Missing data caused estimates to be more scattered, and the uncertainty of predictions increased correspondingly. All methods gave biased results when the noise was temporally correlated or non-Gaussian, or when incorrect model forcing was used. Our results highlight the need for care in choosing the error model in any optimization.