Model selection and timing of acquisition date impacts classification accuracy: A case study using hyperspectral imaging to detect white pine blister rust over time

Marja Haagsma, Gerald F.M. Page, Jeremy S. Johnson, Christopher Still, Kristen M. Waring, Richard A. Sniezko, John S. Selker

Research output: Contribution to journalArticlepeer-review

1 Scopus citations

Abstract

Hyperspectral imaging is useful in identifying plant stress over large areas or with large numbers of individuals. The vast data sets make machine learning indispensable, but the choice of machine-learning model, the accuracy of models in extrapolation over time (dynamic data), and timing of measurements require further elucidation. We assessed two metrics of performance for selection of classification model: the predicted accuracy (PA); and the area under the receiver-operating characteristic curve (AUC), both from a 10-fold cross-validation. These metrics were calculated for 22 models that were trained to track white pine blister rust disease in seedlings of southwestern white pine (Pinus strobiformis) on 16 dates. In static data (training and testing data are randomly picked from all dates) PA was comparable with AUC at ranking the models for tested accuracy (Spearman's rank correlation coefficient, hereafter referred to as Spearman's ρ, were 0.58 and 0.54, respectively). However, for dynamic data (training and testing data came from different dates) AUC was more successful at ranking the models for tested accuracy compared to PA (Spearman's ρ were 0.37 and 0.31, respectively). Classification accuracies were 74.3 % and 75.8 % for the top PA and AUC models when applied to dynamic data. However, using a heterogeneous ensemble output, the accuracies increased to 77.3% (PA) and 77.6% (AUC). In comparison, if we selected the models based on the tested accuracies (which would not be possible in a real-life application), the best accuracy was 77.7% for a support-vector machine with a linear kernel. Classification accuracy was affected by the size of the time gap between training and testing dates as well as the timing of training and test date. The decline in accuracy with time lag was asymmetric, being more pronounced in classifying retrospectively, i.e., when the testing date came before the training date, than vice versa. Thus, for this system training a model on an early date resulted in higher accuracies than training a model on a later date. As for the timing, the highest average accuracies were obtained with a classifier trained on data acquired during the onset of the disease, which in this study was on DOY 116.

Original languageEnglish (US)
Article number106555
JournalComputers and Electronics in Agriculture
Volume191
DOIs
StatePublished - Dec 2021

Keywords

  • Digital phenotyping
  • Heterogeneous ensemble
  • Machine learning
  • Model selection
  • Phenological change

ASJC Scopus subject areas

  • Forestry
  • Agronomy and Crop Science
  • Computer Science Applications
  • Horticulture

Fingerprint

Dive into the research topics of 'Model selection and timing of acquisition date impacts classification accuracy: A case study using hyperspectral imaging to detect white pine blister rust over time'. Together they form a unique fingerprint.

Cite this