Assessment of the statistical significance of classifications in infrared spectroscopy based diagnostic models

Fourier transform infrared (IR) spectroscopy in combination with multivariate data analysis is a versatile tool that can be applied to disease diagnosis. However, a rigorous validation of the obtained models is necessary in order to obtain robust results. This work evaluates the advantages of the use of permutation testing for determining the statistical significance of the misclassification errors obtained from IR based diagnostic models through cross validation (CV). The model performance, estimated by CV, is compared to a distribution of CV-performance values obtained using randomly permuted class labels. The distribution of ‘random CV-values’ is considered as a null distribution and used to establish the significance of the model estimators obtained using real class labels. ATR-FTIR spectra of serum samples were classified using random forest (RF) classifiers according to two criteria, the tag number (a randomly assigned pseudo class membership) and the level of urea (real class). CV errors obtained were compared to the null distribution of CV errors from a permutation test and an independent validation set. The procedure was evaluated testing typical conditions leading to overoptimistic estimations provided by the CV like e.g. the size of subsamples used during CV, variable selection and the use of replicates. Results show that for the tag number (pseudo class), CV indicated classification errors between 23 and 33% depending on the subsample size employed. Those values were even lower when variable selection or replicates were used. However, permutation testing indicated that those CV errors were non-significant. In contrast, for sample classification according to their levels of urea, all cross validation errors were found to be significant. Although the proposed method is computationally intensive, it provides a simple way of calculating an empirical p-value of the CV-estimator, thus establishing the statistical significance and providing a feasibility indicator especially useful for studies where the number of samples is limited.