Comparison of Resampling Methods for Bias-Reduced Estimation of Prediction Error: A Simulation Study Based on Real Datasets from Biomarker Discovery Studies

抄録

Stepwise logistic regression is the traditional and most commonly used method for identifying biomarkers and evaluating the magnitude of their effects based on clinical data. Here, we evaluated the performance of the resampling methods leave-one-out cross-validation, 10-fold cross-validation, bootstrap, and .632+ bootstrap in terms of internal validation of prediction analysis using stepwise logistic regression. We conducted simulation studies to compare the ability of these methods to estimate prediction accuracy based on simulation settings (including statistical models) derived from two real biomarker discovery studies (Ogata et al., Leukemia Research 2012; 36: 1229–1236; Yoshimi et al., Molecular Psychiatry 2016; 21: 1504–1510). The simulation results revealed that leave-one-out cross-validation, 10-fold cross-validation, and .632+ bootstrap were comparable in terms of the root mean square error. We therefore recommend the application of these methods to similar biomarker discovery studies that involve approximately ten biomarkers with or without binary biomarkers (such as sex) and various degrees of correlation between the biomarkers.