Enhanced Accuracy Estimation Method of Models in Production to Accelerate MLOps

Abstract

In MLOps, it is essential to estimate the performance of prediction models in production, especially when the target labels are not immediately available. Existing methods utilize ``check'' models, which check the predictions of the prediction model, trained with the conventional cross-entropy loss. This study presents a novel method to estimate accuracy using a check model based on our proposed GBCE loss, which strictly upper bounds the accuracy estimation error. We analyze the generalization error bounds of both the proposed and the cross-entropy-based methods, thereby demonstrating the theoretical superiority of our method. Through numerical experiments with multiple real-world data sets, we confirm the effectiveness of our proposed method, showing up to 56.3% reduction in accuracy estimation error.