Accuracy Estimation before Obtaining Labels to Accelerate MLOps

Abstract

ML model monitoring is crucial to ensure the reliability of models in operation. However, in real-world use cases, monitoring may not work effectively due to the delays in obtaining labels. This paper proposes a method to estimate the prediction performance of a model for unlabelled data. It trains multiple check models which verify the validity of the model's prediction and utilizes them for accurate estimation. We conducted experiments using various datasets and confirmed that the proposed method outperforms the existing models in estimating accuracy, precision, recall, and F1 score.