Model Calibration Using Expectation and Variance of Decision Loss

Abstract

Classification models produce probabilities for each class as a measure of confidence in their predictions. Calibration is a technique used to adjust these confidence levels in order to better align them with the actual data. This is particularly important for high-performance models like deep neural networks, which may produce confidence levels that differ from reality. Decision Calibration (DC) is a type of calibration method that uses a user's expected loss (decision loss) when making a decision to calibrate the model. When selecting a model, it is important to consider not only the expectation of the decision loss, but also its variance. In this study, we propose a calibration method taking both the expectation and variance of the decision loss into consideration.