Performance Evaluation Criteria for Partial Blur Segmentation in Classification and Segmentation Contexts

Abstract

The performance evaluation of partial blur segmentation algorithms is a crucial part of research. Current performance evaluation criteria for partial blur segmentation, however, are used without sufficient discussion of the target requirements. The validity of performance metrics and statistical methods depends on the context of the particular problem. This study considers the evaluation of partial blur segmentation in the blurred/sharp region classification and segmentation contexts. This paper discusses two important issues regarding the performance evaluation criteria of partial blur segmentation: (1) the performance metrics and (2) the summarization scheme of metric values for accurate comparison. An experimental comparison of performance metrics shows that Informedness and its absolute value are the most suitable metrics to evaluate partial blur segmentation. The research presented in this paper employs the maximum and averaged values of metrics obtained by naive-thresholding using multiple parameters. These values can capture the performance and sensitivities of the parameters of algorithms in detail. This research employs Otsu’s method and Graphcuts to evaluate the extent to which the performance depends on the segmentation algorithms in automatic applications. These approaches ensure a more accurate comparison than the previous evaluation criteria. Four state-of-the-art partial blur segmentation algorithms are evaluated by the proposed criteria. The results revealed detailed conditional dependences of superiority and some overlooked characteristics of algorithms.