An Empirical Evaluation on the Impact of Reviewers' Experience and Review Time to Defect Detection Level in Source Code Patch Review

Abstract

In order to investigate the impact of reviewers' experience and review time to comprehension level of source code in software evolution, we conducted an experiment with software development practitioners using source code written in Java along with four modification tasks. Each task had a change specification and corresponding modified source code. The subjects were asked to judge whether the modified source code meets the corresponding change specification. The subjects recorded the time to review and their reasons for the judgments. Each reason for the judgment was categorized into one of the three comprehension levels. The results of 66 subjects showed that there was no statistically significant difference among comprehension levels and time to review. The results of 65 subjects showed that experience with review had large impact on comprehension level.