Region-based Detection of Essential Differences in Image-based Visual Regression Testing

Abstract

Visual regression testing (VRT) is a useful method for confirming that application screens are correctly displayed. VRT systems detect differences between the screens of an old version and a new version of an application to support the tester in detecting failures on the screen of the new version. One approach to VRT is image-based; i.e., before and after screenshot images are compared. It is particularly promising because screenshots are independent of the application's environment (operating system, web browser, etc.). Existing image-based VRT systems simply compare two images in pixel units and highlight pixels with differences, so if there are changes that affect the entire screen (e.g., parallel movements of screen elements), a large number of unessential differences are detected, and the essential differences are buried within them. An image-based VRT method named ReBDiff is presented that solves this problem. Before and after screen images are each divided into multiple regions, and appropriate matchings are made between corresponding regions in the two images. For each matching, differences such as shift, alteration, and addition, if any, are detected. In addition, suitable views are provided on the basis of the detected differences. By observing these views, the tester can efficiently identify the essential differences even when there are changes that affect the entire screen, e.g., parallel movements of screen elements. Experiments on a prototype system using websites for PCs and smartphones and an application screen of an Electron application demonstrated the effectiveness of the proposed method.