A Quantitative Analysis of Python Test Smells' Impact on Bug-Proneness of Code Under Test

Abstract

In general, unit test cases are prepared as test programs (test code) and organized to facilitate automated testing. Consequently, whenever developers modify their production code, they can also test them automatically. Hence, test code plays an important role in the quality assurance of production code. However, because test code is also a manually written program, it might have a fault and offer inadequate testing. That is, test code can also contain code smells, which are referred to as “test smells.” Test smells may mislead developers by giving inaccurate test results. In other words, developers may feel a false sense of security about test results when the (smelled) test code overlooked latent bugs and did not fail. This paper focuses on test smells in Python test code and quantitatively analyzes relationships between test smell appearances and the bug-proneness of code under test. The analysis results show the following findings: (1) Specific test smells tend to have higher impacts on the bug-proneness of the code under test. (2) Test smells can be useful predictors for predicting latent bugs in the code under test.