Test suites effectiveness evolution in open source systems: empirical study

Abstract

Test suite code coverage is usually used to indicate the capability of a test suite in detecting faults. Earlier research studies, which explored the relationship among test suite effectiveness and code coverage, have not addressed this relationship evolutionally. Moreover, some of these works were studied small or identical domain systems, which make the result generalization process unclear for other systems. Finally, some of these studies were conducted with automatically generated test suites, which might not present the real situation for studied systems, so the results cannot be generalized to real test suites. In this paper, the authors empirically explore three open-source software systems along with their 11 versions. These versions are evolved over time and might have more sources of code and test suites. This work objective is to study the correlation between test suite effectiveness, the size of the test suite, and coverage for three Java programs during their evolution. In this work, the code coverage, test suite LOC and mutation testing coverage are measured to assess the correlation between the effectiveness of fault detection, code coverage, and test suite size. Based on the result we cannot generalize the assumption that test size is always revealing a positive correlation with its effectiveness, but still weak to the high correlation between test effectiveness, test size, and coverage.