How hard does mutation analysis have to be, anyway?

Rahul Gopinath, Amin Alipour, Iftekhar Ahmed, Carlos Jensen, Alex Groce

Research output: Chapter in Book/Report/Conference proceedingConference contribution

22 Scopus citations

Abstract

Mutation analysis is considered the best method for measuring the adequacy of test suites. However, the number of test runs required for a full mutation analysis grows faster than project size, which is not feasible for real-world software projects, which often have more than a million lines of code. It is for projects of this size, however, that developers most need a method for evaluating the efficacy of a test suite. Various strategies have been proposed to deal with the explosion of mutants. However, these strategies at best reduce the number of mutants required to a fraction of overall mutants, which still grows with program size. Running, e.g., 5% of all mutants of a 2MLOC program usually requires analyzing over 100,000 mutants. Similarly, while various approaches have been proposed to tackle equivalent mutants, none completely eliminate the problem, and the fraction of equivalent mutants remaining is hard to estimate, often requiring manual analysis of equivalence. In this paper, we provide both theoretical analysis and empirical evidence that a small constant sample of mutants yields statistically similar results to running a full mutation analysis, regardless of the size of the program or similarity between mutants. We show that a similar approach, using a constant sample of inputs can estimate the degree of stubbornness in mutants remaining to a high degree of statistical confidence, and provide a mutation analysis framework for Python that incorporates the analysis of stubbornness of mutants.

Original languageEnglish (US)
Title of host publication2015 IEEE 26th International Symposium on Software Reliability Engineering, ISSRE 2015
PublisherInstitute of Electrical and Electronics Engineers Inc.
Pages216-227
Number of pages12
ISBN (Electronic)9781509004065
DOIs
StatePublished - Jan 13 2016
Externally publishedYes
Event26th IEEE International Symposium on Software Reliability Engineering, ISSRE 2015 - Gaithersbury, United States
Duration: Nov 2 2015Nov 5 2015

Publication series

Name2015 IEEE 26th International Symposium on Software Reliability Engineering, ISSRE 2015

Conference

Conference26th IEEE International Symposium on Software Reliability Engineering, ISSRE 2015
Country/TerritoryUnited States
CityGaithersbury
Period11/2/1511/5/15

ASJC Scopus subject areas

  • Software
  • Safety, Risk, Reliability and Quality

Fingerprint

Dive into the research topics of 'How hard does mutation analysis have to be, anyway?'. Together they form a unique fingerprint.

Cite this