On the limits of mutation reduction strategies

Rahul Gopinath, Mohammad Amin Alipour, Iftekhar Ahmed, Carlos Jensen, Alex Groce

Research output: Chapter in Book/Report/Conference proceedingConference contribution

38 Scopus citations


Although mutation analysis is considered the best way to evaluate the effectiveness of a test suite, hefty computational cost often limits its use. To address this problem, various mutation reduction strategies have been proposed, all seeking to reduce the number of mutants while maintaining the representativeness of an exhaustive mutation analysis. While research has focused on the reduction achieved, the effectiveness of these strategies in selecting representative mutants, and the limits in doing so have not been investigated, either theoretically or empirically. We investigate the practical limits to the effectiveness of mutation reduction strategies, and provide a simple theoretical framework for thinking about the absolute limits. Our results show that the limit in improvement of effectiveness over random sampling for real-world open source programs is a mean of only 13.078%. Interestingly, there is no limit to the improvement that can be made by addition of new mutation operators. Given that this is the maximum that can be achieved with perfect advance knowledge of mutation kills, what can be practically achieved may be much worse. We conclude that more effort should be focused on enhancing mutations than removing operators in the name of selective mutation for questionable benefit.

Original languageEnglish (US)
Title of host publicationProceedings - 2016 IEEE/ACM 38th IEEE International Conference on Software Engineering Companion, ICSE 2016
PublisherIEEE Computer Society
Number of pages12
ISBN (Electronic)9781450339001, 9781450342056
StatePublished - May 14 2016
Externally publishedYes
Event2016 IEEE/ACM 38th IEEE International Conference on Software Engineering, ICSE 2016 - Austin, United States
Duration: May 14 2016May 22 2016

Publication series

NameProceedings - International Conference on Software Engineering
ISSN (Print)0270-5257


Conference2016 IEEE/ACM 38th IEEE International Conference on Software Engineering, ICSE 2016
Country/TerritoryUnited States


  • Mutation analysis
  • Software testing
  • Statistical analysis
  • Theoretical analysis

ASJC Scopus subject areas

  • Software


Dive into the research topics of 'On the limits of mutation reduction strategies'. Together they form a unique fingerprint.

Cite this