Measuring Effectiveness of Mutant Sets

Rahul Gopinath, Amin Alipour, Iftekhar Ahmed, Carlos Jensen, Alex Groce

Research output: Chapter in Book/Report/Conference proceedingConference contribution

16 Scopus citations

Abstract

Redundancy in mutants, where multiple mutants end up producing the same semantic variant of a program, is a major problem in mutation analysis. Hence, a measure of effectiveness that accounts for redundancy is an essential tool for evaluating mutation tools, new operators, and reduction techniques. Previous research suggests using the size of the disjoint mutant set as an effectiveness measure. We start from a simple premise: test suites need to be judged on both the number of unique variations in specifications they detect (as a variation measure), and also on how good they are at detecting hard-to-find faults (as a measure of thoroughness). Hence, any set of mutants should be judged by how well it supports these measurements. We show that the disjoint mutant set has two major inadequacies - the single variant assumption and the large test suite assumption - when used as a measure of effectiveness in variation. These stem from its reliance on minimal test suites. We show that when used to emulate hard to find bugs (as a measure of thoroughness), disjoint mutant set discards useful mutants. We propose two alternatives: one measures variation and is not vulnerable to either the single variant assumption or the large test suite assumption, the other measures thoroughness. We provide a benchmark of these measures using diverse tools.

Original languageEnglish (US)
Title of host publicationProceedings - 2016 IEEE International Conference on Software Testing, Verification and Validation Workshops, ICSTW 2016
PublisherInstitute of Electrical and Electronics Engineers Inc.
Pages132-141
Number of pages10
ISBN (Electronic)9781509018260
DOIs
StatePublished - Aug 1 2016
Externally publishedYes
Event9th IEEE International Conference on Software Testing, Verification and Validation Workshops, ICSTW 2016 - Chicago, United States
Duration: Apr 10 2016Apr 15 2016

Publication series

NameProceedings - 2016 IEEE International Conference on Software Testing, Verification and Validation Workshops, ICSTW 2016

Conference

Conference9th IEEE International Conference on Software Testing, Verification and Validation Workshops, ICSTW 2016
Country/TerritoryUnited States
CityChicago
Period4/10/164/15/16

Keywords

  • Empirical Analysis
  • Mutation Analysis
  • Software Testing
  • Theoretical Analysis

ASJC Scopus subject areas

  • Software
  • Safety, Risk, Reliability and Quality

Fingerprint

Dive into the research topics of 'Measuring Effectiveness of Mutant Sets'. Together they form a unique fingerprint.

Cite this