Comparing non-adequate test suites using coverage criteria

Milos Gligoric, Alex Groce, Chaoqiang Zhang, Rohan Sharma, Mohammad Amin Alipour, Darko Marinov

Research output: Chapter in Book/Report/Conference proceedingConference contribution

92 Scopus citations

Abstract

A fundamental question in software testing research is how to compare test suites, often as a means for comparing test-generation techniques. Researchers frequently compare test suites by measuring their coverage. A coverage criterion C provides a set of test requirements and measures how many requirements a given suite satisfies. A suite that satisfies 100% of the (feasible) requirements is C-adequate. Previous rigorous evaluations of coverage criteria mostly focused on such adequate test suites: given criteria C and C′, are C-adequate suites (on average) more effective than C′-adequate suites? However, in many realistic cases producing adequate suites is impractical or even impossible. We present the first extensive study that evaluates coverage criteria for the common case of non-adequate test suites: given criteria C and C′, which one is better to use to compare test suites? Namely, if suites T1, T2...Tn have coverage values c1, c2...cn for C and c′1, c′2...c′n for C′, is it better to compare suites based on c1, c2...cn or based on c′1, c′2...c′n? We evaluate a large set of plausible criteria, including statement and branch coverage, as well as stronger criteria used in recent studies. Two criteria perform best: branch coverage and an intra-procedural acyclic path coverage.

Original languageEnglish (US)
Title of host publication2013 International Symposium on Software Testing and Analysis, ISSTA 2013 - Proceedings
Pages302-313
Number of pages12
DOIs
StatePublished - 2013
Externally publishedYes
Event22nd International Symposium on Software Testing and Analysis, ISSTA 2013 - Lugano, Switzerland
Duration: Jul 15 2013Jul 20 2013

Publication series

Name2013 International Symposium on Software Testing and Analysis, ISSTA 2013 - Proceedings

Conference

Conference22nd International Symposium on Software Testing and Analysis, ISSTA 2013
Country/TerritorySwitzerland
CityLugano
Period7/15/137/20/13

Keywords

  • Coverage criteria
  • non-adequate test suites

ASJC Scopus subject areas

  • Software

Fingerprint

Dive into the research topics of 'Comparing non-adequate test suites using coverage criteria'. Together they form a unique fingerprint.

Cite this