Practical Automatic Lightweight Nondeterminism and Flaky Test Detection and Debugging for Python

Alex Groce, Josie Holmes

Research output: Chapter in Book/Report/Conference proceedingConference contribution

3 Scopus citations

Abstract

A critically important, but surprisingly neglected, aspect of system reliability is system predictability. Many soft-ware systems are implemented using mechanisms (unsafe languages, concurrency, caching, stochastic algorithms, environmental dependencies) that can introduce unexpected and unwanted behavioral nondeterminism. Such nondeterministic behavior can result in software bugs and flaky tests as well as causing problems for test reduction, differential testing, and automated regression test generation. We show that lightweight techniques, requiring little effort on the part of developers, can extend an existing testing system to allow detection and debugging of nondeterminism. We show how to make delta-debugging effective for probabilistic faults in general, and that our methods can improve mutation score by 6% for a strong, full differential test harness for a widely used mock file system.

Original languageEnglish (US)
Title of host publicationProceedings - 2020 IEEE 20th International Conference on Software Quality, Reliability, and Security, QRS 2020
PublisherInstitute of Electrical and Electronics Engineers Inc.
Pages188-195
Number of pages8
ISBN (Electronic)9781728189130
DOIs
StatePublished - Dec 2020
Externally publishedYes
Event20th IEEE International Conference on Software Quality, Reliability, and Security, QRS 2020 - Macau, China
Duration: Dec 11 2020Dec 14 2020

Publication series

NameProceedings - 2020 IEEE 20th International Conference on Software Quality, Reliability, and Security, QRS 2020

Conference

Conference20th IEEE International Conference on Software Quality, Reliability, and Security, QRS 2020
Country/TerritoryChina
CityMacau
Period12/11/2012/14/20

Keywords

  • delta debugging
  • flaky tests
  • nondeterminism
  • test generation
  • test reduction

ASJC Scopus subject areas

  • Artificial Intelligence
  • Computer Networks and Communications
  • Safety, Risk, Reliability and Quality
  • Modeling and Simulation
  • Software

Fingerprint

Dive into the research topics of 'Practical Automatic Lightweight Nondeterminism and Flaky Test Detection and Debugging for Python'. Together they form a unique fingerprint.

Cite this