TY - GEN
T1 - Practical Automatic Lightweight Nondeterminism and Flaky Test Detection and Debugging for Python
AU - Groce, Alex
AU - Holmes, Josie
N1 - Publisher Copyright:
© 2020 IEEE.
PY - 2020/12
Y1 - 2020/12
N2 - A critically important, but surprisingly neglected, aspect of system reliability is system predictability. Many soft-ware systems are implemented using mechanisms (unsafe languages, concurrency, caching, stochastic algorithms, environmental dependencies) that can introduce unexpected and unwanted behavioral nondeterminism. Such nondeterministic behavior can result in software bugs and flaky tests as well as causing problems for test reduction, differential testing, and automated regression test generation. We show that lightweight techniques, requiring little effort on the part of developers, can extend an existing testing system to allow detection and debugging of nondeterminism. We show how to make delta-debugging effective for probabilistic faults in general, and that our methods can improve mutation score by 6% for a strong, full differential test harness for a widely used mock file system.
AB - A critically important, but surprisingly neglected, aspect of system reliability is system predictability. Many soft-ware systems are implemented using mechanisms (unsafe languages, concurrency, caching, stochastic algorithms, environmental dependencies) that can introduce unexpected and unwanted behavioral nondeterminism. Such nondeterministic behavior can result in software bugs and flaky tests as well as causing problems for test reduction, differential testing, and automated regression test generation. We show that lightweight techniques, requiring little effort on the part of developers, can extend an existing testing system to allow detection and debugging of nondeterminism. We show how to make delta-debugging effective for probabilistic faults in general, and that our methods can improve mutation score by 6% for a strong, full differential test harness for a widely used mock file system.
KW - delta debugging
KW - flaky tests
KW - nondeterminism
KW - test generation
KW - test reduction
UR - http://www.scopus.com/inward/record.url?scp=85099311629&partnerID=8YFLogxK
UR - http://www.scopus.com/inward/citedby.url?scp=85099311629&partnerID=8YFLogxK
U2 - 10.1109/QRS51102.2020.00035
DO - 10.1109/QRS51102.2020.00035
M3 - Conference contribution
AN - SCOPUS:85099311629
T3 - Proceedings - 2020 IEEE 20th International Conference on Software Quality, Reliability, and Security, QRS 2020
SP - 188
EP - 195
BT - Proceedings - 2020 IEEE 20th International Conference on Software Quality, Reliability, and Security, QRS 2020
PB - Institute of Electrical and Electronics Engineers Inc.
T2 - 20th IEEE International Conference on Software Quality, Reliability, and Security, QRS 2020
Y2 - 11 December 2020 through 14 December 2020
ER -