TY - GEN
T1 - Generating focused random tests using directed swarm testing
AU - Alipour, Mohammad Amin
AU - Groce, Alex
AU - Gopinath, Rahul
AU - Christi, Arpit
N1 - Publisher Copyright:
© 2016 ACM.
PY - 2016/7/18
Y1 - 2016/7/18
N2 - Random testing can be a powerful and scalable method for finding faults in software. However, sophisticated random testers usually test a whole program, not individual components. Writing random testers for individual components of complex programs may require unreasonable effort. In this paper we present a novel method, directed swarm testing, that uses statistics and a variation of random testing to produce random tests that focus on only part of a program, increasing the frequency with which tests cover the targeted code. We demonstrate the effectiveness of this technique using real-world programs and test systems (the YAFFS2 file system, GCC, and Mozilla's SpiderMonkey JavaScript engine), and discuss various strategies for directed swarm testing. The best strategies can improve coverage frequency for targeted code by a factor ranging from 1.1-4.5x on average, and from nearly 3x to nearly 9x in the best case. For YAFFS2, directed swarm testing never decreased coverage, and for GCC and SpiderMonkey coverage increased for over 99% and 73% of targets, respectively, using the best strategies. Directed swarm testing improves detection rates for real SpiderMonkey faults, when the code in the introducing commit is targeted. This lightweight technique is applicable to existing industrial-strength random testers.
AB - Random testing can be a powerful and scalable method for finding faults in software. However, sophisticated random testers usually test a whole program, not individual components. Writing random testers for individual components of complex programs may require unreasonable effort. In this paper we present a novel method, directed swarm testing, that uses statistics and a variation of random testing to produce random tests that focus on only part of a program, increasing the frequency with which tests cover the targeted code. We demonstrate the effectiveness of this technique using real-world programs and test systems (the YAFFS2 file system, GCC, and Mozilla's SpiderMonkey JavaScript engine), and discuss various strategies for directed swarm testing. The best strategies can improve coverage frequency for targeted code by a factor ranging from 1.1-4.5x on average, and from nearly 3x to nearly 9x in the best case. For YAFFS2, directed swarm testing never decreased coverage, and for GCC and SpiderMonkey coverage increased for over 99% and 73% of targets, respectively, using the best strategies. Directed swarm testing improves detection rates for real SpiderMonkey faults, when the code in the introducing commit is targeted. This lightweight technique is applicable to existing industrial-strength random testers.
KW - Random testing
KW - Regression testing
KW - Swarm testing
UR - http://www.scopus.com/inward/record.url?scp=84984916903&partnerID=8YFLogxK
UR - http://www.scopus.com/inward/citedby.url?scp=84984916903&partnerID=8YFLogxK
U2 - 10.1145/2931037.2931056
DO - 10.1145/2931037.2931056
M3 - Conference contribution
AN - SCOPUS:84984916903
T3 - ISSTA 2016 - Proceedings of the 25th International Symposium on Software Testing and Analysis
SP - 70
EP - 81
BT - ISSTA 2016 - Proceedings of the 25th International Symposium on Software Testing and Analysis
A2 - Roychoudhury, Abhik
A2 - Zeller, Andreas
PB - Association for Computing Machinery, Inc
T2 - 25th International Symposium on Software Testing and Analysis, ISSTA 2016
Y2 - 18 July 2016 through 20 July 2016
ER -