TY - GEN
T1 - A Practical, Principled Measure of Fuzzer Appeal
T2 - 20th IEEE International Conference on Software Quality, Reliability, and Security, QRS 2020
AU - Gavrilov, Miroslav
AU - Dewey, Kyle
AU - Groce, Alex
AU - Zamanzadeh, Davina
AU - Hardekopf, Ben
N1 - Publisher Copyright:
© 2020 IEEE.
PY - 2020/12
Y1 - 2020/12
N2 - Fuzzers are important bug-finding tools in both academia and industry. To ensure scientific progress, we need a metric for fuzzer comparison. Bug-based metrics are impractical because (1) the definition of "bug"is vague, and (2) mapping bug-revealing inputs to bugs requires extensive domain knowledge.In this paper, we propose an automated method for comparing fuzzers that alleviates these problems. We replace the question "What bugs can this fuzzer find"with "What changes in program behavior over time can this fuzzer detect". Intuitively, fuzzers which find more behavioral changes are likely to find more bugs. However, unlike bugs, behavioral changes are well-defined and readily detectable. Our evaluation, executed on three targets with several fuzzers, shows that our method is consistent with bug-based metrics, but without associated difficulties. While further evaluation is needed to establish superiority, our results show that our method warrants further investigation.
AB - Fuzzers are important bug-finding tools in both academia and industry. To ensure scientific progress, we need a metric for fuzzer comparison. Bug-based metrics are impractical because (1) the definition of "bug"is vague, and (2) mapping bug-revealing inputs to bugs requires extensive domain knowledge.In this paper, we propose an automated method for comparing fuzzers that alleviates these problems. We replace the question "What bugs can this fuzzer find"with "What changes in program behavior over time can this fuzzer detect". Intuitively, fuzzers which find more behavioral changes are likely to find more bugs. However, unlike bugs, behavioral changes are well-defined and readily detectable. Our evaluation, executed on three targets with several fuzzers, shows that our method is consistent with bug-based metrics, but without associated difficulties. While further evaluation is needed to establish superiority, our results show that our method warrants further investigation.
KW - evaluation methodology
KW - evaluation metrics
KW - fuzzing
UR - http://www.scopus.com/inward/record.url?scp=85099318590&partnerID=8YFLogxK
UR - http://www.scopus.com/inward/citedby.url?scp=85099318590&partnerID=8YFLogxK
U2 - 10.1109/QRS51102.2020.00071
DO - 10.1109/QRS51102.2020.00071
M3 - Conference contribution
AN - SCOPUS:85099318590
T3 - Proceedings - 2020 IEEE 20th International Conference on Software Quality, Reliability, and Security, QRS 2020
SP - 510
EP - 517
BT - Proceedings - 2020 IEEE 20th International Conference on Software Quality, Reliability, and Security, QRS 2020
PB - Institute of Electrical and Electronics Engineers Inc.
Y2 - 11 December 2020 through 14 December 2020
ER -