A Practical, Principled Measure of Fuzzer Appeal: A Preliminary Study

Miroslav Gavrilov, Kyle Dewey, Alex Groce, Davina Zamanzadeh, Ben Hardekopf

Research output: Chapter in Book/Report/Conference proceedingConference contribution

Abstract

Fuzzers are important bug-finding tools in both academia and industry. To ensure scientific progress, we need a metric for fuzzer comparison. Bug-based metrics are impractical because (1) the definition of "bug"is vague, and (2) mapping bug-revealing inputs to bugs requires extensive domain knowledge.In this paper, we propose an automated method for comparing fuzzers that alleviates these problems. We replace the question "What bugs can this fuzzer find"with "What changes in program behavior over time can this fuzzer detect". Intuitively, fuzzers which find more behavioral changes are likely to find more bugs. However, unlike bugs, behavioral changes are well-defined and readily detectable. Our evaluation, executed on three targets with several fuzzers, shows that our method is consistent with bug-based metrics, but without associated difficulties. While further evaluation is needed to establish superiority, our results show that our method warrants further investigation.

Original languageEnglish (US)
Title of host publicationProceedings - 2020 IEEE 20th International Conference on Software Quality, Reliability, and Security, QRS 2020
PublisherInstitute of Electrical and Electronics Engineers Inc.
Pages510-517
Number of pages8
ISBN (Electronic)9781728189130
DOIs
StatePublished - Dec 2020
Event20th IEEE International Conference on Software Quality, Reliability, and Security, QRS 2020 - Macau, China
Duration: Dec 11 2020Dec 14 2020

Publication series

NameProceedings - 2020 IEEE 20th International Conference on Software Quality, Reliability, and Security, QRS 2020

Conference

Conference20th IEEE International Conference on Software Quality, Reliability, and Security, QRS 2020
Country/TerritoryChina
CityMacau
Period12/11/2012/14/20

Keywords

  • evaluation methodology
  • evaluation metrics
  • fuzzing

ASJC Scopus subject areas

  • Artificial Intelligence
  • Computer Networks and Communications
  • Safety, Risk, Reliability and Quality
  • Modeling and Simulation
  • Software

Fingerprint

Dive into the research topics of 'A Practical, Principled Measure of Fuzzer Appeal: A Preliminary Study'. Together they form a unique fingerprint.

Cite this