TY - GEN
T1 - Evaluating Synthetic Bugs
AU - Bundt, Joshua
AU - Fasano, Andrew
AU - Dolan-Gavitt, Brendan
AU - Robertson, William
AU - Leek, Tim
N1 - Funding Information:
DISTRIBUTION STATEMENT A. Approved for public release. Distribution is unlimited. This material is based upon work supported by the Department of Defense under Air Force Contract No. FA8702-15-D-0001. Any opinions, findings, conclusions or recommendations expressed in this material are those of the author?s? and do not necessarily reflect the views of the Department of Defense. Delivered to the U.S. Government with Unlimited Rights, as defined in DFARS Part 252.227-7013 or 7014 ?Feb 2014?. Notwithstanding any copyright notice, U.S. Government rights in this work are defined by DFARS 252.227-7013 or DFARS 252.227-7014 as detailed above. Use of this work other than as specifically authorized by the U.S. Government may violate any copyrights that exist in this work.
Funding Information:
The authors would like to thank Northeastern University’s Research Computing team, the MIT SuperCloud and Lincoln Laboratory Supercomputing Center and the Information Directorate’s AFRL/RITB High Performance Systems Branch for providing HPC resources that contributed to the research reported within this paper. This material is based upon work supported by the National Science Foundation under Grant No. CNS-1916398. Any opinions, findings, and conclusions or recommendations expressed in this material are those of the author?s? and do not necessarily reflect the views of the National Science Foundation. This material is based upon work supported by the Office of Naval Research under Grant No. N00014-19-1-2364. Any opinions, findings, and conclusions or recommendations expressed in this material are those of the author?s? and do not necessarily reflect the views of the Office of Naval Research.
Publisher Copyright:
© 2021 Owner/Author.
PY - 2021/5/24
Y1 - 2021/5/24
N2 - Fuzz testing has been used to find bugs in programs since the 1990s, but despite decades of dedicated research, there is still no consensus on which fuzzing techniques work best. One reason for this is the paucity of ground truth: bugs in real programs with known root causes and triggering inputs are difficult to collect at a meaningful scale. Bug injection technologies that add synthetic bugs into real programs seem to offer a solution, but the differences in finding these synthetic bugs versus organic bugs have not previously been explored at a large scale. Using over 80 years of CPU time, we ran eight fuzzers across 20 targets from the Rode0day bug-finding competition and the LAVA-M corpus. Experiments were standardized with respect to compute resources and metrics gathered. These experiments show differences in fuzzer performance as well as the impact of various configuration options. For instance, it is clear that integrating symbolic execution with mutational fuzzing is very effective and that using dictionaries improves performance. Other conclusions are less clear-cut; for example, no one fuzzer beat all others on all tests. It is noteworthy that no fuzzer found any organic bugs (i.e., one reported in a CVE), despite 50 such bugs being available for discovery in the fuzzing corpus. A close analysis of results revealed a possible explanation: a dramatic difference between where synthetic and organic bugs live with respect to the "main path"discovered by fuzzers. We find that recent updates to bug injection systems have made synthetic bugs more difficult to discover, but they are still significantly easier to find than organic bugs in our target programs. Finally, this study identifies flaws in bug injection techniques and suggests a number of axes along which synthetic bugs should be improved.
AB - Fuzz testing has been used to find bugs in programs since the 1990s, but despite decades of dedicated research, there is still no consensus on which fuzzing techniques work best. One reason for this is the paucity of ground truth: bugs in real programs with known root causes and triggering inputs are difficult to collect at a meaningful scale. Bug injection technologies that add synthetic bugs into real programs seem to offer a solution, but the differences in finding these synthetic bugs versus organic bugs have not previously been explored at a large scale. Using over 80 years of CPU time, we ran eight fuzzers across 20 targets from the Rode0day bug-finding competition and the LAVA-M corpus. Experiments were standardized with respect to compute resources and metrics gathered. These experiments show differences in fuzzer performance as well as the impact of various configuration options. For instance, it is clear that integrating symbolic execution with mutational fuzzing is very effective and that using dictionaries improves performance. Other conclusions are less clear-cut; for example, no one fuzzer beat all others on all tests. It is noteworthy that no fuzzer found any organic bugs (i.e., one reported in a CVE), despite 50 such bugs being available for discovery in the fuzzing corpus. A close analysis of results revealed a possible explanation: a dramatic difference between where synthetic and organic bugs live with respect to the "main path"discovered by fuzzers. We find that recent updates to bug injection systems have made synthetic bugs more difficult to discover, but they are still significantly easier to find than organic bugs in our target programs. Finally, this study identifies flaws in bug injection techniques and suggests a number of axes along which synthetic bugs should be improved.
KW - evaluation
KW - fuzzing
KW - synthetic bugs
UR - http://www.scopus.com/inward/record.url?scp=85108068235&partnerID=8YFLogxK
UR - http://www.scopus.com/inward/citedby.url?scp=85108068235&partnerID=8YFLogxK
U2 - 10.1145/3433210.3453096
DO - 10.1145/3433210.3453096
M3 - Conference contribution
AN - SCOPUS:85108068235
T3 - ASIA CCS 2021 - Proceedings of the 2021 ACM Asia Conference on Computer and Communications Security
SP - 716
EP - 730
BT - ASIA CCS 2021 - Proceedings of the 2021 ACM Asia Conference on Computer and Communications Security
PB - Association for Computing Machinery, Inc
T2 - 16th ACM Asia Conference on Computer and Communications Security, ASIA CCS 2021
Y2 - 7 June 2021 through 11 June 2021
ER -