Bug synthesis: Challenging bug-finding tools with deep faults

Subhajit Roy, Awanish Pandey, Brendan Dolan-Gavitt, Yu Hu

    Research output: Chapter in Book/Report/Conference proceedingConference contribution

    Abstract

    In spite of decades of research in bug detection tools, there is a surprising dearth of ground-truth corpora that can be used to evaluate the efficacy of such tools. Recently, systems such as LAVA and EvilCoder have been proposed to automatically inject bugs into software to quickly generate large bug corpora, but the bugs created so far differ from naturally occurring bugs in a number of ways. In this work, we propose a new automated bug injection system, Apocalypse, that uses formal techniques-symbolic execution, constraint-based program synthesis and model counting-to automatically inject fair (can potentially be discovered by current bug-detection tools), deep (requiring a long sequence of dependencies to be satisfied to fire), uncorrelated (each bug behaving independent of others), reproducible (a trigger input being available) and rare (can be triggered by only a few program inputs) bugs in large software code bases. In our evaluation, we inject bugs into thirty Coreutils programs as well as the TCAS test suite.We find that bugs synthesized by Apocalypse are highly realistic under a variety of metrics, that they do not favor a particular bug-finding strategy (unlike bugs produced by LAVA), and that they are more difficult to find than manually injected bugs, requiring up around 240× more tests to discover with a state-of-the-art symbolic execution tool.

    Original languageEnglish (US)
    Title of host publicationESEC/FSE 2018 - Proceedings of the 2018 26th ACM Joint Meeting on European So ftware Engineering Conference and Symposium on the Foundations of So ftware Engineering
    EditorsAlessandro Garci, Corina S. Pasareanu, Gary T. Leavens
    PublisherAssociation for Computing Machinery, Inc
    Pages224-234
    Number of pages11
    ISBN (Electronic)9781450355735
    DOIs
    StatePublished - Oct 26 2018
    Event26th ACM Joint European Software Engineering Conference and Symposium on the Foundations of Software Engineering, ESEC/FSE 2018 - Lake Buena Vista, United States
    Duration: Nov 4 2018Nov 9 2018

    Publication series

    NameESEC/FSE 2018 - Proceedings of the 2018 26th ACM Joint Meeting on European Software Engineering Conference and Symposium on the Foundations of Software Engineering

    Other

    Other26th ACM Joint European Software Engineering Conference and Symposium on the Foundations of Software Engineering, ESEC/FSE 2018
    CountryUnited States
    CityLake Buena Vista
    Period11/4/1811/9/18

    Keywords

    • Bug Injection
    • Constraintbased Synthesis
    • Program Synthesis
    • Symbolic Execution

    ASJC Scopus subject areas

    • Software
    • Artificial Intelligence

    Fingerprint Dive into the research topics of 'Bug synthesis: Challenging bug-finding tools with deep faults'. Together they form a unique fingerprint.

  • Cite this

    Roy, S., Pandey, A., Dolan-Gavitt, B., & Hu, Y. (2018). Bug synthesis: Challenging bug-finding tools with deep faults. In A. Garci, C. S. Pasareanu, & G. T. Leavens (Eds.), ESEC/FSE 2018 - Proceedings of the 2018 26th ACM Joint Meeting on European So ftware Engineering Conference and Symposium on the Foundations of So ftware Engineering (pp. 224-234). (ESEC/FSE 2018 - Proceedings of the 2018 26th ACM Joint Meeting on European Software Engineering Conference and Symposium on the Foundations of Software Engineering). Association for Computing Machinery, Inc. https://doi.org/10.1145/3236024.3236084