TY - GEN
T1 - Characterizing and Improving Bug-Finders with Synthetic Bugs
AU - Hu, Yu
AU - Shen, Zekun
AU - Dolan-Gavitt, Brendan
N1 - Funding Information:
This research was supported in part by National Science Foundation (NSF) Award 1657199. Any opinions, findings, conclusions, or recommendations expressed are those of the authors and not necessarily of the NSF.
Publisher Copyright:
© 2022 IEEE.
PY - 2022
Y1 - 2022
N2 - Automated bug-finding tools such as KLEE have achieved mainstream success over the last decade, and have proved capable of finding deep bugs even in programs that have received significant manual testing. Some recent works have demonstrated techniques for finding bugs in these bug-finding tools themselves; however, it remains unclear whether these correctness issues have any practical impact on their ability to uncover serious bugs. In this paper, we study this issue by conducting experiments with KLEE 1.4 and 2.2 on several corpora of memory safety bugs. Using automated bug injection, we can automatically find false negatives (i.e., bugs missed by KLEE); moreover, because the bugs we inject come with triggering inputs, we can then use concolic execution to tell which bugs were missed due path explosion and which are caused by soundness issues in KLEE. Our evaluation uncovers several sources of unsoundness, including a limitation in how KLEE detects memory errors, mismatches in the modeling of the C standard library, lack of support for floating point and C++, and issues with calls to external functions. Our results suggest that bug injection and other synthetic corpora can help highlight implementation issues in current tools and illuminate directions for future research in automated software engineering.
AB - Automated bug-finding tools such as KLEE have achieved mainstream success over the last decade, and have proved capable of finding deep bugs even in programs that have received significant manual testing. Some recent works have demonstrated techniques for finding bugs in these bug-finding tools themselves; however, it remains unclear whether these correctness issues have any practical impact on their ability to uncover serious bugs. In this paper, we study this issue by conducting experiments with KLEE 1.4 and 2.2 on several corpora of memory safety bugs. Using automated bug injection, we can automatically find false negatives (i.e., bugs missed by KLEE); moreover, because the bugs we inject come with triggering inputs, we can then use concolic execution to tell which bugs were missed due path explosion and which are caused by soundness issues in KLEE. Our evaluation uncovers several sources of unsoundness, including a limitation in how KLEE detects memory errors, mismatches in the modeling of the C standard library, lack of support for floating point and C++, and issues with calls to external functions. Our results suggest that bug injection and other synthetic corpora can help highlight implementation issues in current tools and illuminate directions for future research in automated software engineering.
KW - bug-finding
KW - software security
KW - symbolic execution
UR - http://www.scopus.com/inward/record.url?scp=85135820658&partnerID=8YFLogxK
UR - http://www.scopus.com/inward/citedby.url?scp=85135820658&partnerID=8YFLogxK
U2 - 10.1109/SANER53432.2022.00115
DO - 10.1109/SANER53432.2022.00115
M3 - Conference contribution
AN - SCOPUS:85135820658
T3 - Proceedings - 2022 IEEE International Conference on Software Analysis, Evolution and Reengineering, SANER 2022
SP - 971
EP - 982
BT - Proceedings - 2022 IEEE International Conference on Software Analysis, Evolution and Reengineering, SANER 2022
PB - Institute of Electrical and Electronics Engineers Inc.
T2 - 29th IEEE International Conference on Software Analysis, Evolution and Reengineering, SANER 2022
Y2 - 15 March 2022 through 18 March 2022
ER -