Sophisticated attackers find bugs in software, evaluate their exploitability, and then create and launch exploits for bugs found to be exploitable. Most efforts to secure software attempt either to eliminate bugs or to add mitigations that make exploitation more difficult. In this paper, we propose a new defensive technique called chaff bugs, which instead targets the bug discovery and exploit creation stages of this process. Rather than eliminating bugs, we instead add large numbers of bugs that are non-exploitable. Attackers who attempt to find and exploit bugs in software will, with high probability, find an intentionally placed non-exploitable bug and waste precious resources in trying to build a working exploit. In a prototype, we demonstrate two strategies for ensuring non-exploitability for memory safety bugs in C/C++ programs and use them to automatically add thousands of non-exploitable bugs to real-world software such as nginx and libFLAC; we show that the functionality of the software is not impaired and demonstrate that our bugs look exploitable to current triage tools. We believe that chaff bugs can serve as an effective deterrent against both human attackers and automated bug-finding tools.