Causal BERT: Improving object detection by searching for challenging groups

Autonomous vehicles (AV) often rely on perception modules built upon neural networks for object detection. These modules frequently have low expected error overall but high error on unknown groups due to biases inherent in the training process. When these errors cause vehicle failure, manufacturers pay humans to comb through the associated images and label what group they are from. Data from that group is then collected, annotated, and added to the training set before retraining the model to fix the issue. In other words, group errors are found and addressed in hindsight. Our main contribution is a method to find such groups in foresight, leveraging advances in simulation as well as masked language modeling in order to perform causal interventions on simulated driving scenes. We then use the found groups to improve detection, exemplified by Diamondback bikes, whose performance we improve by 30 AP points. Such a solution is of high priority because it would greatly improve the robustness and safety of AV systems. Our second contribution is the tooling to run interventions, which will benefit the causal community tremendously.