Evasion-Aware Neyman-Pearson Detectors: A Game-Theoretic Approach

Network security relies heavily on the detection of adversarial behaviors. Traditional detection methods, such as anomaly detection and signature detection, are inadequate for detecting increasingly intelligent, deceptive, and stealthy attacks. They are designed to be capable of evading well-known detection strategies strategically. This work aims to develop an evasion-aware detection theory to counteract such adversaries. We focus on extending a fundamental class of Neyman-Pearson (NP) hypothesis testing techniques, which have been widely used for anomaly detection and intrusion detection problems in network security. We propose game-theoretic models to capture evasion-aware NP detectors. By analyzing both the equilibrium behaviors of the attacker and the NP detector, we characterize their performance using Equilibrium Receiver-Operational-Characteristic (EROC) curves. We show that the evasion-aware NP detectors outperform the passive ones in the way that the former can act strategically against the attacker's behavior and adaptively modify their decision rules based on the received messages. We use a case study of anomaly detection to corroborate the analytical results. This work creates a theoretical underpinning for building next-generation evasion-aware detection systems that can better cope with ever-developing cyber attacks.