In existing literature, various machine learning models have been applied to detect cyber attacks on the power grid. None of them, however, consider the degradation of the model over time due to the distributed and dynamic nature of the power system. At the same time, they also fail to recognize natural events, such as line maintenance, since they are based on binary classification (attack/no attack). In an effort to develop a cyber security protection strategy that will work robustly for an extended period of time, we develop a methodology based on the LightGBM framework, which performs well for a) Training for multi-class events (no attack/natural event/attack), and b) Fast, dynamic retraining with concept drift detection. We use an ensemble learning-based classifier for classifying the events generated through our Real Time Digital Simulatorwith commercial devices in a Hardware-in-The-Loop setup. The proposed novel classification model outperforms binary classifier-based approaches, resulting in an over 97% effectiveness with the inclusion of concept drift detection and retraining.