Knee osteoarthritis (OA) is a chronic degenerative disorder of joints and it is the most common reason leading to total knee joint replacement. Diagnosis of OA involves subjective judgment on symptoms, medical history, and radiographic readings using Kellgren-Lawrence grade (KL-grade). Deep learning-based methods such as Convolution Neural Networks (CNN) have recently been applied to automatically diagnose radiographic knee OA. In this study, we applied Residual Neural Network (ResNet) to first detect knee joint from radiographs and later combine ResNet with Convolutional Block Attention Module (CBAM) to make a prediction of the KL-grade automatically. The proposed model achieved a multi-class average accuracy of 74.81%, mean squared error of 0.36, and quadratic Kappa score of 0.88, which demonstrates a significant improvement over the published results. The attention maps were analyzed to provide insights on the decision process of the proposed model11Code is available at https://github.com/denizlab/OAI-KL-Grade-Classification.