This study proposes a novel diagnosis framework to decrease the early detection miss rate of colorectal cancer (CRC) polyps by using a hypersensitive vision-based tactile sensor (HySenSe) and a deep residual neural network. The HySenSe generates high-resolution 3D textural images of 160 realistic polyp phantoms for accurate classification via the proposed deep learning (DL) architecture. The DL module explores lightweight dilated convolutions, residual neural network architecture, and transfer learning to overcome the challenge of a small dataset of 229 images. Results show that the proposed architecture outperforms state-of-the-art DL models (i.e., EfficientNet and DenseNet) with a 94% accuracy, offering a promising solution for improving early detection of CRC polyps. The proposed framework can be used as a diagnostic module within tele-assessment medical robots, highlighting the potential of advanced technology and deep learning to revolutionize the early detection and treatment of CRC.