Hybrid CNN–transformer model with BM3D and YOLOv8 for early detection of lung cancer in low-dose CT scans

Abstract Lung cancer remains the primary cause of cancer-related deaths throughout the world. The main reason behind this is late diagnosis and the restrictions in the manual interpretation of imaging data. In these days Low-Dose Computed Tomography (LDCT) has been widely adopted for early screening. LDCT contains Low Dose x-rays as compared to the normal CT scan. But the existence of noise and subtle nodular patterns often impairs diagnostic accuracy. In this study, authors proposed a novel hybrid deep learning model which uses BM3D for pre-processing and YOLOv8 for segmentation. Further this model integrates Convolutional Neural Networks (CNNs) with Transformer Encoders to enhance the early detection of lung cancer using LDCT scan images. The model powers the spatial feature extraction with the help of CNNs and the contextual reasoning capability of Transformers to achieve superior classification performance. In this work, during the training of model BM3D filtering (advanced image preprocessing technique) are applied to reduce noise and enhance structural details. Further YOLOv8 is used for segmentation. The proposed hybrid model achieved 93.8% sensitivity, 95.1% accuracy, 94.4% F1-Score, 96.2% Specificity, 0.92 Dice Metric and 0.97 AUC for classification. Experimental results demonstrate that the proposed model outperforms existing models in terms of accuracy, precision, recall, AUC, and Dice coefficient. These findings suggest that the hybrid model holds strong potential as a robust tool for early lung cancer screening and clinical decision support.