AI-Powered Defense: Leveraging Deep Learning for Effective Malware Detection

Traditional malware detection techniques frequently fail to detect and stop malicious activity in an era where cyber threats are becoming more complex. Any software that enters a computer system without the administrator’s consent is considered malicious software. In order to improve malware detection systems, this paper offers a comparative framework that combines deep learning and machine learning techniques for enhanced malware detection, featuring a novel Temporal–TabNet–LSTM hybrid model. Addressing the challenge of converting static tabular features into meaningful sequences, the model leverages temporal attributes (e.g., process timestamps) and fuses TabNet’s attention mechanism for dynamic feature selection with LSTM’s sequential processing to reduce noise and computational overhead. Evaluated on the Ember 2018 dataset (achieving 97.9% accuracy, surpassing vanilla LSTM at 96.8% and LightGBM at 97.5%) and the Kaggle Malware Detection baseline (99.7%), with validation on the Microsoft Malware Prediction dataset, the framework incorporates traditional ML models (logistic regression, extra trees, K-nearest neighbors, Naive Bayes, and support vector machine) and DL models (TabNet, ANN, LSTM, and sequential Keras). Rigor is ensured through SMOTE for class imbalance mitigation, correlation-based feature selection to address multicollinearity, GridSearchCV hyperparameter tuning, ablation studies, and robustness testing against noise and adversarial perturbations using the Adversarial Robustness Toolbox (ART). Cross-validation and robustness tests confirm generalizability, though we discuss limitations for real-world evasion. These findings contribute benchmarks and a lightweight hybrid for scalable detection.