A Comprehensive Review of Deep Learning Techniques for Malware Detection and Classification
Main Article Content
Abstract
The rapid growth of sophisticated cyber threats has made malware detection and classification a critical challenge in modern cybersecurity systems. Traditional signature-based malware detection techniques are effective against known threats but often fail to identify new, polymorphic, metamorphic, and zero-day malware variants. Deep learning (DL) has emerged as a promising approach for addressing these limitations because of its ability to automatically learn complex patterns and discriminative features from large-scale malware datasets. This paper presents a comprehensive review of deep learning techniques employed for malware detection and classification, including Convolutional Neural Networks (CNN), Recurrent Neural Networks (RNN), Long Short-Term Memory (LSTM), Autoencoders, Deep Belief Networks (DBN), Generative Adversarial Networks (GAN), Transformer-based models, and hybrid deep learning architectures. The review examines different malware analysis approaches, including static, dynamic, and hybrid analysis, along with commonly used datasets, feature extraction methods, preprocessing techniques, and performance evaluation metrics such as accuracy, precision, recall, F1-score, and area under the ROC curve. Furthermore, the study compares the capabilities of different DL architectures in identifying malware families and distinguishing malicious software from benign applications. Key challenges, including data imbalance, adversarial attacks, obfuscated malware, computational complexity, model interpretability, and limited availability of representative datasets, are also discussed. Finally, emerging research directions such as explainable artificial intelligence, transfer learning, federated learning, graph neural networks, and Transformer-based malware analysis are highlighted. The review provides a comprehensive foundation for researchers and cybersecurity practitioners seeking to develop accurate, robust, and adaptive deep learning-based malware detection and classification systems.
Article Details
This work is licensed under a Creative Commons Attribution-NonCommercial 4.0 International License.
References
M. Alshoulie and A. Mehmood, “Deep Learning Approaches for Malware Detection: A Comprehensive Review of Techniques, Challenges, and Future Directions,” IEEE Access, vol. 13, pp. 118652–118677, 2025, doi: 10.1109/ACCESS.2025.3582875.
P. Maniriho, A. N. Mahmood, and M. J. M. Chowdhury, “A Survey of Recent Advances in Deep Learning Models for Detecting Malware in Desktop and Mobile Platforms,” ACM Computing Surveys, vol. 56, no. 6, Art. 145, pp. 1–41, 2024, doi: 10.1145/3638240.
H. Wang, B. Cui, Q. Yuan, R. Shi, and M. Huang, “A Review of Deep Learning Based Malware Detection Techniques,” Neurocomputing, vol. 598, Art. 128010, 2024, doi: 10.1016/j.neucom.2024.128010.
A. Bensaoud, J. Kalita, and M. Bensaoud, “A Survey of Malware Detection Using Deep Learning,” Machine Learning with Applications, vol. 16, Art. 100546, 2024, doi: 10.1016/j.mlwa.2024.100546.
A. R. Redhu, P. Choudhary, K. S. Srinivasan, and T. K. Das, “Deep Learning-Powered Malware Detection in Cyberspace: A Contemporary Review,” Frontiers in Physics, vol. 12, 2024, Art. 1349463, doi: 10.3389/fphy.2024.1349463.
Omer Aslan and Abdullah Asim Yilmaz, “A New Malware Classification Framework Based on Deep Learning Algorithms”, IEEE Access 2021.
K. Bhargavi N. Vadivelan S. Kodati and M. Nalini "M. Secure internet of battlefield from malicious software using deep eigenspace learning" In AIP Conference Proceedings vol. 2405 no. 1 pp. 030002 2022.
M. D. AlGarni R. AlRoobaea J. Almotiri S. S. Ullah S. Hussain and F. Umar "An efficient convolutional neural network with transfer learning for malware classification" Wireless Communications and Mobile Computing pp. 1-8 2022.
S. Kumar "MCFT-CNN: Malware classification with fine-tune convolution neural networks using traditional and transfer learning in Internet of Things" Future Generation Computer Systems vol. 125 pp. 334-351 2021.
Ö. Aslan and R. Samet, ‘‘A comprehensive review on malware detection approaches,’’ IEEE Access, vol. 8, pp. 6249–6271, Jan. 2020.
R. Komatwar and M. Kokare, ‘‘A survey on malware detection and classification,’’ J. Appl. Secur. Res., pp. 1–31, Aug. 2020.
A. A. Yilmaz, M. S. Guzel, E. Bostanci, and I. Askerzade, ‘‘A novel action recognition framework based on deep-learning and genetic algorithms,’’ IEEE Access, vol. 8, pp. 100631–100644, 2020.
A. A. Yilmaz, M. S. Guzel, I. Askerbeyli, and E. Bostanci, ‘‘A vehicle detection approach using deep learning methodologies,’’ in Proc. Int. Conf. Theor. Appl. Comput. Sci. Eng., Nov. 2018, pp. 64–71.
A. A. Yılmaz, M. S. Guzel, I. Askerbeyli, and E. Bostancı, ‘‘A hybrid facial emotion recognition framework using deep learning methodologies,’’ in Human-Computer Interaction, T. Özseven, Eds. Hauppauge, NY, USA: Nova Science Publishers, 2020.
S. A. Roseline, S. Geetha, S. Kadry, and Y. Nam, ‘‘Intelligent vision-based malware detection and classification using deep random forest paradigm,’’ IEEE Access, vol. 8, pp. 206303–206324, 2020.
M. Nisa, J. H. Shah, S. Kanwal, M. Raza, M. A. Khan, R. Damaševičius, and T. Blažauskas, ‘‘Hybrid malware classification method using segmentation-based fractal texture analysis and deep convolution neural network features,’’ Appl. Sci., vol. 10, no. 14, p. 4966, 2020.
J. Love. (2018). A Brief History of Malware—Its Evolution and Impact. Accessed: Mar. 20, 2021. [Online]. Available: https://www.lastline. com/blog/history-of-malware-its-evolution-and-impact/
D. Palmer. (2017). Ransomware: Security Researchers Spot Emerging New Strain of Malware. Accessed: Mar. 20, 2021.
Ö. Aslan, ‘‘Performance comparison of static malware analysis tools versus antivirus scanners to detect malware,’’ in Proc. Int. Multidisciplinary Stud. Congr. (IMSC), 2017, pp. 1–6.
S. K. Pandey and B. M. Mehtre, ‘‘Performance of malware detection tools: A comparison,’’ in Proc. IEEE Int. Conf. Adv. Commun., Control Comput. Technol., May 2014, pp. 1811–1817.