A Hybrid HMM-LSTM Model for Advanced Cyber Attack Detection
DOI:
https://doi.org/10.64137/31079911/IJMST-V2I2P102Keywords:
Cybersecurity, Network Security, Intrusion Detection, Machine Learning, Deep Learning, Hidden Markov Model (HMM), Long Short-Term Memory (LSTM), Anomaly DetectionAbstract
The growth, development and connectedness of digital networks and systems have resulted in a dramatic increase in the complexity of cyber attacks, presenting new challenges for conventional intrusion detection systems (IDS). Traditional methods may fail to effectively model temporal correlations and attack patterns in network traffic. To overcome these limitations, we present a hybrid approach in this work that combines the probabilistic nature of Hidden Markov Models (HMM) with the temporal feature learning of Long Short-Term Memory (LSTM) networks. Experiments are conducted on a variety of attack vectors present in benchmark datasets such as the CICIDS2017 dataset and NSL-KDD dataset, which include both contemporary and traditional attacks. Our findings show that the hybrid model outperforms individual HMM and LSTM models in terms of accuracy, precision, recall and F1-score. The combination of statistical and deep learning approaches enhances the capabilities to identify various types of cyber threats, while minimizing false positives. This approach offers a scalable and robust approach for next-generation cloud-based intrusion detection in real-world network settings.
References
[1] Y. Lai, Z. Wang, Z. Lin, Y. Cao, Z. Li, and Q. Ye, “An efficient network intrusion detection model based on beta mixture models,” Knowledge-Based Systems, vol. 330, p. 114506, Nov. 2025, doi: https://doi.org/10.1016/j.knosys.2025.114506.
[2] M. G. Karthik et al., “Energy-efficient intrusion detection with a protocol-aware transformer–spiking hybrid model,” Scientific Reports, vol. 16, no. 1, Feb. 2026, doi: https://doi.org/10.1038/s41598-026-37367-4.
[3] A. Villafranca, Kyaw Min Thant, I. Tasic, and M.-D. Cano, “AI-Enabled IoT Intrusion Detection: Unified Conceptual Framework and Research Roadmap,” Machine Learning and Knowledge Extraction, vol. 7, no. 4, pp. 115–115, Oct. 2025, doi: https://doi.org/10.3390/make7040115.
[4] R. M. Al-Khatib, L. Heilat, W. Qudah, S. Alhatamleh, and A. Al-Khateeb, “A novel improved deep learning model based on Bi-LSTM algorithm for intrusion detection in WSN,” Networks and Heterogeneous Media, vol. 20, no. 2, pp. 532–565, 2025, doi: https://doi.org/10.3934/nhm.2025024.
[5] X. Yuan, J. Wan, D. An, and H. Pei, “A novel encrypted traffic detection model based on detachable convolutional GCN-LSTM,” Scientific Reports, vol. 15, no. 1, Jul. 2025, doi: https://doi.org/10.1038/s41598-025-13397-2.
[6] Afrah Gueriani, Hamza Kheddar, and A. C. Mazari, “Enhancing IoT Security with CNN and LSTM-Based Intrusion Detection Systems,” arXiv (Cornell University), pp. 1–7, Apr. 2024, doi: https://doi.org/10.1109/pais62114.2024.10541178.
[7] A. Farabi et al., “IntrusionX: A Hybrid Convolutional-LSTM Deep Learning Framework with Squirrel Search Optimization for Network Intrusion Detection,” arXiv, 2025. Doi: https://doi.org/10.48550/arXiv.2510.00572
[8] Abdulhakim Alsaiari, and Mohammad Ilyas, “Deep Learning for Smart Grid Intrusion Detection: a Hybrid Cnn-lstm-based Model,” arXiv, 2025. Doi: https://dx.doi.org/10.2139/ssrn.4851226
[9] S. Poddar, S. Aswani, Ram Chandra Sachan, Venkata Nedunoori, and U. Patel, “Enhancing Cloud Network Security With Hybrid Cnn-Lstm Models for Intrusion Detection,” 2021 IEEE 8th Uttar Pradesh Section International Conference on Electrical, Electronics and Computer Engineering (UPCON), pp. 1–5, Nov. 2024, doi: https://doi.org/10.1109/upcon62832.2024.10983857.
[10] X. Wang and R. Stadler, “IT Intrusion Detection Using Statistical Learning and Testbed Measurements,” NOMS 2024-2024 IEEE Network Operations and Management Symposium, Seoul, Korea, pp. 1-7, 2024. Doi: https://doi.org/10.1109/NOMS59830.2024.10575087
[11] A. Ganesan et al., “Hidden Markov mixture models for pattern recognition,” Pattern Analysis and Applications, 2024.
[12] L. R. Rabiner, “A tutorial on hidden Markov models and selected applications in speech recognition,” Proceedings of the IEEE, vol. 77, no. 2, pp. 257–286, 1989, doi: https://doi.org/10.1109/5.18626.
[13] W. Rajeh et al., “Deep maxout network-based IDS for smart city security,” PeerJ Computer Science, 2025.
[14] Mert Nakıp and Erol Gelenbe, “Online Self-Supervised Deep Learning for Intrusion Detection Systems,” IEEE transactions on information forensics and security, pp. 1–1, Jan. 2024, doi: https://doi.org/10.1109/tifs.2024.3402148.
[15] T. Ali et al., “Hybrid deep learning models for network security,” Springer, 2024.
[16] N. Moustafa and J. Slay, “UNSW-NB15: a comprehensive data set for network intrusion detection systems (UNSW-NB15 network data set),” IEEE Xplore, Nov. 01, 2015. https://ieeexplore.ieee.org/document/7348942
[17] I. Sharafaldin, A. Habibi Lashkari, and A. A. Ghorbani, “Toward Generating a New Intrusion Detection Dataset and Intrusion Traffic Characterization,” Proceedings of the 4th International Conference on Information Systems Security and Privacy, 2018, doi: https://doi.org/10.5220/0006639801080116.
[18] O. F. Jeelani et al., “Intrusion detection in IoT healthcare using ML,” IEEE Conference, 2025.
[19] Ian Goodfellow, Yoshua Bengio, and Aaron Courville, Deep Learning, MIT Press, pp. 1-777, 2016.
[20] R. Vinayakumar, M. Alazab, K. P. Soman, P. Poornachandran, A. Al-Nemrat, and S. Venkatraman, “Deep Learning Approach for Intelligent Intrusion Detection System,” IEEE Access, vol. 7, pp. 41525–41550, 2019, doi: https://doi.org/10.1109/access.2019.2895334.
[21] M. Lin et al., “Hybrid ensemble learning for network intrusion detection,” International Journal of Systems Science, 2024.
[22] Jahongir Azimjonov and T. Kim, “Designing accurate lightweight intrusion detection systems for IoT networks using fine-tuned linear SVM and feature selectors,” Computers & security, vol. 137, pp. 103598–103598, Feb. 2024, doi: https://doi.org/10.1016/j.cose.2023.103598.
[23] M. Fatima, O. Rehman, S. Ali, and M. F. Niazi, “ELIDS: Ensemble Feature Selection for Lightweight IDS against DDoS Attacks in Resource-Constrained IoT Environment,” Future Generation Computer Systems, vol. 159, pp. 172–187, Oct. 2024, doi: https://doi.org/10.1016/j.future.2024.05.013.
[24] H. Ding et al., “GAN-based intrusion detection system,” IEEE Transactions on Information Forensics, 2024.
[25] K. Swathi et al., “Deep learning-based IDS for IoT networks,” Knowledge-Based Systems, 2024.
