Enhancing Multi-Agent Reinforcement Learning Intrusion Detection Systems Using Random Forest Q-Value Estimation
Received: 24 March 2025 | Revised: 29 April 2025 and 5 May 2025 | Accepted: 10 May 2025 | Online: 2 August 2025
Corresponding author: Ricky Aurelius Nurtanto Diaz
Abstract
Intrusion Detection Systems (IDSs) analyze network traffic and system activity to identify anomalies or suspicious attack patterns. Various artificial intelligence-based approaches have been explored, including Deep Learning (DL) and Multi-Agent Reinforcement Learning (MARL) to increase their accuracy. This study combines MARL with Random Forest (RF) for Q-value estimation and utilizes two agents, a Detector and a Classifier. The proposed method was evaluated on three public datasets, including UNSW-NB15, NSL-KDD, and UKM-IDS20. The experimental results showed that the Detector Agent achieved higher accuracy (99.95%) compared to the Classifier Agent (80.63%) for the UNSW-NB15 dataset. On the NSL-KDD dataset, both agents performed similarly, with the Detector Agent achieving 99.82% accuracy and the Classifier Agent 99.80%. In contrast, for the UKM-IDS20 dataset, the Classifier Agent slightly outperformed the Detector Agent, with accuracies of 99.98% and 99.94%, respectively. These findings demonstrate the effectiveness of MARL-based IDS and highlight variations in agent performance across different datasets.
Keywords:
IDS, MARL, RF, q-value estimationDownloads
References
C. Yue, L. Wang, D. Wang, R. Duo, and X. Nie, "An Ensemble Intrusion Detection Method for Train Ethernet Consist Network Based on CNN and RNN," IEEE Access, vol. 9, pp. 59527–59539, 2021. DOI: https://doi.org/10.1109/ACCESS.2021.3073413
J. Ghasemi, J. Esmaily, and R. Moradinezhad, "Intrusion detection system using an optimized kernel extreme learning machine and efficient features," Sādhanā, vol. 45, no. 1, Dec. 2019, Art. no. 2. DOI: https://doi.org/10.1007/s12046-019-1230-x
M. A. Khan and J. Kim, "Toward Developing Efficient Conv-AE-Based Intrusion Detection System Using Heterogeneous Dataset," Electronics, vol. 9, no. 11, Nov. 2020, Art. no. 1771. DOI: https://doi.org/10.3390/electronics9111771
P. Rajesh Kanna and P. Santhi, "Unified Deep Learning approach for Efficient Intrusion Detection System using Integrated Spatial–Temporal Features," Knowledge-Based Systems, vol. 226, Aug. 2021, Art. no. 107132. DOI: https://doi.org/10.1016/j.knosys.2021.107132
J. Yang, T. Li, G. Liang, W. He, and Y. Zhao, "A Simple Recurrent Unit Model Based Intrusion Detection System With DCGAN," IEEE Access, vol. 7, pp. 83286–83296, 2019. DOI: https://doi.org/10.1109/ACCESS.2019.2922692
T. T. H. Le, H. Kim, H. Kang, and H. Kim, "Classification and Explanation for Intrusion Detection System Based on Ensemble Trees and SHAP Method," Sensors, vol. 22, no. 3, Jan. 2022, Art. no. 1154. DOI: https://doi.org/10.3390/s22031154
K. H. Le, M. H. Nguyen, T. D. Tran, and N. D. Tran, "IMIDS: An Intelligent Intrusion Detection System against Cyber Threats in IoT," Electronics, vol. 11, no. 4, Jan. 2022, Art. no. 524. DOI: https://doi.org/10.3390/electronics11040524
V. Gowdhaman and R. Dhanapal, "An intrusion detection system for wireless sensor networks using deep neural network," Soft Computing, vol. 26, no. 23, pp. 13059–13067, Dec. 2022. DOI: https://doi.org/10.1007/s00500-021-06473-y
A. M. Aleesa, A. A. Mohammed, and N. M. Sahar, "Deep-intrusion detection system with enhanced UNSW-NB15 dataset based on deep learning techniques," Journal of Engineering Science and Technology, vol. 16, no. 1, pp. 711–727, 2021.
J. Gu and S. Lu, "An effective intrusion detection approach using SVM with naïve Bayes feature embedding," Computers & Security, vol. 103, Apr. 2021, Art. no. 102158. DOI: https://doi.org/10.1016/j.cose.2020.102158
N. V. Sharma and N. S. Yadav, "An optimal intrusion detection system using recursive feature elimination and ensemble of classifiers," Microprocessors and Microsystems, vol. 85, Sep. 2021, Art. no. 104293. DOI: https://doi.org/10.1016/j.micpro.2021.104293
R. Panigrahi et al., "A Consolidated Decision Tree-Based Intrusion Detection System for Binary and Multiclass Imbalanced Datasets," Mathematics, vol. 9, no. 7, Jan. 2021, Art. no. 751. DOI: https://doi.org/10.3390/math9070751
A. Al-Bakaa and B. Al-Musawi, "Improving the Performance of Intrusion Detection System through Finding the Most Effective Features," in 2021 International Congress of Advanced Technology and Engineering (ICOTEN), Taiz, Yemen, Jul. 2021, pp. 1–9. DOI: https://doi.org/10.1109/ICOTEN52080.2021.9493564
S. Rawat, A. Srinivasan, V. Ravi, and U. Ghosh, "Intrusion detection systems using classical machine learning techniques vs integrated unsupervised feature learning and deep neural network," Internet Technology Letters, vol. 5, no. 1, 2022, Art. no. e232. DOI: https://doi.org/10.1002/itl2.232
M. Ramaiah, V. Chandrasekaran, V. Ravi, and N. Kumar, "An intrusion detection system using optimized deep neural network architecture," Transactions on Emerging Telecommunications Technologies, vol. 32, no. 4, 2021, Art. no. e4221. DOI: https://doi.org/10.1002/ett.4221
S. N. Mighan and M. Kahani, "A novel scalable intrusion detection system based on deep learning," International Journal of Information Security, vol. 20, no. 3, pp. 387–403, Jun. 2021. DOI: https://doi.org/10.1007/s10207-020-00508-5
F. Medjek, D. Tandjaoui, N. Djedjig, and I. Romdhani, "Fault-tolerant AI-driven Intrusion Detection System for the Internet of Things," International Journal of Critical Infrastructure Protection, vol. 34, Sep. 2021, Art. no. 100436. DOI: https://doi.org/10.1016/j.ijcip.2021.100436
K. N. Rao, K. V. Rao, and P. V. G. D. Prasad Reddy, "A hybrid Intrusion Detection System based on Sparse autoencoder and Deep Neural Network," Computer Communications, vol. 180, pp. 77–88, Dec. 2021. DOI: https://doi.org/10.1016/j.comcom.2021.08.026
Y. N. Kunang, S. Nurmaini, D. Stiawan, and B. Y. Suprapto, "Attack classification of an intrusion detection system using deep learning and hyperparameter optimization," Journal of Information Security and Applications, vol. 58, May 2021, Art. no. 102804. DOI: https://doi.org/10.1016/j.jisa.2021.102804
S. K. Sahu, D. P. Mohapatra, J. K. Rout, K. S. Sahoo, Q. V. Pham, and N. N. Dao, "A LSTM-FCNN based multi-class intrusion detection using scalable framework," Computers and Electrical Engineering, vol. 99, Apr. 2022, Art. no. 107720. DOI: https://doi.org/10.1016/j.compeleceng.2022.107720
A. U. H. Qureshi, H. Larijani, M. Yousefi, A. Adeel, and N. Mtetwa, "An Adversarial Approach for Intrusion Detection Systems Using Jacobian Saliency Map Attacks (JSMA) Algorithm," Computers, vol. 9, no. 3, Sep. 2020, Art. no. 58. DOI: https://doi.org/10.3390/computers9030058
Y. Yang, K. Zheng, C. Wu, and Y. Yang, "Improving the Classification Effectiveness of Intrusion Detection by Using Improved Conditional Variational AutoEncoder and Deep Neural Network," Sensors, vol. 19, no. 11, Jan. 2019, Art. no. 2528. DOI: https://doi.org/10.3390/s19112528
S. Thakur, A. Chakraborty, R. De, N. Kumar, and R. Sarkar, "Intrusion detection in cyber-physical systems using a generic and domain specific deep autoencoder model," Computers & Electrical Engineering, vol. 91, May 2021, Art. no. 107044. DOI: https://doi.org/10.1016/j.compeleceng.2021.107044
A. Basati and M. M. Faghih, "APAE: an IoT intrusion detection system using asymmetric parallel auto-encoder," Neural Computing and Applications, vol. 35, no. 7, pp. 4813–4833, Mar. 2023. DOI: https://doi.org/10.1007/s00521-021-06011-9
F. Laghrissi, S. Douzi, K. Douzi, and B. Hssina, "Intrusion detection systems using long short-term memory (LSTM)," Journal of Big Data, vol. 8, no. 1, May 2021, Art. no. 65. DOI: https://doi.org/10.1186/s40537-021-00448-4
"IDS 2018 Intrusion CSVs (CSE-CIC-IDS2018)." [Online]. Available: https://www.kaggle.com/datasets/solarmainframe/ids-intrusion-csv.
"UNSW_NB15." [Online]. Available: https://www.kaggle.com/datasets/mrwellsdavid/unsw-nb15.
"elifnurkarakoc/CICIDS2017." [Online]. Available: https://github.com/elifnurkarakoc/CICIDS2017.
Downloads
How to Cite
License
Copyright (c) 2025 Ricky Aurelius Nurtanto Diaz, I. Ketut Gede Darma Putra, Made Sudarma, I. Made Sukarsa, I. Wayan Budi Sentana, Ni Luh Gede Pivin Suwirmayanti
This work is licensed under a Creative Commons Attribution 4.0 International License.
Authors who publish with this journal agree to the following terms:
- Authors retain the copyright and grant the journal the right of first publication with the work simultaneously licensed under a Creative Commons Attribution License that allows others to share the work with an acknowledgement of the work's authorship and initial publication in this journal.
- Authors are able to enter into separate, additional contractual arrangements for the non-exclusive distribution of the journal's published version of the work (e.g., post it to an institutional repository or publish it in a book), with an acknowledgement of its initial publication in this journal.
- Authors are permitted and encouraged to post their work online (e.g., in institutional repositories or on their website) after its publication in ETASR with an acknowledgement of its initial publication in this journal.
