Spark-MLlib intrusion detection mechanism using machine learning models

Asra Sarwath, Raafiya Gulmeher, Zeenath Sultana


Typically, a single method is employed in machine learning (ML) based intrusion detection to identify intrusion information. However, this approach lacks flexibility, has a low detection rate, and struggles to handle high-dimensional data. Consequently, it is not efficient in addressing these challenges. This study proposes a new intrusion detection architecture that utilizes Spark and ensures resilient data dissemination across the platform to improve its effectiveness. It consists of preprocessing module, a label encoder module, a feature extraction module, a classification module and a database module. The preprocessing module compresses information by utilizing the module for label encoding. This generates a lower-dimensional reconstruction and classification characteristic. The database module has the capability to store the compressed characteristics of all traffic. This enables the classifier to be tested and then returns these features back into the original traffic, facilitating retraining. In order to evaluate the efficacy of the framework, simulations were conducted using the CICIDS 2017 dataset to accurately replicate the network traffic. Based on the test findings, the accuracy of both multiclass and binary classification surpasses that of earlier studies. High precision was achieved for the traffic that was restored. The possible application of the proposed architecture for edge/fog networks is discussed in the conclusion.


Classification; Deep learning; Intrusion detection system; Machine learning; PySpark; Spark Mllib

Full Text:




  • There are currently no refbacks.

Creative Commons License
This work is licensed under a Creative Commons Attribution-ShareAlike 4.0 International License.

The Indonesian Journal of Electrical Engineering and Computer Science (IJEECS)
p-ISSN: 2502-4752, e-ISSN: 2502-4760
This journal is published by the Institute of Advanced Engineering and Science (IAES) in collaboration with Intelektual Pustaka Media Utama (IPMU).

shopify stats IJEECS visitor statistics