Spark-MLlib intrusion detection mechanism using machine learning models

Abstract

Typically, a single method is employed in machine learning (ML) based intrusion detection to identify intrusion information. However, this approach lacks flexibility, has a low detection rate, and struggles to handle high-dimensional data. Consequently, it is not efficient in addressing these challenges. This study proposes a new intrusion detection architecture that utilizes Spark and ensures resilient data dissemination across the platform to improve its effectiveness. It consists of preprocessing module, a label encoder module, a feature extraction module, a classification module and a database module. The preprocessing module compresses information by utilizing the module for label encoding. This generates a lower-dimensional reconstruction and classification characteristic. The database module has the capability to store the compressed characteristics of all traffic. This enables the classifier to be tested and then returns these features back into the original traffic, facilitating retraining. In order to evaluate the efficacy of the framework, simulations were conducted using the CICIDS 2017 dataset to accurately replicate the network traffic. Based on the test findings, the accuracy of both multiclass and binary classification surpasses that of earlier studies. High precision was achieved for the traffic that was restored. The possible application of the proposed architecture for edge/fog networks is discussed in the conclusion.