Anomaly Detection in Log Records

Abstract

Complex software systems are continuously generating application and server logs for the events which had occurred in the past. These logs generated and can be utilized for anomaly and intrusion detection. These log files can be used for anomaly or outlier detection. Certain types of abnormalities or exceptions such as spikes in HTTP requests, number of exceptions raised in logs, etc. All these events are logged into the log files for further analysis. These types of events are generally used for predicting the anomalies in future. The developed prototype assumes that the user inputs log records in a standard apache log format. At first the user uploads the log file for outlier detection. Next, a prototype is developed to get the number of HTTP requests for outlier detection. Then anomalies in number of HTTP requests are detected using three techniques namely InterQuartileRange method, Moving averages and Median Absolute deviation. Once the outliers are detected, these outliers are removed from the current dataset. This output is given as input to the Multilayer Perceptron model to predict the number of HTTP requests at the next timestamp