Multi-Agent based MapReduce Model for Efficient Utilization of System Resources

Abstract

Recently with increased adoption of big data, Internet of Things and sensor technology by various organization for provisioning smart intelligent services for various application uses. Data processing on real-time social media and sensor data is been a key area of research in recent times and these data are massive and continuous. Smart application using sensor and social media data can be classified into three class: 1) online processing of streaming data; 2) online processing of historical data; and 3) hybrid processing of both. The existing model are designed considering stream or batch processing. For provisioning real-time processing MapReduce framework using Hadoop framework is considered by state-of-art technique for data inflow forecasting. However, the Hadoop based forecasting model are not efficient in fully utilizing system resource. Agent based MapReduce forecasting model is adopted by state-of-art technique to utilize system efficiently. However, they incurs high computation overhead, thus increase cost of computing cost. To overcome this work present an agent based Data Inflow Forecasting (DIF) model for both stream and non-stream (historical) data by using Multivariate Gaussian Mixture (MGM) model. This work present an Agent based MapReduce (AMR) framework to process data in real-time and utilize system resource efficiently. To provide scalability for processing social media and sensor data DIF-AMR model adopts cloud computing architecture. Experiment are conducted to evaluate performance of DIF-AMR of over existing model shows significant performance improvement in terms of computation time.