Research on distributed processing of network monitoring information flow based on Hadoop

Research on distributed processing of network monitoring information flow based on Hadoop

Flexor zhi Jian Guoliang chen

A new distributed cluster processing method based on Hadoop cloud computing framework is proposed to solve the problem of access of large datasets in intelligent power grid dispatching system. By analyzing the characteristics of information flow in power grid monitoring system, extracting 3 kinds of critical information flow, using the Distributed File system HDFs and Mapping aggregation model Map/reduce, establishing the distributed processing platform of Cluster group, and realizing the high efficient parallel processing of the monitoring data. The results show that the processing efficiency of the method is about 1.4 times times higher than that of the traditional client/server database method, the larger the cluster scale, the better the effect, and the more effective avoidance of the processing response delay of the large data set.

Research on distributed processing of network monitoring information flow based on Hadoop