Data processing of wide-area measurement system based on Hadoop

Data processing of wide-area measurement system based on Hadoop

Chichiang Julie Zhang Shilin

In order to solve the problem of data redundancy and low processing efficiency in the mass data processing of wide area Measurement system (WAMS), a WAMS data processing cloud computing platform based on Hadoop is designed and implemented. First, the architecture of the platform is given. Secondly, a wams mass data loading method based on Hadoop Distributed File System (HDFS) is designed and the MapReduce model is used to realize the parallel extraction, transformation and loading (ETL) operation process of multiple file data. A Mpapriori data mining algorithm combined with MapReduce is proposed, which can be used to effectively excavate the interaction between each site in a cascading failure. Finally, through the wams of the actual data of the regional power grid, it verifies the high efficiency of the Hadoop processing the massive data. This platform is suitable for high performance local area network connected computer cluster to data processing of massive power grid. More restores

Keywords-cloud computing; data processing; Wide area measurement system; MapReduce

Temp_12092112388299.pdf