Research and application of parallel K meteorological data mining based on cloud computing

Research and application of parallel K meteorological data mining based on cloud computing

Nanjing University of Information Engineering Pan Wubin

Clustering analysis is an important research field in data mining, it divides the data into several kinds of clustering, which makes the objects in the same cluster more similar, and the objects in different clustering are very big. K algorithm is considered to be the most classical clustering method based on partition, the error squared and criterion function is used as the clustering criterion, and the algorithm is relatively scalable and efficient in processing data sets. However, in the face of large scale data, the computation also increases, and the resource and time cost of operation increase obviously. In order to break through this bottleneck, through analysis and research, compared with the traditional parallel computing model, the MapReduce distributed computing model encapsulates the details of parallel processing, fault-tolerant processing, localization computation, load balancing and so on, and provides a simple and powerful programming interface, which greatly simplifies the difficulty of parallel programming, When using MapReduce to develop parallel programs, users only need to focus on the parallel computing task that they want to solve. Therefore, using Hadoop platform to deal with the advantages of massive data, an improved algorithm is proposed to realize the MapReduce parallelization of K algorithm on Hadoop platform. Finally, the parallel K algorithm based on MapReduce is evaluated through the validity, speedup and scalability of the clustering results.

Research and application of parallel K meteorological data mining based on cloud computing