The experiment of meteorological data processing and tuning based on MapReduce calculation model

The experiment of meteorological data processing and tuning based on MapReduce calculation model

Yang Runzhi Shen Wenhai Xiaoweizing Hu Kaixi Yang Xin Wang

Cloud computing uses distributed computing technology to realize the computational power and computational efficiency of parallel computing, and solves the problem of low computing power of stand-alone servers. The climatic standard values calculated based on the long sequence history data are of great significance to the real time service, quasi real time service and scientific research in the meteorological field. Because of the large amount of data and the complexity of the computational logic, it takes a long time to reorganize the traditional single node computing platform. Based on the framework of Hadoop distributed computing, this paper constructs a cloud computing platform of cluster model, takes the long sequence historical data as the source data, realizes partial reorganization algorithm based on MapReduce calculation model, and improves the calculation time limitation. At the same time, because the data source itself has many files and a small number of files, the data source storage form and the size of the data file are reconstructed, respectively, using Sequencefile mode and text file merging method to the same scene for the calculation of aging contrast test, respectively, tested 10 file merging, 100 file Merging two cases, so that the timeliness of a greater degree of improvement.

The experiment of meteorological data processing and tuning based on MapReduce calculation model