Design and implementation of a large amount of small XML data file processing technology based on Hadoop

Design and implementation of a large amount of small XML data file processing technology based on Hadoop

University Kong Xin

This paper focuses on the following: 1 a distributed mass of small XML data processing system (distributed massive smallxml files SYSTEM,DMSX) designed specifically for processing large quantities of small data XML files. The main idea of the system is to efficiently handle the large amount of small data XML files in the Hadoop system. 2 The system can improve the whole processing ability of the system by using producer-consumption mode, multithreading concurrency technology and multilevel caching technology. 3 The system improves the efficiency of the system in processing large amount of small data XML by storing the merged files in a segmented mode without affecting the working module while efficiently utilizing the network resources and concurrent with the data processing module. In this paper, the function modules of the system are described in detail, and the efficiency of the system is analyzed experimentally. The performance of the system is validated by the test method of actual deployment, multiple tests and average evaluation. It is proved that the method proposed in this paper can effectively improve the system resource utilization and the system response speed, and solve the problem of the current Hadoop in processing large amount of small data XML files.

Design and implementation of a large amount of small XML data file processing technology based on Hadoop