[Documentation] design and application of data analysis scheme based on Hadoop platform

Design and application of data analysis scheme based on Hadoop platform

Jiang, Sinyang, Chen Linshun

Absrtact: In the face of the massive data on the Internet, a single host can not meet its storage and computing requirements, and the application of distributed storage and distributed computing becomes an inevitable trend. Hadoop is one of the most widely used distributed storage and computing frameworks. Under this platform, through the data statistic analysis of the thousands user search log in a domestic search engine within two months, the design idea and example of the corresponding Map/reduce program are given, and some design and performance optimization methods of Map/reduce Distributed program are proposed, and the experimental results show that The methods proposed in this paper can simplify the Map/reduce program design and improve the program performance effectively.

Keywords: distributed computing; map/reduce; text processing; Data analysis

Temp_12042716011861.pdf