Design and implementation of recommendation system based on Hadoop

Design and implementation of recommendation system based on Hadoop

Tangjin of the

The main work contents are as follows: 1. Based on the research of the running mechanism of Hadoop and the principle of MapReduce programming, combined with recommendation system and recommendation algorithm, especially the network recommendation algorithm represented by the material diffusion recommendation algorithm and the heat conduction recommendation algorithm, the paper designs and realizes the MapReduce programming scheme of network recommendation algorithm, The complex computing task of this algorithm is decomposed into a series of mapreduce operation Flow, so as to make the distributed parallel processing on Hadoop and cloud computing platform, the algorithm has good parallelism and expansibility in the cluster through a series of experimental tests. 2. On the basis of the MapReduce scheme of network recommendation algorithm, using the optimization methods of combiner function and sequence file input output type, the paper analyzes the characteristics of the long tail distribution data of the dataset and the implementation details of the algorithm calculation process, By using the pair and stripe scheme in the calculation of resource allocation matrix and the optimization design idea of cutting the extremely active users, the problem of performance bottleneck of the algorithm is effectively solved, and the MapReduce scheme of network recommendation algorithm based on Hadoop is improved, The experimental results show that the improved method can improve the efficiency of the algorithm. 3. Studied the installation, deployment and usage of a series of related open source software, including Hadoop, Mahout, Sqoop, ganglia, combined with the MapReduce scheme of the network recommendation algorithm designed in this paper, through system requirements, system framework and process design, The system implementation and the system test several steps, has designed, implemented and deployed the recommended prototype system based on Hadoop in the cluster environment composed of many computers.

Design and implementation of recommendation system based on Hadoop