Hadoop applications in Baidu

Baidu, as the world's largest Chinese Search Engine Company, provides a variety of Search Engine-based products, including functional search based on network search; community search based on discount bars; vertical searches, MP3 music searches, and Encyclopedias targeting regions and industries cover almost all the search needs in the Chinese online world.

Baidu has a high requirement on massive data processing. It needs to analyze the data online, complete the processing within the specified time, and report it to the platform. As shown in the requirements of Baidu's platform in the Internet field, hadoop is a good choice.

In Baidu, hadoop is mainly used in the following aspects:

• Log storage and statistics;
• Analysis and mining of webpage data;
• Business analysis, such as user behavior and advertising attention;
• Feedback on online data to get Click information of online advertisements in a timely manner;
• Cluster user web pages to analyze user recommendations and user relevance.

Mapreduce is mainly an idea that cannot solve computing-related problems in all fields. Baidu researchers believe that a better model should be shown in. HDFS achieves shared storage, some computing uses mapreduce, some computing uses MPI, and some computing needs to be processed together. Because mapreduce is suitable for processing large data that is suitable for partitioning, you can use mapreduce to filter such data to obtain the basic vector matrix, after MPI is further processed, the returned results are returned. Only the integrated technology can better solve the problem.

Hadoop applications in Baidu