Read "Hadoop core technology" author Zhou Wei: My indissoluble bond with Hadoop concludes

Original URL: http://www.csdn.net/article/1970-01-01/2824661

1.Hadoop in Baidu to use

The main applications of Hadoop in Baidu include: Big Data Mining and analysis, log analysis platform, data Warehouse system, user behavior Analysis system, advertising platform and other storage and computing services.

At present, the size of the Hadoop cluster of Baidu is more than dozens of, the number of single-cluster nodes more than 5000 units, the amount of data processed more than 8000TB a day. At the same time, Baidu also developed its own log analysis platform, data Warehouse system, and unified C + + programming interface on the basis of Hadoop, and developed a more powerful HCE (Hadoop C + + Extend Systerm) system for the deep transformation of Hadoop.

The Learning Path of 2.Hadoop

As early as 2009, we tried to use Hadoop technology for China Telecom Research Institute, Mobile Research Institute and other institutions to build big data cloud platform and as a production platform for online business to provide cloud storage and computing Analysis Services. This was one of the first practitioners of the country to apply Hadoop to the ground.

I was not a core member when I was initially involved in the development of these projects, and all I had to do was to constantly learn and accumulate knowledge about Hadoop and big data in the project, and most importantly to prove myself in the project development process to demonstrate that I had enough knowledge to accumulate and to complete the project and exceed expectations. Of course, in this process will encounter some difficulties and setbacks, such as the output of data and expectations, system performance bottlenecks and other issues, in the face of these problems, the first is not afraid of themselves can solve these problems to have enough confidence, and then is to seriously analyze the problem, step by step debug fixed position problem, It's a hard job to finally get rid of these problems, but I enjoy the joy of success, because I like the technical challenges.

Read "Hadoop core technology" author Zhou Wei: My indissoluble bond with Hadoop concludes