Mapreduce application scenarios

In typical application scenarios of mapreduce, log analysis is widely used, as well as search index and machine learning.AlgorithmPackage mahout is also one of them. Of course there are many things it can do, such as data mining and information extraction.

Mapreduce is widely used in Distributed sorting, Web connection diagram reversal, and Web access log analysis.

Google has established a mapreduce-based search index system. In essence, this index is composed of sequential batch processing operations. It distributes large-scale operations on datasets to each node on the network for computation, and each node periodically reports the completed work and status updates back to the primary computation.

Lipkovitz first talked about how Google handles mapreduce-based file indexing systems. "We have to deal with a very large data system. Before that, we had to wait 8 hours for the computing to get the complete results, then we will publish it to the index system. In the past, we kept repeating this time-consuming and labor-consuming job ."

Mapreduce is only a batch processing method. google gave up mapreduce because it could not provide Google with the desired index speed. Especially with the advent of the real-time retrieval era, what Google needs is to refresh the index content within several seconds, instead of 8 hours.

The indexing system is Google's largest mapreduce ApplicationProgram.

Let's take a look at Doug cutting's business use of hadoop:

Doug cutting (DC): Yahoo regularly uses hadoop in its search business to improve its products and services, such as ranking functions and target advertisements. In addition, there are also some cases where data is directly generated using hadoop. Hadoop's long-term goal is to provide world-class distributed computing tools and web-scale services that support next-generation services (such as search result analysis.

In general, mapreduce can be used to include distributed grep, distributed sorting, Web access log analysis, reverse index construction, Document Clustering, machine learning, and statistical-based machine translation, generate the index of the entire Google search and other large-scale data processing work. For Real-Time Parallel Computing, such as time-consuming computing for parallel processing, mapreduce may not be an ideal choice. In this case, you may need to consider other directions, such: MPI, OpenMP, and hybrid Cuda.

More: http://blog.donews.com/me1105/archive/2011/02/10/116.aspx