Google's three core technologies (ii) Google mapreduce Chinese version

Google's three core technologies (ii) Google mapreduce Chinese version

Google mapreduce Chinese version

Translator: Alex

Summary

MapReduce is a programming model and a related implementation of an algorithmic model for processing and generating very large datasets. The user first creates a map function that processes a data set based on the key/value pair, outputs the middle of the data set based on the Key/value pair, and then creates a reduce function to combine all the intermediate value values with the same intermediate key value. In the real world there are many examples that satisfy the above-mentioned processing model, this paper will describe this model in detail.

The MapReduce architecture program enables parallelization on a large number of commonly configured computers. This system is only concerned when it is running: How to divide the input data, the scheduling on a large number of computers, the error handling of the computer in the cluster, and the necessary communication between the computers in the cluster. Using the MapReduce architecture enables programmers who have no experience in parallel computing and distributed processing systems to effectively utilize the rich resources of distributed systems.

Our MapReduce implementations run on a cluster of ordinary machines that can be flexibly scaled: A typical mapreduce calculation is often comprised of thousands of machines that process terabytes of data. Programmers find the system very useful: Hundreds of MapReduce programs have been implemented, and in Google's clusters, more than 1000 mapreduce programs are executing every day.

1. Introduction

Over the past 5 years, many of Google's programmers, including the authors of this article, have implemented hundreds of dedicated computing methods to deal with massive amounts of raw data. These methods are used to deal with a large amount of raw data, such as document fetching (like a web crawler), Web request logs, and so on, and to calculate and process various types of derivative data, such as inverted indexes, the various representations of the graph structure of Web documents, the number of pages crawled by the crawler on each host, A collection of the most requested queries per day, and so on. Most of these data processing operations are conceptually easy to understand. However, because of the large amount of data being entered, it is only possible to distribute these calculations across hundreds of hosts if you want to complete the operation in an acceptable time period. How to handle parallel computations, how to distribute data, and how to handle errors. All of these problems are combined and require a lot of code processing, which makes it difficult to deal with the original simple operations.

To solve these complex problems, we design a new abstract model, using this abstract model, we can simply express the simple operations we want to perform, without having to worry about the complex details of parallel computing, fault tolerance, data distribution, load balancing, and so on, which are encapsulated in a library. The design of this abstract model is inspired by the original language of the map and reduce of Lisp and many other functional languages. We realize that most of our operations include the operation of applying the map operation on the "logical" record of the input data to derive an intermediate Key/value pair collection, and then apply the reduce operation on all value values with the same key value to achieve the merging of the intermediate data, The purpose of getting a desired result. Using the MapReduce model, combined with the user-implemented map and reduce functions, we can easily implement large-scale parallelization computation, and through the "Re-Execution" (re-execution) function of the MapReduce model, it also provides a primary disaster-tolerant implementation scheme.

The main contribution of this work (implementing a MapReduce framework model) is to automate parallelism and large-scale distributed computing through a simple interface, using the MapReduce model interface for high-performance computing on a large number of common PC machines.

The second section describes the basic programming model and some use cases. The third part describes a tailored mapreduce implementation that is tailored to our cluster-based computing environment. Part IV describes some of the practical techniques we think of in the MapReduce programming model. Part V measures the performance of our MapReduce implementations for a variety of different tasks. Part VI reveals how we use MapReduce as a basis for rewriting our indexing system products, including some other experiences using MapReduce. Part VII discusses relevant and future work.

2. Programming Model

The MapReduce programming model works by using an input Key/value pair collection to produce an output of the Key/value pair collection. Users of the MapReduce Library Express this calculation in two functions: map and reduce.

The user-defined map function accepts an input key/value pair value and then produces a set of intermediate key/value pair values. The MapReduce library passes all the intermediate value values with the same intermediate key value I to the reduce function.

The user-defined reduce function accepts a collection of the value I of an intermediate key and a value associated with it. The reduce function merges these value values to form a small collection of value values. In general, each reduce function call produces only 0 or 1 output value values. Usually we supply the middle value value to the reduce function through an iterator, so that we can handle the collection of a large number of value values that cannot be put into memory all.

2.1. Example

For example, to calculate the number of occurrences of each word in a large document collection, the following is a pseudo-code snippet:
Map (string key, String value):
Key:document Name
Value:document contents
For each word w in value:
Emitintermediate (W, "1″");
Reduce (String key, Iterator values):
Key:a Word
Values:a List of Counts
int result = 0;
For each V in values:
Result + = parseint (v);
Emit (asstring (result));

The map function outputs each word in the document, and the number of occurrences of the word (in this simple example, 1). The reduce function adds up the count of each particular word produced by the map function.

In addition, the user writes code, uses the name of the input and output files, optional tuning parameters to complete an object conforming to the MapReduce model specification, and then calls the MapReduce function and passes the canonical object to it. The user's code and the MapReduce library are linked together (implemented in C + +). Appendix A contains all the program code for this instance.

2.2. Type

Although the input and output values in string representations are used in the pseudo-code of the preceding example, conceptually, the user-defined map and the reduce function have the associated type:
Map (K1,V1)->list (K2,V2)
Reduce (K2,list (v2))->list (v2)
For example, the input key and value values differ from the fields that the output key and value values derive on the type. In addition, the intermediate key and value values are the same as the fields derived from the output key and value values on the type.

(Alex Note: The meaning of this domain in the original text is not very clear, I refer to Hadoop, KFS and other implementations, both map and reduce are used generics, so I translated domain into type derivation of the field).
Our C + + uses string types as input and output for user-defined functions, and the user makes the appropriate type conversions for strings in their own code.

2.3. More examples

Here are some interesting simple examples that can easily be represented using the MapReduce model:

· The distributed Grep:map function output matches one line of a pattern, and the reduce function is an identity function that copies intermediate data to the output.

· Calculate URL Access frequency: The map function processes the record of the Web page request in the log, and then outputs (url,1). The reduce function sums up the value of the same URL, producing the result (URL, total number of records).

· Reverse the network link graph: The map function searches for all the link targets (target) in the source page and outputs as (Target,source). The reduce function combines the links of a given link target with a list of outputs (Target,list (source)).

· Vector of search terms for each host: Retrieves a word vector using a list of (words, frequencies) to summarize some of the most important words appearing in a document or set of documents. The map function is output for each input document (host name, search word vector), where the hostname comes from the URL of the document. The reduce function receives a vector of the search terms for all the documents of a given host, and adds these search terms vectors together, discarding the low-frequency search term, and outputting a final (hostname, search-word vector).

· Inverted index: The map function parses a list of each document output (Word, document number), the input of the reduce function is all (Word, document number) of a given word, sorts all document numbers, outputs (Word, list (document number)). All the output sets form a simple inverted index that tracks the position of the word in the document in a simple algorithm.

· Distributed ordering: The map function extracts key, output (Key,record) from each record. The reduce function does not change any value. This operation relies on the partitioning mechanism (described in 4.1) and the sort attribute (described in 4.2).

3. Realize

The MapReduce model can be implemented in many different ways. How the right choice depends on the specific environment. For example, one implementation is for a small, shared-memory machine, and the other is for multiprocessor hosts with large NUMA architectures, and some implementations are better suited for large network-connected clusters.

This section describes an implementation for a widely used computing environment within Google: a large cluster of ordinary PCs connected by an Ethernet switch. In our environment, we include:
1.x86 architecture, machines running the Linux operating system, dual processors, 2-4GB memory.
2. Ordinary network hardware devices, the bandwidth of each machine is hundred trillion or gigabit, but far less than half the average bandwidth of the network. (Alex Note: This needs to be explained by a network expert)
3. The cluster contains hundreds or thousands of machines, so machine failures are the norm.
4. Store as a cheap built-in IDE hard drive. An internal distributed file system is used to manage the data stored on these disks. The file system ensures the reliability and validity of data on unreliable hardware by means of data replication.
5. The user submits the work (job) to the dispatch system. Each job contains a series of tasks that the dispatch system dispatches to multiple available machines in the cluster.

3.1. Executive Summary

The map call is distributed across multiple machines by automatically splitting the input data of the map invocation into a collection of M data fragments. The input data fragments can be processed in parallel on different machines. Use the partition function to divide the intermediate key value produced by the map call into R different partitions (for example, hash (key) mod R), and the reduce call is distributed across multiple machines. The number of partitions (R) and partition functions are specified by the user.

Figure 1 shows the full flow of operations in our MapReduce implementation. When the user calls the MapReduce function, the following sequence of actions occurs (the following sequence number corresponds to the ordinal number one by one in Figure 1):
1. The MapReduce library that the user program calls first divides the input file into m-data slices, each of which is typically 16MB to 64MB in size (the size of each piece of data can be controlled by an optional parameter). The user program then creates a large number of copies of the program in the cluster.(Alex:copies of the program is really hard to translate)
2. There is a special program –master in the copy of these procedures. The other programs in the replica are worker programs, and the master assigns tasks. There are m map tasks and R reduce tasks to be assigned, and master assigns a map task or reduce task to an idle worker.
3. The worker program assigned the map task reads the relevant input data fragment, resolves the key/value pair from the input data fragment, and then passes the Key/value pair to the user-defined map function, which is generated and output by the map function in the middle key/ Value pair, and is cached in memory.
4. The key/value pair in the cache is partitioned into r regions by partitioning functions and then periodically written to the local disk. The storage location of the cached Key/value pair on the local disk will be passed back to master, which is the responsibility of the master to transfer these storage locations to the reduce worker.
5. When the reduce worker program receives the data storage location information from the master program, it uses RPC to read the cached data from the disk on the host where the map worker resides. When the reduce worker reads all the intermediate data, the data with the same key value is aggregated by sorting the key. Because many different key values are mapped to the same reduce task, they must be sorted. If the intermediate data is too large to be sorted in memory, then the order must be sorted externally.
The 6.Reduce worker program iterates through the sorted intermediate data, and for each unique intermediate key value, the reduce worker program passes this key value and its associated set of intermediate value values to the user-defined Reduce function. The output of the reduce function is appended to the output file of the owning partition.
7. When all the map and reduce tasks are completed, Master wakes up the user program. At this point, the call to MapReduce in the user program is returned.

After successful completion of the task, the output of MapReduce is stored in the R output file (corresponding to each reduce task produces an output file with the file name specified by the user). In general, users do not need to use this r