Distributed Basic Learning (2) Distributed Computing System (MAP/REDUCE)

Two . Distributed Computing ( Map/reduce )

Distributed computing, too, is a broad concept, where it narrowly refers to a distributed framework designed by the Google Map/reduce framework. In Hadoop, distributed file systems, to a large extent, are served by a variety of distributed computing needs. We say that distributed file systems are distributed file systems, and similar definitions are generalized to distributed computing, and we can think of them as computational functions that add distributed support. From a computational standpoint, the MAP/REDUCE framework accepts key values in various formats as input, and after reading the calculation, it eventually generates a custom formatted output file. From a distributed perspective, distributed computing input files are often large and distributed across multiple machines, single-machine computing is completely unsustainable and inefficient, so the map/reduce framework needs to provide a mechanism to extend this calculation to an unlimited number of machine clusters. According to this definition, our understanding of the whole map/reduce can also be seen along these two processes.

In the Map/reduce framework, each calculation request is called a job. In the distributed Computing Map/reduce Framework, in order to complete this job, it carries out a two-step strategy, the first is to split it into a number of map tasks, assigned to different machines to execute, each map task takes a part of the input file as its own input, after some calculations, Produces an intermediate file of a format that exactly matches the desired file format, but contains only a subset of the data. So, when all the map tasks are completed, it goes to the next step to merge the intermediate files to get the final output file. At this point, the system generates several reduce tasks, which are also assigned to different machines to execute, and the goal is to summarize the intermediate files generated by several map tasks into the final output file. Of course, this rollup is not always as straightforward as 1 + 1 = 2, which is where the value of the reduce task is. After the steps as above, eventually, the job is completed and the desired target file is generated. The key to the whole algorithm is to add a process of intermediate file generation, which greatly improves the flexibility and ensures the distributed extensibility ...

I. Terminology Control

Like Distributed file Systems, Google, Hadoop, and .... I, each in a way to express a unified concept, in order to ensure its unity, the following table is unique ...

Translation in text	Hadoop terminology	Google terminology	Related explanations
Homework	Job	Job	Each calculation request for a user is called a job.
Job Server	Jobtracker	Master	The server that the user submits the job at the same time, it also is responsible for each job task assignment, manages all task server.
Task Server	Tasktracker	Worker	The worker bees are responsible for carrying out specific tasks.
Task	Task	Task	Each job, need to split up, to multiple servers to complete, split out the execution unit, called the task.
Backup tasks	Speculative Task	Buckup Task	Each task is likely to fail or slow, in order to reduce the cost of this, the system will be a proactive implementation on the other task server to perform the same task, which is the backup task.

II. Basic architecture

Similar to Distributed file systems, map/reduce clusters are composed of three types of servers. Where the job server, called Job Tracker in Hadoop, is called Master in Google paper. The former tells us that the job server is responsible for managing all the jobs under this framework, which tells us that it is also the core of assigning tasks to each job. Similar to the HDFs master server, it also exists as a single point, simplifying the responsible synchronization process. Specifically responsible for the implementation of user-defined operations, is the task server, each job is split into a number of tasks, including map tasks and reduce tasks, the task is the implementation of the basic unit, they need to be assigned to the appropriate task server to execute, The task server reports the status of individual tasks to the job server on one side, helping the job server understand the overall situation of job execution, assigning new tasks, etc...

In addition to the manager performer of the job, there is a need for a task submitter, which is the client. Like the Distributed file system, the client is not a separate process, but a set of APIs, the user needs to customize the content of their own, through the client-related code, the job and its related content and configuration, to the job server, and always monitor the status of execution ...

With the same implementation as Hadoop, Hadoop Map/reduce also uses protocol interfaces to communicate between servers. As the RPC server, the caller is called through the RPC proxy, so that most of the communication is done, the architecture of the specific server, and the state of the various protocols in which it runs, see. As you can see, there are few related protocols compared to HDFS, and there is no direct communication relationship between the client and the task server and the task server. This does not mean that the client does not need to understand the execution of the specific tasks, nor does it mean that the task server does not need to know the other task execution situation, but because the whole cluster of machines more complex than HDFS, direct communication is too difficult to maintain, so, are unified by the job server collation and forwarding. In addition, from this picture can be seen, the task server is not a person in the fight, it will be like the Monkey King to recruit a group of babies to help their specific implementation of the task. The benefits of doing so, personally think, should have security considerations, after all, the task of the code is user-submitted, the data is user-specified, this quality naturally mixed, in the event of a sabotage, the entire task server process to kill, pound foolish. Therefore, in a separate site, the love of the land, but also the responsibility is clear ...

Compared with the Distributed file system, the Map/reduce framework has another feature, which is the strong customization. Many of the algorithms in the file system are fixed and intuitive, and do not change too much because of the content that is stored. As a general-purpose computing framework, the problems that need to be faced are much more complex, and it is difficult to have a package of cures for all kinds of diseases, between different problems, different inputs and different needs. As a map/reduce framework, we should try to extract the public demand as much as possible and realize it. More importantly, it is necessary to provide a good extensible mechanism to meet the needs of users to customize the various algorithms. Hadoop is implemented by Java, so it is a bit of a piece of cake to implement custom extensions by reflection. In the Jobconf class, a large number of interfaces are defined, which is essentially a centralized display of all the customizable content of the Hadoop map/reduce framework. In jobconf, there are a number of set interfaces that accept a class parameter, usually it has a default implementation of the class, if the user is not satisfied, you can customize the implementation ...

Iii. Calculation Process

If everything is done on a step-by-step basis, then the entire job's calculation process should be the assignment of the map task, assignment and execution of the job's submit and execute, and the completion of the job. In the execution of each task, it also contains the output of the execution of the input preparation---------three sub-steps. Along this process, we can quickly clean up the execution of the job under the whole map/reduce framework ...

1, the submission of the job

A job, before committing, you need to configure all the things should be configured, because once submitted to the job server, is caught in a fully automated process, users in addition to wait and see, at most can play a role in supervision, punish some do not work hard task ...

Basically, the work that the user needs to do in submitting the code phase is this:

First, write all your custom code, and at the very least, you need to have map and reduce execution code. In Hadoop, map needs to derive from the Mapper interface, and reduce needs to derive from the reducer interface. This is all used generics to support different key-value types. Both interfaces have only one method, one is map, the other is reduce, both methods accept four parameters directly, the first two are input key and value related data structure, the third is as output related data structure, the last one, is an instance of reporter class, When implemented, it can be used to count some counts. In addition to these two interfaces, there are a number of interfaces that can be derived, such as the split Partitioner interface ...

Then, you need to write the code of the main function, the most important thing is to instantiate an object of the Jobconf class, and then call its rich setxxx interface, set the desired content, including the input and output file path, map and reduce class, Even include the format support classes required to read the file, and so on ...

Finally, call Jobclient's Runjob method to commit this Jobconf object. The Runjob method invokes the Submitjob method defined by the Jobsubmissionprotocol interface and submits the job to the job server. Then, Runjob begins the loop, calls the Jobsubmissionprotocol Gettaskcompletionevents method, obtains the object instance of the Taskcompletionevent class, and understands the execution status of each task of this job ...

2 . Distribution of Map tasks

When a job is submitted to the job server, the job server generates several map tasks, each of which is responsible for converting a portion of the input into an intermediate file in the same format as the final format. Usually a job input is based on Distributed File System files (of course, in a single-machine environment, the file system can also be single-machine ...) Because, it can be very natural and distributed computing to generate connections. For a map task, its input is often a block of data in the input file, or part of the data block, but usually not across the data block. Because, once the data block is spanned, it may involve multiple servers, which brings unnecessary complexity ...

When a job is submitted to the job server from the client, the job server generates an Jobinprogress object, which is the corresponding identity for administration. When a job is split into several map tasks, it is pre-hung in the task server topology tree on the job server. This is divided according to the location of the distributed file data block, such as a map task needs a block of data, the data block has three copies, then, on the three servers will hang on this task, can be considered as a pre-allocation ...

With regard to the implementation of most of the real functions and logic of task management and allocation, Jobinprogress relies on subclasses of Jobinprogresslistener and TaskScheduler. TaskScheduler, as the name implies, is the policy class for task assignment (to simplify the description, use it to refer to all TaskScheduler subclasses ... ）。 It will master all the task information of the job, its assigntasks function, accept a tasktrackerstatus as a parameter, according to the status of this task server and the existing task status, assign it a new task. In order to master all job-related tasks, TaskScheduler will register several jobinprogresslistener to Jobtracker, and when new assignments arrive, remove, or update, Jobtracker will inform all Jobinprogresslistener so that they can be handled accordingly ...

Task allocation is an important part, so-called task assignment, is to assign appropriate tasks to the appropriate task on the server. It's easy to see that there are two steps involved, first selecting the job and then selecting the task in this job. As with all assignments, task assignment is also a complex job. A less-than-good assignment of tasks can lead to increased network traffic, an inefficient load on some task servers, and so on. Furthermore, task allocation is a problem with no consistent pattern, and different business backgrounds may require different algorithms to meet the requirements. So, in Hadoop, there are a lot of taskscheduler subclasses, like Facebook,yahoo, that contribute their own algorithms. In Hadoop, the default task allocator is the Jobqueuetaskscheduler class. It chooses the basic order of the jobs: Map clean up task (the MAP Task Server Cleanup task, used to clean up related outdated files and environments ...)  -> map setup task (MAP Task Server Installation task, responsible for configuring the relevant environment ...) )  -> map tasks -> reduce clean up task -> reduce  setup task -> reduce tasks. In this premise, specific to the map task assigned to come up. When a task server is working and looking for a new task, Jobqueuetaskscheduler is assigned from the highest priority job according to the priority of each job. Each assigned one, will also leave a margin for it, has been a rainy hour. For example: The system currently has priority 3, 2, 1 of three jobs, each job has a assignable map task, a task server to apply for a new task, it also has the ability to carry out the execution of 3 tasks, Jobqueuetaskscheduler will first favorably The first Level 3 job takes a task to assign to it, and then leaves out a margin of 1 tasks. At this point, the system can only assign tasks of priority 2 jobs to this server, not priority 1 tasks. Such a strategy, the basic idea is that everything for high-priority job services, priority allocation does not say, allocated a good reserve for a rainy life, so preferential, enough to let the high priority of the operation of the joy and cry, so that the low priority of the job is not only born yoga Sheng, even starve to death.

After deciding which job to extract the task from, the specific allocation algorithm, after a series of calls, is actually done by Jobinprogress's findnewmaptask function. Its algorithm is very simple, is to make every effort to this server mismatch and the best possible allocation of tasks, that is, as long as there are assignable tasks, will be assigned to it, regardless of the later. The job server will start with the server closest to it and see if there are any unassigned tasks (pre-allocated), from near to far, if all the tasks are assigned, then see if there are multiple executions, and if so, consider reassigning the unfinished task again. ）。。。

Assigning a task to a job server does not mean that it is completely liberated, and that the task can be ignored. Because tasks can fail on the task server and may perform slowly, this requires the job server to help them do it again. So in the task, the record has a taskattemptid, for the task server, they run each time, in fact, is only a attempt, reduce the task only need to take a letter of output, the other is white busy ...

3.Map Task Execution

Similar to HDFs, the task server reports to the job server the status of each task being performed at this point in time, and requests new tasks from the job server, through a heartbeat message. Implementation, is Tasktracker call Intertrackerprotocol protocol heartbeat method to do. This method takes a Tasktrackerstatus object as a parameter, which describes the state of this task server at this time. When it has the ability to accept a new task, it also passes the parameter acceptnewtasks to True, indicating that the job server is expected to be entrusted. After the jobtracker receives the relevant parameters, it is processed and returns a Heartbeatresponse object. In this object, a set of tasktrackeraction is defined that directs the task server to the next step. The system has defined a bunch of its tasktrackeraction sub-class, some of the parameters carried to expand, and some just marked the next ID, the specific unknown to write, a look will know ...

When Tasktracker receives the tasktrackeraction that contains the launchtaskaction, it starts executing the assigned new task. In Tasktracker, there is a tasktracker.tasklauncher thread (literally two, a map task, a reduce task), and they are waiting for the new mission to come. Once this is done, the Createrunner method of the task is eventually called, constructing a Taskrunner object and creating a new thread to execute. For a map task, it corresponds to the runner of the Taskrunner subclass Maptaskrunner, but the core is within the Taskrunner implementation. Taskrunner will download all the required files and pack them up and record them in a global cache, which is a global directory that can be used by all the tasks of this job. It will use some soft links to link some file names to this cache. Then, depending on the parameters, a JVM execution environment is configured, which corresponds to the object of the Jvmenv class.

Next, Taskrunner calls Jvmmanager's Launchjvm method and submits it to jvmmanager processing. The Jvmmanager is used to manage all the task child processes running on the tasktracker. In the current implementation, the way to try is pooling. There are several fixed slots, if the slot is not full, then start a new sub-process, otherwise, to find the idle process, if the same job directly put in, otherwise kill the process, with a new process instead. Each process is managed by Jvmrunner, which is also in a separate thread. But from the implementation, this mechanism seems to have not been deployed, the child process is a dead loop waiting, and not blocked on the parent process on the relevant thread, the parent thread's variable has not been adjusted, once assigned, always in a busy state.

The real execution vector, which is the child, contains a main function, the process executes, the relevant parameters will be passed in, it will disassemble the parameters, and construct the relevant task instance, call its run function to execute. Each child process can execute a specified number of tasks, which is the pooled configuration described above. However, this mechanism in my view, and did not run, each process actually did not have the opportunity to do new tasks, just silly waiting for the process pool full, and was killed by a knife. Perhaps my old eye dim, did not see the realization of the clue ...

4,Reduce the assignment and implementation of the task

Compared to the map task, the distribution of reduce and its simplicity, basically all map tasks completed, there are idle task servers, come on to assign a job task. Because the map task results are dotted, and changeable, really want to engage in a global optimization algorithm, is definitely not worth the candle. And the process of the implementation of the reduce task is structured and distributed, and the map is basically exactly the same, there is nothing to say ...

In fact, the biggest difference between the reduce task and the map task is that the map task's files are isolated locally, and the reduce task needs to be collected everywhere. This process is the task server that the job server is working on by the reduce task, tells the process that the reduce task is executing, it needs the server address that the map task executes, and the reduce task server contacts the original Map task server (of course, the local is exempt ...). ), via FTP service, download it over. This implicit direct data connection is the biggest difference between performing a reduce task and performing a map task ...

5, the completion of the work

When all the reduce tasks are complete and the required data is written to the distributed file system, the entire job is formally completed. This, involves a lot of classes, a lot of files, a lot of servers, so it is very laborious, say, a schematic thousand words, said so much, I still draw two pictures, completely express it ...

First, it is a time series diagram. It simulates a job execution process consisting of 3 map tasks and a reduce task. As we can see, in the process of execution, as long as someone is too slow, or fails, an attempt is made to gain the fastest execution time. Once all the map tasks are complete, reduce starts to work (in fact, not necessarily ...). ), for each map task, only perform to the reduce task to download the data above it to complete the success, otherwise, is a failure, need to re-try ...

And the second picture, not I draw, do not reprint, see here, it describes the entire Map/reduce server condition diagram, including the overall process, the server process, input and output, see this picture, the basic flow of map/reduce should be able to completely run through. There are these points, may be described in the diagram is not clear enough to mention, one is in HDFs, in fact, there are log files, the picture is not marked, and the other is step 5, in fact, by the Tasktracker initiative to pull instead of jobtracker push over; and steps 8 and 11, The created Maptask and Reducetask are all running on separate processes in Hadoop ...

Iv. Map Task details please

From above, you can learn the whole process of the map and the reduce task, and the later is verbose, is the details of the implementation. The input to a map task is a file on a distributed file system that contains key-value pairs of information. In order to specify input for each map task, we need to master the file format to divide it into chunks and separate the key value information from each piece. In HDFs, the input file format is represented by the InputFormat class, and in Jobconf, its default value is the Textinputformat class (see Getinputformat), which is the Fileinputformat subclass of the special class, And Fileinputformat is the InputFormat subclass. With this kind of relationship we can easily understand that the default file format is a text file, and the key is the longwritable type (shaping number), the value is the text type (string). Just knowing the file type is not enough, we also need to separate each piece of data in the file into a key-value pair, this work is recordreader to do. In the Getrecordreader method of Textinputformat, we can see that the default companion with Textinputformat is the Linerecordreader class, which is a subclass of the recordreader of the special type, It takes each row as a record, the starting position as the key, and the string of the entire row as the value. With the format, the key values are separated, and you need to cut each map task. The input for each map task is represented by the Inputsplit interface, and for a file input the implementation is Filesplit, which contains the file name, start location, length, and the set of server addresses that store it ...

When the map task gets to its inputsplit, it begins to read the records and calls the mapper for the definition, which is calculated (see Maprunner and Maptask Run methods), and then output. Maptask is passed to mapper a Outputcollector object as the data structure of the output. It defines a collect function that accepts a key-value pair. In Maptask, the subclass of two outputcollector is defined, one is maptask.directmapoutputcollector, the person is the name, its implementation is really direct, straightforward. It will use a Recordwriter object, collect a call, directly call Recordwriter's Write method, write to the local file. If the recordwriter appears very abrupt, then look at the Recordreader mentioned in the previous paragraph, basically, the data structure is the corresponding, one is the input is the output. The output is symmetrical also by Recordwriter and OutputFormat to work together, its default implementation is Linerecordwriter and Textoutputformat, how familiar ah ...

In addition to this very straightforward implementation, there is a more complex implementation of Maptask in the Maptask.mapoutputbuffer. Saying goes simple overwhelming, then why there is a very simple implementation, to ponder a complex it. The reason is that it looks beautiful often with a thorn, simple output implementation, every call to write a file once collect, frequent hard disk operation is likely to lead to the inefficiency of this scenario. In order to solve this problem, this complex version, it first open a memory cache, and then set a scale to do the threshold, open a thread to monitor this cache. Collect to the content, first write to the cache, when the monitoring thread found that the content of the cache exceeded the threshold, suspend all write operations, build a new file, the contents of the cache to batch brush to this file, empty the cache, re-open, accept continue collect ...

Why do you want to brush it into a file? Because this is not a simple scripted simple copy process, the memory in the cache, sorted, merged (Combiner), is written before it is written. If you think combiner this noun sounds too unfamiliar, then consider Reducer,combiner is a reducer class, through the jobconf Setcombinerclass to set up, in the common configuration, Combiner is often the reducer subclass defined by the user for the reduce task. Only, combiner is only a smaller range of services, it is in the map task execution of the server local, according to the map processed by the small amount of data, first do a reduce operation, so that you can compress the size needed to transfer content, improve speed. Every time the cache is cached, a new file is opened, and when all the inputs are processed, a number of sequential, merged output files are available. The system will put together these files, and then make a multi-way merge outside the row, while using the merger merge, finally, the unique, ordered, merged intermediate file (note: The number of files equal to the number of categories, when not considered the classification, simply as a ... ）。 It is the input file that the reduce task is dreaming of ...

Besides doing the merging, the complex version of Outputcollector also has the function of classification. Classification is defined by Partitioner, and the default implementation is Hashpartitioner, which the job submitter can customize by jobconf Setpartitionerclass. What is the meaning of classification, simply, is to divide the output of the map task into several files (usually equal to the number of reduce tasks), so that each reduce task can process a certain type of file. The benefits are big, give an example to illustrate this. For example, a job is a word statistic, the middle result of its map task should be the word as the key, the number of words to the value of the file. If there is only one reduce task at this time, it is good to say, collect the files from all the map tasks, and statistically get the final output file. However, if a single reduce task cannot host this load or is inefficient, multiple reduce tasks are required to execute in parallel. At this point, there is a problem with the previous pattern. Each reduce task obtains input files from a subset of map tasks, but the final output is not correct, because the same word may be counted in different reduce tasks, and the method needs to be counted together to get the final result, so there is no map/ The role of reduce is fully played out. At this point, the classification needs to be used. If there are two reduce tasks at this point, then the output is divided into two categories, a class of words that hold a higher alphabetical order, a class of words with low alphabet sorting, and each reduce task obtains a class of intermediate files from all the map tasks to get its own output. The final result, only need to put the various reduce task output, stitching together on it. Essentially, this is the input of the reduce task, which is split vertically into horizontal segmentation. The function of Partitioner is to accept a key value and return the ordinal of a category. It will be in the brush from the cache to the file before the work, in fact, just a choice of a file name, no other logic need to change ...

In addition to caching, merging, sorting, and other additional work, the complex version of the Outputcollector also support the Skip function of error data, in the later distributed will be the wrong time, will also mention, mark a bit, press no table ...

V. Reduce Task Details

In theory, the entire execution process of the reduce task is more verbose than the map task, because it needs to collect the input files before it can be processed. Reduce tasks, there are three main steps: Copy, Sort, reduce (see Reducetask's Run method). The so-called copy is from the server that executes each map task, acquisition to Local. The copy task, which is owned by the Reducetask.reducecopier class, has an inline class called mapoutputcopier, which is responsible for copying the files on a map task server within a single thread. Remote copy of the content (of course, it can also be local ...) ), as a Mapoutput object exists, it can also be serialized on disk in memory, which is automatically adjusted according to memory usage. The entire copy process is a dynamic process, that is, it is not a good time to give all the input information will not change. It will not stop calling the Getmapcompletionevents method of the Taskumbilicalprotocol protocol, Ask the parent tasktracker about the completion status of the map task for this job (Tasktracker to ask Jobtracker and then tell it ...) ）。 Once the information for the server is executed on the relevant map task, a thread is opened to do the specific copy work. At the same time, there is a memory merger thread and a file merger threads are working synchronously, they will be freshly downloaded files (possibly in memory, simply collectively referred to as files ...). ), do a merge sort, in order to save time, reduce the number of input files, for the subsequent sort of work burden ...

Sort, sorting work, is equivalent to a continuation of the above sort work. It will be done after all the files have been copied, because although the synchronization has to do with the work of merging, but may keep the tail, not done thoroughly. After this process, it was completely complete, and a new file that merged all the required map task output files was born. And those thousand lines of pain from other servers to snare the map task output file, quickly ended their historical mission, was swept out swept away, all deleted ...

In the end, the final phase of the reduce task is the reduce itself. It will also prepare a outputcollector to collect the output, unlike Maptask, this outputcollector is simpler, just open a recordwriter,collect once, write once. The biggest difference is that the incoming Recordwriter file system is basically a distributed file system, or HDFs. In terms of input, Reducetask calls a bunch of custom classes, such as Getmapoutputkeyclass, Getmapoutputvalueclass, Getoutputkeycomparator, and so on, from jobconf. Constructs the type of key required for the reducer, and the iteration type of the value iterator (a key is usually the corresponding set of values). Concrete implementation is rather roundabout, suggest to look at merger.mergequeue,rawkeyvalueiterator,reducetask.reducevaluesiterator and so on implementation. With the input, with the output, the loop calls the custom reducer, and finally, the reduce phase is complete ...

Vi. Distributed Support 1, server correctness guarantee

Hadoop map/reduce Server Status and HDFs are similar, so that the way to heal the same is the same. Don't say much nonsense, just cut to the chase. As a client, Map/reduce's client simply submits the job, starts moving a bench to see the play, and does not occupy Manger's action. So, once it's hung up, it's dead, and it doesn't hurt. While the task server also needs to keep the heartbeat contact with the job server at any time, once there is a problem, the job server can hand over the task it is running on to the person who completes it. The job server, as a single point, is very similar to recovering with a restore point (which is equivalent to an HDFS mirror) and a history record (equivalent to an HDFS log). On top of that, you need to persist content for recovery, including job status, task status, work status of individual task attempts, and so on. With this content, plus the dynamic registration of the task server, even if you move a nest, it is easy to recover. Jobhistory is a static class of historical records, originally, it is a dry write log live, but in the implementation of Hadoop, the log write to the object-oriented package, but also a large number of observer patterns to do some embedding, making it look not so intuitive. Essentially, it is to open several log files and use various interfaces to write content. But, these logs, will put in the Distributed file system, do not need like HDFs, to a secondxxx with time life, this shows, there is a giant stepping on the foot, really good. The Jobtracker.recoverymanager class is used in the job server for recovery-related things, and when the job server starts, it calls its recover method to recover the contents of the log file. The steps, written in the comments are very clear, please see for yourself ...

2. The correctness and speed of task execution

The entire operation process, adhering to the principle of the barrel. The slowest map task and the reduce task, which determines the overall execution time of the system (of course, if the execution time is very small in the overall process, it may be insignificant ...) ）。 Therefore, as much as possible to speed up the slowest task execution speed, become the key to improve overall speed. The strategy used, simple but not simple, is that a task executes multiple times. When all the unfinished tasks have been assigned, and the part of the task that is rich first has been completed, and the task server has been working tirelessly to obtain the task, the job server will start to fry the leftovers, put those tasks that are Chi Chi on a server, and then assign the task to a new task server. Execute simultaneously. Two servers each do their own power, into a king defeated, the results of the first end will be adopted. Such a strategy, implied by a hypothesis, we believe that the input file segmentation algorithm is fair, a task execution is slow, not because the task itself is too heavy, but because the server does not bear the burden of too much capacity or is about to abandon the West, give it a new environment, people move the dead tree to move more and more ...

Of course, there must be a choking task, no matter on which server, can not be completed successfully. This means that the problem is not on the server, but on the task itself. Where is the flaw? Each job, the function code is the same, the other task succeeds, that is, the task is unsuccessful, obviously, the problem is in the input there. There are illegal input entries in the input, which causes the program to be unrecognizable and can only departs farewell. Speaking of this, the solution has surfaced, 36 of the position, can not afford to, still can afford to. In the maptask of Maptask.skippingrecordreader and Reducetask in the Reducetask.skippingreducevaluesiterator, are used to do this thing. Their principle is simple, that is, before reading a record, the current position information, encapsulated into a Sortedranges.range object, through the task of the Reportnextrecordrange method submitted to tasktracker up. Tasktracker will put the content in the TaskStatus object, with the heartbeat message, reported to Jobtracker above. In this way, the job server can be at any time to understand clearly, each task is reading in that location, once the error, re-execution, in the assigned task information to add a set of sortedranges information. When the Maptask or reducetask reads, they will look at these areas and skip reading if the current area is exactly in the above mined area. So repeated, is the road twists and turns, the future bright ah ...

Vii. Summary for Map/reduce, the real difficulty is to improve their adaptability and to create a framework of implementation that is capable of treating all ills. Hadoop has done a great job, but you can help it do better if you really understand the whole process ...

Distributed Basic Learning (2) Distributed Computing System (MAP/REDUCE)