Big Data Learning-hadoop-fourth lesson

MapReduce Learning

MAP ": The main node reads input data, divides it into small chunks that can be solved in the same way (here is a divide-and-conquer idea), and distributes these small chunks to different working nodes (Worder nodes), each working node (worder node) Loop to do the same thing, this is going to be a tree-row structure (many of the models in distributed computing are related to graph theory, PageRank is also), and each leaf node has to process every small chunk of data, and then return the results to the parent node. Reduce ": The main node gets the results of all child nodes and then combines all the results and returns to the output

<code class= "Hljs r has-numbering" style= "display:block; padding:0px; Color:inherit; Box-sizing:border-box; font-family: ' Source Code Pro ', monospace;font-size:undefined; White-space:pre; border-radius:0px; Word-wrap:normal; background:transparent; " > (<span class= "Hljs-number" style= "Color:rgb (0, 102, 102); box-sizing:border-box;" >1</span>). The client submits an Mr Jar package to jobclient (Submit by: Hadoop jar <span class= "Hljs-keyword" style= "Color:rgb (0, 0, 136); Box-sizing:border-box; " >...</span>) </code><ul class= "pre-numbering" style= "Box-sizing:border-box; Position:absolute; width:50px; top:0px; left:0px; margin:0px; padding:6px 0px 40px; border-right-width:1px; Border-right-style:solid; Border-right-color:rgb (221, 221, 221); List-style:none; Text-align:right; Background-color:rgb (238, 238, 238); " ><li style= "Box-sizing:border-box; padding:0px 5px; " >1</li></ul><ul class= "pre-numbering" style= "Box-sizing:border-box; PositioN:absolute; width:50px; top:0px; left:0px; margin:0px; padding:6px 0px 40px; border-right-width:1px; Border-right-style:solid; Border-right-color:rgb (221, 221, 221); List-style:none; Text-align:right; Background-color:rgb (238, 238, 238); " ><li style= "Box-sizing:border-box; padding:0px 5px; " >1</li></ul>

<code class= "Hljs avrasm has-numbering" style= "display:block; padding:0px; Color:inherit; Box-sizing:border-box; font-family: ' Source Code Pro ', monospace;font-size:undefined; White-space:pre; border-radius:0px; Word-wrap:normal; background:transparent; " > (<span class= "Hljs-number" style= "Color:rgb (0, 102, 102); box-sizing:border-box;" >2</span>) <span class= "Hljs-preprocessor" style= "Color:rgb (68, 68, 68); Box-sizing:border-box; " . Jobclient</span> communicates via RPC and Jobtracker, returning an address (HDFS) that holds the jar package and Jobid</code><ul class= "pre-numbering" Style= "Box-sizing:border-box; Position:absolute; width:50px; top:0px; left:0px; margin:0px; padding:6px 0px 40px; border-right-width:1px; Border-right-style:solid; Border-right-color:rgb (221, 221, 221); List-style:none; Text-align:right; Background-color:rgb (238, 238, 238); " ><li style= "Box-sizing:border-box; padding:0px 5px; " >1</li></ul><ul class= "pre-numbering" style= "box-sizing: Border-box; Position:absolute; width:50px; top:0px; left:0px; margin:0px; padding:6px 0px 40px; border-right-width:1px; Border-right-style:solid; Border-right-color:rgb (221, 221, 221); List-style:none; Text-align:right; Background-color:rgb (238, 238, 238); " ><li style= "Box-sizing:border-box; padding:0px 5px; " >1</li></ul>

<code class= "Hljs fix has-numbering" style= "display:block; padding:0px; Color:inherit; Box-sizing:border-box; font-family: ' Source Code Pro ', monospace;font-size:undefined; White-space:pre; border-radius:0px; Word-wrap:normal; background:transparent; " ><span class= "Hljs-attribute" style= "Box-sizing:border-box;" > (3). Client writes the jar package to HDFs (path </span>=<span class= "hljs-string" style= "Color:rgb (0, 136, 0); box-sizing : Border-box; " > address on HDFs + jobId) </span></code><ul class= "pre-numbering" style= "Box-sizing:border-box; Position:absolute; width:50px; top:0px; left:0px; margin:0px; padding:6px 0px 40px; border-right-width:1px; Border-right-style:solid; Border-right-color:rgb (221, 221, 221); List-style:none; Text-align:right; Background-color:rgb (238, 238, 238); " ><li style= "Box-sizing:border-box; padding:0px 5px; " >1</li></ul><ul class= "pre-numbering" style= "Box-sizing:border-box; Position:absolute; Width:50px; top:0px; left:0px; margin:0px; padding:6px 0px 40px; border-right-width:1px; Border-right-style:solid; Border-right-color:rgb (221, 221, 221); List-style:none; Text-align:right; Background-color:rgb (238, 238, 238); " ><li style= "Box-sizing:border-box; padding:0px 5px; " >1</li></ul>

<code class= "Hljs has-numbering" style= "Display:block"; padding:0px; Color:inherit; Box-sizing:border-box; font-family: ' Source Code Pro ', monospace;font-size:undefined; White-space:pre; border-radius:0px; Word-wrap:normal; background:transparent; " > (4). Start Submitting tasks (description information for tasks, not jars, including Jobid,jar location, configuration information, etc.) </code><ul class= "pre-numbering" style= "box-sizing : Border-box; Position:absolute; width:50px; top:0px; left:0px; margin:0px; padding:6px 0px 40px; border-right-width:1px; Border-right-style:solid; Border-right-color:rgb (221, 221, 221); List-style:none; Text-align:right; Background-color:rgb (238, 238, 238); " ><li style= "Box-sizing:border-box; padding:0px 5px; " >1</li></ul><ul class= "pre-numbering" style= "Box-sizing:border-box; Position:absolute; width:50px; top:0px; left:0px; margin:0px; padding:6px 0px 40px; border-right-width:1px; Border-right-style:solid; Border-right-color:rgb (221, 221, 221); List-style:none; Text-align: Right; Background-color:rgb (238, 238, 238); " ><li style= "Box-sizing:border-box; padding:0px 5px; " >1</li></ul>

<code class= "Hljs avrasm has-numbering" style= "display:block; padding:0px; Color:inherit; Box-sizing:border-box; font-family: ' Source Code Pro ', monospace;font-size:undefined; White-space:pre; border-radius:0px; Word-wrap:normal; background:transparent; " > (<span class= "Hljs-number" style= "Color:rgb (0, 102, 102); box-sizing:border-box;" >5</span>) <span class= "Hljs-preprocessor" style= "Color:rgb (68, 68, 68); Box-sizing:border-box; " . Jobtracker</span> the initialization task (put into the scheduler) </code><ul class= "pre-numbering" style= "Box-sizing:border-box; Position:absolute; width:50px; top:0px; left:0px; margin:0px; padding:6px 0px 40px; border-right-width:1px; Border-right-style:solid; Border-right-color:rgb (221, 221, 221); List-style:none; Text-align:right; Background-color:rgb (238, 238, 238); " ><li style= "Box-sizing:border-box; padding:0px 5px; " >1</li></ul><ul class= "pre-numbering" style= "Box-sizing:border-box; Position:absolutE width:50px; top:0px; left:0px; margin:0px; padding:6px 0px 40px; border-right-width:1px; Border-right-style:solid; Border-right-color:rgb (221, 221, 221); List-style:none; Text-align:right; Background-color:rgb (238, 238, 238); " ><li style= "Box-sizing:border-box; padding:0px 5px; " >1</li></ul>

<code class= "Hljs has-numbering" style= "Display:block"; padding:0px; Color:inherit; Box-sizing:border-box; font-family: ' Source Code Pro ', monospace;font-size:undefined; White-space:pre; border-radius:0px; Word-wrap:normal; background:transparent; " > (6). Read the files to be processed on the HDFS and begin to compute the input shards, each of which corresponds to a mappertask</code><ul class= "pre-numbering" style= "box-sizing: Border-box; Position:absolute; width:50px; top:0px; left:0px; margin:0px; padding:6px 0px 40px; border-right-width:1px; Border-right-style:solid; Border-right-color:rgb (221, 221, 221); List-style:none; Text-align:right; Background-color:rgb (238, 238, 238); " ><li style= "Box-sizing:border-box; padding:0px 5px; " >1</li></ul><ul class= "pre-numbering" style= "Box-sizing:border-box; Position:absolute; width:50px; top:0px; left:0px; margin:0px; padding:6px 0px 40px; border-right-width:1px; Border-right-style:solid; Border-right-color:rgb (221, 221, 221); List-style:none; Text-align:right; Background-color:rgb (238, 238, 238); " ><li style= "Box-sizing:border-box; padding:0px 5px; " >1</li></ul>

<code class= "Hljs avrasm has-numbering" style= "display:block; padding:0px; Color:inherit; Box-sizing:border-box; font-family: ' Source Code Pro ', monospace;font-size:undefined; White-space:pre; border-radius:0px; Word-wrap:normal; background:transparent; " > (<span class= "Hljs-number" style= "Color:rgb (0, 102, 102); box-sizing:border-box;" >7</span>) <span class= "Hljs-preprocessor" style= "Color:rgb (68, 68, 68); Box-sizing:border-box; " . Tasktracker</span> the task (description of the task) through the heartbeat mechanism </code><ul class= "pre-numbering" style= "box-sizing: Border-box; Position:absolute; width:50px; top:0px; left:0px; margin:0px; padding:6px 0px 40px; border-right-width:1px; Border-right-style:solid; Border-right-color:rgb (221, 221, 221); List-style:none; Text-align:right; Background-color:rgb (238, 238, 238); " ><li style= "Box-sizing:border-box; padding:0px 5px; " >1</li></ul><ul class= "pre-numbering" style= "Box-sizing:border-box; Position:aBsolute; width:50px; top:0px; left:0px; margin:0px; padding:6px 0px 40px; border-right-width:1px; Border-right-style:solid; Border-right-color:rgb (221, 221, 221); List-style:none; Text-align:right; Background-color:rgb (238, 238, 238); " ><li style= "Box-sizing:border-box; padding:0px 5px; " >1</li></ul>

<code class= "Hljs has-numbering" style= "Display:block"; padding:0px; Color:inherit; Box-sizing:border-box; font-family: ' Source Code Pro ', monospace;font-size:undefined; White-space:pre; border-radius:0px; Word-wrap:normal; background:transparent; " > (8). Download the required jar, config file etc. </code><ul class= "pre-numbering" style= "Box-sizing:border-box; Position:absolute; width:50px; top:0px; left:0px; margin:0px; padding:6px 0px 40px; border-right-width:1px; Border-right-style:solid; Border-right-color:rgb (221, 221, 221); List-style:none; Text-align:right; Background-color:rgb (238, 238, 238); " ><li style= "Box-sizing:border-box; padding:0px 5px; " >1</li></ul><ul class= "pre-numbering" style= "Box-sizing:border-box; Position:absolute; width:50px; top:0px; left:0px; margin:0px; padding:6px 0px 40px; border-right-width:1px; Border-right-style:solid; Border-right-color:rgb (221, 221, 221); List-style:none; Text-align:right; Background-color:rgb (238, 238, 238); " ><li style= "Box-sizing:border-box; padding:0px 5px; " >1</li></ul>

<code class= "Hljs avrasm has-numbering" style= "display:block; padding:0px; Color:inherit; Box-sizing:border-box; font-family: ' Source Code Pro ', monospace;font-size:undefined; White-space:pre; border-radius:0px; Word-wrap:normal; background:transparent; " > (<span class= "Hljs-number" style= "Color:rgb (0, 102, 102); box-sizing:border-box;" >9</span>) <span class= "Hljs-preprocessor" style= "Color:rgb (68, 68, 68); Box-sizing:border-box; " . Tasktracker</span> initiates a Java child subprocess to perform a specific task (Mappertask or reducertask) </code><ul class= " Pre-numbering "style=" Box-sizing:border-box; Position:absolute; width:50px; top:0px; left:0px; margin:0px; padding:6px 0px 40px; border-right-width:1px; Border-right-style:solid; Border-right-color:rgb (221, 221, 221); List-style:none; Text-align:right; Background-color:rgb (238, 238, 238); " ><li style= "Box-sizing:border-box; padding:0px 5px; " >1</li></ul><ul class= "pre-numbering" style= "box-sizing:border-box; Position:absolute; width:50px; top:0px; left:0px; margin:0px; padding:6px 0px 40px; border-right-width:1px; Border-right-style:solid; Border-right-color:rgb (221, 221, 221); List-style:none; Text-align:right; Background-color:rgb (238, 238, 238); " ><li style= "Box-sizing:border-box; padding:0px 5px; " >1</li></ul>

<code class= "Hljs has-numbering" style= "Display:block; padding:0px; color:inherit; box-sizing:border-box;  font-family: ' Source Code Pro ', monospace;font-size:undefined; White-space:pre; border-radius:0px; Word-wrap:normal; background:transparent; " > (10). Write the results into HDFs </code>

Big Data Learning-hadoop-fourth lesson