Analysis of mapreduce in nutch [zz]

 

Nutch was the first project to use mapreduce (hadoop was actually part of it). The plug-in mechanism of nutch draws on Eclipse's plug-in design idea. In nutch, The mapreduce programming method occupies the majority of its core structure. From the inserted URL list (inject), generate the capture list (generate), capture the content (FETCH), analyze the processed content (PARSE), update the crawl DB database (update ), the conversion Link (invert links) until the index is created, mapreduce is used. Looking at the source code of nutch, we can learn more about how to use mapreduce to handle the problems encountered in our programming.

From obtaining the download list to creating an index:

Insert the URL list to the crawl dB to guide the following crawling program loop:-generate some URL lists from the crawl dB;-capture content;-analyze and process captured content;-update the crawl DB database. convert the external part of each page to create an index for its link

Technical Implementation Details:

1. Insert URL list (inject)

Mapreduce program 1: objective: to convert the input to the crawler format. input: URL File map (line) → <URL, crawldatum> reduce () Merge multiple URLs. output: Temporary crawldatum file. mapreduce2: Target: Merge the temporary files generated in the previous step into the new DB input. Filter duplicate URLs using the crawler map () output by the previous mapreduce operation. reduce: merge two crawldatum to a new DB output.

2. Generate a capture list (generate)

Mapreduce program 1: Target: Select capture list input: Crawl dB file map () → if the current capture time is later than the current time, replace it with the <crawldatum, URL> format. partition: Use the host of the URL to ensure that the same site is distributed to the same reduce program. reduce: obtains n links at the top. mapreduce Program 2: Target: Prepare to capture map () and replace it with <URL, crawler,> Format distributor (partition): use the host output of the URL: <URL, crawler> File

3. Fetch)

Mapreduce: Target: capture content input: <URL, crawldatum>, divided by host, sorted by hash map (URL, crawldatum) → output <URL, fetcheroutput> multithreading, call the crawling protocol plug-in of nutch to capture the output <crawldatum, content> output: <URL, crawldatum>, <URL, content> two files

4. Analysis and Processing content (PARSE)

Mapreduce: Target: Processing captured content input: the captured <URL, content> map (URL, content) → <URL, parse> calls the parsing plug-in of nutch, the output format is <parsetext, parsedata> output: <URL, parsetext>, <URL, parsedata> <URL, crawldatum>.

5. Update the crawl Database)

Mapreduce: Target: integrate fetch and parse into DB input: <URL, crawler ldatum> Add the fetch and parse output to the existing dB, and combine the above three DB into a new DB output: new crawling DB

6. Conversion Link (invert links)

Mapreduce: Target: Count external pages. Enter the following link on the current page: <URL, parsedata>, which contains the Link Map (srcurl, parsedata >→< desturl, inlinks> collect external links to this page in the inlinks format: <srcurl, anchortext> reduce () add inlinks output: <URL, inlinks>

7. Create an index)

Mapreduce: Target: Generate Lucene index input: extract the <URL, parsedata> processed by Parse in multiple file formats, such as the title and metadata information processed by parse, parsetext> extract the <URL, inlinks> extracted <URL, crawldatum> extracted capture time after the text content is converted to the link. map () uses objectwritable to wrap the preceding content reduce () to call the index plug-in of nutch to generate Lucene Document Output: Output Lucene Index