Build a hadoop cluster using the nuttk

1. Apache nutch

Apache nutch is an open-source framework for Web search. It provides all the tools we need to run our own search engine, including full-text search and web crawler.

1.1. Component Structure of nutch

Webdb: Stores webpage data and connection information

Fetch lists: Divides the connections stored in webdb into multiple groups for Distributed Retrieval.

Fetchers: Retrieves the content in the fetch list and downloads it to the local device. There are two outputs: The connected update

Information and content

Updates: Update the page retrieval status of webdb

Webdb, updates, fetch lists, and fetchers constitute a loop structure that keeps running to ensure that the obtained Web image is up-to-date.

Content: Interface content. After obtaining the content, the content can be used to create indexes and perform query operations.

Indexers: Create an index for the target content. When the index content is large, you can divide the index into multiple index fragments and assign them to different seracher for parallel search.

Searchers: When the query function is implemented, content is also cached.

Webservers: There are two roles:

1. Process User Interaction requests (nutch search client)

2. Obtain query results from searchers (HTTP server)

Note: The Operations corresponding to the fetchers and searchers nodes can be completed in the distributed environment (hadoop ).

You can use the SOLR framework to create indexes and query data.

1.2. Data Structure of the nutch:

The nutch data contains three directory structures:

1. crawldb: used to store the URL Information to be searched by the nutch and the retrieval status (whether or not to retrieve and when to retrieve)

2. linkdb: used to store the hyperlink information (including the anchor) contained in each URL)

3. segments: a set of URLs. As a retrieval unit, segments can be used for Distributed Retrieval.

The Segment directory contains the following sub-directories:

(1) crawl_generate: defines the URL set to be retrieved (the file type is sequencefile)

(2) crawl_fetch: stores the retrieval status of each URL (the file type is mapfile)

(3) content: stores the binary byte stream corresponding to each URL (the file type is mapfile)

(4) parse_text: stores the text content parsed by each URL (the file type is mapfile)

(5) parse_data: stores the metadata parsed by each URL (the file type is mapfile)

(6) crawl_parse: Used to promptly update the content in the crawler LDB (for example, the URL to be retrieved does not exist)-the file type is sequencefile

Note: In view of the data structure and component structure of nutch, crawldb is equivalent to webdb, while segment is equivalent to fetchlists.

In the distributed crawl process, each mapreduce job generates a segment named by time.

2. Apache hadoop

However, the local collection method is not complex. However, when the collected data sources are large, it is difficult for a single machine to meet performance requirements, therefore, the common practice is to integrate nutch into the hadoop environment to achieve the effect of distributed collection and Distributed Query (deploy mode ).

The hadoop framework consists of three sub-frameworks:

Mapreduce: used for Distributed Parallel Computing

HDFS: used for Distributed Storage

Common: encapsulate the worker classes required by HDFS and mapreduce

2.1 mapreduce Workflow

1. Cut the Input Source (inputfiles) into different segments. The size of each segment is usually between 16-64 MB (which can be configured by parameters), and then start the cloud program.

2. mapreduce programs are deployed based on the master/slaves method. Select a machine on the cloud to run the master program. The roles include: scheduling tasks assigned to slaves and monitoring the execution of tasks.

3. in the figure, the form of slave is worker. When a worker receives a map task, it reads the Input Source segment and parses the key/value Key-value pair, and passed as a parameter to the User-Defined map function. The output value of the map function is also a key/value Key-value pair. These key-value pairs are temporarily cached in the memory.

4. after caching, the program regularly writes cached key-value pairs to the local hard disk (local write operation executed) and returns the storage address to the master, so that the master can record their locations for the reduce operation.

5. When the worker is notified to execute the reduce operation, the master sends the address stored in the corresponding map output data to the worker so that it can obtain the data through remote calls. After the data is obtained, reduce worker organizes the records with the same key value to achieve the sorting effect.

6. Reduce worker transmits the sorted data as parameters to the custom reduce function, and the output results of the function are stored persistently in the output file.

7. After all the map tasks and reduce tasks are completed, the master node will wake up the user master program again. At this point, a mapreduce operation call is completed.

2.2 HDFS Component Structure

Similar to the mapreduce deployment structure, HDFS also has the master/slaves Master/Slave structure.

1. namenode acts as the master, including: managing the namespace of the document system; adjusting the client to access the required files (files stored in datenode)

Note: namespace-directory structure of the ing File System

2. datanodes acts as slaves. Generally, only one datenode is deployed on one machine to store the data required by mapreduce programs.

Namenode regularly receives heartbeat and blockreport feedback from datanodes.

Heartbeat feedback is used to ensure that the datanode has no function exception;

Blockreport contains the block set stored by datanode

2.3 hadoop Resources

1 Tib

3. Environment Construction

3.1 prepare

3.1.1 two or more Linux machines)

The name of one machine is set to master, and the name of the other machine is set to slave01. The two machines have the same logon username, And the ETC/hosts files of the two machines are set to the same content, for example:
192.168.7.11 master

192.168.7.12 slave01

......

In this way, you can find the corresponding machine through the host name

3.1.2 set up an SSH Environment

To install SSH, run the following command:

$ Sudo apt-Get Install SSH

$ Sudo apt-Get install rsync

3.1.3Install JDK

$ Apt-Get install openjdk-6-jdkopenjdk-6-jre

3.1.4Download the latest versions of hadoop and nutch

:

Hadoop:

Http://www.apache.org/dyn/closer.cgi/hadoop/common/

Nutch: http://www.apache.org/dyn/closer.cgi/nutch/

3.2 set up configuration

3.2.1ssh logon Configuration

(1) run the following command on the master machine to generate the Certificate file authorized_keys:

$ Ssh-keygen-t rsa-p'-f ~ /. Ssh/id_rsa

$ Cat ~ /. Ssh/id_rsa.pub> ~ /. Ssh/authorized_keys

(2) copy the Certificate file to the user's home directory of another machine

$ SCP/home/nutch/. Ssh authorized_keys
Nutch @ slave01:/home/nutch/. Ssh/authorized_keys

With the above two steps, the master machine can SSH to the slave01 machine without a password.

3.2.2hadoop Configuration

Similar to the SSH Login certificate configuration, hadoop configuration is also completed on the master machine, and then copied to the slave machine to ensure that the hadoop environment of each machine is the same

$ Hadoop_home/conf directory:

(1) hadoop-env.sh documents

        export HADOOP_HOME=/PATH/TO/HADOOP_HOME

        export JAVA_HOME=/PATH/TO/JDK_HOME

        export HADOOP_LOG_DIR=${HADOOP_HOME}/logs

(2) core-site.xml documents

        <configuration>

           <property>

               <name>fs.default.name</name>

               <value>hdfs://master:9000</value>

           </property>

        </configuration>

(3) hdfs-site.xml documents

        <configuration>

            <property>

                <name>dfs.name.dir</name>

               <value>/nutch/filesystem/name</value>

           </property>

            <property>

               <name>dfs.data.dir</name>

                <value>/nutch/filesystem/data</value>

            </property>

            <property>

               <name>dfs.replication</name>

               <value>1</value>

            </property>

        </configuration>

(4) mapred-site.xml documents

       <configuration>

            <property>

               <name>mapred.job.tracker</name>

                <value>master:9001</value>

           </property>

           <property> 

               <name>mapred.map.tasks</name>

               <value>2</value>

           </property> 

           <property> 

               <name>mapred.reduce.tasks</name>

               <value>2</value>

           </property> 

           <property>

               <name>mapred.system.dir</name>

               <value>/nutch/filesystem/mapreduce/system</value>

           </property>

           <property>

               <name>mapred.local.dir</name>

               <value>/nutch/filesystem/mapreduce/local</value>

           </property>

      </configuration>

(5) Configure masters and slaves

Add the corresponding machine IP address to the corresponding configuration file.

3.2.3 configuration of nutch

$ Nutch_home/conf directory

(1) nutch-site.xml documents

      <property>

               <name>http.agent.name</name>

               <value>Nutch Spider</value>

      </property>

      (2)regex-urlfilter.txt

Add the URL to be retrieved

      +^http://([a-z0-9]*\.)*nutch.apache.org/

(3) Put the modified file in the maid/runtime/deploy/nutch-*. Job.

3.3 start running

3.3.1 start hadoop

1. format the namenode Node

Bin/hadoop namenode-format

2. Start the hadoop Process

Bin/start-all.sh

After successful startup, you can view the namenode and mapreduce running status through the following URL

Namenode: http: // master: 50070/

Mapreduce: http: // master: 50030/

3. Add test data to HDFS

$ Bin/hadoop FS-put conf Input

4. run the test.

$ Bin/hadoop jar hadoop-examples-*. Jar grep input output 'dfs [A-Z.] +'

5. Disable the hadoop Process

Bin/stop-all.sh

3.3.2 run nutch

1. Prerequisites:

(1) hadoop has been started successfully.

(2) Add the hadoop_home/bin path to the environment variable so that the hadoop command can be found in the nutch.

By modifying the/etc/enviroment configuration file

(3) execute the export java_home =/path/to/Java command on the console

2. store the data to be retrieved into HDFS

    $ bin/hadoop fs -put urldir urldir

Note: The first urldir is a local folder that stores the URL data file. Each line contains a URL

The second urldir is the HDFS storage path.

3. Start the nutch command

Run the following command in the directory of nutch_hone/runtime/deploy:

    $ bin/nutch crawl urldir –dir crawl -depth 3 –topN 10

After the command is successfully executed, the crawl directory is generated in HDFS.

Note: You must execute this command in the deploy directory. In the local directory, the command is collected by a single machine, instead of using the hadoop environment.