Configuration of nutch0.9 in Windows

I. Introduction

Nutch is an open-source Web search engine.

It consists of two parts: crawler and query searcher. The interface between the two is index.

 

2. Required Software

• Jdk1.6

• Tomcat6.0

• Cygwin

  Cause of use: running the script command provided by nutch requires a Linux environment. cygwin is used to simulate the environment. Cygwin is a UNIX simulated environment running on Windows.

• Nutch0.9

Iii. installation and configuration 1. JDK

• Download JDK

• Installation Directory: D:/Java/jdk1.6.0 _ 21

• Modify environment variables:

Java_home = D:/Java/jdk1.6.0 _ 21

Classpath = D:/Java/jdk1.6.0 _ 21/lib/dt. jar; D:/Java/jdk1.6.0 _ 21/lib/tool. Jar

Path = % java_home %/bin

• Test command: Java-version

2. tomcat6.0

• Download Tomcat
• Installation Directory: D:/tomcat6

• Installation test:

  A) start Tomcat: under DOS, cd d:/tomcat6/bin/startup. bat

  B) enter http: // localhost: 8080 in the address bar of the browser. The Tomcat homepage is displayed successfully.

• Change D:/tomcat6/CONF/tomcat-users.xml content

 

<Tomcat-users>

<Role rolename = "manager"/>

<User Username = "Tomcat" Password = "Tomcat" roles = "manager"/>

</Tomcat-users>

 

3. cygwin

• : Http://www.cygwin.cn/

• Installation Directory: D:/cygwin

  Note:On the Select Download Site you need to enter the following URL and at the end select local disk installation: http://www.cygwin.cn/pub/

• Test cygwin

4. nutch

• Download nutch version: nutch-0.9

• Set the environment variable of nutch:

Add the environment variable MAID: Maid = D:/Java/jdk1.6.0 _ 21

 

Modify the environment variable path: Path = % java_home %/bin; % nutch_java_home %/bin

• Decompress the nutch and place it in the/home/user name directory of cygwin. The decompression method is as follows:

1. For ease of operation, we recommend that you copy the installation package of nutch 0.9 to the cygwin/home/user name/directory. This is because cygwin is used to enter the default directory/home/user name on the operation interface. We can use the PWD command to view the current directory.

2. Run the Dir command to view the files in the current directory. Then we can see the installation package of nutch 0.9. Then we can decompress it. Input command: Tar xvf nutch-0.9.tar.gz

 

• Run the command bin/nutch to perform the test. The successful interface is as follows:

 

4. Capture webpage data

In order to capture the http://www.163.com as an example

(1) specify crawler rules

 

Create a folder myurl under the username directory, create a new text file in the folder, and enter a http://www.163.com in the text file to save. In addition, create a folder named mylog under the Directory, which is used to store web page information captured by using nutch.

We also need to modify several files below. We recommend that you use a text editor. For example, I use ultredit.

 

A) Open the nutch-0.9/CONF/nutch-site.xml file. Insert the following content in the middle of <configuration> </configuration>:<Property> <br/> <Name> HTTP. agent. name </Name> <br/> <value> XXXX </value> <br/> <description> HTTP 'user-agent' request header. must not be empty please set this to a single word uniquely related to your organization. </P> <p> Note: You shoshould also check other related properties: </P> <p> HTTP. robots. agents <br/> HTTP. agent. description <br/> HTTP. agent. URL <br/> HTTP. agent. email <br/> HTTP. agent. version </P> <p> and set their values appropriately. </P> <p> </description> <br/> </property> </P> <p> <property> <br/> <Name> HTTP. agent. description </Name> <br/> <value> XXXX </value> <br/> <description> further description of our bot-this text is used in <br/> user-Agent header. it appears in parenthesis after the agent name. <br/> </description> <br/> </property> </P> <p> <property> <br/> <Name> HTTP. agent. URL </Name> <br/> <value> XXXX </value> <br/> <description> a URL to advertise in the User-Agent header. this will <br/> appear in parenthesis after the agent name. custom dictates that this <br/> shocould be a URL of a page explaining the purpose and behavior of this <br/> crawler. <br/> </description> <br/> </property> </P> <p> <property> <br/> <Name> HTTP. agent. email </Name> <br/> <value> XXXX </value> <br/> <description> An email address to advertise in the HTTP 'from 'request <br/> header and User-Agent header. A good practice is to mangle this <br/> address (e.g. 'info at example dot com ') to avoid spamming. <br/> </description> <br/> </property> 

Modify the value in the middle of <value> </value>. The setting here is because nutch complies with the robots protocol and submits relevant information to the crawled website when obtaining the response, for identification. So you can set it to any value you like.

 

B) Open the Nutch-0.9/CONF/crawl-urlfilter.txt file and find+ ^ Http: // ([a-z0-9] */.) * My. domain. Name/Delete 'my. domain. Name/'directly.

(2) start crawling

Run the following command in cygwin:

Bin/nutch crawl myurl-Dir mydir-depth 2> & mylog/crawl1.log

 

 

Here, crawl is followed by directories and files used to store the URLs of the websites you want to crawl. -DIR is followed by the directory used to store the content of the website crawled by the nutch. Note: This directory must not exist currently. It is used to indicate the directory that stores the crawling results. -Depth indicates the crawling depth. The last parameter specifies the Directory and file for storing log files. After running, you can open the log file to view detailed crawler running processes.

 

5. Deploy Web Front-end

• Copy the nutch-0.9.war to Tomcat/webapps;
• Enter http: // localhost: 8080/nutch-0.9/in the browser, Tomcat will automatically unzip the nutch-0.9.war to the webapps directory;
• Modify the tomcat/webapps/nutch-0.9/WEB-INF/classes/nutch-site.xml file and add the following in <configuration> </configuration>:<Property> <br/> <Name> searcher. dir </Name> <br/> <value> D:/cygwin/home/user name/mydir </value> <br/> </property> 

The value in the middle of <value> </value> must be the directory where the previous crawling website is used to store the results.

 

• Modify tomcat/CONF/server. XML to support Chinese search. Find the corresponding location and change it to the following content:

<Connector Port = "8080" protocol = "HTTP/1.1"

Connectiontimeout = "20000"

Redirectport = "8443" uriencoding = "UTF-8" usebodyencodingforuri = "true"/>

 

• Enter http: // localhost: 8080/nutch-0.9 in the browser