Nutch-1.2 configuration under Ubuntu

1. Download nutch1.2 to a specified directory and open eclipse to create a new java project. And select "Createprojectfromexistingsource" to point to the nutch directory. 2. Next, switch to "Libraries" and select "AddClassFolder... "button, select" conf "from the list, continue operation: switch to" OrderandExport "find" conf "and move it to the top.

1. Download nutch1.2 to a specified directory and open eclipse to create a new java project. And select "Create project from existing source" to point to the nutch directory.

2. Next, switch to "Libraries" and select "Add Class Folder... "button, select" conf "from the list, continue operation: switch to" Order and Export "find" conf ", and move it to the top.

3. Go to "Source" and set output folder to Nutch/bin/tmp_build. Click finish to finish the import.

4. configuration file: Maid, fig.
1) nutch-default.xml
Modify here:

Plugin. folders
./Src/plugin
Directories where nutch plugins are located. Each
Element may be a relative or absolute path. If absolute, it is used
As is. If relative, it is searched for on the classpath.

2) nutch-site.xml

In Add:

Http. agent. name

My nutch agent

Http. agent. version

1.0

 

32.16crawl-urlfilter.txt

Delete: MY. DOMAIN. NAME

Adding: + ^ http: // ([a-z0-9] */.) * qq.com/

The following is your practice: directly configure accept anything else to "+ .".

Create weburls.txt in the nutchdirectory and add the entry address: http://www.qq.com

5. Capture

Run crawl executable class
Configure the runtime environment:
Program arguments:
Bin/nuttch crawl weburls.txt-dir localweb-depth 50-topN 100-threads 2

VM arguments:
-DHadoop. log. dir = logs-Dhadoop. log. file = hadoop. log

6) Some errors may occur, for example, the class cannot be found. At this time, you may need to re-compile the nutch with ant, switch to the installation root directory of the nutch, and then execute the ant command. After compilation, try again.

PS: the blue part of this article is your own practice. For other references, see other articles.