Install nutch1.2 in myeclipse in Windows (no error is found at last)

Install nutch1.2 in myeclipse in Windows (no error is found at last)

 

1. Download and install cygwin. The installation and environment configuration are not detailed. Add % cygwin_home % \ bin to path.


2. Import to eclipse


① Add File> New> Project> JAVA project in eclipse.

Project name casually, select "Create project from existing source", in the Browse select the extract path of the nutch, such as D: \ nutch-1.2


② Select conf in "add class folder" File Folder.


③ Define a "Default ouput folder" with any name. Note that you cannot select the bin folder, because if you select the bin folder as the default Output Folder, the folder will be cleared during compilation, and other files under the bin will be deleted, leading to other problems.


④ Finish.


3. modify the configuration file of the nutch. Here we take the capture of www.163.com as an example.


① Modify the nutch-1.2 configuration under D: \ nutch-site.xml \ conf

2. <? XML version = "1.0"?>
   
4. <? XML-stylesheet href = "configuration. XSL"?>
   
6. <! -- Put site-specific property overrides in this file. -->
   
8. <Configuration>
   
10. 
    
12. <Property>
    
14. <Name> HTTP. Agent. Name </Name>
    
16. <Value> nutch-1.2 </value>
    
18. <Description> HTTP 'user-agent' </description>
    
20. </Property>
    
22. 
    
24. <Property>
    
26. <Name> searcher. dir </Name>
    
28. <Value> D: \ nutch-1.2 \ crawl </value>
    
30. <Description> path to root of Crawl. </description>
    
32. </Property>
    
34. 
    
36. </Configuration>

CopyCode

② Modify the crawler l-urlfilter.txt under D: \ nutch-1.2 \ conf

2. # Accept hosts in my. domain. Name
   
4. + ^ Http: // ([a-z0-9] * \.) * 163.info/
   
6. # Skip everything else

Copy code

③ Modify the nutch-1.2 under D: \ nutch-default.xml \ conf

2. <Property>
   
4. <Name> plugin. Folders </Name>
   
6. <Value>./src/plugin </value>
   
8. <Description> directories where nutch plugins are located. Each
   
10. element may be a relative or absolute path. if absolute, it is u SED
    
12. As is. If relative, It is searched for on the classpath. </description>
    
14. </Property>

Copy code

(4) In D: \ nutch-1.2 \, create a file named urlsfolder, and create url.txt text in the file folder, write

1. Http://www.163.com/

Copy code

4. Run nutch in eclipse


① Run-Open Run Dialog


② Name casually written


③ Fill in the main class

1. Org.Apache. Nutch. Crawl. Crawl

Copy code

④ Fill in arguments

1. URLs-Dir crawl-depth 3-topn 50

Copy code

⑤ Fill in VM arguments

1. -Dhadoop. log. dir = logs-dhadoop. log. File = hadoop. Log

Copy code

OK. Run the job. Check if the job is crawling.