How to import myeclipse and eclipse

How to import myeclipse and eclipse

 

Today, I spent a whole day on this. At the beginning, I used nutch1.1 to import data to myeclipse and eclipse. I kept trying, debugging, and reading too much online.ArticleThey found that errors could occur in each of their methods. After finally communicating with others, they tried it in their own way. okay, it's not a waste of time. next, let's talk about how to import nutch1.0 without errors.

 

Preparations and import steps:

1. Download cygwin tool: http://www.cygwin.com and install cygwin. After installation is successful, remember to configure environment variables in my computer properties:

Edit the path attribute and addD: cygwin/binTo path.

2. Download The nutch software package:

Http://labs.renren.com/apache-#//nutch/general binpackage can be downloaded.

3. decompress the downloaded nutch package. For example, the decompression location is:D: \ nutch-1.04. Create a new one in eclipse or myeclipseJava ProjectProject, name defined by yourself (nutch). Select"Create project from existing source", Pointing to yourselfNutch-1.0 directory.

 

4. ClickNext step, Switch"Libraries"Select"Add class folder..."Button, select from the list"Conf". -->Default output floder-->Brower-À CongCreate new folder... Create a new folder output. (This step is different from all the methods on the Internet .).

 

Note:: ModifyOutputFolderNutch-site.xmlFile:

 

 

 

XML Code

2. <Property>
4. <Name> HTTP. Agent. Name </Name>
6. <Value> HD nutch agent </value>
8. <Description> </description>
10. </Property>
14. <Property>
16. <Name> HTTP. Agent. Description </Name>
18. <Value> hpjianhua </value>
20. <Description> </description>
22. </Property>
26. <Property>
28. <Name> HTTP. Agent. url </Name>
30. <Value> http://www.163.com </value>
32. <Description> </description>
34. </Property>
38. <Property>
40. <Name> HTTP. Agent. Email </Name>
42. <Value> hpjianhua@163.com </value>
44. <Description> </description>
46. </Property>

 

 

 

5. ClickFinishTo complete the import of nutch1.0.

 

InMyeclipseOrEclipseModerateNutch1.0Modify it to remove the errors prompted by the project.:

 

1.ModifyConfFiles under the folder:

 

1.1 modifyNutch-site.xmlFile:

 

 

XML Code

2. <Configuration>
4. <Property>
6. <Name> HTTP. Agent. Name </Name>
8. <Value> nutch </value>
10. <Description> </description>
12. </Property>
16. <Property>
18. <Name> HTTP. Agent. Description </Name>
20. <Value> hpjianhua </value>
22. <Description> </description>
24. </Property>
28. <Property>
30. <Name> HTTP. Agent. url </Name>
32. <Value> http://www.163.com </value>
34. <Description> </description>
36. </Property>
40. <Property>
42. <Name> HTTP. Agent. Email </Name>
44. <Value> hpjianhua@163.com </value>
46. <Description> </description>
48. </Property>
50. </Configuration>

 

1.2 modifyNutch-defaul.xmlFile:

 

 

 

XML Code

2. <Property>
4. <Name> HTTP. Agent. Name </Name>
6. <Value> HD nutch agent </value>
8. <Description> HTTP 'user-agent' request header. Must not be empty-
10. Please set this to a single word uniquely related to your organization.
14. Note: You shoshould also check other related properties:
18. HTTP. Robots. Agents
20. HTTP. Agent. Description
22. HTTP. Agent. url
24. HTTP. Agent. Email
26. HTTP. Agent. Version
30. And set their values appropriately.
34. </Description>
36. </Property>

 

 

1.3 modifyCrawl-urlfilter.txtFile:

 

 

 

XML Code

1. # Accept hosts in my. domain. Name
2. + ^ Http: // ([a-z0-9] * \.) * 163.com/

 

 

 

If yesNutch1.1You can skip the following2, 3, 4Step,Directly go to5Step 2!

 

2.Download the jar files of MP3 and RTF

Http://nutch.cvs.sourceforge.net/nutch/nutch/src/plugin/parse-mp3/lib,

Http://nutch.cvs.sourceforge.net/nutch/nutch/src/plugin/parse-rtf/lib/

Copy to src/plugin/parse-mp3/lib and src/plugin/parse-RTF/lib/respectively

 

 

3.Refresh a few times, right-click the project folder, select build path-> Configure build path... In the displayed window, switch to libraries and select Add jars ..., Add the downloaded JAR file to the project.

 

4In this step, there will be two errors in the general project. In the official 1.0 Release Version of nutch, these two problems are not fixed because the licensing issues.

 

The next step is the most important part..

 

 

 

Java code

2. Modify rtfparsefactory. Java under -- SRC \ plugin \ parse-RTF \ SRC \ Java \ org \ apache \ nutch \ parse \ RTF
4. Add -- import org. Apache. nutch. parse. parseresult;
6. -- Public parse getparse (content ){
8. Change to -- Public parseresult getparse (content ){
10. -- Return New parsestatus (parsestatus. failed,
12. Parsestatus. failed_exception,
14. E. tostring (). getemptyparse (CONF );
16. Change to -- return New parsestatus (parsestatus. failed,
18. Parsestatus. failed_exception,
20. E. tostring (). getemptyparseresult (content. geturl (), getconf ());
22. -- Return New parseimpl (text,
24. New parsedata (parsestatus. STATUS_SUCCESS,
26. Title,
28. Outlinkextractor. getoutlinks (text, this. conf ),
30. Content. getmetadata (),
32. Metadata ));
34. Change to -- Return parseresult. createparseresult (content. geturl (),
36. New parseimpl (text,
38. New parsedata (parsestatus. STATUS_SUCCESS,
40. Title,
42. Outlinkextractor. getoutlinks (text, this. conf ),
44. Content. getmetadata (),
46. Metadata )));
48. Modify testrtfparser. Java under -- SRC \ plugin \ parse-RTF \ SRC \ test \ org \ apache \ nutch \ parse \ RTF
50. -- Parse = new parseutil (CONF). parsebyextensionid ("parse-rtf", content );
52. Change to -- parse = new parseutil (CONF). parsebyextensionid ("parse-rtf", content). Get (urlstring). At this step, the eclipse project will not be wrong.

 

5. Create a new folder named URLs in the directory of nutch1.0, and then create a text file URL in URLs. Write the link. Note that "/" will be available later.

 

6. Run nutch1.0:

 

Choose run-> Run as-> JAVA application on the pop-up select Java application select Crawl-org.apache.nutch.crawl. Next,

 

Choose run> run commands... In the left-side Java application, there will be the crawl item. Select it,

 

Switch to arguments. The content of program arguments is the parameter to be set. Fill in URLs-Dir crawl-depth 3-topn 50 (the URL is the link based on your actual situation)

 

Enter-dhadoop. log. dir = logs-dhadoop. log. File = hadoop. log.

 

Run it directly: Check whether there is any information on your console.

Note:: For Java heap size problems, check logs/hadoop. log or console output. If a statement similar to Java. Lang. outofmemoryerror: Java heap space appears,

 

Solution:

 

SetEclipse->Window->Preferences->Java->Installed jres->Edit->Default VM arguments

 

7. Set to-xms256m-xmx1024m. XMS indicates the minimum memory, and xmx indicates the maximum memory.