Nutch 2.2+mysql+solr4.2 Implementation of Web site content crawling and indexing

Original address: http://blog.sina.com.cn/s/blog_3c9872d00101p4f0.html

Nutch 2.2.1 was released nearly two months ago, and this version has changed significantly compared to the previous version of Nutch, especially the installation and configuration process with MySQL, which is prone to error in many places. I also encountered a lot of trouble in the installation process, most of the problems through Baidu and Google also did not find a solution, their own only by looking at the code and analysis of the log and trial and error, finally solved the various problems encountered, the important installation and configuration process is now organized as follows.

1. mysql Database configuration

L my.ini configuration

Add under [client], [MySQL], respectively

Default-character-set=utf8

Under [Mysqld], add:

Character-set-server=utf8

　　

L Permission Grant

Mysql–u root–p xxxxgrant All privileges on * * to [email protected] '% ' identified by  ' XXXX ';

　　

L Create DATABASE and table

Manually create the database Nutch and data table webpage "If you do not want the default library name and table name to be modified in the relevant configuration file after Nutch installation, see the following instructions", where the table structure of webpage is as follows:

CREATE TABLE ' webpage ' (' ID ' varchar (767) Not NULL, ' headers ' blob, ' text ' longtext default NULL, ' status ' int (one) ' Default NU ' LL, ' markers ' blob, ' parsestatus ' blob, ' Modifiedtime ' bigint () default null, ' Prevmodifiedtime ' bigint (default null)  , ' score ' float DEFAULT null, ' typ ' varchar (+) CHARACTER set latin1 DEFAULT null, ' batchid ' varchar (+) CHARACTER set Latin1 Default NULL, ' BASEURL ' varchar (767) default NULL, ' content ' Longblob, ' title ' varchar (2048) default NULL, ' Reprurl ' varchar (767) default NULL, ' Fetchinterval ' int (one) default null, ' Prevfetchtime ' bigint (default) null, ' InLinks ' Mediumblob, ' prevsignature ' blob, ' outlinks ' Mediumblob, ' Fetchtime ' bigint () DEFAULT NULL, ' retriessincefetch ' int (11 Default NULL, ' Protocolstatus ' blob, ' signature ' blob, ' metadata ' blob,primary KEY (' id ')) engine=innodb default charset= UTF8;

　　

Note: The fields in the table are set according to the Nutch conf file "Gora-sql-mapping". You can also generate databases and tables automatically: After you have configured "Gora-sql-mapping", "gora.properties" and other files, you can automatically generate databases and tables by running "Bin/nutch inject URLs" for the first time. However, you may encounter problems in auto-generation, but it doesn't matter, by checking the Hadoop.log file in time you will find many problems (such as one) related to MySQL supported data type, data length, only need to make changes according to the log hints, Debug (You can use the Navicat tool like SQL Server to easily manipulate the database), and then repeat the auto-build process until it succeeds.

2. Installation and configuration of Nutch

1) get nutch 2.2.x: download from official website http://www.apache.org/dyn/closer.cgi/nutch/, then unzip to local installation directory, such as local root directory ${nutch_home};

2) Configure nutch support for MySQL , Modify the ${apache_nutch_home}/ivy/ivy.xml file, respectively:

L will cancel the comment in the following line

<dependency org= "MySQL" name= "Mysql-connector-java" rev= "5.1.18″conf=" *->default

　　

L Modify the line to the downside. From the default

<dependency org= "Org.apache.gora" name= "Gora-core" rev= "0.3" conf= "*->default"/>

Change into

<dependency org= "Org.apache.gora" name= "Gora-core" rev= "0.2.1" conf= "*->default"/>

　　

L will cancel the comment in the following line

<dependency org= "Org.apache.gora" name= "Gora-sql" rev= "0.1.1-incubating" conf= "*->default"/>

Note: In the 2nd and 3rd above, if you do not modify by default, you will encounter the following error when crawling Web pages.

Exception in thread "main" Java.lang.ClassNotFoundException:org.apache.gora.sql.store.SqlStore

3) database connection Configuration

Edit the ${nutch_home}/conf/gora.properties file, comment out the default database connection configuration, and add the following configuration content:

################################ Default MySQL Properties    ################################ Gora.sqlstore.jdbc.driver=com.mysql.jdbc.drivergora.sqlstore.jdbc.url=jdbc:mysql://localhost:3306/nutch? Createdatabaseifnotexist=truegora.sqlstore.jdbc.user=xxxx (mysql user name) gora.sqlstore.jdbc.password=xxxx (mysql password)

　　

4) data Table mapping configuration

The main is to modify the ${nutch_home}/conf/gora.properties file, where the modification is proposed in accordance with the previous method of automatic generation of data table to modify, online said to the length of the PrimaryKey from 512 to 767, namely < PrimaryKey column= "id" length= "767″/>. Anyway, I did not succeed in this operation (should be affected by the encoding format), and finally changed to <primarykey column= "id" length= "255″/> done.

5) Modify the nutch-site configuration file

My practice is to save the Nutch-default file as Nutch-site directly, and then modify the Nutch-site content, including:

L Add the value of Http.agent.name

<property><name>http.agent.name</name><value>yournutchspider</value></ Property>

　　

L Add the following at the end of the file

<property> <name>http.accept.language</name> <value>ja-jp, En-us,en-gb,en;             Q=0.7,*;q=0.3</value> <description>value of the Accept-language request header field.             This allows selecting Non-english language as the default one to retrieve.             It is a useful setting for search engines build for certain national group. </description> </property> <property> <name>storage.data.store.class< /name> <value>org.apache.gora.sql.store.SqlStore</value> <description>the Gor             A DataStore class for storing and retrieving data.             Currently the following stores is available:. </description> </property> <property> <name>parser.character.encoding.default&lt ;/name> <value>utf-8</value> <description>the character EncoDing to fall back to where no other information is available</description></property> 

　　

L Special Add the following content

<property>    <name>generate.batch.id</name>    <value>*</value></property >

If you do not add this item, you will see the following error in the log when crawling Web pages through "Bin/nutch crawl urls–threads n–depths n":

Java.lang.NullPointerException
At Org.apache.avro.util.utf8.<init> (utf8.java:37)
At Org.apache.nutch.crawl.GeneratorReducer.setup (generatorreducer.java:100)
At Org.apache.hadoop.mapreduce.Reducer.run (reducer.java:174)
At Org.apache.hadoop.mapred.ReduceTask.runNewReducer (reducetask.java:649)
At Org.apache.hadoop.mapred.ReduceTask.run (reducetask.java:418)
At Org.apache.hadoop.mapred.localjobrunner$job.run (localjobrunner.java:398)

and " Nutch-site "file needs to be saved as Utf-8 format, otherwise in the execution Nutch command, the following error occurs.

Exception in thread "main" java.lang.RuntimeException: Byte 1 of the UTF-8 sequence of the com.sun.org.apache.xerces.internal.impl.io.malformedbytesequenceexception:1 byte is invalid.

6) compile nutch 2.2

To ensure that Ant is installed (not installed in the online Baidu Ant installation method), go back to the NUTCH root directory, using ant compile ${nutch_home}. If you follow the above configuration step by step, the compilation process will be completed successfully. At this point, the installation of Nutch 2.2 has been completed, the next can be configured to configure Web page crawl information, Web page crawl.

3. Web Crawl Configuration

1) set Crawl site

CD ${nutch_home}/runtime/local mkdir-p URLs echo ' http://www.tianya.cn ' > Urls/seed.txt

　　

2) perform a crawl operation

Bin/nutch Crawl urls-depth 3-TOPN 5

　　

When you're done with MySQL, you can see what crawlers crawl, such as: