Nutch 2.2+mysql installation on the Unbunt

Reference documents

http://nlp.solutions.asia/?p=362

http://blog.csdn.net/fby98710/article/details/10367175

http://blog.csdn.net/itufo/article/details/21519593

Environments that require jdk1.7

1. mysql Database configuration

L My.ini Configuration

Add "Default-character-set=utf8" under [client], [MySQL], respectively;
Added under [mysqld]: Character-set-server=utf8

L Permission Grant

Mysql–u root–p xxxx
GRANT all privileges on * * to [email protected] '% ' identified by ' XXXX ';

L Create DATABASE and table
手动创建数据库nutch:CREATE DATABASE nutch DEFAULT CHARACTER SET utf8mb4 DEFAULT COLLATE utf8mb4_unicode_ci;and data table webpage "If you do not want to use the default library name and table name can also be modified in the relevant configuration file after Nutch installation, see the following instructions", where webpage table structure is as follows:

CREATE TABLE ' webpage ' (
' ID ' varchar (255) Not NULL,//If fill 767 appears
' Headers ' blob,
' Text ' Longtext DEFAULT NULL,
' Status ' int (one) DEFAULT NULL,
' Markers ' blob,
' Parsestatus ' blob,
' Modifiedtime ' bigint () DEFAULT NULL,
' Prevmodifiedtime ' bigint () DEFAULT NULL,
' Score ' float DEFAULT NULL,
' Typ ' varchar (+) CHARACTER SET latin1 DEFAULT NULL,
' BatchId ' varchar (+) CHARACTER SET latin1 DEFAULT NULL,
' BASEURL ' varchar (767) DEFAULT NULL,
' Content ' Longblob,
' title ' varchar (2048) DEFAULT NULL,
' Reprurl ' varchar (767) DEFAULT NULL,
' Fetchinterval ' int (one) DEFAULT NULL,
' Prevfetchtime ' bigint () DEFAULT NULL,
' InLinks ' Mediumblob,
' Prevsignature ' blob,
' Outlinks ' Mediumblob,
' Fetchtime ' bigint () DEFAULT NULL,
' Retriessincefetch ' int (one) DEFAULT NULL,
' Protocolstatus ' blob,
' Signature ' blob,
' Metadata ' blob,
PRIMARY KEY (' id ')
) Engine=innodb DEFAULT Charset=utf8;

Note: The fields in the table are set according to the Nutch conf file "Gora-sql-mapping". You can also generate databases and tables automatically: After you have configured "Gora-sql-mapping", "gora.properties", and other files, you can automatically generate databases and tables by running "Bin/nutch Injecturls" for the first time. However, you may encounter problems in auto-generation, but it doesn't matter, by checking the Hadoop.log file in time you will find a lot of problems related to MySQL supported data types, data length, only need to modify and debug according to the log hints (with the help of navicat tools like SQL Server to operate the database), and then repeat the automatic generation process until it succeeds.
2. Installation and configuration of Nutch

1) Get Nutch 2.2.x: Download from the official website http://www.apache.org/dyn/closer.cgi/nutch/, then extract to the local installation directory, such as the local root directory is ${nutch_home};
2) Configure NUTCH support for MySQL, modify the ${apache_nutch_home}/ivy/ivy.xml file, respectively:

L Uncomment the following lines:

<dependency org= "MySQL" name= "Mysql-connector-java" rev= "5.1.18″conf=" *->default

L Modify the following line:

From the default <dependencyorg= "Org.apache.gora" name= "Gora-core" rev= "0.3" conf= "*->default"/&GT; change to <dependencyorg= "Org.apache.gora" name= "Gora-core" rev= "0.2.1" conf= "*->default"/>

L Uncomment the following lines:

<dependency org= "Org.apache.gora" name= "Gora-sql" rev= "0.1.1-incubating" conf= "*->default"/>
3) Database Connection Configuration

Edit the ${nutch_home}/conf/gora.properties file, comment out the default database connection configuration, and add the following configuration content:

###############################
#Default MySQL Properties #
###############################

Gora.sqlstore.jdbc.driver=com.mysql.jdbc.driver
Gora.sqlstore.jdbc.url=jdbc:mysql://localhost:3306/nutch?createdatabaseifnotexist=true
Gora.sqlstore.jdbc.user=xxxx (mysql user name)
gora.sqlstore.jdbc.password=xxxx (mysql password)

Edit the ${apache_nutch_home}/conf/gora-sql-mapping.xml file changing the length of the PrimaryKey from $767 in both Places.
<primarykey column= "id" length= "767″/>

Configure ${apache_nutch_home}/conf/nutch-site.xml to put in a name in the Value field under Http.agent.name. It can anything but cannot is left blank. Add additional languages If you want (I had added Japanese ja-jp below) and utf-8 as default as well. You must specify Sqlstore.

<property>
<name>http.agent.name</name>
<value>YourNutchSpider</value>
</property>

<property>
<name>http.accept.language</name>
<VALUE>JA-JP, en-us,en-gb,en;q=0.7,*;q=0.3</value>
<description>value of the "Accept-language" Request header field.
This allows selecting Non-english language as the default one to retrieve.
It is a useful setting for search engines build for certain national group.
</description>
</property>

<property>
<name>parser.character.encoding.default</name>
<value>utf-8</value>
<description>the character encoding to fall back to when no other information
Is available</description>
</property>

<property>
<name>storage.data.store.class</name>
<value>org.apache.gora.sql.store.SqlStore</value>
<description>the Gora DataStore class for storing and retrieving data.
Currently the following stores is available: ....
</description>
</property>

<property>
<name>parser.character.encoding.default</name>
<value>utf-8</value>
<description>thecharacter encoding to fall back to when no other information
Isavailable</description>
</property>

L Special Add the following content
<property>
<name>generate.batch.id</name>
<value>*</value>
</property>

If you do not add this item, you will see the following error in the log when crawling Web pages through "Bin/nutch crawl urls–threadsn–depths N":
Java.lang.NullPointerException
Atorg.apache.avro.util.utf8.<init> (utf8.java:37)
and the "Nutch-site" file needs to be saved in utf-8 format, otherwise the following error occurs when you execute the nutch command.
Exception in thread "main" java.lang.RuntimeException: Byte 1 of the UTF-8 sequence of the com.sun.org.apache.xerces.internal.impl.io.malformedbytesequenceexception:1 byte is invalid.
6) Compiling Nutch 2.2
To ensure that Ant is installed (not installed in the online Baidu Ant installation method), go back to the Nutch root directory, using ant to compile the ant build. This process can take several hours. If you follow the above configuration step by step, the compilation process will be completed successfully. At this point, the installation of Nutch 2.2 has been completed, the next can be configured to configure Web page crawl information, Web page crawl.
cd ${APACHE_NUTCH_HOME}/runtime/local
mkdir -p urls
echo ‘http://nutch.apache.org/‘ > urls/seed.txt


Start crawling (you'll want to create your own script later but manually just to see what's happening type the Followin G into the command line)

bin/nutch inject urls

bin/nutch generate -topN 20
bin/nutch fetch -all
bin/nutch parse -all
bin/nutch updatedb

Repeat the last four commands (generate, Fetch, parse and updatedb) again.

For the Generate command, TopN are the max number of links you want to actually parse each time. The first time there is the only one URL (the one we injected from Seed.txt) and after that there be many more. Note, however, Nutch keeps track of all links it encounters in the webpage table. It just limits the amount it actually parses to TopN so don ' ts be surprised by seeing many more rows in the webpage table T Han you expect by limiting with TopN.

Check Your crawl results by looking on the webpage table in the Nutch database.

mysql -u xxxxx -p
use nutch;
SELECT * FROM nutch.webpage;

can only use Mysql-connector-java-5.1.18.jar, with other JAR file will be error.

Need to load Gora-sql-0.2.1.jar

http://search.maven.org/remotecontent?filepath=org/apache/gora/gora-sql/

Gora-sql-0.1.1-incubating.jar

Nutch 2.2+mysql installation on the Unbunt