Nutch 2.2+MySQL在unbunt上的安裝

標籤：nutch

參考文檔

http://nlp.solutions.asia/?p=362

http://blog.csdn.net/fby98710/article/details/10367175

http://blog.csdn.net/itufo/article/details/21519593

需要jdk1.7的環境

1.  MySql資料庫配置

l  my.ini配置

分別在[client]、[mysql]下添加“default-character-set=utf8”；
在[mysqld]下添加：character-set-server=utf8

l  許可權授予

mysql –u root –p xxxx
GRANT ALL PRIVILEGES ON *.* TO [email protected]"%" IDENTIFIED BY  "xxxx";

l  建立資料庫與表
手動建立資料庫nutch:CREATE DATABASE nutch DEFAULT CHARACTER SET utf8mb4 DEFAULT COLLATE utf8mb4_unicode_ci;和資料表webpage【如果不想用預設的庫名和表名也可在nutch安裝後的相關設定檔中進行修改，見後續說明】，其中webpage的表結構如下：

CREATE TABLE `webpage` (
`id` varchar(255) NOT NULL, //如果填767會出現
`headers` blob,
`text` longtext DEFAULT NULL,
`status` int(11) DEFAULT NULL,
`markers` blob,
`parseStatus` blob,
`modifiedTime` bigint(20) DEFAULT NULL,
`prevModifiedTime` bigint(20) DEFAULT NULL,
`score` float DEFAULT NULL,
`typ` varchar(32) CHARACTER SET latin1 DEFAULT NULL,
`batchId` varchar(32) CHARACTER SET latin1 DEFAULT NULL,
`baseUrl` varchar(767) DEFAULT NULL,
`content` longblob,
`title` varchar(2048) DEFAULT NULL,
`reprUrl` varchar(767) DEFAULT NULL,
`fetchInterval` int(11) DEFAULT NULL,
`prevFetchTime` bigint(20) DEFAULT NULL,
`inlinks` mediumblob,
`prevSignature` blob,
`outlinks` mediumblob,
`fetchTime` bigint(20) DEFAULT NULL,
`retriesSinceFetch` int(11) DEFAULT NULL,
`protocolStatus` blob,
`signature` blob,
`metadata` blob,
PRIMARY KEY (`id`)
) ENGINE=InnoDB DEFAULT CHARSET=utf8;

註：表中的欄位根據nutch的conf檔案“gora-sql-mapping”進行設定。同時也可通過自動方式產生資料庫和表：配置好“gora-sql-mapping”、“gora.properties”及其它檔案後，首次通過運行”bin/nutch injecturls”即可自動產生資料庫和表，不過或許在自動產生的時候你會遇到問題，不過沒有關係，通過及時查看hadoop.log檔案你便會發現很多問題與MySQL支援的資料類型、資料長度有關，只需要根據日誌提示做修改、調試（可藉助navicat工具像SQL Server方便操作資料庫），然後再重複自動產生過程，直到成功為止。
2.  Nutch的安裝與配置

1)擷取nutch 2.2.x：從官網http://www.apache.org/dyn/closer.cgi/nutch/下載，然後解壓至本地安裝目錄，如本地根目錄為${NUTCH_HOME}；
2)配置nutch對mysql的支援，修改${APACHE_NUTCH_HOME}/ivy/ivy.xml檔案，分別：

l uncomment以下行:

<dependency org=”mysql” name=”mysql-connector-java”rev=”5.1.18″ conf=”*->default”/>

l 修改以下行:

從預設的<dependencyorg="org.apache.gora" name="gora-core" rev="0.3"conf="*->default"/>，改成<dependencyorg="org.apache.gora" name="gora-core"rev="0.2.1"conf="*->default"/>

l uncomment以下行:

<dependency org="org.apache.gora"name="gora-sql" rev="0.1.1-incubating"conf="*->default" />
3)資料庫連接配置

編輯${NUTCH_HOME}/conf/gora.properties檔案，注釋掉預設的資料庫連接配置，同時添加以下配置內容：

###############################
#Default MySQL properties     #
###############################

gora.sqlstore.jdbc.driver=com.mysql.jdbc.Driver
gora.sqlstore.jdbc.url=jdbc:mysql://localhost:3306/nutch?createDatabaseIfNotExist=true
gora.sqlstore.jdbc.user=xxxx（MySQL使用者名稱）
gora.sqlstore.jdbc.password=xxxx（MySQL密碼）

Edit the ${APACHE_NUTCH_HOME}/conf/gora-sql-mapping.xml file changing the length of the primarykey from 512 to 767 in both places.
<primarykey column=”id” length=”767″/>

Configure ${APACHE_NUTCH_HOME}/conf/nutch-site.xml to put in a name in the value field under http.agent.name. It can be anything but cannot be left blank. Add additional languages if you want (I have added Japanese ja-jp below) and utf-8 as default as well. You must specify Sqlstore.

<property>
<name>http.agent.name</name>
<value>YourNutchSpider</value>
</property>

<property>
<name>http.accept.language</name>
<value>ja-jp, en-us,en-gb,en;q=0.7,*;q=0.3</value>
<description>Value of the “Accept-Language” request header field.
This allows selecting non-English language as default one to retrieve.
It is a useful setting for search engines build for certain national group.
</description>
</property>

<property>
<name>parser.character.encoding.default</name>
<value>utf-8</value>
<description>The character encoding to fall back to when no other information
is available</description>
</property>

<property>
<name>storage.data.store.class</name>
<value>org.apache.gora.sql.store.SqlStore</value>
<description>The Gora DataStore class for storing and retrieving data.
Currently the following stores are available: ….
</description>
</property>

<property>
     <name>parser.character.encoding.default</name>
     <value>utf-8</value>
     <description>Thecharacter encoding to fall back to when no other information
         isavailable</description>
</property>

l  特別添加以下內容
<property>
<name>generate.batch.id</name>
<value>*</value>
</property>

如果不添加此項內容，則通過”bin/nutch crawl urls –threadsn –depths n”爬取網頁時，在日誌中會看到以下錯誤：
java.lang.NullPointerException
atorg.apache.avro.util.Utf8.<init>(Utf8.java:37)
並且“nutch-site”檔案需要儲存為utf-8格式，否則在執行nutch命令時會出現以下錯誤。
Exception in thread “main”java.lang.RuntimeException:com.sun.org.apache.xerces.internal.impl.io.malformedByteSequenceException: 1位元組的UTF-8序列的位元組 1 無效。
6) 編譯nutch 2.2
在保證已安裝ant的情況下（沒有安裝的可在網上baidu下ant的安裝方法），回到nutch根目錄，使用ant編譯 ant build。這個過程可能耗時幾個小時。如果都按上述配置一步步做了，則編譯過程將順利完成。至此，Nutch 2.2的安裝也已完成，接下來就可以根據需要配置網頁抓取資訊，進行網頁抓取了。
cd ${APACHE_NUTCH_HOME}/runtime/local
mkdir -p urls
echo ‘http://nutch.apache.org/‘ > urls/seed.txt


Start crawling (you will want to create your own script later but manually just to see what is happening type the following into the command line)

bin/nutch inject urls

bin/nutch generate -topN 20
bin/nutch fetch -all
bin/nutch parse -all
bin/nutch updatedb

Repeat the last four commands (generate, fetch, parse and updatedb) again.

For the generate command, topN is the max number of links you want to actually parse each time. The first time there is only one URL (the one we injected from seed.txt) but after that there are many more. Note, however, Nutch keeps track of all links it encounters in the webpage table. It just limits the amount it actually parses to TopN so don’t be surprised by seeing many more rows in the webpage table than you expect by limiting with TopN.

Check your crawl results by looking at the webpage table in the nutch database.

mysql -u xxxxx -p
use nutch;
SELECT * FROM nutch.webpage;

// 只能用mysql-connector-java-5.1.18.jar，用其他jar檔案會報錯。

// 需要載入gora-sql-0.2.1.jar

http://search.maven.org/remotecontent?filepath=org/apache/gora/gora-sql/

// gora-sql-0.1.1-incubating.jar

Nutch 2.2+MySQL在unbunt上的安裝