nutch2.1抓取中文網站

標籤：des   style   code   c   tar   ext   

對nutch添加中文網站抓取功能。

1、中文網頁抓取

    A、調整mysql配置,避免存入mysql的中文出現亂碼。修改 ${APACHE_NUTCH_HOME} /runtime/local/conf/gora.properties

     

###############################

# MySQL properties            #

###############################

gora.sqlstore.jdbc.driver=com.mysql.jdbc.Driver

gora.sqlstore.jdbc.url=jdbc:mysql://10.10.11.252:3306/nutch? useUnicode=true&characterEncoding=utf8&autoReconnect=true&zeroDateTimeBehavior=convertToNull

gora.sqlstore.jdbc.user=devuser

gora.sqlstore.jdbc.password=devuser

    B、修改 ${APACHE_NUTCH_HOME} /runtime/local/conf/nutch-site.xml檔案

        <property>

<name>http.accept.language</name>

<value>ja-jp, en-us, zh-cn,en-gb,en;q=0.7,*;q=0.3</value>

<description>Value of the “Accept-Language” request header field.

   This allows selecting non-English language as default one to retrieve.

   It is a useful setting for search engines build for certain national group.

</description>

    </property>