Nutch 1.3 學習筆記1

Nutch 1.3 學習筆記1
--------------------
1. Nutch是什麼?

Nutch是一個開源的網頁抓取工具，主要用於收集網頁資料，然後對其進行分析，建立索引，以提供相應的介面來對其網頁資料進行查詢的一套工具。其底層使用了Hadoop來做分散式運算與儲存，索引使用了Solr分布式索引架構來做，Solr是一個開源的全文索引架構，從Nutch 1.3開始，其整合了這個索引架構

2. 在哪裡要可以下載到最新的Nutch?

在下面地址中可以下載到最新的Nutch 1.3二進位包和原始碼
http://mirror.bjtu.edu.cn/apache//nutch/

3. 如何配置Nutch?   3.1 對下載後的壓縮包進行解壓，然後cd $HOME/nutch-1.3/runtime/local   3.2 配置bin/nutch這個檔案的許可權，使用chmod +x bin/nutch    3.3 配置JAVA_HOME，使用export JAVA_HOME=$PATH4. 抓取前要做什麼準備工作?

4.1 配置http.agent.name這個屬性，如下

<property> <name>http.agent.name</name> <value>My Nutch Spider</value></property>

4.2 建立一個地址目錄,mkdir -p urls

   在這個目錄中建立一個url檔案，寫上一些url，如

http://nutch.apache.org/

4.3 然後運行如下命令

bin/nutch crawl urls -dir crawl -depth 3 -topN 5

注意，這裡是不帶索引的，如果要對抓取的資料建立索引，運行如下命令

bin/nutch crawl urls -solr http://localhost:8983/solr/ -depth 3 -topN 5

5. Nutch的抓取流程是什麼樣子的？5.1 初始化crawlDb，注入初始url

bin/nutch inject Usage: Injector <crawldb> <url_dir>

在我本地運行這個命令後的輸出結果如下：

lemo@debian:~/Workspace/java/Apache/Nutch/nutch-1.3$ bin/nutch inject db/crawldb urls/Injector: starting at 2011-08-22 10:50:01Injector: crawlDb: db/crawldbInjector: urlDir: urlsInjector: Converting injected urls to crawl db entries.Injector: Merging injected urls into crawl db.Injector: finished at 2011-08-22 10:50:05, elapsed: 00:00:03

5.2 產生新的抓取urls

bin/nutch generateUsage: Generator <crawldb> <segments_dir> [-force] [-topN N] [-numFetchers numFetchers] [-adddays numDays] [-noFilter] [-noNorm][-maxNumSegments num]

本機輸出結果如下：

lemo@debian:~/Workspace/java/Apache/Nutch/nutch-1.3$ bin/nutch generate db/crawldb/ db/segmentsGenerator: starting at 2011-08-22 10:52:41Generator: Selecting best-scoring urls due for fetch.Generator: filtering: trueGenerator: normalizing: trueGenerator: jobtracker is 'local', generating exactly one partition.Generator: Partitioning selected urls for politeness.Generator: segment: db/segments/20110822105243   // 這裡會產生一個新的segmentGenerator: finished at 2011-08-22 10:52:44, elapsed: 00:00:03

5.3 對上面產生的url進行抓取

bin/nutch fetchUsage: Fetcher <segment> [-threads n] [-noParsing]

這裡是本地的輸出結果：

lemo@debian:~/Workspace/java/Apache/Nutch/nutch-1.3$ bin/nutch fetch db/segments/20110822105243/Fetcher: Your 'http.agent.name' value should be listed first in 'http.robots.agents' property.Fetcher: starting at 2011-08-22 10:56:07Fetcher: segment: db/segments/20110822105243Fetcher: threads: 10QueueFeeder finished: total 1 records + hit by time limit :0fetching http://www.baidu.com/-finishing thread FetcherThread, activeThreads=1-finishing thread FetcherThread, activeThreads=-finishing thread FetcherThread, activeThreads=1-finishing thread FetcherThread, activeThreads=1-finishing thread FetcherThread, activeThreads=0-activeThreads=0, spinWaiting=0, fetchQueues.totalSize=0-activeThreads=0Fetcher: finished at 2011-08-22 10:56:09, elapsed: 00:00:02

我們來看一下這裡的segment目錄結構

lemo@debian:~/Workspace/java/Apache/Nutch/nutch-1.3$ ls db/segments/20110822105243/content  crawl_fetch  crawl_generate

5.4 對上面的結果進行解析

bin/nutch parseUsage: ParseSegment segment

本機輸出結果：

lemo@debian:~/Workspace/java/Apache/Nutch/nutch-1.3$ bin/nutch parse db/segments/20110822105243/ParseSegment: starting at 2011-08-22 10:58:19ParseSegment: segment: db/segments/20110822105243ParseSegment: finished at 2011-08-22 10:58:22, elapsed: 00:00:02

我們再來看一下解析後的目錄結構

lemo@debian:~/Workspace/java/Apache/Nutch/nutch-1.3$ ls db/segments/20110822105243/content  crawl_fetch  crawl_generate  crawl_parse  parse_data  parse_text

這裡多了三個解析後的目錄。

5.5 更新外連結資料庫

bin/nutch updatedbUsage: CrawlDb <crawldb> (-dir <segments> | <seg1> <seg2> ...) [-force] [-normalize] [-filter] [-noAdditions]

本機輸出結果：

lemo@debian:~/Workspace/java/Apache/Nutch/nutch-1.3$ bin/nutch updatedb db/crawldb/ -dir db/segments/CrawlDb update: starting at 2011-08-22 11:00:09CrawlDb update: db: db/crawldbCrawlDb update: segments: CrawlDb update: additions allowed: trueCrawlDb update: URL normalizing: falseCrawlDb update: URL filtering: falseCrawlDb update: Merging segment data into db.CrawlDb update: finished at 2011-08-22 11:00:10, elapsed: 00:00:01

這時它會更新crawldb連結庫，這裡是放在檔案系統中的，像taobao抓取程式的連結庫是用redis來做的，一種key-value形式的NoSql資料庫。

5.6 計算反向連結

bin/nutch invertlinksUsage: LinkDb <linkdb> (-dir <segmentsDir> | <seg1> <seg2> ...) [-force] [-noNormalize] [-noFilter]

本地輸出結果：

lemo@debian:~/Workspace/java/Apache/Nutch/nutch-1.3$ bin/nutch invertlinks db/linkdb -dir db/segments/LinkDb: starting at 2011-08-22 11:02:49LinkDb: linkdb: db/linkdbLinkDb: URL normalize: trueLinkDb: URL filter: trueLinkDb: adding segment: file:/home/lemo/Workspace/java/Apache/Nutch/nutch-1.3/db/segments/20110822105243LinkDb: finished at 2011-08-22 11:02:50, elapsed: 00:00:01

5.7 使用Solr為抓取的內容建立索引

bin/nutch solrindexUsage: SolrIndexer <solr url> <crawldb> <linkdb> (<segment> ... | -dir <segments>

Nutch端的輸出如下：

lemo@debian:~/Workspace/java/Apache/Nutch/nutch-1.3$ bin/nutch solrindex http://127.0.0.1:8983/solr/ db/crawldb/ db/linkdb/ db/segments/*

SolrIndexer: starting at 2011-08-22 11:05:33

SolrIndexer: finished at 2011-08-22 11:05:35, elapsed: 00:00:02

Solr端的部分輸出如下：

INFO: SolrDeletionPolicy.onInit: commits:num=1        commit{dir=/home/lemo/Workspace/java/Apache/Solr/apache-solr-3.3.0/example/solr/data/index,segFN=segments_1,version=1314024228223,generation=1,filenames=[segments_1]Aug 22, 2011 11:05:35 AM org.apache.solr.core.SolrDeletionPolicy updateCommitsINFO: newest commit = 1314024228223Aug 22, 2011 11:05:35 AM org.apache.solr.update.processor.LogUpdateProcessor finishINFO: {add=[http://www.baidu.com/]} 0 183Aug 22, 2011 11:05:35 AM org.apache.solr.core.SolrCore executeINFO: [] webapp=/solr path=/update params={wt=javabin&version=2} status=0 QTime=183Aug 22, 2011 11:05:35 AM org.apache.solr.update.DirectUpdateHandler2 commitINFO: start commit(optimize=false,waitFlush=true,waitSearcher=true,expungeDeletes=false)

5.8 在Solr的用戶端查詢
在瀏覽器中輸入 

http://localhost:8983/solr/admin/

查詢條件為baidu

輸出的XML結構為

如果你要以HTML結構顯示把Solr的設定檔solrconfig.xml中的content改為如下就可以
<field name="content" type="text" stored="true" indexed="true"/>

<response><lst name="responseHeader"><int name="status">0</int><int name="QTime">0</int><lst name="params"><str name="indent">on</str><str name="start">0</str><str name="q">baidu</str><str name="version">2.2</str><str name="rows">10</str></lst></lst><result name="response" numFound="1" start="0"><doc><float name="boost">1.0660036</float><str name="digest">7be5cfd6da4a058001300b21d7d96b0f</str><str name="id">http://www.baidu.com/</str><str name="segment">20110822105243</str><str name="title">百度一下，你就知道</str><date name="tstamp">2011-08-22T14:56:09.194Z</date><str name="url">http://www.baidu.com/</str></doc></result></response>
   

    

    
6 參考
    
http://wiki.apache.org/nutch/RunningNutchAndSolr