提高nutch爬取效率

Here are the things that could potentially slow down fetching

下面這些是潛在的影響爬取效率的內容：

 

1) DNS setup
2) The number of crawlers you have, too many, too few.
3) Bandwidth limitations
4) Number of threads per host (politeness)
5) Uneven distribution of urls to fetch and politeness.
6) High crawl-delays from robots.txt (usually along with an uneven distribution of urls).

7) Many slow websites (again usually with an uneven distribution).
8) Downloading lots of content (PDFS, very large html pages, again possibly an uneven distribution).

9) Others

 

1）DNS設定
2）你的爬蟲數量，太多或太少
3）頻寬節流設定
4）每一主機的線程數
5）要抓取的urls的分配不均勻
6） robots.txt中的高爬取延時（通常和urls的分配不均勻同時出現）
7）有很多比較慢的網頁（通常和分配不均勻同時出現）
8）要下載太多的內容（PDF，大的html頁面，通常和分配不均勻同時出現）
9）其它

 

Now how do we fix them
那現在怎樣改善它們？

1) Have a DNS setup on each local crawling machine, if multiple crawling machines and a single centralized DNS it

can act like a DOS attack on the DNS server slowing the entire system. We always did a two layer setup hitting

first to the local DNS cache then to a large DNS cache like OpenDNS or Verizon.

 

 

1）在每一個本地的爬蟲機器上設定DNS，如果是多個爬取機器和一個單獨的DNS中心這種情況，那麼它就會像有DOS攻擊在DNS服務

器上那樣，使整個系統變慢。我們經常設定兩層，首先命中本地DNS緩衝，然後就是大的DNS緩衝，就像OpenDNS或Verizon。

 

 

2) This would be number of map tasks * fetcher.threads.fetch. So 10 map tasks * 20 threads = 200 fetchers at

once. Too many and you overload your system, too few and other factors and the machine sites idle. You will need

to play around with this setting for your setup.

 

 

2)這將是map任務數乘以fetcher.threads.fetch屬性值的數量。所以10個map任務*20個線程=一次200個爬取列表。太多的話會超

過你系統的負擔，太少的話就會使一些機器閑置。你需要認真考慮在你的環境下如何設定這些屬性。

 

 

3) Bandwidth limitations. Use ntop, ganglia, and other monitoring tools to determine how much bandwidth you are

using. Account for in and out bandwidth. A simple test, from a server inside the fetching network but not itself

fetching, if it is very slow connecting to or downloading content when fetching is occurring, it is a good bet

you are maxing out bandwidth. If you set http timeout as we describe later and are maxing your bandwidth, you

will start seeing many http timeout errors.

 

 

3）頻寬節流設定，用ntop，ganglia和其它監視軟體來測定你使用了多少的頻寬。計算輸入和輸出的頻寬。可以做一個簡單的測試，

用抓取網路中一台不用作爬蟲的伺服器中，如果它與其中一台爬蟲機器串連時或當那台機器抓取時從中下載資訊時非常慢，這時

你就可以加大頻寬。如果你像我後來說的那樣設定http的逾時時間並且增加了你的頻寬，你會開始看到很多http逾時的錯誤。

 

 

4) Politeness along with uneven distribution of urls is probably the biggest limiting factor. If one thread is

processing a single site and there are a lot of urls from that site to fetch all other threads will sit idle

while that one thread finishes. Some solutions, use fetcher.server.delay to shorten the time between page fetches

and use fetcher.threads.per.host to increase the number of threads fetching for a single site (this would still

be in the same map task though and hence the same JVM ChildTask process). If increasing this > 0 you could also

set fetcher.server.min.delay to some value > 0 for politeness to min and max bound the process.

 

 

4）urls分配的不均勻很有可能是限制效能的一個最大的因素。如果一個線程正在處理一個網站並且那個網站還有很多url等待抓

取，那麼其它線程就會閑置直到那個線程完成抓取。一些解決方案是，使用fetcher.server.delay來縮短網頁抓取之間的時間間

隔，和使用fetcher.threads.per.host來增加同一網站抓取的線程數（這仍然在同一個map任務中，因此也是在同一個JVM中的子

任務中處理）。如果把這些屬性都設定為大於0，你也可以設定fetcher.server.min.delay屬性大於0來設定處理的最小和最大的

界限。

 

 

5) Fetching a lot of pages from a single site or a lot of pages from a few sites will slow down fetching

dramatically. For full web crawls you want an even distribution so all fetching threads can be active. Setting

generate.max.per.host to a value > 0 will limit the number of pages from a single host/domain to fetch.

 

 

5）在一個網站上抓取大量的網頁或在少量網站上抓取大量的網頁將顯著地降低抓取的速度。對於全網爬取，你希望用分布式環境

來使所有抓取線程活動。設定generate.max.per.host大於0將限制在同一網站/網域名稱抓取網頁的數量。

 

 

6) Crawl-delay can be used and is obeyed by nutch in robots.txt. Most sites don't use this setting but a few

(some malicious do). I have seen crawl-delays as high as 2 days in seconds. The fetcher.max.crawl.delay variable

will ignore pages with crawl delays > x. I usually set this to 10 seconds, default is 30. Even at 10 seconds if

you have a lot of pages from a site from which you can only crawl 1 page every 10 seconds it is going to be slow.

On the flip side, setting this to a low value will ignore and not fetch those pages.

 

 

6）爬取延遲。大多數網站不使用這些設定只有少數使用（一些惡意的網站）。我見過爬取延遲每秒最長延遲2天的。

fetcher.max.crawl.delay屬性將忽略爬取延遲大於x的頁面。我經常把它設定成10秒，預設是30秒。儘管設定為10秒，如果你在

某個網站上有大量的頁面要爬取，但你只能每10秒爬取一個頁面，這樣也是很慢的。另一方面，把它的值設定過小將忽略該頁面

並且不抓取這些網頁。

 

 

7) Sometimes, manytimes websites are just slow. Setting a low value for http.timeout helps. The default is 10

seconds. If you don't care and want as many pages as fast as possible, set it lower. Some websites, digg for

instance, will bandwidth limit you on their side only allowing x connections per given time frame. So even if you

only have say 50 pages from a single site (which I still think is to many). It may be waiting 10 seconds on each

page. The ftp.timeout can also be set if fetching ftp content.

 

 

7）有時，網頁剛好很慢。設定http.timeout一個低點的值就有助於這種情況。它的預設值為10秒。如果你不在意並想所有網頁都

儘可能的快，設定得小點。一些網站。例如digg，會在網站中限制你的頻寬並且只允許在某個時間段記憶體在x個到你機器的串連。所以即使你只在一個網站中爬取50個網頁（我仍然認為太多了）。這樣將在每一頁面中等待10秒。ftp.timeout也可以用來設定抓取ftp的內容時的時間間隔。

 

 

8) Lots of content means slower fetching. If downloading PDFs and other non-html documents this is especially

true. To avoid non-html content you can use the url filters. I prefer the prefix and suffix filters. The

http.content.limit and ftp.content.limit can be used to limit the amount of content downloaded for a single

document.

 

 

8）大量的內容意味著要降低抓取的速度。特別是下載PDF或其它非html的檔案時。為了避免下載非html的內容，你可以使用url過

濾器。我更喜歡prefix和suffix過濾器。http.content.limit和ftp.content.limit屬性可以限制一個文檔中下載資料的多少。

 

 

9) Other things that could be causing slow fetching:

Max the number of open sockets/files on a machine. You will start seeing IO errors or can't open socket errors.

Poor routing. Bad routers or home routers might not be able to handle the number of connections going through at

once. An incorrect routing setup could also be causing problems but those are usually much more complex to

diagnose. Use network trace and mapping tools if you think this is happening. Upstream routing can also be a

problem from your network provider.
Bad network cards. I have seen network cards flip once they reach a certain bandwidth point. This was more

prevalent on, at the time, newer gigabit cards. Not usually my first thought but always a possibility. Use

tcpdump and network monitoring tools on the single interface.

 

9）其它可能導致抓取變慢的因素：

一台機器最大可開啟的socket或檔案的多少。你可能會開始看到IO錯誤或不能開啟socket的錯誤。低效的路由。壞的或家裡的路

由不能控制同一時間大量串連的建立。一個錯誤的路由設定也可能導致問題但這些問題通常很難發現。如果你認為是這個問題，

可以用網路跟蹤和映射工具來尋找。反向的路由則可能是你網路供應商的問題。壞的網卡。我曾經見過一些網卡突然達到了某個

頻寬值。這個問題在使用新的網卡時更加普遍。這通常不是我首先想到的但是通常是可能會出現的。可以使用tcpdump和網路監控

工具來檢查那個網路介面。

That is about it from my perspective. Feel free to add anything if anybody else thinks of other things.

這些是我的觀點。如果大家還想到其它的因素，歡迎補充。

 

 

 

本文翻譯自：http://wiki.apache.org/nutch/OptimizingCrawls

英文水平有限，有錯誤歡迎指出。