Some new discovery about nutch

1 nutch common is'

Bin/nutch crawl <urldir> [-Dir D] [-threads N] [-depth I] [-topn]',

 

Nutch will generate segmentEach depth, And topn means each layer will collect topn URLs. Generally each layer has one single segment, it depends onMaxnumsegments (1 is the default value) in generator. java, so we may find 'fetcher. fetch (segs [0]... '(from crawl. java) Just use the first value of segment array for each layer.

And I also find a variant maxcount, in my mind maxcount is the maximum number of links from the same host can be stored in each segment, if the value is-1 that means no limit here.