Storage Structure of nutch-hbase

 
Webpage field description

Field value changes in various stages of webpage


 

 
Row COM.2345. Www: HTTP/Col F: fi//FetchintervalCol F: TS//Fetchtime
 

 
Id primary key, which is generated based on the webpage URL (Format: reversed domain name: Protocol: Port and Path). Therefore, nutch2 can only save the status of the current webpage, however, historical information cannot be saved. (if this is not the case, historical versions will be retained. Each row has a timestamp, and a certain number of historical versions will be retained. If it is too low, it may be deleted ).
 

Rowkey isReversed domain name: Protocol: Port and path example: COM.2345. www: HTTP/
 

Basically, the URL is used as the rowkey to remove the URL. Then, we can use "fetchtime" to check whether it has reached the generate date.

 

Storage Structure of nutch-hbase