Set the crawling interval policy for the nutch

HTTP://Caols.diandian.com/post/2012-06-05/40028026285


Http://blog.csdn.net/witsmakemen/article/details/7799546. This is related.CodeAnalysis
 

 

I saw an error yesterday. In fact, for successfully crawled URLs, in the update () phase, Program The fetchtime + Fetchinterval is the final next fetchtime. This fetchtime does not represent the time when the webpage successfully fetch, but the next fetchtime. If the URL is crawled within a time earlier than the new fetchtime, the program filters out the URL. Reduce function in crawldbreducer: [Java] view plaincopy  Case Crawldatum. status_fetch_success: //  Succesful fetch      Case Crawldatum. status_fetch_redir_temp://  Successful fetch, redirected      Case  Crawldatum. status_fetch_redir_perm:  Case Crawldatum. status_fetch_notmodified: //  Successful fetch, notmodified  //  Determine the modification status        Int Modified = Fetchschedule. status_unknown;  If (Fetch. getstatus () =Crawldatum. status_fetch_notmodified) {modified = Fetchschedule. status_notmodified ;}  Else  {  If (Oldset & old. getsignature ()! = Null & Signature! = Null  ){  If (Signaturecomparator. _ compare (old. getsignature (), signature )! = 0 ) {Modified = Fetchschedule. status_modified ;} Else  {Modified = Fetchschedule. status_notmodified ;}}}  //  Set the schedule System. Err. println ("1: result. fetchtime =" + Result. getfetchtime (); Result = Schedule. setfetchschedule (text) Key, result, prevfetchtime, prevmodifiedtime, Fetch. getfetchtime (), Fetch. getmodifiedtime (), modified );  //  Set the result status and signature System. Err. println ("2: result. fetchtime =" +Result. getfetchtime ());  If (Modified = Fetchschedule. status_notmodified) {result. setstatus (crawldatum. status_db_notmodified );  If  (Oldset) result. setsignature (old. getsignature ());}  Else  {  Switch  (Fetch. getstatus ()){  Case  Crawldatum. status_fetch_success: result. setstatus (crawldatum. status_db_fetched ); Break  ;  Case  Crawldatum. status_fetch_redir_perm: result. setstatus (fig. status_db_redir_perm );  Break  ;  Case  Crawldatum. status_fetch_redir_temp: result. setstatus (fig. status_db_redir_temp );  Break  ;  Default  : Log. Warn ( "Unexpected status:" + fetch. getstatus () + "resetting to old status .");  If  (Oldset) result. setstatus (old. getstatus ());  Else  Result. setstatus (fig. status_db_unfetched);} result. setsignature (signature );  If (Metafromparse! = Null  ){  For (Entry <writable, writable> E: metafromparse. entryset () {result. getmetadata (). Put (E. getkey (), E. getvalue ());}}}  // If fetchinterval is larger than the system-wide maximum, trigger  //  An unconditional recrawl. This prevents the page to be stuck  //  Notmodified state, when the old fetched copy was already removed  //  Old segments.        If (Maxinterval < Result. getfetchinterval () Result = Schedule. forcerefetch (text) Key, result, False  ); Break  By tracking and printing the fetchtime value of the result, we can find that the program is calling schedule. after the setfetchschedule () function, the value of fetchtime has changed, so it must have changed the fetchtime status of the current URL. In the crawldbreducer class, the fetchschedule called is extended to the defaultfetchschedule class.Source code: [Java] view plaincopy  Public   Class Defaultfetchschedule Extends  Abstractfetchschedule {@ override  Public  Crawldatum setfetchschedule (Text URL, crawldatum datum,  Long Prevfetchtime, Long Prevmodifiedtime,  Long Fetchtime, Long Modifiedtime, Int  State ){  //  System. err. println ("++ 555555555555555 ++ >>>> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>> "); Datum = Super  . Setfetchschedule (URL, datum, prevfetchtime, prevmodifiedtime, fetchtime, modifiedtime, State ); If (Datum. getfetchinterval () = 0 ) {Datum. setfetchinterval (defaultinterval);} datum. setfetchtime (fetchtime + ( Long ) Data. getfetchinterval () * 1000 ); Datum. setmodifiedtime (modifiedtime );  Return  Datum ;}} we can see that there is only one method in this class, setfetchschedule (), this function will eventually set the value of datum fetchtime to datum. setfetchtime (fetchtime + ( Long ) Datum. getfetchinterval () * 1000 );

References
1 http://caols.diandian.com/post/2012-06-05/40028026285
2 http://blog.csdn.net/witsmakemen/article/details/7799546
3. Read the adaptivefetchschedule Class documentation.

Key Attribute parsing: interval
1 injectmapper interval = jobconf. getint ("DB. Fetch. interval. Default", 2592000 );
2 int custominterval = interval; custominterval = integer. parseint (metavalue); or set
3 crawldatum datum = new crawldatum (crawldatum. status_injected, custominterval); it is placed in the crawldataum
4.
Result = schedule. setfetchschedule (text) Key, result, prevfetchtime, prevmodifiedtime, Fetch. getfetchtime (), Fetch. getmodifiedtime (), modified );
5 schedule: defaultfetchschedule is called by default. Here you can set the next collection date by setting the index page expiration time 1800 (half an hour) to set the content page expiration time (7776000 90 days)

<Property>
<Name> dB. Fetch. interval. Default </Name>
<Value> 86400 </value>
<Description> This usage change to be the interval of index page
This property has wasted by wqj 2013-1-18 the default number of seconds between re-fetches of a page (30 days ).
</Description>
</Property>

<Property>
<Name> dB. Fetch. interval. content </Name>
<Value> 7776000 </value>
<Description> interval for content page, default is (90 days ).
</Description>
</Property>


/* Modify time: 2013-01-18 author wqj */
Configuration confcurrent = super. getconf ();
Int interval_index = confcurrent. getint ("DB. Fetch. interval. Default", 86400); // The default value is 24 hours.
Int interval_content = confcurrent. getint ("DB. Fetch. interval. Content", 7776000); // The default value is 90 days.
String Regi = datum. getmetadata (). Get (new text ("Regi"). tostring ();
If (URL. tostring (). Matches (REGI )){
Datum. setfetchinterval (interval_index );
} Else {
Datum. setfetchinterval (interval_content );
}
Datum. setfetchtime (fetchtime + (long) datum. getfetchinterval ()
* 1000 );
Datum. setmodifiedtime (modifiedtime );
Return datum;


Collection Time: around
Http://glcx.moc.gov.cn/CsckManageAction/cxckMoreInfo.do? Byname = lkbs_newroad & page = 9 version: 7
Status: 2 (db_fetched)
Fetch time: Fri Jan 18 14:59:53 CST 2013 the collection month is still in February, but it has been added for half an hour.
Modified Time: Thu Jan 01 08:00:00 CST 1970
Retries since fetch: 0
Retry Interval: 1800 seconds (0 days)

Http://glcx.moc.gov.cn/CsckManageAction/cxckInformationAction.do? Infoid = 8a8181d532b43eb40132b9076dd20253 version: 7
Status: 2 (db_fetched)
Fetch time: Thu Apr 18 14:26:26 CST 2013 month of collection is 2013 February this fetchtime is changed to February this shows that this time is added with Retry Interval
Modified Time: Thu Jan 01 08:00:00 CST 1970
Retries since fetch: 0
Retry Interval: 7776000 seconds (90 days)
Score: 0.009259259
Signature: e901870589199bb918c45c9c9fad0782

We can see from the above
Fetchtime is actually calculated during the last collection.
Next time, the system will directly read the <Current Time URL from the crawlerdb for collection.






Extended:
If you want to view the content page update for example, the content has changed after a period of time, but we use the URL bloom filter to de-duplicate the content, even if there is an update, it will not be reflected in the database.
Check whether regexdumpparser has any special mark on the dump file of the updated webpage. Article Change to update.