源碼:injectedScore()初讀

來源:互聯網
上載者:User

Inject類下的InjectMapper中有一句:

try {
         scfilters.injectedScore(value, datum);
        } catch (ScoringFilterException e) {
         if (LOG.isWarnEnabled()) {
         LOG.warn("Cannot filter injected score for url " + url
         + ", using default (" + e.getMessage() + ")");
         }
        }

裡面調用的是ScoringFilters類的執行個體裡面的方法:public void injectedScore(Text url, CrawlDatum datum) throws ScoringFilterException

作用是:通過分數作為度量,計算(初始化)每個通過substitution和filter後的種子url分數。

接下來,說說ScoringFilters類,首先它實現了ScoringFilter介面,並且裡面有一個private ScoringFilter[] filters;變數

再看看public void injectedScore(Text url, CrawlDatum datum) throws ScoringFilterException方法的原始碼:

 /** Calculate a new initial score, used when injecting new pages. */
  public void injectedScore(Text url, CrawlDatum datum) throws ScoringFilterException {
    for (int i = 0; i < this.filters.length; i++) {
      this.filters[i].injectedScore(url, datum);
    }
  }

可見ScoringFilters類實際上的injectedScore的工作是通過調用private ScoringFilter[] filters;變數中的每個filter的injectedScore()方法來完成的。

所以,很有必要知道怎麼樣去指定private ScoringFilter[] filters中的內容的。

留意到ScoringFilters類的public
ScoringFilters(Configuration conf) 建構函式。

下面是public
ScoringFilters(Configuration conf) 的原始碼:(注意為了實驗,作者增加了一些測試代碼)

public
ScoringFilters(Configuration conf) {
    super(conf);
    ObjectCache objectCache = ObjectCache.get(conf);

//讀取conf檔案(conf/nutch-default.xml或者conf/nutch-site.xml)下的設定檔選項(scoring.filter.order)擷取次序
    String order = conf.get("scoring.filter.order");

//查看緩衝中的內容,看是否存在filters(已經包含了次序的);比如,第二次使用時候,就有用了
    this.filters = (ScoringFilter[]) objectCache.getObject(ScoringFilter.class.getName());

    if (this.filters == null) {//第一次啟用ScoringFilters類

//通過conf檔案(conf/nutch-default.xml或者conf/nutch-site.xml)下的設定檔選項(scoring.filter.order)擷取次序

// String[]
是一個數組,使用次序的;
      String[] orderedFilters = null;
      if (order != null && !order.trim().equals("")) {
        orderedFilters = order.split("\\s+");
      }

      try {
      //test by kaiwii
      System.out.println("try block");
        ExtensionPoint point = PluginRepository.get(conf).getExtensionPoint(ScoringFilter.X_POINT_ID);
        if (point == null) throw new RuntimeException(ScoringFilter.X_POINT_ID + " not found.");
        Extension[] extensions = point.getExtensions();
        HashMap<String, ScoringFilter> filterMap =
          new HashMap<String, ScoringFilter>();
        for (int i = 0; i < extensions.length; i++) {//通過ScoringFilter.X_POINT_ID擷取擴充點,從而擷取所有的ScoringFilter的實作類別

  
                                                                     //ScoringFilter的實作類別與擴充點之間的對應關係,通過遍曆外掛程式中的plugin.xml中獲得(詳情查看PluginRepository類的實現)

  
       Extension extension = extensions[i];
          //for test by kaiwii
          System.out.println("extension_id "+i+extension.getId());
          System.out.println("extension_clazz "+i+extension.getClazz());
          //因為外掛程式的實現採用了lazy_load的方式,所以上面擷取extension時,只是擷取了一個plugindescriptor(包含的僅是某個plugin的資訊而已)

  
       //這裡才真正進行執行個體化
          ScoringFilter filter = (ScoringFilter) extension.getExtensionInstance();
          if (!filterMap.containsKey(filter.getClass().getName())) {//用一個map來組織起所有filter的執行個體
            filterMap.put(filter.getClass().getName(), filter);
          }
        }
        if (orderedFilters == null) {//當conf檔案(conf/nutch-default.xml或者conf/nutch-site.xml)下的設定檔選項(scoring.filter.order)為空白的時候
          objectCache.setObject(ScoringFilter.class.getName(), filterMap.values().toArray(new ScoringFilter[0]));
        } else {
          ScoringFilter[] filter = new ScoringFilter[orderedFilters.length];
          for (int i = 0; i < orderedFilters.length; i++) {//按照orderedFilters指定的順序,把filter放進一個臨時變數中

  
         filter[i] = filterMap.get(orderedFilters[i]);
          }
          objectCache.setObject(ScoringFilter.class.getName(), filter);
        }
      } catch (PluginRuntimeException e) {
        throw new RuntimeException(e);
      }

//實際就是上面filter的值而言
      this.filters = (ScoringFilter[]) objectCache.getObject(ScoringFilter.class.getName());
      //for test by kaiwii
      System.out.println("filters content:kaiwii want to know:");
      for(ScoringFilter v:this.filters){
      System.out.println("filter:"+v.toString());
      }
    }
  }

到此,至於怎麼樣的ScoringFilter實作類別會放到裡面去,應該大概有個明白了吧?

其實,還要留意一下conf檔案(conf/nutch-default.xml或者conf/nutch-site.xml)下的設定檔選項(scoring.filter.order):

<!--
scoring filters properties -->

<property>
  <name>scoring.filter.order</name>
  <value></value>
  <description>The order in which scoring filters are applied.
  This may be left empty (in which case all available scoring
  filters will be applied in the order defined in plugin-includes
  and plugin-excludes), or a space separated list of implementation
  classes.
  </description>
</property>

通過上面的英文介紹,你應該可以知道為空白的時候,怎麼樣的ScoringFilter實作類別會被調用,次序怎麼樣,就是 plugin.includes和plugin-excludes說了算。

就以預設的設定來看看吧,首先,scoring.filter.order為空白;

然後 plugin.includes設定為:(相關內容高亮出來了!)

<property>
  <name>plugin.includes</name>
  <value>protocol-http|urlfilter-regex|parse-(text|html|js|tika)|index-(basic|anchor)|query-(basic|site|url)|response-(json|xml)|summary-basic|scoring-opic|urlnormalizer-(pass|regex|basic)</value>

最後,plugin.excludes為空白。

結合修改後的測試代碼,看看run
inject類的console怎麼樣:(相關內容高亮出來!)

Injector:
starting at 2011-09-04 09:11:13
Injector: crawlDb: crawldb
Injector: urlDir: urls.txt
Injector: Converting injected urls to crawl db entries.
try block
extension_id 0org.apache.nutch.scoring.opic.OPICScoringFilter
extension_clazz 0org.apache.nutch.scoring.opic.OPICScoringFilter
filters content:kaiwii want to know:
filter:org.apache.nutch.scoring.opic.OPICScoringFilter@12a3793
using method:regexNormalize()
kaiwii want to know confiFilenull
use global rules
Injector: Merging injected urls into crawl db.
Injector: finished at 2011-09-04 09:11:19, elapsed: 00:00:05

通過,上面的console顯示得知,只有org.apache.nutch.scoring.opic.OPICScoringFilter被調用了。

所以,要使用相關的ScoringFilter實作類別,就要按照下面的操作進行了:

The order in which scoring filters are applied.
  This may be left empty (in which case all available scoring
  filters will be applied in the order defined in plugin-includes
  and plugin-excludes), or a space separated list of implementation
  classes.

總結:熟悉外掛程式機制!!!!!!!!!!!!!!!!




聯繫我們

該頁面正文內容均來源於網絡整理,並不代表阿里雲官方的觀點,該頁面所提到的產品和服務也與阿里云無關,如果該頁面內容對您造成了困擾,歡迎寫郵件給我們,收到郵件我們將在5個工作日內處理。

如果您發現本社區中有涉嫌抄襲的內容,歡迎發送郵件至: info-contact@alibabacloud.com 進行舉報並提供相關證據,工作人員會在 5 個工作天內聯絡您,一經查實,本站將立刻刪除涉嫌侵權內容。

A Free Trial That Lets You Build Big!

Start building with 50+ products and up to 12 months usage for Elastic Compute Service

  • Sales Support

    1 on 1 presale consultation

  • After-Sales Support

    24/7 Technical Support 6 Free Tickets per Quarter Faster Response

  • Alibaba Cloud offers highly flexible support services tailored to meet your exact needs.