Inject類下的InjectMapper中有一句:
try {
scfilters.injectedScore(value, datum);
} catch (ScoringFilterException e) {
if (LOG.isWarnEnabled()) {
LOG.warn("Cannot filter injected score for url " + url
+ ", using default (" + e.getMessage() + ")");
}
}
裡面調用的是ScoringFilters類的執行個體裡面的方法:public void injectedScore(Text url, CrawlDatum datum) throws ScoringFilterException
作用是:通過分數作為度量,計算(初始化)每個通過substitution和filter後的種子url分數。
接下來,說說ScoringFilters類,首先它實現了ScoringFilter介面,並且裡面有一個private ScoringFilter[] filters;變數
再看看public void injectedScore(Text url, CrawlDatum datum) throws ScoringFilterException方法的原始碼:
/** Calculate a new initial score, used when injecting new pages. */
public void injectedScore(Text url, CrawlDatum datum) throws ScoringFilterException {
for (int i = 0; i < this.filters.length; i++) {
this.filters[i].injectedScore(url, datum);
}
}
可見ScoringFilters類實際上的injectedScore的工作是通過調用private ScoringFilter[] filters;變數中的每個filter的injectedScore()方法來完成的。
所以,很有必要知道怎麼樣去指定private ScoringFilter[] filters中的內容的。
留意到ScoringFilters類的public
ScoringFilters(Configuration conf) 建構函式。
下面是public
ScoringFilters(Configuration conf) 的原始碼:(注意為了實驗,作者增加了一些測試代碼)
public
ScoringFilters(Configuration conf) {
super(conf);
ObjectCache objectCache = ObjectCache.get(conf);
//讀取conf檔案(conf/nutch-default.xml或者conf/nutch-site.xml)下的設定檔選項(scoring.filter.order)擷取次序
String order = conf.get("scoring.filter.order");
//查看緩衝中的內容,看是否存在filters(已經包含了次序的);比如,第二次使用時候,就有用了
this.filters = (ScoringFilter[]) objectCache.getObject(ScoringFilter.class.getName());
if (this.filters == null) {//第一次啟用ScoringFilters類
//通過conf檔案(conf/nutch-default.xml或者conf/nutch-site.xml)下的設定檔選項(scoring.filter.order)擷取次序
// String[]
是一個數組,使用次序的;
String[] orderedFilters = null;
if (order != null && !order.trim().equals("")) {
orderedFilters = order.split("\\s+");
}
try {
//test by kaiwii
System.out.println("try block");
ExtensionPoint point = PluginRepository.get(conf).getExtensionPoint(ScoringFilter.X_POINT_ID);
if (point == null) throw new RuntimeException(ScoringFilter.X_POINT_ID + " not found.");
Extension[] extensions = point.getExtensions();
HashMap<String, ScoringFilter> filterMap =
new HashMap<String, ScoringFilter>();
for (int i = 0; i < extensions.length; i++) {//通過ScoringFilter.X_POINT_ID擷取擴充點,從而擷取所有的ScoringFilter的實作類別
//ScoringFilter的實作類別與擴充點之間的對應關係,通過遍曆外掛程式中的plugin.xml中獲得(詳情查看PluginRepository類的實現)
Extension extension = extensions[i];
//for test by kaiwii
System.out.println("extension_id "+i+extension.getId());
System.out.println("extension_clazz "+i+extension.getClazz());
//因為外掛程式的實現採用了lazy_load的方式,所以上面擷取extension時,只是擷取了一個plugindescriptor(包含的僅是某個plugin的資訊而已)
//這裡才真正進行執行個體化
ScoringFilter filter = (ScoringFilter) extension.getExtensionInstance();
if (!filterMap.containsKey(filter.getClass().getName())) {//用一個map來組織起所有filter的執行個體
filterMap.put(filter.getClass().getName(), filter);
}
}
if (orderedFilters == null) {//當conf檔案(conf/nutch-default.xml或者conf/nutch-site.xml)下的設定檔選項(scoring.filter.order)為空白的時候
objectCache.setObject(ScoringFilter.class.getName(), filterMap.values().toArray(new ScoringFilter[0]));
} else {
ScoringFilter[] filter = new ScoringFilter[orderedFilters.length];
for (int i = 0; i < orderedFilters.length; i++) {//按照orderedFilters指定的順序,把filter放進一個臨時變數中
filter[i] = filterMap.get(orderedFilters[i]);
}
objectCache.setObject(ScoringFilter.class.getName(), filter);
}
} catch (PluginRuntimeException e) {
throw new RuntimeException(e);
}
//實際就是上面filter的值而言
this.filters = (ScoringFilter[]) objectCache.getObject(ScoringFilter.class.getName());
//for test by kaiwii
System.out.println("filters content:kaiwii want to know:");
for(ScoringFilter v:this.filters){
System.out.println("filter:"+v.toString());
}
}
}
到此,至於怎麼樣的ScoringFilter實作類別會放到裡面去,應該大概有個明白了吧?
其實,還要留意一下conf檔案(conf/nutch-default.xml或者conf/nutch-site.xml)下的設定檔選項(scoring.filter.order):
<!--
scoring filters properties -->
<property>
<name>scoring.filter.order</name>
<value></value>
<description>The order in which scoring filters are applied.
This may be left empty (in which case all available scoring
filters will be applied in the order defined in plugin-includes
and plugin-excludes), or a space separated list of implementation
classes.
</description>
</property>
通過上面的英文介紹,你應該可以知道為空白的時候,怎麼樣的ScoringFilter實作類別會被調用,次序怎麼樣,就是 plugin.includes和plugin-excludes說了算。
就以預設的設定來看看吧,首先,scoring.filter.order為空白;
然後 plugin.includes設定為:(相關內容高亮出來了!)
<property>
<name>plugin.includes</name>
<value>protocol-http|urlfilter-regex|parse-(text|html|js|tika)|index-(basic|anchor)|query-(basic|site|url)|response-(json|xml)|summary-basic|scoring-opic|urlnormalizer-(pass|regex|basic)</value>
最後,plugin.excludes為空白。
結合修改後的測試代碼,看看run
inject類的console怎麼樣:(相關內容高亮出來!)
Injector:
starting at 2011-09-04 09:11:13
Injector: crawlDb: crawldb
Injector: urlDir: urls.txt
Injector: Converting injected urls to crawl db entries.
try block
extension_id 0org.apache.nutch.scoring.opic.OPICScoringFilter
extension_clazz 0org.apache.nutch.scoring.opic.OPICScoringFilter
filters content:kaiwii want to know:
filter:org.apache.nutch.scoring.opic.OPICScoringFilter@12a3793
using method:regexNormalize()
kaiwii want to know confiFilenull
use global rules
Injector: Merging injected urls into crawl db.
Injector: finished at 2011-09-04 09:11:19, elapsed: 00:00:05
通過,上面的console顯示得知,只有org.apache.nutch.scoring.opic.OPICScoringFilter被調用了。
所以,要使用相關的ScoringFilter實作類別,就要按照下面的操作進行了:
The order in which scoring filters are applied.
This may be left empty (in which case all available scoring
filters will be applied in the order defined in plugin-includes
and plugin-excludes), or a space separated list of implementation
classes.
總結:熟悉外掛程式機制!!!!!!!!!!!!!!!!