Some views on the Nutch2.1 abstract storage Layer-

Nutch2.1 extends the storage layer through Gora, supporting Http://www.aliyun.com/zixun/aggregation/13713.html ">hbase, Accumulo, Cassandra, MySQL, Datafileavrostore, Avrostore and other storage methods. In my repeated tests found that Nutch2.1 than 1.6 of the performance is much worse, the most important thing is not long-term stable operation. Here are a few different ways to store each:

HBase, the input segmentation is supported, and the region is the smallest partition unit. With the increase of data scale, the advantage of parallel processing is embodied, so it is suitable for large data application. However, the maintenance of hbase cluster is a big problem, more complex than HDFs, memory consumption is also very scary.

Accumulo crawl after 3 rounds of abnormal exit, prompted Unsupportedoperationexception.

Cassandra, it should be noted that hosts localhost cannot be mapped to 127.0.0.1. The biggest problem with Cassandra is that it does not support input segmentation, even if the data scale is large and only one map, completely lose the parallelism.

MySQL, only one server as a data source, then as the size of the data, how to face MySQL? So MySQL for small-scale simple vertical search and other applications are more appropriate.

Datafileavrostore,injection job throws NullPointerException, see https://issues.apache.org/jira/browse/NUTCH-1477.

Avrostore, and Datafileavrostore the same problem.

From the above analysis, the current Gora also need to improve. For the pursuit of the ultimate performance of friends, nutch2.1 is not stable, the proposed use of nutch1.6, using HDFS and MapReduce data localization and natural parallelism, can be optimized to very fast speed.