Differences between indexes and indexes folders in the crawl result folder

 

1,Nutch-In 1.x, it is in crawl. java. Finally, there is such a sentence.

?IndexEr.Index(IndexEs,
Crawldb, linkdb, arrays. aslist (hadoopfsutil. getpaths (fstats )));

It indexes crawldb, linkdb, and segments. Generate a fileIndexEs, which contains part-00000,... depends on your own depth.

2,

Dedup. dedup (new path [] {IndexEs });
Fstats = FS. liststatus (IndexEs, hadoopfsutil. getpassdirectoriesfilter (FS ));
Merger. Merge (hadoopfsutil. getpaths (fstats ),Index,
Tmpdir );

Mainly forIndexMerge the es indexes (MD5 ). Delete duplicate indexes. You can modify your business as needed.

2. The distributed index can also be specified at the specified time.IndexElasticsearchIndexPart. You should know the difference between the two!