Possible errors during the running of nutch1.2 and Solutions

Error 1. Caused by the maximum number of files allowed to be opened in Linux

Error message:
Java. io. ioexception: Background merge hit exception: _ 0: c500-> _ 0 _ 1: c500-> _ 0 _ 2: c500-> _..... [optimize]
At org. Apache. Lucene. Index. indexwriter. Optimize (indexwriter. Java: 2310)
At org. Apache. Lucene. Index. indexwriter. Optimize (indexwriter. Java: 2249)
At org. Apache. Lucene. Index. indexwriter. Optimize (indexwriter. Java: 2219)
At org. Apache. nutch. indexer. Lucene. javasewriter. Close (javasewriter. Java: 237)
At org. Apache. nutch. indexer. indexeroutputformat $1. Close (indexeroutputformat. Java: 48)
At org. Apache. hadoop. mapred. reducetask. runoldreducer (reducetask. Java: 474)
At org. Apache. hadoop. mapred. reducetask. Run (reducetask. Java: 411)
At org. Apache. hadoop. mapred. Child. Main (child. Java: 170)
Caused by: Java. Io. filenotfoundexception:/var/lib/crawlzilla/nutch-crawler/mapred/local/index/_ 682243155/_ 6a. frq (too submit open files)
At java. Io. randomaccessfile. Open (native method)
At java. Io. randomaccessfile. (randomaccessfile. Java: 212)
At org. Apache. Lucene. Store. simplefsdirectory $ simplefsindexinput $ descriptor. (simplefsdirectory. Java: 76)
At org. Apache. Lucene. Store. simplefsdirectory $ simplefsindexinput. (simplefsdirectory. Java: 97)
At org. Apache. Lucene. Store. niofsdirectory $ niofsindexinput. (niofsdirectory. Java: 87)
At org. Apache. Lucene. Store. niofsdirectory. openinput (niofsdirectory. Java: 67)
At org. Apache. Lucene. Index. segmentreader $ corereaders. (segmentreader. Java: 129)
At org. Apache. Lucene. Index. segmentreader. Get (segmentreader. Java: 576)
At org. Apache. Lucene. Index. indexwriter $ readerpool. Get (indexwriter. Java: 609)
At org. Apache. Lucene. Index. indexwriter. mergemiddle (indexwriter. Java: 4239)
At org. Apache. Lucene. Index. indexwriter. Merge (indexwriter. Java: 3917)
At org. Apache. Lucene. Index. concurrentmergeschedge. domerge (concurrentmergeschedge. Java: 231)
At org. Apache. Lucene. Index. concurrentmergeschedread $ mergethread. Run (concurrentmergeschedread. Java: 288)

 

Cause and solution:
The Java program runs in a Unix/Linux environment and requires a lot of operations on the file.
There are restrictions on file handles in Unix/Linux environments. You can use ulimit-N to view the number of file handles allowed to be opened in the current environment. The default value is 1024. However, when our Java program is concurrent with or out of 1024 and frequently reads and writes files

The preceding exception is solved by increasing the limit on file handles as needed.
Command:
Ulimit-N 32768

Error 2. Insufficient disk space

Error message:
Error: Java. Io. ioexception: no space left on Device
At java. Io. fileoutputstream. writebytes (native method)
At java. Io. fileoutputstream. Write (fileoutputstream. Java: 260)
At org. Apache. hadoop. fs. rawlocalfilesystem $ localfsfileoutputstream. Write (rawlocalfilesystem. Java: 190)
At java. Io. bufferedoutputstream. Write (bufferedoutputstream. Java: 105)
At org. Apache. hadoop. fs. fsdataoutputstream $ positioncache. Write (fsdataoutputstream. Java: 49)
At java. Io. dataoutputstream. Write (dataoutputstream. Java: 90)
At org. Apache. hadoop. mapred. ifileoutputstream. Write (ifileoutputstream. Java: 84)
At org. Apache. hadoop. fs. fsdataoutputstream $ positioncache. Write (fsdataoutputstream. Java: 49)
At java. Io. dataoutputstream. Write (dataoutputstream. Java: 90)
At org. Apache. hadoop. mapred. ifile $ writer. append (ifile. Java: 218)
At org. Apache. hadoop. mapred. Merger. writefile (merger. Java: 157)
At org. Apache. hadoop. mapred. reducetask $ reducecopier $ localfsmerger. Run (reducetask. Java: 2454)

 

Cause and solution:
When the hard disk is insufficient, nutch will wait for available space. If it is distributed, a running node can be added.

 

Error 3. HDFS cannot be started due to namenode ID conflict

Error message:
10:12:11, 987 error org. Apache. hadoop. HDFS. server. datanode. datanode: Java. Io. ioexception: incompatible namespaceids in

/Home/admin/Joe. wangh/hadoop/data/dfs. Data. dir: namenode namespaceid = 898136669; datanode namespaceid = 2127444065

At org. Apache. hadoop. HDFS. server. datanode. datastorage. dotransition (datastorage. Java: 233)
At org. Apache. hadoop. HDFS. server. datanode. datastorage. recovertransitionread (datastorage. Java: 148)
At org. Apache. hadoop. HDFS. server. datanode. datanode. startdatanode (datanode. Java: 288)
At org. Apache. hadoop. HDFS. server. datanode. datanode. <init> (datanode. Java: 206)
At org. Apache. hadoop. HDFS. server. datanode. datanode. makeinstance (datanode. Java: 1239)
At org. Apache. hadoop. HDFS. server. datanode. datanode. instantiatedatanode (datanode. Java: 1194)
At org. Apache. hadoop. HDFS. server. datanode. datanode. createdatanode (datanode. Java: 1202)
At org. Apache. hadoop. HDFS. server. datanode. datanode. Main (datanode. Java: 1324)
......

 

Cause and solution:
Each time namenode format creates a new namenodeid, while tmp/dfs/data contains the ID of the last format, namenode format clears the data of namenode, but there is no clear data under datanode, resulting in failure at startup, all you need to do is

Before each fotmat, clear all the TMP directories.
Solution 1:
Stop the hadoop cluster and delete the local datanode data folder. The folder location is configured in CONF/hdfs-site.xml and put in/usr/local/hadoop-datastore/hadoop-hadoop by default (this is the login of the machine

Name)/dfs/data, but my default is/tmp/hadoop-Dev (My hadoop startup username)/dfs/Data
Solution 2 (recommended ):
Stop the datanode node, find <DFS. Data. dir>/current/version, and change the namenode ID in it to the one shown in the error message. In my example, It is 898136669.

 

Error 4. configuration file Error

Error message:
[Fatal error] hadoop-site.xml: 15: 7: The content of elements must consist of well

-Formed character data or markup.

Exception in thread "Main" Java. Lang. runtimeexception: org. xml. Sax. saxparseexception:

The content of elements must consist of well-formed character data or mark up.

Cause and Solution
One of the tags in the configuration file such as nutch-site.xml </property> is preceded by an angle bracket.