Hadoop hdfs cannot be restarted after the space is full. hadoophdfs

Hadoop hdfs cannot be restarted after the space is full. hadoophdfs

When the server checks, it finds that files on HDFS cannot be synchronized and hadoop is stopped. Restart failed.

View hadoop logs:

2014-07-30 14:15:42,025 INFO org.apache.hadoop.hdfs.server.namenode.FSNamesystem: Roll Edit Log from 192.168.206.133<span style="color:#ff0000;">2014-07-30 14:15:42,026 WARN org.apache.hadoop.hdfs.server.namenode.FSNamesystem: Cannot roll edit log, edits.new files already exists in all healthy directories:  /home/hadoop/hdfs/name/current/edits.new</span><span style="color:#ff0000;">2014-07-30 14:17:17,853 ERROR org.apache.hadoop.hdfs.server.namenode.FSNamesystem: Unable to sync edit log.java.io.IOException: No space left on device</span>at sun.nio.ch.FileChannelImpl.force0(Native Method)at sun.nio.ch.FileChannelImpl.force(FileChannelImpl.java:348)at org.apache.hadoop.hdfs.server.namenode.FSEditLog$EditLogFileOutputStream.flushAndSync(FSEditLog.java:215)at org.apache.hadoop.hdfs.server.namenode.EditLogOutputStream.flush(EditLogOutputStream.java:89)at org.apache.hadoop.hdfs.server.namenode.FSEditLog.logSync(FSEditLog.java:1017)at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.startFile(FSNamesystem.java:1190)at org.apache.hadoop.hdfs.server.namenode.NameNode.create(NameNode.java:628)at sun.reflect.GeneratedMethodAccessor300.invoke(Unknown Source)at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)at java.lang.reflect.Method.invoke(Method.java:597)at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:563)at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:1388)at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:1384)at java.security.AccessController.doPrivileged(Native Method)at javax.security.auth.Subject.doAs(Subject.java:396)at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1121)at org.apache.hadoop.ipc.Server$Handler.run(Server.java:1382)2014-07-30 14:17:17,853 FATAL org.apache.hadoop.hdfs.server.namenode.FSNamesystem: No edit streams are accessiblejava.lang.Exception: No edit streams are accessibleat org.apache.hadoop.hdfs.server.namenode.FSEditLog.fatalExit(FSEditLog.java:388)at org.apache.hadoop.hdfs.server.namenode.FSEditLog.exitIfNoStreams(FSEditLog.java:407)at org.apache.hadoop.hdfs.server.namenode.FSEditLog.removeEditsAndStorageDir(FSEditLog.java:432)at org.apache.hadoop.hdfs.server.namenode.FSEditLog.removeEditsStreamsAndStorageDirs(FSEditLog.java:470)at org.apache.hadoop.hdfs.server.namenode.FSEditLog.logSync(FSEditLog.java:1030)at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.startFile(FSNamesystem.java:1190)at org.apache.hadoop.hdfs.server.namenode.NameNode.create(NameNode.java:628)at sun.reflect.GeneratedMethodAccessor300.invoke(Unknown Source)at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)at java.lang.reflect.Method.invoke(Method.java:597)at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:563)at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:1388)at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:1384)at java.security.AccessController.doPrivileged(Native Method)at javax.security.auth.Subject.doAs(Subject.java:396)at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1121)at org.apache.hadoop.ipc.Server$Handler.run(Server.java:1382)

The space is full.

Run the command df-h to view one of the datanode.

However, you can use du-s to view the data, but only a few hundred Gbit/s is used.

This is indeed confusing, so we have to restart the server where datanode is located and re-mount it. The system returns as is and the capacity is displayed normally.

However, at this time, the hadoop cluster still cannot be restarted and the following error is reported:

014-07-30 16:27:46,610 ERROR org.apache.hadoop.hdfs.server.namenode.FSNamesystem: FSNamesystem initialization failed.java.io.IOException: Incorrect data format. logVersion is -32 but writables.length is 0. at org.apache.hadoop.hdfs.server.namenode.FSEditLog.loadFSEdits(FSEditLog.java:560)at org.apache.hadoop.hdfs.server.namenode.FSImage.loadFSEdits(FSImage.java:1026)at org.apache.hadoop.hdfs.server.namenode.FSImage.loadFSImage(FSImage.java:839)at org.apache.hadoop.hdfs.server.namenode.FSImage.recoverTransitionRead(FSImage.java:377)at org.apache.hadoop.hdfs.server.namenode.FSDirectory.loadFSImage(FSDirectory.java:100)at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.initialize(FSNamesystem.java:388)at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.<init>(FSNamesystem.java:362)at org.apache.hadoop.hdfs.server.namenode.NameNode.initialize(NameNode.java:276)at org.apache.hadoop.hdfs.server.namenode.NameNode.<init>(NameNode.java:496)at org.apache.hadoop.hdfs.server.namenode.NameNode.createNameNode(NameNode.java:1279)at org.apache.hadoop.hdfs.server.namenode.NameNode.main(NameNode.java:1288)2014-07-30 16:27:46,612 ERROR org.apache.hadoop.hdfs.server.namenode.NameNode: java.io.IOException: Incorrect data format. logVersion is -32 but writables.length is 0. at org.apache.hadoop.hdfs.server.namenode.FSEditLog.loadFSEdits(FSEditLog.java:560)at org.apache.hadoop.hdfs.server.namenode.FSImage.loadFSEdits(FSImage.java:1026)at org.apache.hadoop.hdfs.server.namenode.FSImage.loadFSImage(FSImage.java:839)at org.apache.hadoop.hdfs.server.namenode.FSImage.recoverTransitionRead(FSImage.java:377)at org.apache.hadoop.hdfs.server.namenode.FSDirectory.loadFSImage(FSDirectory.java:100)at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.initialize(FSNamesystem.java:388)at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.<init>(FSNamesystem.java:362)at org.apache.hadoop.hdfs.server.namenode.NameNode.initialize(NameNode.java:276)at org.apache.hadoop.hdfs.server.namenode.NameNode.<init>(NameNode.java:496)at org.apache.hadoop.hdfs.server.namenode.NameNode.createNameNode(NameNode.java:1279)at org.apache.hadoop.hdfs.server.namenode.NameNode.main(NameNode.java:1288)

No solution, so we have to try to recover from secondrynamenode

1. Back up namenode and secondrynamenode data first

2. Run the checkpoint data: hadoop namenode-importCheckpoint

Failed ......

Finally, I checked the method on the Internet (one is to modify the code and re-compile the code ):
Printf "\ xff \ xee \ xff"> edits

Failed to restart. Continue:
Printf "\ xff \ xee \ xff"> edits. new

Reboot successful.

However, the problem is that the data is lost because the latest log Content of the edits file is cleared.

In general, there is no way to do this.

Fortunately, we have two locations for synchronization and backup. We have synchronized the data in the HDFS cluster of another data center and recovered successfully.

If there is no remote backup, I will be cautious about the above operations.

Hadoop HDFS provides an interface for viewing the total size and remaining size of space.

Hadoop 1.2.x
Check src/webapps/hdfs/dfshealth. jsp, which corresponds to the page shown in 50070.
For example, Configured Capacity corresponds
Org. apache. hadoop. hdfs. server. namenode. FSNamesystem. getCapacityTotal ()

You can find all the remaining items in the same way.

Hadoop installation problem: the HDFS file system cannot be formatted

1. The hadoop command is not found because you have not added the hadoop bin directory to the PATH (only programs added to the PATH environment variable can directly access it ).
You can
A) cd $ HADOOP_HOME (IF configured) then run bin/hadoop.
B) simply use the full path
C) add the bin directory of hadoop to PATH.
2. jps is a java tool. Enter jps to display 8813 Jps, indicating that no java program is running.
To install hadoop, you only need to decompress the package. You must at least configure JAVA_HOME in the hadoop-env.xml file in the conf directory to run.