Checksum of HDFS data

Datanode verifies the data checksum before actually storing the data.

The client writes data to datanode through pipeline. The last datanode checks the checksum.

When the client reads data from datanode, it also checks and compares the checksum of the actual data and the checksum on the datanode of the contract.

 

Each datanode stores a persistent log for checksum verification. The log contains the last update time of each block of the current datanode.

When the client successfully verifies a block, it will tell datanode, and then datanode will update its log.

Saving this information helps detect bad disks.

 

In addition to verifying the block when the client reads data, each datanode also runs a datablockdown thread in the background to periodically verify all

Block on datanode. This is used to prevent "bit attenuation" on the physical storage media ".

 

Because HDFS stores multiple copies of the same block, it can replace the damaged data copies with good copies.

If a client detects a data error when reading data, it reports information to namenode before throwing a checksumexception.

And datanode. namenode will mark the specified block as "upt", and the client will no longer locate this block, and this block will not

After being copied to another datanode, namenode schedules a normal copy of the block to ensure load balancing. After that, the damaged block copy will be deleted.

 

You can call setverifychecksum () to disable checksum detection before calling open () for filesystem.

You can also specify the-ignorecrc option when running the-Get and-copytolocal commands in shell.

 

 

Localfilesystem has been tested and generated on the client side. If you write a file filename to HDFS, the client will

In the same path as filename, a hidden file named. filename. CRC containing all block checksum information is created transparently. The chunk size of the checksum is specified through the bytes. Per. checksum attribute,

The default value is 512 bytes. The chunk size is also stored in the metadata of the. CRC file, so the file can be correctly parsed even if bytes. Per. checksum is changed later.

The Checksum check is performed when the file is read. If an error occurs, localfilesystem throws a checksumexception.

 

From http://www.cnblogs.com/xuxm2007/archive/2011/06/28/2091887.html