The consistency of HDFs

　　The file system Consistency model describes the visibility of file read/write. HDFs sacrifices some POSIX requirements to compensate for performance, so some operations may be different from traditional file systems.

When you create a file, it is visible in the namespace of the file system and the code is as follows:

Path p = new Path ("P");

Fs.create (P);

Assertthat (Fs.exists (P), is (true));

However, any write operation to this file is not guaranteed to be visible, even if the data stream has been refreshed, the length of the file will be displayed as 0:

Path p = new Path ("P");
OutputStream out = Fs.create (p);

Out.write ("Content". GetBytes ("UTF-8"));

Out.flush ();

Assertthat (Fs.getfilestatus (p), Getlen (), is (0L));

Once a block of data is written, the new request will be visible to the block, and the block that is currently written is invisible to everyone. HDFs provides a way to force the synchronization of data between all caches and Datanode, this method

is the sync () function in Fsdataoutputstream. When the sync () function returns successfully, HDFS guarantees that the file data written at this time is consistent and visible to all new users. Even if there is a conflict between the HDFS clients, the data is lost and the code is as follows:
Path p = new Path ("P");
Fsdataoutputstream out = Fs.create (p);
Out.write ("Content". GetBytes ("UTF-8"));

Out.flush ();
Out.sync ();

Assertthat (Fs.getfilestatus (p), Getlen (), is ((long) content. Length ()));

This operation is similar to the Fsync system call in a UNIX system, committing cached data for a file descriptor, writing local data using the Java API, so that you can see the data that refreshes the stream and then synchronizes, with the following code:

FileOutputStream out = new Fileoutstream (localfile);

Out.write ("Content". GetBytes ("UTF-8"));

Out.flush (); Flush to Operatig system

OUT.GETFD (). sync (); Sync to disk

Assertthat (Fs.getfilestatus (p), Getlen (), is ((long) content. Length ()));

Closing a file in HDFs also implicitly executes the sync () function, with the following code:

Path p = new Path ("P");
OutputStream out = Fs.create (p);

Out.write ("Content". GetBytes ("UTF-8"));

Out.close ();

Assertthat (Fs.getfilestatus (p), Getlen (), is ((long) content. Length ()));

Here's how the consistency model is important to your application design. File system consistency is related to the way the application is designed. If you do not call sync (), you need to be prepared to lose some of the data due to a client or system failure. For most applications, this is unacceptable, so you need to call Sync () in the right place, for example, after writing a certain amount of data. Although sync () is designed to minimize the burden of HDFS, it still has negligible overhead, so there is a tradeoff between data robustness and throughput, and a good point of reference is to test the application to choose the best balance between the different sync () frequencies.

The consistency of HDFs