The authoritative guide to Hadoop (fourth edition) highlights translations (4)--chapter 3. The HDFS (1-4)

Filesystems that manage the storage across a network of machines is called distributed filesystems. Since They is network based, all the complications of the network programming kick in, thus making distributed filesystems mo Re complex than regular disk filesystems.
A file system stored across multiple computers in a management network is called a distributed file system. Because it is based on the network, it introduces the complexity of network programming, so the Distributed file system is more complicated than the ordinary disk file system.
1) The Design of HDFS
A) HDFS is a filesystem designed for storing very large files with streaming data access patterns, running on clusters of Commodity hardware.
HDFs is designed to store very large files in streaming data access mode, running on a commercial hardware cluster.
b) HDFS is built around the idea, the most efficient data processing pattern is a writeonce,read-many-times pattern. A dataset is typically generated or copied from Source,and then various analyses be performed on this dataset over time. Each analysis would involve a large proportion, if not all, the of the datasets, so the time to read the whole dataset was more I Mportant than the latency in reading the first record.
HDFs is built on the idea of a single write, multiple reads of such a most efficient data processing mode. Datasets typically have a data source generated or copied from a data source, followed by lengthy data analysis operations on this dataset. Each analysis involves a large part of the data, even the entire data set, so it is more important to read the entire data set than to read the first record's delay.
c) Hadoop doesn ' t require expensive, highly reliable hardware. It's designed to run on clusters of commodity hardware (commonly available hardware so can be obtained from multiple ven Dors) for which the chance of node failure across the cluster are high, at least for large clusters. HDFS is designed to carry on working without a noticeable interruption to the user in the face of such failure.
Hadoop does not require expensive, high-reliability hardware, Hadoop runs on a commercial hardware cluster (common hardware can be obtained from a variety of vendors), so the chances of failure for the entire cluster node are high, at least for large clusters.
2) HDFS Concepts
A) a disk has a block size, which are the minimum amount of data that it can read or write. Filesystems for a single disk build on the "dealing with data" in blocks, which is an integral multiple of the disk Blo CK size. Filesystem blocks is typically a few kilobytes in size, whereas disk blocks is normally bytes.
Each disk has a default block size, which is the minimum amount of data that the disk can read and write, and a single disk file management system is built on top of the processing disk block data, which is an integer multiple of the disk block size. The block size of the disk file system is typically several kilobytes, however the disk block size is typically 512 bytes.
b) Unlike a filesystem for a single disk, a file in HDFS which is smaller than a single block does not occupy a full block ' s worth of underlying storage.
Unlike a single disk file system, files that are smaller than disk blocks in HDFs are not covered with potential storage space for the entire block.
c) HDFS blocks is large compared to disk blocks, and the reason was to minimize the cost of seeks. IF The block is large enough, the time it takes to transfer the data from the disk can be significantly longer than the TI Me to seek to the start of the block. Thus, transferring a large file made of multiple blocks operates at the disk transfer rate.
The block of HDFs is larger than the disk block, which minimizes the search cost. If the block is large enough, the time it takes to transfer the data from the disk is much longer than the start of the search block, so a large file consisting of multiple blocks depends on the disk transfer rate
D) have a block abstraction for a distributed filesystem brings several benefits. The first benefit is the most obvious:a file can being larger than any single disk in the network. Second, making the unit of abstraction a block rather than a file simplifies the storage subsystem. Furthermore, blocks fit well with replication for providing fault tolerance and availability.
Abstracting the blocks of a distributed file system can bring several benefits. First and foremost, the size of a file can be larger than the capacity of any disk in the network. Second, the storage subsystem can be simplified by using abstract blocks rather than the entire file. Finally, abstract blocks are well suited for backup, which can improve fault tolerance and usability.
e) An HDFS cluster have types of nodes operating in a master?worker pattern:a Namenode (the master) and a number of DA Tanodes (workers). The Namenode manages the filesystem namespace.
The HDFs cluster working in host-slave mode has two types of nodes that can operate: one Namenode (on the host) and several Datanode (from the machine). Namenode manages the namespace of the entire file system.
f) Datanodes is the workhorses of the filesystem. They store and retrieve blocks when they is told to (by clients or the Namenode), and they report back to the Namenode PE Riodically with lists of blocks that they is storing.
Datanode is the direct working point of the file system. They store and retrieve blocks of data (subject to client or Namenode notifications) and periodically report to namenode the block list information they store.
g) For this reason, it's important to make the namenode resilient to failure, and Hadoop provides both mechanisms for this . The first and the "to" back "the Files" and "the" persistent state of the filesystem metadata.
For this reason, it is important to ensure that the namenode is resilient to failure, and Hadoop provides two mechanisms. The first mechanism is to back up files that are made up of file system metadata persistence states.
h) It is also possible to run a secondary namenode, which despite its name does not act as a namenode. Its main role is-periodically merge the namespace image with the edit log to prevent the edit log from becoming too lar Ge.
Another possible approach is to run an auxiliary namenode, although it will not be used as a namenode. Its primary role is to prevent the editable log from being too large by fusing namespace mirrors periodically through editable logs.
i) However, the state of the secondary namenode lags that of the primary, so in the event of total failure of the primary, Data loss is almost certain. The usual course of action in the "is" to copy the Namenode's metadata files that's on NFS to the secondary and run It as the new primary.
Then, the state in the secondary namenode always lags behind the primary node, so the data loss is almost certain in the overall failure event of the master node. In such cases, it is common practice to copy the metadata files stored on NFS to the secondary Namenode and run as a new primary node, Namenode.
j) Normally a Datanode reads blocks from disk, but for frequently accessed files The blocks is explicitly cached in th E Datanode ' s memory, in an off-heap block cache.
Typically a node reads block data from disk, but for frequently accessed files, its block data may be cached in the node's memory, a non-heap block cache.
k) HDFS federation,introduced in the 2.x release series, allows a cluster to scale by adding namenodes, each of which mana GES a portion of the filesystem namespace.
The Federation in HDFs, is in 2. Introduced in the X-series, it allows a cluster to be extended by adding namenode nodes, each of which manages a portion of the file System namespace in the Namenode node.
L) Hadoop 2 remedied this situation by adding support for HDFS high Availability (HA). In this implementation, there is a pair of namenodes in an Active-standby configuration. In the event of the failure of the active Namenode, the standby takes over it duties to continue servicing client request s without a significant interruption. A few architectural changes is needed to allow this to happen:
Hadoop 2 corrects this by increasing support for HA, and in this implementation, there will be 2 Namenode for two-machine hot backup. In the event of a primary active node failure, the backup master node can take over the responsibility of continuing to respond to client requests without significant disruption. The following structural changes are allowed to occur:
m) The Namenodes must use highly available GKFX storage to share the edit log. Datanodes must send block reports to both Namenodes because the block mappings is stored in a Namenode ' s memory, and not On disk. Clients must be configured to handle Namenode failover, using a mechanism that's transparent to users. The secondary Namenode ' s role is subsumed by the standby, which takes periodic checkpoints of the active Namenode ' s NAMESP Ace.
Namenode must use high-availability shared storage to enable the sharing of editable logs, because the block mapping information is stored in Namenode memory instead of on disk, so Datanode must send block information to the namenode of a dual-machine hot backup, The client must be configured for failover operations, which can be achieved through a mechanism that is transparent to the user. The role of the secondary node is included in the backup, which contains periodic checkpoint information for the active master node namespace.
N) There is choices for the highly available GKFX Storage:an NFS filer, or a quorum journal manager (QJM). The QJM is a dedicated HDFS implementation, designed for the sole purpose of providing a highly available edit log, and is The recommended choice for most HDFS installations.
There are two options for high-availability shared storage: NFS files, QJM (quorum journal manager). QJM focuses on the implementation of HDFs, and its sole purpose is to provide a highly practical editable log, which is also recommended for most HDFS installations.
O) If The active Namenode fails, the standby can take over very quickly (in a few tens of seconds) because it has the late St state available in Memory:both the latest edit log entries and an up-to-date block mapping.
If the active Namenode fails, the backup node quickly takes over the task (in seconds) because the in-memory backup node has the most recent available state, including the most recent editable logging and block mapping information.
p) The transition from the active Namenode to the standby are managed by a new entity in the system called the Failover con Troller. There is various failover controllers, but the default implementation uses ZooKeeper to ensure that's only one namenode is Active. Each namenode runs a lightweight failover controller process whose job it's to monitor its namenode for failures and trig Ger a failover should a namenode fail.
Failover from the active master node to the backup node is managed by a new entity-failover controller in the system. Although there are multiple versions of the failover controller, Hadoop defaults to zookeeper, which also ensures that only one namenode is active. Each Namenode node runs a lightweight failover controller process, which is tasked with monitoring namenode failures and triggering a failover once the namenode fails.
Q) The HA implementation goes to great lengths to ensure, the previously active namenode is prevented from doing any D Amage and causing corruption-a method known as fencing.
The HA implementation will make every effort to ensure that the previous active primary node does not make any harmful actions that cause a failure, which is fencing.
R) The QJM only allows one namenode-to-write to the edit-log at one time; However, it's still possible for the previously active namenode to serve stale read requests to clients SSH fencing command that would kill the Namenode ' s process is a good idea.
QJM only allows a namenode to write to an editable log at the same time; however, it is possible to respond to a stale read request service from the client for the previously active node, so it is a good way to establish a fencing command that can kill the namenode process.
3) The command-line Interface
A) You can type Hadoop fs-help to get detailed help on every command.
You can use Hadoop fs–help on every command to get detailed help.
b) Let's copy the file back to the local filesystem and check whether it ' s the same:

% hadoop fs -copyToLocal quangle.txt quangle.copy.txt% md5 input/docs/quangle.txt quangle.copy.txtMD5 (input/docs/quangle.txt) = e7891a2627cf263a079fb0f18256ffb2MD5 (quangle.copy.txt) = e7891a2627cf263a079fb0f18256ffb2

The MD5 digests is the same, showing that the file survived its-trip-to-HDFS and is-back intact.
Let's copy this file to local and check if they are the same file. The MD5 is the same, indicating that the file was transferred to HDFs and returned intact.
4) Hadoop filesystems
A) Hadoop has an abstract notion of filesystems, of which HDFS is just one implementation. The Java abstract class Org.apache.hadoop.fs.FileSystem represents the client interface to a FileSystem in Hadoop, and the Re is several concrete implementations.
Hadoop has an abstract concept for file systems, and HDFs is just one of those implementations. Java's abstract class Org.apache.hadoop.fs.FileSystem defines the interface between the client and the file system, and there are several specific implementations of the abstract class.
b) Hadoop is written in Java, so most of Hadoop filesystem interactions is mediated through the Java API.
Hadoop is written in Java, so the interaction of most Hadoop file systems is done through the Java API.
c) by exposing their filesystem interface as a Java API, Hadoop makes it awkward for Non-java applications to access HDFS. The HTTP REST API exposed by the WEBHDFS protocol makes it easier for other languages to interact with HDFS. Note that the HTTP interface are slower than the native Java client, so should being avoided for very large data transfers if Possible.
The use of the FileSystem interface as a Java API makes it cumbersome for non-Java applications to access HDFs. The HTTP REST API, implemented through the WEBHDFS protocol, makes it easy for other languages to interact with HDFs. Note that the HTTP interface is slower than the local Java client, so it is possible to avoid using the interface for large file data transfers.

The authoritative guide to Hadoop (fourth edition) highlights translations (4)--chapter 3. The HDFS (1-4)