Distributed data processing with Hadoop, part 2nd

The real strength of the Hadoop distributed Computing architecture is its distribution. In other words, the ability to distribute multiple nodes in parallel to work enables Hadoop to be applied to large infrastructure and to processing large amounts of data. In this paper, we first decompose a distributed Hadoop architecture and then discuss the distributed configuration and usage.

Distributed Hadoop Architecture

Based on distributed data processing with Hadoop, part 1th: Getting Started, all Hadoop daemons run on the same host. Despite the parallelism of Hadoop, this pseudo distributed configuration provides an easy way to test the functionality of Hadoop with minimal setup. Now, let's use the machine cluster to explore the parallelism of Hadoop.

According to the 1th part, the Hadoop configuration defines that all Hadoop daemons are run on one node. So let's first look at how Hadoop is naturally distributed to perform parallel operations. In a distributed Hadoop setting, you have a master node and some from nodes (see Figure 1).

Figure 1. The decomposition of Hadoop master-slave node

As shown in Figure 1, the master node includes the name node, the subordinate name node, and the Jobtracker daemon (the so-called Master daemon). In addition, this is the node that you use to manage the cluster for this demo (using the Hadoop utility and the browser). Includes Tasktracker and data nodes (subordinate daemon) from nodes. The difference between the two settings is that the master node includes daemons that provide the management and coordination of the Hadoop cluster, while the nodes include daemons that implement the Hadoop file system (HDFS) storage and MapReduce functions (data processing capabilities).

For this demo, create a master node and two from nodes on a single LAN. The settings are shown in Figure 2. Now, let's explore the installation and configuration of Hadoop for multi-node distribution.

Figure 2. Hadoop cluster configuration

To simplify deployment, the technology has several benefits, using virtualization technology. Although the use of virtualization technology in this setting does not see performance benefits, it can create a Hadoop installation and then clone the installation for other nodes. To do this, your Hadoop cluster should appear as follows: the master-slave node is run as a virtual machine (VM) in the context of a hypervisor on a host (see Figure 3).

Figure 3. Hadoop cluster configuration in virtual environments