Resource extension technology for Hadoop virtualization extension (hve)

Background

In virtualized cloud environments, Hadoop can have better "resiliency", which is an important advantage of cloud computing, such as Amazon's EMR (elasticmapreduce) service, where users can quickly deploy a Hadoop cluster in the cloud to run computing tasks, And users can dynamically add or remove compute nodes to the cluster.

There is a potential problem, HAODOP data nodes are not inherently "resilient", and deleting a data node means replicating the data stored in this node, which is a slow process and takes up network bandwidth.

Figure 1: Storage calculation separation

Storage compute separation can solve this problem very well, as shown in the figure above, we can statically deploy a data node for each physical node, dynamically deploy and adjust the compute nodes on each of the physical nodes. In addition, virtual machines can be used for better isolation, providing a separate compute node for each tenant and providing multiple tenant (multi-tenancy) services.

HVE's resource expansion technology is the expansion of the Hadoop computing layer, by increasing the dynamic of the vertical expansion and horizontal expansion of Hadoop, cooperating with the resource sharing, and finally achieving the goal of optimizing resource utilization.

Technology

Resource overload configuration (Resource overcommitment)

Resource sharing between virtual machines is typically achieved through an excessive allocation of resources. Most virtual machine hypervisor, such as VMware and Xen, support excessive allocation of resources. For example, VMware ESX allows you to configure reserved resources (reservation), maximum resources (LIMIT), and shares (Share) for each virtual machine to control resource sharing between multiple virtual machines on the same physical machine, as well as allocation policies when resources are low.

Figure 2: Excessive resource allocation

With this technology, multiple applications can share resources, such as when other applications are busy, allowing the Hadoop compute node to take up a fraction of the resources, while other applications are idle, the Hadoop node can allocate more resources.

Hadoop compute Node Vertical extension

Only through the hypervisor resource scheduling is not good to achieve the purpose of resource sharing, the current Hadoop resource allocation is static:

1. In MapReduce v1, the resource is configured with the slot number of each tasktracker, and the slot is a quantitative CPU and memory resource abstraction

2. In MapReduce v2, resources are configured directly through the CPU and memory configuration entries for each NodeManager

These configurations have been started and cannot be dynamically adjusted, so that Hadoop will still allocate excessive tasks to this node in the case of insufficient resources for the node virtual machine, and the execution speed of the task will be greatly reduced, thus affecting the performance of the entire cluster.

Hve's resource-extension technology solves this problem by dynamically configuring the Hadoop resource configuration:

1. During the operation of the compute node, the user can dynamically set the slot number (MRV1) or the number of resources (MRV2) for each compute node.

2. When computing nodes are low on resources, users or VHM (Virtual Hadoop Manager) can automatically adjust their resource allocations to minimize the performance degradation caused by resource competition

In MRv1, when the slot is adjusted, the running task will not be killed if it exceeds the slot number, but it will not accept the new task until the redundant task is complete, so the resource recycle may be more slow

In MRv2, the community is developing task preemption, which is expected to increase the speed of resource recovery with preemption capabilities

Dynamic resource adjustment of graph 3:MRV1