Linux LXD container to build Hadoop cluster

Configuration requirements

• Host Memory 4GB.
• Disk more than GB.
• HOST Machine installs common Linux distributions.

Linux Container (LXD)

Take the host Ubuntu 16.04 as an example.

• Install LXD.

  sudo Install  sudo lxd init

• To view the available image sources, if you use the default image, you can skip the next two steps and go directly to the back of the launch.

  $ LXC Remote List

• Select the image you like in the previous step, copy the link, and add the image source below.

  $ LXC remote Add images <URL>

• The
• can now get and load the mirror.

   $ lxc Launch <images_name>:<version> # images_name is the image source name, is the last step named images, what can be taken. # version is an image. # HOST_NAME is the host name of the instance you want to create. # to use the default image source, build Hadoop as an example, here should be: $ lxc launch ubuntu -daily:16.04  master 

• OK, the instance has started to run.

  $ lxc List+--------+---------+--------------------+-----------------------------------------------+---- --------+-----------+|  NAME  |  State  |        IPV4        |                     IPV6                      |    TYPE    10.71.  16.37 (eth0) | FD16:E204:21D5:5295:2160         |+--------+---------+--------------------+----- ------------------------------------------+------------+-----------+

  You can now see that only the master node is running.
• Let's go into the container of Ubuntu.

  $ LXC EXEC master--/bin/bash

  If you enter successfully, congratulations! The first step is open.

Hadoop Cluster

• Update the system.

  $ LXC EXEC master--/bin/bash[email protected]# apt-get update[email protected]# apt-get upgrade-y

• Configure the environment, because LXD when the instance is created, the host name is set, the firewall is disabled, and the LXD Network module makes Dynamic host mapping for each virtual host. So the following is OK.

  $hostnamearch$ LXC exec master--/bin/Bash[email protected]:~#Pingarchping Arch (127.0.1.1) About( -) bytes of data. -Bytes from Rain-mark (127.0.1.1): icmp_seq=1Ttl= -  Time=0.020Ms -Bytes from Rain-mark (127.0.1.1): icmp_seq=2Ttl= -  Time=0.031Ms^c

  We configured the environment on the master node and cloned multiple nodes using the LXD container clone. Hostname-based network access can be done directly between each node.
• Configure the Java environment.

  file push ~/downloads/jdk-8u111-linux-x64. tar. GZ master/root/jdk-8u111-linux-x64. Tar  --/bin/tar XF jdk-8u111-linux-x64. tar. gz-c/usr/echo"export java_home=/usr/local/jdk1.8.0_111" >> ~/. Bashrc[email protected]# Exit

• Configure Hadoop.

  $ lxcfilePush ~/downloads/hadoop-2.7.3.Tar. GZ master/root/hadoop-2.7.3.Tar. gz$ LXC exec Master--/bin/Bash[email protected]#mkdir-p/home/root/Hd_data[email protected]#TarXF hadoop-2.7.3.Tar. gz-c/usr/local[email protected]# CD/usr/local/hadoop-2.7.3[email protected]:/usr/local/hadoop-2.7.3#Cat<< EOF > etc/core-Site.xml<?xml version="1.0"encoding="UTF-8"? ><?xml-stylesheet type="text/xsl"href="configuration.xsl"?><configuration> <property> <name>hadoop.tmp.dir</name> <value>/home/root/HD_data</value> </property> <property> <nam E>fs.defaultfs</name> <value>hdfs://master:8020</value></property></configuration>Eof[email protected]:/usr/local/hadoop-2.7.3#Cat<< EOF > etc/yarn-Site.xml<?xml version="1.0"?><configuration> <property> <name>yarn.resourcemanager.hostname</name> <value>master</value> </property> <property> <name>yarn.no Demanager.aux-services</name> <value>mapreduce_shuffle</value> </property></configura Tion>Eof[email protected]:/usr/local/hadoop-2.7.3#Cat<< EOF > etc/mapred-Site.xml<?xml-stylesheet type="text/xsl"href="configuration.xsl"?><configuration> <property> <name>mapreduce.framework.name</name> &LT;VALUE&G T;yarn</value> </property></configuration>Eof[email protected]:/usr/local/hadoop-2.7.3#./bin/hdfs Namenode-Format[email protected]:/usr/local/hadoop-2.7.3# exit

• Now that the Hadoop basic configuration file is ready, we clone out two slave nodes.

  $ lxc Copy Master slave0$ lxc start slave0$ lxc copy master slave1$ lxc start slave1

  OK, now that the two slave nodes are the same as master, we eliminate the hassle of duplicate configuration.
• Start the Hadoop cluster.

  $ LXC EXEC master--/bin/bash[email protected]# CD/usr/local/hadoop-2.7.3[email protected]:/usr/local/hadoop-2.7.3#./sbin/hadoop-daemon.SHstart namenodestarting Namenode, logging to/usr/local/hadoop-2.7.3/logs/hadoop-root-namenode-Master.out[email protected]:/usr/local/hadoop-2.7.3#./sbin/yarn-daemon.SHstart ResourceManager starting ResourceManager, logging to/usr/local/hadoop-2.7.3/logs/yarn-root-resourcemanager-Master.out[email protected]:/usr/local/hadoop-2.7.3#/USR/LOCAL/JDK1.8.0_111/bin/JPS506NameNode604ResourceManager829Jps[email protected]:/usr/local/hadoop-2.7.3# exit$ LXC exec slave0--/bin/Bash[email protected]:~# cd/usr/local/hadoop-2.7.3/[email protected]:/usr/local/hadoop-2.7.3#./sbin/hadoop-daemon.SHstart datanodestarting Datanode, logging to/usr/local/hadoop-2.7.3/logs/hadoop-root-datanode-Slave0.out[email protected]:/usr/local/hadoop-2.7.3#./sbin/yarn-daemon.SHstart nodemanagerstarting NodeManager, logging to/usr/local/hadoop-2.7.3/logs/yarn-root-nodemanager-Slave0.out[email protected]:/usr/local/hadoop-2.7.3#/USR/LOCAL/JDK1.8.0_111/bin/JPS433DataNode538NodeManager670Jps[email protected]:/usr/local/hadoop-2.7.3# exit# need to slave1 also launch Datanode as well as NodeManager. 

  At this point, the cluster has been built! We can see the status of the cluster through the HOST's browser.
• View the status of individual nodes.

  $ LXC List+--------+---------+--------------------+-----------------------------------------------+------------+---------  --+|  NAME |        State |                     IPV4 |    IPV6 | TYPE | Snapshots |+--------+---------+--------------------+-----------------------------------------------+------------ +-----------+| Master | RUNNING |10.71.16.37(eth0) | FD16:E204:21D5:5295:216: 3eff:fec9:dd16 (eth0) | Persistent |0|+--------+---------+--------------------+-----------------------------------------------+------------+-------- ---+| Slave0 | RUNNING |10.71.16.22(eth0) | FD16:E204:21D5:5295:216: 3eff:fe8e:8e57 (eth0) | Persistent |0|+--------+---------+--------------------+-----------------------------------------------+------------+-------- ---+| slave1 | RUNNING |10.71.16.31(eth0) | FD16:E204:21D5:5295:216: 3eff:fe5a:ef1 (eth0) | Persistent |0|+--------+---------+--------------------+-----------------------------------------------+------------+-------- ---+

  OK, the host browser input master node ip:50070 can see the HDFS status, as well as ip:8088 can see Yarn information. We can also see how many nodes are running on a Web page.

Summary

Based on the LXD virtualization container, we can make it easy and resource-efficient for Hadoop cluster building exercises with 3 nodes and even more nodes under the Linux host. It is very advantageous for the students and teachers who have insufficient hardware resources to demonstrate. For more resources on LXD and resources for Hadoop, please refer to the respective official documentation.

Original articles, reproduced please indicate the source, or the relevant responsibility!

Linux LXD container to build Hadoop cluster