Using Docker to build Hadoop distributed clusters

Source: Internet
Author: User
Tags mkdir ssh centos docker run

Deploying Hadoop distributed clusters using Docker
Looking for a long time on the internet did not find the use of Docker to build a Hadoop distributed cluster of documents, no way, can only write one.
One: Environment preparation:
1: First, you have a CENTOS7 operating system that can be installed in the virtual machine.
2: The version of Docker,docker installed in Centos7 is 1.8.2
The installation steps are as follows:
<1> installation of the developed version of Dockeryum Install-y Docker-1.8.2-10.el7.centos
<2> may error when installing, need to delete this dependency

Rpm-e lvm2-7:2.02.105-14.el7.x86_64
   
   
    
    1
   
   
   
   
    
    1
   
   

Start Docker

Service Docker start
   
   
    
    1
   
   
   
   
    
    1
   
   

To verify the installation results:

<3> Execute Docker info after startup and see two lines of warning below

Need to shut down the firewall and reboot the system

Systemctl Stop firewalld
systemctl disable FIREWALLD
Note: Restart the system after executing the above command
reboot-h (restart the system)
   
   
    
    1
    
    2
    
    3
    
    4
   
   
   
   
    
    1
    
    2
    
    3
    
    4
   
   

<4> Running containers may error

Need to close SELinux
Workaround:

1:setenforce 0 (Effective immediately without restarting the operating system)
2: Modify the selinux=disabled in the/etc/selinux/config file, and then restart the system to take effect the
recommended two steps are executed, This ensures that SELinux is also off after the system restarts
   
   
    
    1
    
    2
    
    3
   
   
   
   
    
    1
    
    2
    
    3
   
   

3: You need to build a basic image of Hadoop first and build it using the Dockerfile file method.
First, build an SSH-capable image to facilitate later use. (This will have an impact on the security of the container)
Note: The root user's password in this image is root
Mkdir Centos-ssh-root
Cd Centos-ssh-root
Vi Dockerfile

# Select an existing OS image as the basis from the author of the CentOS # Image Maintainer Crxy # Install the openssh-server and sudo packages and set the sshd usepam parameter to no RUN yum I Nstall-y openssh-server sudo run sed-i ' s/usepam yes/usepam no/g '/etc/ssh/sshd_config #安装openssh-clients RUN Yum I Nstall-y openssh-clients # Add test user root, password root, and add this user to Sudoers RUN echo "Root:root" |  Chpasswd run echo "root all= (All) All" >>/etc/sudoers # The following two sentences are special, must have on the CENTOS6, otherwise the created container sshd cannot log on to RUN   
Ssh-keygen-t dsa-f/etc/ssh/ssh_host_dsa_key RUN ssh-keygen-t rsa-f/etc/ssh/ssh_host_rsa_key # start sshd service and expose 22 ports
    
    RUN mkdir/var/run/sshd EXPOSE + CMD ["/usr/sbin/sshd", "-D"] 1 2 3 4
    
    5 6 7 8 9 10 11 12 13 14
   
   
   
   
    
    15 16 17 18 19 20 21 22 23
    
 1 2 3) 4 5   6 7 8 9 10 11 12 13 14 15 16
   
    17 18 19 20 21 22 23

Build command:
Docker build-t= "Crxy/centos-ssh-root".

Query the image that was just built successfully


4: Build a mirror with JDK based on this image
Note: The JDK is using the 1.7 version of the
Mkdir CENTOS-SSH-ROOT-JDK
Cd CENTOS-SSH-ROOT-JDK
Cp.. /jdk-7u75-linux-x64.tar.gz.
Vi Dockerfile

From Crxy/centos-ssh-root
ADD jdk-7u75-linux-x64.tar.gz/usr/local/
RUN mv/usr/local/jdk1.7.0_75/usr/local /jdk1.7
env java_home/usr/local/jdk1.7
env PATH $JAVA _home/bin: $PATH
   
   
    
    1
    
    2
    
    3
    
    4
    
    5
   
   
   
   
    
    1
    
    2
    
    3
    
    4
    
    5
   
   

Build command:
Docker build-t= "Crxy/centos-ssh-root-jdk".

Query builds a successful image

5: Build a mirror with Hadoop based on this JDK image
Note: Hadoop is using the 2.4.1 version.
Mkdir Centos-ssh-root-jdk-hadoop
Cd Centos-ssh-root-jdk-hadoop
Cp.. /hadoop-2.4.1.tar.gz.
Vi Dockerfile

From Crxy/centos-ssh-root-jdk
ADD hadoop-2.4.1.tar.gz/usr/local
RUN mv/usr/local/hadoop-2.4.1/usr/local/ Hadoop
env hadoop_home/usr/local/hadoop
env PATH $HADOOP _home/bin: $PATH
   
   
    
    1
    
    2
    
    3
    
    4
    
    5
   
   
   
   
    
    1
    
    2
    
    3)
    
    4
    
    5
   
   

Build command:
Docker build-t= "Crxy/centos-ssh-root-jdk-hadoop".

Query builds a successful image

Two: Build Hadoop distributed cluster
1: Cluster planning
Ready to build a cluster with three nodes, one master two from
Master node: hadoop0 ip:192.168.2.10
From node 1:hadoop1 ip:192.168.2.11
From node 2:hadoop2 ip:192.168.2.12

But because the IP changes after the Docker container restarts, we need to set up a fixed IP for Docker. To set a fixed IP for a docker container using pipework
2: Start three containers, respectively, as a hadoop0 hadoop1 hadoop2
Execute the following command on the host, set the host name and container name for the container, and open ports 50070 and 8088 in Hadoop0

Docker run--name hadoop0--hostname hadoop0-d-p-p 50070:50070-p 8088:8088 Crxy/centos

Contact Us

The content source of this page is from Internet, which doesn't represent Alibaba Cloud's opinion; products and services mentioned on that page don't have any relationship with Alibaba Cloud. If the content of the page makes you feel confusing, please write us an email, we will handle the problem within 5 days after receiving your email.

If you find any instances of plagiarism from the community, please send an email to: info-contact@alibabacloud.com and provide relevant evidence. A staff member will contact you within 5 working days.

A Free Trial That Lets You Build Big!

Start building with 50+ products and up to 12 months usage for Elastic Compute Service

  • Sales Support

    1 on 1 presale consultation

  • After-Sales Support

    24/7 Technical Support 6 Free Tickets per Quarter Faster Response

  • Alibaba Cloud offers highly flexible support services tailored to meet your exact needs.