Multi-node Hadoop cluster on Docker

In the last article you've seen how easy it is to create a single point Hadoop cluster in your devbox.

Now we're raising the bar and creating a Docker Hadoop cluster on the top. Before you start, make sure you have the latest Ambari mirrors:

Line command

Once you have the latest image, you can start the Docker container. We've created several shell functions to help you enter the Docker command, avoiding the typing of lengthy commands like Docker run [options] image [command].

With these features, creating 3-node Hadoop clusters requires only the following line of code:

Default parameter values can be changed according to need, such as blueprint, cluster size, etc. ... The header file of the Shellj.mp/docker-ambari function function has a parameter list.

It is implemented in the following steps:

Custom

If you have a well-defined script, you can put it in gist and run Ambarishell. Start Ambarishell First:

Ambarishell Start Premise:

Ambarishell's hint command can help http://www.aliyun.com/zixun/aggregation/7155.html "> developers realize the automatic complement and congruent function."

Automatic completion includes:

Full command line (for example, the cluster command is not available without this help)

Add Required Parameters

Add Alternative parameters:--plus <TAB>

Add arguments, such as parameter names, host names, and so on ...

Summary

Basically, when we started using Docker, we already used the multi-terminal Hadoop feature – the limit for running 3 to 4 clusters on notebooks is much less than the Sandbox VM.

We used the Docker model to simplify the usage patterns of Hadoop – we can find our latest progress on Cloudbreak in LinkedIn – The open source cloud Hadoop as a Service API application and built on Docker.

Hopefully this article will help you simplify your development process – if you have any questions about using Hadoop on Docker, welcome to communicate.