Spark Combat 1: Create a spark cluster based on GettyImages Spark Docker image

1, first download the image to local. https://hub.docker.com/r/gettyimages/spark/

~$ Docker Pull Gettyimages/spark

2, download from https://github.com/gettyimages/docker-spark/blob/master/docker-compose.yml to support the spark cluster DOCKER-COMPOSE.YML file

Start it

$ docker-compose Up

$ docker-compose Up

Creating spark_master_1

Creating spark_worker_1

Attaching to Spark_master_1, spark_worker_1

master_1 | 16/10/29 13:25:05 INFO Master. master:started daemon with process name: [Email protected]

master_1 | 16/10/29 13:25:05 INFO util. Signalutils:registered signal handler for term

master_1 | 16/10/29 13:25:05 INFO util. Signalutils:registered Signal handler for HUP

master_1 | 16/10/29 13:25:05 INFO util. Signalutils:registered Signal handler for INT

master_1 | 16/10/29 13:25:06 WARN util. nativecodeloader:unable to load Native-hadoop library for your platform ... using Builtin-java classes where applicable

master_1 | 16/10/29 13:25:06 INFO Spark. Securitymanager:changing View ACLs To:root

master_1 | 16/10/29 13:25:06 INFO Spark. Securitymanager:changing Modify ACLS To:root

master_1 | 16/10/29 13:25:06 INFO Spark. Securitymanager:changing View ACLs groups to:

master_1 | 16/10/29 13:25:06 INFO Spark. Securitymanager:changing Modify ACLs groups to:

master_1 | 16/10/29 13:25:06 INFO Spark. SecurityManager:SecurityManager:authentication Disabled; UI ACLs Disabled; Users with view permissions:set (root); Groups with view Permissions:set (); Users with modify Permissions:set (root); Groups with modify Permissions:set ()

worker_1 | 16/10/29 13:25:06 INFO worker. worker:started daemon with process name: [Email protected]

worker_1 | 16/10/29 13:25:06 INFO util. Signalutils:registered signal handler for term

worker_1 | 16/10/29 13:25:06 INFO util. Signalutils:registered Signal handler for HUP

worker_1 | 16/10/29 13:25:06 INFO util. Signalutils:registered Signal handler for INT

master_1 | 16/10/29 13:25:07 INFO util. Utils:successfully started service ' Sparkmaster ' on port 7077.

master_1 | 16/10/29 13:25:07 INFO Master. Master:starting Spark Master at spark://master:7077

master_1 | 16/10/29 13:25:07 INFO Master. Master:running Spark Version 2.0.1

master_1 | 16/10/29 13:25:07 INFO util.log:Logging initialized @2542ms

master_1 | 16/10/29 13:25:07 INFO Server. Server:jetty-9.2.z-snapshot

master_1 | 16/10/29 13:25:07 INFO handler. contexthandler:started [Email protected]{/app,null,available}

master_1 | 16/10/29 13:25:07 INFO handler. contexthandler:started [Email protected]{/app/json,null,available}

master_1 | 16/10/29 13:25:07 INFO handler. contexthandler:started [Email protected]{/,null,available}

master_1 | 16/10/29 13:25:07 INFO handler. contexthandler:started [Email protected]{/json,null,available}

master_1 | 16/10/29 13:25:07 INFO handler. contexthandler:started [Email protected]{/static,null,available}

master_1 | 16/10/29 13:25:07 INFO handler. contexthandler:started [Email protected]{/app/kill,null,available}

master_1 | 16/10/29 13:25:07 INFO handler. contexthandler:started [Email protected]{/driver/kill,null,available}

master_1 | 16/10/29 13:25:07 INFO Server. serverconnector:started [Email protected]{http/1.1}{0.0.0.0:8080}

master_1 | 16/10/29 13:25:07 INFO Server. Server:started @2742ms

master_1 | 16/10/29 13:25:07 INFO util. Utils:successfully started service ' Masterui ' on port 8080.

master_1 | 16/10/29 13:25:07 INFO UI. Masterwebui:bound Masterwebui to 0.0.0.0, and started at http://localhost:8080

master_1 | 16/10/29 13:25:07 INFO Server. Server:jetty-9.2.z-snapshot

master_1 | 16/10/29 13:25:07 INFO handler. contexthandler:started [Email protected]{/,null,available}

master_1 | 16/10/29 13:25:07 INFO Server. serverconnector:started [Email protected]{http/1.1}{master:6066}

master_1 | 16/10/29 13:25:07 INFO Server. Server:started @2779ms

master_1 | 16/10/29 13:25:07 INFO util. Utils:successfully started service on port 6066.

master_1 | 16/10/29 13:25:07 INFO Rest. standalonerestserver:started REST Server for submitting applications on port 6066

worker_1 | 16/10/29 13:25:07 WARN util. nativecodeloader:unable to load Native-hadoop library for your platform ... using Builtin-java classes where applicable

worker_1 | 16/10/29 13:25:08 INFO Spark. Securitymanager:changing View ACLs To:root

worker_1 | 16/10/29 13:25:08 INFO Spark. Securitymanager:changing Modify ACLS To:root

worker_1 | 16/10/29 13:25:08 INFO Spark. Securitymanager:changing View ACLs groups to:

worker_1 | 16/10/29 13:25:08 INFO Spark. Securitymanager:changing Modify ACLs groups to:

worker_1 | 16/10/29 13:25:08 INFO Spark. SecurityManager:SecurityManager:authentication Disabled; UI ACLs Disabled; Users with view permissions:set (root); Groups with view Permissions:set (); Users with modify Permissions:set (root); Groups with modify Permissions:set ()

master_1 | 16/10/29 13:25:08 INFO handler. contexthandler:started [Email protected]{/metrics/master/json,null,available}

master_1 | 16/10/29 13:25:08 INFO handler. contexthandler:started [Email protected]{/metrics/applications/json,null,available}

master_1 | 16/10/29 13:25:08 INFO Master. MASTER:I have been elected leader! New state:alive

worker_1 | 16/10/29 13:25:08 INFO util. Utils:successfully started service ' Sparkworker ' on port 8881.

worker_1 | 16/10/29 13:25:08 INFO worker. Worker:starting Spark worker 172.17.0.3:8881 with 2 cores, 1024.0 MB RAM

worker_1 | 16/10/29 13:25:08 INFO worker. Worker:running Spark Version 2.0.1

worker_1 | 16/10/29 13:25:08 INFO worker. Worker:spark Home:/usr/spark-2.0.1

worker_1 | 16/10/29 13:25:08 INFO util.log:Logging initialized @2625ms

worker_1 | 16/10/29 13:25:08 INFO Server. Server:jetty-9.2.z-snapshot

worker_1 | 16/10/29 13:25:08 INFO handler. contexthandler:started [Email protected]{/logpage,null,available}

worker_1 | 16/10/29 13:25:08 INFO handler. contexthandler:started [Email protected]{/logpage/json,null,available}

worker_1 | 16/10/29 13:25:08 INFO handler. contexthandler:started [Email protected]{/,null,available}

worker_1 | 16/10/29 13:25:08 INFO handler. contexthandler:started [Email protected]{/json,null,available}

worker_1 | 16/10/29 13:25:08 INFO handler. contexthandler:started [Email protected]{/static,null,available}

worker_1 | 16/10/29 13:25:08 INFO handler. contexthandler:started [Email protected]{/log,null,available}

worker_1 | 16/10/29 13:25:08 INFO Server. serverconnector:started [Email protected]{http/1.1}{0.0.0.0:8081}

worker_1 | 16/10/29 13:25:08 INFO Server. Server:started @2749ms

worker_1 | 16/10/29 13:25:08 INFO util. Utils:successfully started service ' Workerui ' on port 8081.

worker_1 | 16/10/29 13:25:08 INFO UI. Workerwebui:bound Workerwebui to 0.0.0.0, and started at http://localhost:8081

worker_1 | 16/10/29 13:25:08 INFO worker. Worker:connecting to Master master:7077 ...

worker_1 | 16/10/29 13:25:08 INFO handler. contexthandler:started [Email protected]{/metrics/json,null,available}

worker_1 | 16/10/29 13:25:08 INFO Client. Transportclientfactory:successfully created connection to master/172.17.0.2:7077 after-MS (0 Ms spent in bootstraps)

master_1 | 16/10/29 13:25:09 INFO Master. Master:registering worker 172.17.0.3:8881 with 2 cores, 1024.0 MB RAM

worker_1 | 16/10/29 13:25:09 INFO worker. worker:successfully registered with Master spark://master:7077

This article is from "Fish xiang Shallow Bottom" blog, please be sure to keep this source http://yanwe.blog.51cto.com/10031173/1869988

Spark Combat 1: Create a spark cluster based on GettyImages Spark Docker image