Remember once marathon is black process

Open the MESOS platform this morning and find a killed mission, mesos_task_id= HYAKUHEI.A318E232-28D9-11E6-BC8F-96BED1F124A2, the name is very strange, not I run Ah, and then go to Marathon to see, without this task container in the run, may have been deleted, view Mesos log, found in two Slav The e-node ran over the task, logged in to slave Docker ps-a, and saw the image name scare Jump:

# docker ps -acontainer id        image                                                                  COMMAND                 CREATED              STATUS                       PORTS                names0ef6eeda359a        linuxkonsult/ Kali-metasploit                                          "Bash"                   2 hours ago         Exited  (0)  2  hours ago                           loving_payne                                                                             f9de5a11f30e         linuxkonsult/kali-metasploit                                          "chmod ug+rx  ' tail -   13  hours ago                                                           mesos-025bd996-0430-46b3-afee-7d4b4248482b-s1.c8e9b672-ebda-4ade-98d5-4d7780f6686c

A little bit of security knows what the Kali is, what the Metasploit is, how to look at the last record, and see the other IP addresses where no one is logged in successfully, and it is normal for the key program of the system to be compared to the other nodes:

Rpm-vf/bin/ls

Rpm-vf/usr/sbin/sshd

Rpm-vf/sbin/ifconfig

Rpm-vf/usr/bin/docker

Zabbix looked at the record, out of the I/O a bit high, no other exception, I was curious about the Docker start f9de5a11f30e see what is inside, try Docker start, Find the boot error (0ef6eeda359a is my own boot), then go to the Docker directory below to check the configuration information of this task:

The container information after being dispatched by Mesos can be found in this directory:

650) this.width=650; "src=" Http://s5.51cto.com/wyfs02/M02/82/53/wKiom1dREquC-W83AAFL3FpkKiY080.png "title=" 11.png "alt=" Wkiom1drequc-w83aafl3fpkkiy080.png "/>

Then combined with the name of Docker ps-a, the container name directory that has been used by hackers is this:

F9DE5A11F30E4A88E186CDB443A72D0A8B66A8EB151A688D89A7EA9B8160DF77, to the directory to see there is a config.json, inside the contents are as follows:

{"state": {"Running": false, "Paused": false, "restarting": false, "oomkilled": false, "Dead": false, "Pid": 0, "ExitCode": -1, "Error": "[8] System error:exec: \" chmod ug+rx ' tail-f/dev/null; ' \u0026\u0026 exec ' tail-f/dev/null; ' \ ': Stat CHM Od ug+rx ' tail-f/dev/null; ' \u0026\u0026 exec ' tail-f/dev/null; ': No such file or directory ', ' Startedat ': ' 0001-01-01T 00:00:00z "," Finishedat ":" 0001-01-01t00:00:00z "}," ID ":" F9de5a11f30e4a88e186cdb443a72d0a8b66a8eb151a688d89a7ea9b8160df77 "," Created ":" 2016-06-02t15:50:31.8048387z "," Path ":" chmod ug+rx ' tail-f/dev/null; ' \u0026\u0026 exec ' tail-f/dev/null; ' "," Args ": []," Config ": {" Hostname ":" Bastion "," Domainname ":" Shanker "," User ":" "," Attachstdin ": false," Attachstdout " : True, "Attachstderr": True, "portspecs": null, "Exposedports": null, "Tty": false, "Openstdin": false, "stdinonce": false , "ENV": ["libprocess_port=0", "mesos_agent_endpoint=192.168.0.33:5051", "Mesos_checkpoint=1", "mesos_directory=/ tmp/mesos/slaves/025bd996-0430-46b3-afee-7d4b4248482b-s1/frameworks/d4bb23e8-a0b8-4dee-8500-27f663613ba0-0000/executors/marathon-hyakuhei.a318e232-28d9-11e6-bc8f-96bed1f124a2/runs/ C8e9b672-ebda-4ade-98d5-4d7780f6686c "," mesos_executor_id= Marathon-hyakuhei.a318e232-28d9-11e6-bc8f-96bed1f124a2 "," Mesos_executor_shutdown_grace_period=5secs "," MESOS_ framework_id=d4bb23e8-a0b8-4dee-8500-27f663613ba0-0000 "," mesos_native_java_library=/usr/lib/libmesos-0.28.1.so "," mesos_native_library=/usr/lib/libmesos-0.28.1.so "," Mesos_recovery_timeout=15mins "," MESOS_SLAVE_ID= 025bd996-0430-46b3-afee-7d4b4248482b-s1 "," Mesos_slave_pid=slave (1) @192.168.0.33:5051 "," MESOS_SUBSCRIPTION_ Backoff_max=2secs "," marathon_app_version=2016-06-02t15:48:42.608z "," host=192.168.0.33 "," MARATHON_APP_RESOURCE_ cpus=1.0 "," port_10006=31027 "," Marathon_app_docker_image=linuxkonsult/kali-metasploit "," MESOS_TASK_ID= Hyakuhei.a318e232-28d9-11e6-bc8f-96bed1f124a2 "," port=31027 "," marathon_app_resource_mem=128.0 "," PORTS=31027 "," marathon_app_resource_disk=0.0 "," marathon_app_labels= "," Marathon_app_id=/hyakuhei "," PORT0=31027"," Mesos_sandbox=/mnt/mesos/sandbox "," mesos_container_name= Mesos-025bd996-0430-46b3-afee-7d4b4248482b-s1.c8e9b672-ebda-4ade-98d5-4d7780f6686c "," PATH=/usr/local/sbin:/usr /local/bin:/usr/sbin:/usr/bin:/sbin:/bin "," debian_frontend=noninteractive "]," CMD ": [" chmod ug+rx ' tail-f/dev/null ; ' \u0026\u0026 exec ' tail-f/dev/null; ' "]," Image ":" Linuxkonsult/kali-metasploit "," Volumes ": null," Volumedriver ":" "," Workingdir ":" "," entrypoint ": null," Networkdisabled ": false," MacAddress ":" "," onbuild ": null," Labels ": {}}," Image ":" 5284900a1876c960190c0d789f9562f285ac8231ecedbf533fa2371d1d6edc26 "," networksettings ": {" Bridge ":" "," EndpointID ": "", "Gateway": "", "globalipv6address": "", "Globalipv6prefixlen": 0, "Hairpinmode": false, "IPAddress": "", "Ipprefixlen ": 0," Ipv6gateway ":" "," linklocalipv6address ":", "Linklocalipv6prefixlen": 0, "MacAddress": "," Networkid ":", " Portmapping ": null," Ports ": null," Sandboxkey ":" "," secondaryipaddresses ": null," secondaryipv6addresses ": null}," Resolvconfpath ":"/var/lib/docker/containers/f9de5a11f30e4a88e186cdb443a72d0a8b66a8eb151a688d89a7ea9b8160df77/resolv.conf "," Hostnamepath ":"/var/lib/docker/containers/ F9de5a11f30e4a88e186cdb443a72d0a8b66a8eb151a688d89a7ea9b8160df77/hostname "," Hostspath ":"/var/lib/docker/ Containers/f9de5a11f30e4a88e186cdb443a72d0a8b66a8eb151a688d89a7ea9b8160df77/hosts "," LogPath ":"/var/lib/docker/ containers/f9de5a11f30e4a88e186cdb443a72d0a8b66a8eb151a688d89a7ea9b8160df77/ F9de5a11f30e4a88e186cdb443a72d0a8b66a8eb151a688d89a7ea9b8160df77-json.log "," Name ":"/ Mesos-025bd996-0430-46b3-afee-7d4b4248482b-s1.c8e9b672-ebda-4ade-98d5-4d7780f6686c "," Driver ":" Devicemapper "," Execdriver ":" native-0.2 "," Mountlabel ":" "," Processlabel ":" "," Restartcount ": 0," Updatedns ": false," mountpoints ": {" /mnt/mesos/sandbox ": {" Name ":" "," Destination ":"/mnt/mesos/sandbox "," Driver ":" "," RW ": True," Source ":"/tmp/mesos/ slaves/025bd996-0430-46b3-afee-7d4b4248482b-s1/frameworks/d4bb23e8-a0b8-4dee-8500-27f663613ba0-0000/executors/ marathon-hyakuhei.a318e232-28d9-11e6-bc8f-96bed1f124a2/runs/c8e9b672-ebda-4ade-98d5-4d7780f6686c "," Relabel ":" "}}," Volumes ": {"/mnt/mesos/sandbox ":"/tmp/mesos/slaves/ 025bd996-0430-46b3-afee-7d4b4248482b-s1/frameworks/d4bb23e8-a0b8-4dee-8500-27f663613ba0-0000/executors/ Marathon-hyakuhei.a318e232-28d9-11e6-bc8f-96bed1f124a2/runs/c8e9b672-ebda-4ade-98d5-4d7780f6686c "}," VolumesRW " : {"/mnt/mesos/sandbox": true}, "Apparmorprofile": ""}

Looks messy, careful patience to see the first line found that the other this instance is not successfully started, the first start failure, then the hacker restarted, and then failed, and then to another node ran a time, or failed, so this time did not cause a great loss, but let us see what he has executed some orders, It's also an experience to make Docker safe.

The startup instance name is: Hyakuhei

1 cpu,128m memory allocated, no disk added

31027 of Port Mappings

The image used is: Linuxkonsult/kali-metasploit

Command executed: CMD ": [" chmod ug+rx ' tail-f/dev/null; ' \u0026\u0026 exec ' tail-f/dev/null; ' "

This pair of strange \u0026 is compiled by Java results, restore back is two & symbols, so the command is

chmod ug+rx ' tailf-f/dev/null; ' && exec ' tail-f/dev/null; '

It doesn't really make sense to see that some of the commands are compiled by Java.

The final review of Mesos's records also proves my judgment that hackers did not successfully perform this task:

I0602 15:49:42.988018 29647 master.cpp:4763] Status update task_failed (uuid:4310d90d-12fd-4d47-a723-a4f52691a99c) for Task HYAKUHEI.7B94EAB0-28D9-11E6-BC8F-96BED1F124A2 of the framework d4bb23e8-a0b8-4dee-850-27f663613ba0-0000 from slave 025BD996-0430-46B3-AFEE-7D4B4248482B-S4 at Slave (1) @192.168.0.32:5051 (192.168.0.32)

Now think about the good scared, it seems that everyone is worried about the Docker security problem is really a big problem, and then blame me, Mesos and Marathon when the start of the time no certification, viewing the official documents, marathon when the start of the--HTTP_ Credentials can, and then Mesos start time with--authenticate--credentials parameters, let Mesos slave connected to master when added authentication. I hope everyone lesson, do not talk about marathon burst into the public network, and to add certification, and do not monitor the operation of the Docker regularly.

Welcome to add!

This article is from "Tianya Horizon" blog, please be sure to keep this source http://shanker.blog.51cto.com/1189689/1785797

Remember once marathon is black process