Deploying Apache Mesos on CentOS7

Overview

Apache Mesos is an open source cluster Management suite based on multi-resource (memory, disk, CPU, port, etc.) scheduling that makes fault-tolerant and distributed systems easier.

Working principle

Apache Mesos uses the master/slave structure to simplify the design, making master as lightweight as possible, preserving only the state information of the various computing frameworks (framework) and Mesos slave. These states are easily refactored when Mesos fails, and Mesos can also use zookeeper to solve the master single point of failure.

Mesos Master acts as the global resource scheduler, using a strategy algorithm to assign idle resources on a slave to a framework, while various frameworks are registered with the master through their own schedulers for access. Mesos Master is the executor that collects task status and launches various frameworks. Working principle:

Apache Mesos Basic Terminology

• Mesos Master: Manages the various frameworks and slave and assigns resources on the slave to each framework.
• Mesos Slave: Responsible for managing each Mesos Task on this node, assigning resources to each executor.
• Framework: Computing frameworks, such as Hadoop, spark, etc., can be accessed via mesosschedulerdiver.
• Executor: Actuator, installed on master slave, to start a task in the compute frame.

Experimental requirements

1. Must use CENTOS7 system, requires kernel 3.10 and above

2. Virtual machine memory must be 2GB and above

Experimental environment table

Host name	IP Address	Install packages
Master	192.168.126.156	Mesos-0.25.0.tar.gz;jdk-8u91-linux-x64.tar.gz
Slave	192.168.126.160	mesos-0.25.0.tar.gz;jdk-8u91-linux-x64.tar.gz; Docker

Installation steps

1. Configuring the Java Environment

[[email protected] ~]# tar zxvf jdk-8u91-linux-x64.tar.gz -C /usr/local/  #解包[[email protected] ~]# cd /usr/local/[[email protected] ~]# mv jdk1.8.0_91/ java   #重命名[[email protected] ~]# vim /etc/profileexport JAVA_HOME=/usr/local/javaexport PATH=$JAVA_HOME/bin:$PATHexport CLASSPATH=$JAVA_HOME/jre/lib/ext:$JAVA_HOME/lib/tools.jar[[email protected] ~]# source /etc/profile

2. Install the relevant environment

1) Install Development tools

[[email protected] ~]# yum groupinstall -y "Development Tools"

2) Add Apache-maven Source to provide support for Mesos project management and build automation tools.

[[email protected] ~]# wget http://repos.fedorapeople.org/repos/dchen/apache-maven/epel-apache-maven.repo -O /etc/yum.repos.d/epel-apache-maven.repo

3) Install dependent packages

[[email protected] ~]# yum install -y apache-maven python-devel zlib-devel libcurl-devel openssl-devel cyrus-sasl-devel cyrus-sasl-md5 apr-devel apr-util-devel subversion-devel

4) Configure WANDISCOSVN network source

[[email protected] ~]# vim /etc/yum.repos.d/wandisco-svn.repo添加以下内容[WANdiscoSVN]name=WANdisco SVN Repo 1.9enabled=1baseurl=http://opensource.wandisco.com/centos/7/svn-1.9/RPMS/$basearch/gpgcheck=1gpgkey=http://opensource.wandisco.com/RPM-GPG-KEY-WANdisco

3. Configuring Mesos Environment variables

[[email protected] ~]# vim /etc/profile在行尾添加以下内容export MESOS_NATIVE_JAVA_LIBRARY=/usr/local/lib/libmesos.soexport MESOS_NATIVE_LIBRARY=/usr/local/lib/libmesos.so[[email protected] ~]# source /etc/profile

4. Build Mesos

This is where the source code is used to compile the installation Mesos

[[email protected] ~]# wget http://www.apache.org/dist/mesos/0.25.0/mesos-0.25.0.tar.gz[[email protected] ~]# tar zxvf mesos-0.25.0.tar.gz -C /opt/[[email protected] ~]# cd /opt/mesos-0.25.0/[[email protected] ~]# mkdir build[[email protected] ~]# cd build/[[email protected] ~]# ../configure   [[email protected] ~]# make   

Installation Mesos time is longer, so the experimental environment installed a mesos after the rest of the clone can be.

Configuring single Mesos-master and Mesos-slave

1. Configure Mesos-master

1) Mesos-master is responsible for maintaining the heartbeat of the slave cluster, extracting resource information from slave. The corresponding parsing work should be done before configuration.

[[email protected] ~]# hostnamectl set-hostname master  #更改主机名称为master[[email protected] ~]# init 6   #重启虚拟机[[email protected] ~]# vim /etc/hosts  192.168.126.156  master192.168.126.160 slave[[email protected] ~]# ln -sf /root/mesos-0.25.0/build/bin/mesos-master.sh /usr/sbin/mesos-master  #建立软链接

2) Simple with start mesos-master

[[email protected] ~]# mesos-master --work_dir=/home/q/mesos/data --log_dir=/home/q/mesos/logs --no-hostname_lookup --ip=0.0.0.0 减配参数：--work_dir ：运行期数据存放路径，包含了sandbox、slave、meta等信息，建议修改。--log_dir  ：Mesos日志存放路径，建议修改。--no-hostname_lookup ：是否从DNS获取主机名，本例中关闭了此配置，直接显示IP。--ip ：Mesos进程绑定的IP。

3) After the configuration is complete, you can use the browser to access the local 5050 port for authentication, 1 shows:

2. Configure Mesos-slave

1) Mesos-slave is responsible for receiving and executing the tasks from the Mesos-master and monitoring the status of tasks, collecting tasks using the system, the configuration should be done before the corresponding parsing work.

[[email protected] ~]# hostnamectl set-hostname slave  #更改主机名称为master[[email protected] ~]# init 6   #重启虚拟机[[email protected] ~]# vim /etc/hosts  192.168.126.156  master192.168.126.160 slave[[email protected] ~]# ln -sf /root/mesos-0.25.0/build/bin/mesos-slave.sh /usr/sbin/mesos-slave   #建立软链接

2) Install and start the Docker container on the Mesos-slave side

[[email protected] ~]# yum install docker -y[[email protected] ~]# systemctl start docker.service[[email protected] ~]# systemctl enable docker.service

3) Simple with start mesos-slave

[[email protected] ~]# mesos-slave --containerizers="mesos,docker" --work_dir=/home/q/mesos/data --log_dir=/home/q/mesos/logs --master=192.168.126.156:5050 --no-hostname_lookup --ip=0.0.0.0

4) After you close the Mesos-master firewall and use your browser to verify the 5050 port of master again, you can see the status of slave on the left side of the Mesos Web page, as shown in 1.1:

5) Click on the Slaves link in the menu bar to see the hardware information and registration time of the slave host, as shown in 1.2:

3. Single Mesos-master Configuration zookeeper

Zookeeper is an open source distributed Application Coordination Service that provides consistent service software for distributed applications, including configuration maintenance, domain name services, distributed synchronization, group services, and more.

The purpose of zookeeper is to encapsulate complex and error-prone critical services to provide users with efficient, stable and easy-to-use systems.

1) Once the zookeeper is downloaded, you only need to rename the profile template to use it.

[[email protected] ~]# wget http://mirrors.cnnic.cn/apache/zookeeper/zookeeper-3.4.6/zookeeper-3.4.6.tar.gz[[email protected] ~]# tar zxvf zookeeper-3.4.6.tar.gz -C /home/q/[[email protected] ~]# cd /home/q/zookeeper-3.4.6/[[email protected] zookeeper-3.4.6]# mv conf/zoo_sample.cfg conf/zoo.cfg

2) Start Zookeeper service

[[email protected] zookeeper-3.4.6]# ./bin/zkServer.sh start conf/zoo.cfg JMX enabled by defaultUsing config: conf/zoo.cfgStarting zookeeper ... STARTED

3) stand-alone mode zookeeper in standalone state

[[email protected] zookeeper-3.4.6]# ./bin/zkServer.sh status conf/zoo.cfg JMX enabled by defaultUsing config: conf/zoo.cfgMode: standalone

4) After the zookeeper service is started, you can use the zookeeper client to connect to the test.

[[email protected] zookeeper-3.4.6]# ./bin/zkCli.sh Connecting to localhost:21812018-08-15 22:16:03,625 [myid:] - INFO  [main:[email protected]] - Client environment:zookeeper.version=3.4.6-1569965, built on 02/20/2014 09:09 GMT2018-08-15 22:16:03,631 [myid:] - INFO  [main:[email protected]] - Client environment:host.name=master.....//省略[zk: localhost:2181(CONNECTED) 1] ls /     #查看根节点[zookeeper]

4. Background running Mesos-master and Mesos-slave

Zookeeper abbreviation for ZK, in the entire Apache Mesos, mainly used to store Mesos-master address, convenient mesos-slave read. When Mesos-slave obtains an address from ZK, it can use the Mesos-master address and the port connection Mesos-master directly.

1) The Nohup command can ignore all hang-up (SIGHUP) signals and run Mesos-master and Mesos-slave as a daemon.

[[email protected] ~]# nohup mesos-master --work_dir=/home/q/mesos/data --log_dir=/home/q/mesos/logs --no-hostname_lookup --ip=0.0.0.0 --zk=zk://192.168.126.163:2181/mesos --quorum=1 &>/dev/null &[1] 3272

配置参数：--zk ：ZooKeeper地址，用于Leader选举。指定zk端口号--zk_session_timeout ：根据网络环境调整zk session超时时间（默认10s）。--quorum ：Master replica logs多写数量，多Master场景下此值要超过Master数量的一半。--credential ：提供密钥对，介入集群时用于验证。

2) At this point, Mesos-slave uses the ZK address and port number to connect the mesos-master.

[[email protected] ~]# nohup mesos-slave --containerizers="mesos,docker" --work_dir=/home/q/mesos/data --log_dir=/home/q/mesos/logs--master=zk://192.168.126.163:2181/mesos --no-hostname_lookup --ip=0.0.0.0 &>/dev/null &[1] 3996

3) Use the browser for authentication, 1.3 as shown:

Deploying Apache Mesos on CentOS7