CentOS7 Deploying Apache Mesos

CentOS7 Deploying Apache Mesos

Apache Mesos is the first open source cluster management software developed by Amplab of the University of California, Berkeley, to support application architectures such as Hadoop, ElasticSearch, Spark, Storm, and Kafka. Mesos uses rules similar to those of the Linux kernel to construct, just the difference between different levels of abstraction. Mesos extracts CPU, memory, storage and other computing resources from devices (physical or virtual machines), making fault-tolerant and resilient distributed systems easier to use. The Mesos core runs on every machine, providing API interfaces for resource management and resource load to applications (Hadoop, Spark, Kafka, Elastic Serarch, and so on) across the entire data center and cloud environment.

How Apache Mesos Works

Apache Mesos uses the master/slave structure to simplify the design, making master as lightweight as possible, preserving only the state information of the various computing frameworks (framework) and Mesos slave, which are easily refactored in the event of Mesos failure. , Mesos can also use zookeeper to solve the master single point of failure.

Mesos Master acts as the global resource scheduler, using a strategy algorithm to assign idle resources on a slave to a framework, while various frameworks are registered with the master through their own schedulers for access. Mesos Slave is a executor that collects task status and launches various frameworks. The working schematic is as follows.

Apache Mesos Configuration Requirements

This experiment is using the CENTOS7 system because the Mesos configuration requires a minimum of 3.10 cores.

[[email protected] ~]# uname -r3.10.0-693.el7.x86_64

Test environment

Host name	IP Address
Master	192.168.58.152
Slave	192.168.58.162

Installation steps

The current Mesos supports a scheduler written in three languages, namely C + +, Java, and Python, which can provide a unified access to different schedulers.

Configuring the Java Environment

First download the JDK software package from the official website, the address is https://www.oracle.com/technetwork/java/, you can choose the appropriate version to download, I use the jdk-8u91-linux-x64.tar.gz here.

[[email protected] ~]# tar zxvf jdk-8u91-linux-x64.tar.gz -C /usr/local/#将jdk包解压到指定目录中[[email protected] ~]# vim /etc/profileexport JAVA_HOME=/usr/local/jdk1.8.0_91export PATH=$JAVA_HOME/bin:$PATHexport CLASSPATH=$JAVA_HOME/jre/lib/ext:$JAVA_HOME/lib/tools.jar[[email protected] ~]# source /etc/profile#将java添加到环境变量中，方便识别

Installation related environment

• Installing development tools

  [[email protected] ~]# yum groupinstall "Development tools" -y

  After installation, use the Yum Group list to view the installed tool groups.
  

• Add Apache-maven Source
  Provides support for project management and building automation tools for Mesos

  [[email protected] ~]# wget http://repos.fedorapeople.org/repos/dchen/apache-maven/epel-apache-maven.repo -O /etc/yum.repos.d/epel-apache-maven.repo

• Install dependent packages

  [[email protected] ~]# yum install -y apache-maven python-devel zlib-devel libcurl-devel > openssl-devel cyrus-sasl-devel cyrus-sasl-md5 apr-devel apr-util-devel subversion-devel

• Configuring WANDISCOSVN Network sources

[[email protected] ~]# vim /etc/yum.repos.d/wandisco-svn.repo[WANdiscoSVN]name=WANdisco SVN Repo 1.9enabled=1baseurl=http://opensource.wandisco.com/centos/7/svn-1.9/RPMS/$basearch/gpgcheck=1gpgkey=http://opensource.wandisco.com/RPM-GPG-KEY-WANdisco

• Configuring MESOS Environment variables

[[email protected] ~]# vim /etc/profileexport MESOS_NATIVE_JAVA_LIBRARY=/usr/local/lib/libmesos.soexport MESOS_NATIVE_LIBRARY=/usr/local/lib/libmesos.so[[email protected] ~]# source /etc/profile

• Build Mesos
  We are using the source code to compile the way to install, of course, can also use the Yum Warehouse installation method

  [[email protected] ~]# wget http://archive.apache.org/dist/mesos/0.25.0/mesos-0.25.0.tar.gz#首先先下载mesos软件包[[email protected] ~]# tar xf mesos-0.25.0#将mesos软件包解压到当前目录[[email protected] ~]# cd mesos-0.25.0/#进入到解压完的目录中[[email protected] mesos-0.25.0]# mkdir build/#创建一个build目录，具体为什么这么操作，参考http://mesos.mydoc.io/官方文档[[email protected] mesos-0.25.0]# cd build/[[email protected] build]# ../configure #进入到build目录中，注意是../configure，因为configure在上一层目录中[[email protected] build]# make#注意这个环节时间特别长，要有耐心[[email protected] build]# make check#make编译完后，进行测试，最后可能会有两个报错，不过不影响[[email protected] build]# make install#最后就是安装

  In addition, there may be errors in the make process, which is due to network reasons, causing some files can not be downloaded, resulting in timeout exit, the build directory files are deleted, recompile installation.
  
  Because the installation of Mesos time is longer, so it is recommended to install one, the others use cloning, wmworkstation14 cloning is very convenient, to be good at using.

  Configuring a single Mesos-apache and Mesos-slave configuration Mesos-master

  Mesos-master is responsible for maintaining the heartbeat of the slave cluster, extracting the resource information from the slave, and the corresponding parsing work should be done before configuration.

  [[email protected] ~]# hostnamectl set-hostname master#将本机的主机名改为master[[email protected] ~]# vim /etc/hosts192.168.58.152 master192.168.58.162 slave#在hosts文件中写入ip和主机名的对应关系[[email protected] ~]# ping slavePING slave (192.168.58.162) 56(84) bytes of data.64 bytes from slave (192.168.58.162): icmp_seq=1 ttl=64 time=0.262 ms^C--- slave ping statistics ---1 packets transmitted, 1 received, 0% packet loss, time 0msrtt min/avg/max/mdev = 0.262/0.262/0.262/0.000 ms#可以使用主机名，直接ping通[[email protected] ~]# ln -sf /root/mesos-0.25.0/build/bin/mesos-master.sh /usr/sbin/mesos-master#这是建立软链接，方便使用mesos的相关命令

  Configuring Startup Mesos-master

  [[email protected] ~]# mesos-master --work_dir=/home/q/mesos/data --log_dir=/home/q/mesos/logs --no-hostname_lookup --ip=0.0.0.0 #--work-dir：运行期间数据的存放路径，包含sanbox、slave meta信息#--log——dir：日志存放路径#--no-hostname_lookup：是否从DNS获取主机名，这里是关闭这个功能，直接从本地hosts文件获取#--ip：Mesos进程绑定的IP

  After the configuration is complete, you can use the browser to access the local 5050 port for verification, remember to turn off the firewall and SELinux.
  

  VM14 Cloning Introduction

  The cloning of the VM14 is powerful, and first the cloned virtual machine needs to be shut down.
  
  
  
  
  
  

  Configure Mesos-slave

  Mesos-slave is responsible for accepting and executing the tasks from the Mesos-master and monitoring the status of the task, the use of mobile phone task system, the configuration before the corresponding resolution must be done.

  [[email protected] ~]# hostnamectl set-hostname slave#将Mesos-slave服务器主机名改为slave[[email protected] ~]# vim /etc/hosts192.168.58.152 master192.168.58.162 slave#和Mesos-master一样

  Mesos 0.20.0 started to support the launch of the task via Docker mirroring, while also supporting some of the Docker parameters. Of course we plan to support more parameters in the future. The user can start the Docker image as a task or as a Executor. In order to run slave that supports Docker containers, you must use Docker as one of the Containerizer (containerized) options when starting slave.

  [[email protected] ~]# yum install docker -y#先安装docker服务[[email protected] ~]# systemctl start docker.service[[email protected] ~]# systemctl enable docker.service#启动docker服务已经设置为开机启动

  Configuring Startup Mesos-slave

  [[email protected] ~]# mesos-slave --containerizers="mesos,docker" --work_dir=/home/q/mesos/data > --log_dir=/home/q/mesos/logs --master=192.168.58.152:5050 --no-hostname_lookup --ip=0.0.0.0#--containerizers 支持docker虚拟化的slave#--master:指定mesos-master地址及端口

  After running, refresh the http://192.168.58.152:5050 and you can see that there is an active mesos-slave.
  
  

CentOS7 Deploying Apache Mesos