Lagstash + Elasticsearch + kibana 3 + Kafka Log Management System Deployment 02

As a result of the company's data security and analysis needs, so research GlusterFS + Lagstash + elasticsearch + Kibana 3 + Redis integrated log Management application:

installation, configuration process, usage, etc. continued

One, Glusterfs Distributed File System deployment: Description: The company wants to do the website business log and system log unified collection and management, after the MFS, Fastdfs and other Distributed File system research, finally selected Glusterfs, because Gluster has high scalability, high performance, High availability, scale-out elasticity features, no meta data server design so that Glusterfs no single point of failure, official website: www.gluster.org

1. System Environment Preparation:

Centos 6.4 Server: 192.168.10.101 192.168.10.102 192.168.10.188 192.168.10.189 client: 192.168.10.103 epel source and Glusterfs source Add Epel source and Glusterfs source, Epel source contains Glusterfs, version is older, relatively stable, this test uses the latest 3.5.0 version.

     rpm -Uvh http://dl.fedoraproject.org/pub/epel/6/x86_64/epel-release-6-8.noarch.rpm     wget -P /etc/yum.repos.dhttp://download.gluster.org/pub/gluster/glusterfs/LATEST/CentOS/glusterfs-epel.repo

2. Deployment process

     Server-side installation: yum-y install Glusterfs glusterfs-fuseglusterfs-server chkconfig glusterd on service glusterd STA     RT Server Configuration: 4 storage nodes to form a cluster, this article in the first node execution, only need to execute on any node OK. [[email protected] ~]# Gluster peer probe192.168.10.102 probe successful [[email protected] ~]# Gluster Peer probe192.168.10.188 probe successful [[email protected] ~]# Gluster peer probe 192.168.10.189 probe Successful view the node information for the cluster: [[email protected] ~]# Gluster Peer status number of Peers:3 hostname:192.168     .10.102 Uuid:b9437089-b2a1-4848-af2a-395f702adce8 State:peer in cluster (connected) hostname:192.168.10.188 uuid:ce51e66f-7509-4995-9531-4c1a7dbc2893 State:peer in cluster (connected) hostname:192.168.10.189 Uui D:66d7fd67-e667-4f9b-a456-4f37bcecab29 State:peer in cluster (connected) with/data/gluster as a shared directory, create a volume named Test-volume, copy Number for 2:sh cmd.sh "Mkdir/data/gluster" [[email protected] ~]# gluster voLume Create Test-volume replica 2192.168.10.101:/data/gluster 192.168.10.102:/data/gluster192.168.10.188:/data/ Gluster 192.168.10.189:/data/gluster Creation of volume Test-volume has beensuccessful.     Please start the volume to access data.     Boot volume: [[email protected] ~]# Gluster Volume Starttest-volume starting volume Test-volume has beensuccessful     View volume Status: [[email protected] ~]# gluster Volume info Volume name:test-volume type:distributed-replicate status:started number of Bricks:2 x 2 = 4 Transport-type:tcp bricks:brick1:192.168.10.101:/data/g Luster Brick2:192.168.10.102:/data/gluster Brick3:192.168.10.188:/data/gluster brick4:192.168.10.189:/data/ Gluster

3. Client Installation configuration:

     安装：     yum -y installglusterfs glusterfs-fuse     挂载：     mount -t glusterfs 192.168.10.102:/test-volume/mnt/ （挂载任意一个节点即可）推荐用这种方式。     mount -t nfs -o mountproto=tcp,vers=3192.168.10.102:/test-volume /log/mnt/ （使用nfs挂载，注意远端的rpcbind服务必须开启）     echo "192.168.10.102:/test-volume/mnt/ glusterfs defaults,_netdev 0 0" >> /etc/fstab (开机自动挂载)

4. Testing

     检查文件正确性     dd if=/dev/zero of=/mnt/1.img bs=1Mcount=1000 # 在挂载客户端生成测试文件     cp /data/navy /mnt/  # 文件拷贝到存储上     宕机测试。     使用glusterfs-fuse挂载，即使目标服务器故障，也完全不影响使用。用NFS则要注意挂载选项，否则服务端故障容易导致文件系统halt住而影响服务！     # 将其中一个节点停止存储服务service glusterd stop     service glusterfsd stop# 在挂载客户端删除测试文件     rm -fv /mnt/navy# 此时在服务端查看，服务被停止的节点上navy并未被删除。此时启动服务：serviceglusterd start# 数秒后，navy就被自动删除了。新增文件效果相同！

5. Operations Common commands:

     Deleting a volume Gluster volume stop test-volume gluster volume delete Test-volume move the machine out of the cluster Gluster peer detach 192 The. 168.10.102 only allows 172.28.0.0 network access glusterfs gluster volume set Test-volumeauth.allow 192.168.10.* to add new machines to the volume (due to the secondary This number is set to 2, add at least 2 (4, 6, 8.) machines) Gluster peer probe 192.168.10.105 Gluster peer probe 192.168.10.106 Gluster volume Add-brick test-volume192.168.10.105:/data/gluster 192.168.10.106:/data/gluster Shrink Volume # shrink the volume before gluster need to move data to another location G Luster Volume Remove-brick test-volume192.168.10.101:/data/gluster/test-volume192.168.10.102:/data/gluster/ Test-volume Start # View the Migration status Gluster volume Remove-brick test-volume192.168.10.101:/data/gluster/test-volume192.168. 10.102:/data/gluster/test-volume Status # Gluster volume Remove-brick Test-volume192.168.10.101:/data/glus submitted after migration complete     Ter/test-volume192.168.10.102:/data/gluster/test-volume Commit Migration Volume # migrate 192.168.10.101 data to, first add 192.168.10.107 to the cluster    Gluster Peer probe 192.168.10.107 Gluster Volume Replace-bricktest-volume 192.168.10.101:/data/gluster/test-volume192.168.10.107:/data/gluster/ Test-volume Start # View migration status Gluster volume Replace-brick test-volume192.168.10.101:/data/gluster/test-volume192.168.10 .107:/data/gluster/test-volume Status # Submitted Gluster volume Replace-brick test-volume192.168.10.101:/data/after data migration Gluster/test-volume192.168.10.107:/data/gluster/test-volume Commit # If the machine 192.168.10.101 fails to run, Perform a forced commit and then ask Gluster to perform a synchronization immediately gluster volume Replace-bricktest-volume 192.168.10.101:/data/gluster/ Test-volume192.168.10.102:/data/gluster/test-volume Commit-force gluster Volume heal Test-volumes full 24007

Two Log Collection System Deployment

Description Simple solution:

系统各部分应用介绍Logstash：做系统log收集，转载的工具。同时集成各类日志插件,对日志查询和分析的效率有很大的帮助.一般使用shipper作为log收集、indexer作为log转载.Logstash shipper收集log 并将log转发给redis 存储Logstash indexer从redis中读取数据并转发给elasticsearchredis：是一个db，logstash shipper将log转发到redis数据库中存储。Logstash indexer从redis中读取数据并转发给elasticsearch。kafka：这里我们把redis换作用kafka，主要用于处理活跃的流式数据，高吞吐率，显式分布式，支持数据并行加载Elasticsearch：开源的搜索引擎框架,前期部署简单,使用也简单,但后期需要做必要的优化具体请参照博客http://chenlinux.com/categories.html#logstash-ref  中logstash部分.可进行多数据集群，提高效率。从redis中读取数据，并转发到kibana中Kibana： 开源web展现。

Log collection system architecture diagram:

Virtual Server Preparation:

    192.168.10.143    logstash  shipper        日志数据生产端    192.168.10.144   logstash indexer  kafka    日志消费端将日志写入elasticsearch集群    192.168.10.145    elasticsearch-node1  kibana3   kibana展显elasticsearch中的数据    192.168.10.146   elasticsearch-node2

1. Three hosts to install JDK 1.7 recommended Oracle JDK 1.7+ version Java-version set Java environment variables, such as

    vim ~/.bashrc    >>    JAVA_HOME=/usr/java/jdk1.7.0_55    PATH=$PATH:/$JAVA_HOME/bin    CLASSPATH=.:$JAVA_HOME/lib    JRE_HOME=$JAVA_HOME/jre    export JAVA_HOME PATH CLASSPATH JRE_HOME     >>    source ~/.bashrc

2. Install Kafka (192.168.10.144) wget http://mirrors.hust.edu.cn/apache/kafka/0.8.1.1/kafka2.9.2-0.8.1.1. Tgz tar zxvf Kafka2.9.2-0.8.1.1.tgz ln–s Kafka_2.9.2-0.8.1.1/usr/local/kafka vim/usr/local/kafka/config/ Server.properties broker.id=10-144 host.name=kafka-10-144 echo "192.168.10.144 kafka-10-144" >>/etc/hosts Note Kafka start depending on the lazy zookeeper, need to install Zookeeper-server cdh5.2 Source, configure the Yum source can be yum install zookeeper-server–y vim/etc/zookeeper/ Conf/zoo.cfg datadir=/data/zookeeper Configure zookeeper data storage path start zookeeper and Kafka/etc/init.d/zookeeper-server start nohup/ Usr/local/kafka/bin/kafka-server-start.sh/usr/local/kafka/config/server.properties &

3. Installing Elasticsearch (192.168.10.145)

    wget https://download.elasticsearch.org/elasticsearch/elasticsearch/elasticsearch-1.4.1.tar.gz     elasticsearch解压即可使用非常方便，接下来我们看一下效果，首先启动ES服务，切换到elasticsearch目录，运行bin下的elasticsearch    tar zxvf elasticsearch-1.4.1.tar.gz    ln –s elasticsearch-1.4.1 /usr/local/es    cd  /usr/local/es/    vim config/elasticsearch.yml  添加如下配置，要不然kibana调es时会报错（es 1.4 和kibana 3.1.2版本的问题）    cluster.name: elasticsearch   打开前面注释    node.name: "G1-logs-es02"  打开前面注释，根据主机名配置，做集群    http.cors.enabled: true    http.cors.allow-origin: "*"    nohup bin/elasticsearch &    访问默认的9200端口    curl -X GET http://localhost:9200        Elasticsearch（192.168.10.146）的安装同上：

4. Install Logstash (192.168.10.143, 192.168.10.144) both the production and consumer ends are installed here for reference http://blog.csdn.net/xzwdev/article/ details/41278033 Https://github.com/joekiller/logstash-kafka git clone https://github.com/joekiller/logstash-kafka

                    cd  /usr/local/src/logstash-kafka                    make tarball    编译kakfa支持logstash 时间比较长，大概两个小时                    会在/usr/local/src/logstash-kafka/build/ 目录生成logstash-1.4.2.tar.gz 文件，                logstash-1.4.2.tar.gz文件后面会在日志生产端和消费日志传输用，

Production log end configuration and startup (192.168.10.143)

配置收集haproxy日志示例：                tar zxvf logstash-1.4.2.tar.gz                ln –s logstash-1.4.2 /usr/local/logstash                vim /usr/local/logstash/conf/ logstash_shipper_haproxy.conf                input{                    file{            path => "/data/application/haproxy-1.4.18/logs/haproxy.log"   指定所收集的日志文件路径            type => "haproxylog"                                      日志所属业务服务名称        }    }    output{        kafka{        broker_list => "192.168.10.144:9092"    kafka服务地址        topic_id => "logstash-haproxylog"     标识消费端取日志的id        }    }                    启动生产端日志收集服务                nohup  /usr/local/logstash/bin/logstash –f /usr/local/logstash/conf/ logstash_shipper_haproxy.conf &

Consumer-Side Server configuration:

    tar zxvf logstash-1.4.2.tar.gz    ln –s logstash-1.4.2 /usr/local/logstash    vim /usr/local/logstash/consumer_conf/logstash_haproxylog_es.conf    input{        kafka{            zk_connect => "192.168.10.144:2181"            group_id  => ‘logstash-haproxylog‘            topic_id  => ‘logstash-haproxylog‘        }    }    output{        elasticsearch{        host => "192.168.10.145"        port => "9300"        index => "haproxy-5-13-%{+YYYY.MM.dd}"        }    }

Start consumer Services

    nohup /usr/local/logstash/bin/logstash –f /usr/local/logstash/consumer_conf/logstash_haproxylog_es.conf  &    

Optimization Supplement:

    1,es in the establishment of different business index, need to do if judgment, here is configured in the Logstash consumer, such as: input{kafka{zk_connect = "192.168.35.130:2            181 "group_id =" g1.api.test.com "topic_id = g1.api.test.com '} kafka{ Zk_connect = "192.168.35.130:2181" group_id = "go.clientinner.test.com" topic_id = "Go.clientinner.test.com"} kafka{zk_connect = "192.168.35.130:2181" group_id =&G T "Api7.mobile.test.com_app" topic_id = "Api7.mobile.test.com_app"}} filter {Ruby {init = "@kname = [' Time ', ' uid ', ' IP ', ' uname ', ' stime ', ' etime ', ' exec_time ', ' url ', ' ua ', ' module ', ' Response_sta            Tus ', ' http_status ', ' query_string '] "code =" Event.append (hash[@kname. zip (event[' message '].split (' | ')]) "            } mutate {convert = = ["Exec_time", "float"]} geoip { Database = "/data/application/logstash/patterns/geolitecity.dat "Source =" IP "Fields" and "Country_name", "City        _name "]} useragent {Source =" ua "target =" useragent "} } output{if [type] = = "Go.clientinner.test.com" {elasticsearch{template = = "/u Sr/local/logstash/conf/logstash.json "Template_overwrite" = true # modifies the URL of the non-participle property, host = "192.168.35.131" port = "9300" index = "go.clientinner.test.com-%{+yyyy. MM.DD} "}} else if [type] = =" G1.api.test.com "{elasticsearch{template ="/u "            Sr/local/logstash/conf/logstash.json "Template_overwrite" = True host = "192.168.35.131" Port = "9300" index "=" g1.api.test.com-%{+yyyy. MM.DD} "}}else if [type] = =" Api7.mobile.test.com_app "{Elasticsearch{template = "/usr/local/logstash/conf/logstash.json" Template_overwrite = True Host = "192.168.35.131" port = "9300" index = "api7.mobile.test.com_app-%{+yyyy. MM.DD} "}}}

2,logstash to ES data, the daily establishment of the time index, the default will be in UTC time, is 8 in the morning to establish, resulting in the current log data into yesterday's index, the following changes need to be made:

    修改logstash/lib/logstash/event.rb 可以解决这个问题    第226行    .withZone(org.joda.time.DateTimeZone::UTC)    修改为    .withZone(org.joda.time.DateTimeZone.getDefault())

5 Installing Kibana (192.168.10.145)

    logstash的最新版已经内置kibana，你也可以单独部署kibana。kibana3是纯粹JavaScript+html的客户端，所以可以部署到任意http服务器上。    wget http://download.elasticsearch.org/kibana/kibana/kibana-latest.zip    unzip kibana-latest.zip    cp -r  kibana-latest /var/www/html    可以修改config.js来配置elasticsearch的地址和索引。    修改以下行。    elasticsearch: "http://192.168.10.145:9200",

6, the final realization such as:

7, log system maintenance:

    1,elsasticsearch cluster extension Here is the main point to add new ES node es elsasticsearch installation reference above, before adding a new node, you need to execute the following command 1) First pause the cluster's Shard auto-equalization:  Curl-xput http://192.168.35.131:9200/_cluster/settings-d ' {"transient" on the master node: {"Cluster.routing.allocation.enable": "None"} ' closes other nodes and Master nodes: Curl-xpost http://192.168.35.132:9200/_cluster/nodes/_local/_shutdown curl-xpost http ://192.168.35.131:9200/_cluster/nodes/_local/_shutdown 2) Start Master node, and other slave nodes, 3) Add new node, start and configure reference other slave node 2,kafka+zookeeper cluster Extension: Installation package: Kafka_2.9.2-0.8.1.1.tgz elasticsearch-1.4.1.tar.gz Kafka+zookeeper is configured as follows: Cat Kafka/config/server.propert IES main configuration broker.id=35125 host.name=192.168.35.125 advertised.host.name=192.168.35.125 Log.dirs=/data/kafka-lo    GS zookeeper.connect=192.168.35.130:2181,192.168.35.124:2181,192.168.35.125:2181 CAT/ETC/ZOOKEEPER/CONF/ZOO.CFG Datadir=/data/zookeeper node Information storage Directory clientport=2181 # zookeeper cluster server.35130=g1-logs-kafka:2888:3888 s Erver.35124=bj03-bi-pro-tom01:2888:3888 server.35125=bj03-bi-pro-tom02:2888:3888 start Service Chown zookeeper.zookeeper/data/zookeeper-r /etc/init.d/zookeeper-server Init needs to initialize/data/zookeeper directory on first boot Echo "35130" >/data/zookeeper/myid Chown Zookee Per.zookeeper/data/zookeeper-r/etc/init.d/zookeeper-server start starts zookeeper Nohup first./bin/kafka-server-start.sh Config/server.properties >/data/kafka-logs/kafka.log & Restart Kafka

8,kibana Login Authentication Installation configuration:

    应用说明：        Nginx:  记录日志,做es的反向代理      Nodejs: 跑 kibana-authentication-proxy    Kibana: 把原来kibana目录链接到kibana-authentication-proxy 下    kibana-authentication-proxy：用户认证和代理请求es，

8.1,nginx Installation Configuration #wget http://nginx.org/download/nginx-1.2.9.tar.gz # yum-y install zlib zlib-devel OpenSSL Openssl--devel pcre pcre-devel #tar zxvf nginx-1.2.9.tar.gz # cd nginx-1.2.9 #./configure--prefix=/usr/local/nginx #make && make Install

    The configuration is as follows: #cat/usr/local/nginx/conf/nginx.conf user web;    Worker_processes 4;    Error_log Logs/error.log Info;    PID Logs/nginx.pid;    Events {Worker_connections 1024;    Use Epoll;    } http {include mime.types;    Default_type Application/octet-stream; Log_format Main ' $remote _addr-$remote _user [$time _local] "$request" "$status $body _bytes_sent" $http _refe    RER "'" $http _user_agent "" $http _x_forwarded_for ";    Access_log Logs/access.log Main;    Sendfile on;    #tcp_nopush on;    Keepalive_timeout 65;        Upstream Kianaca {server 192.168.35.131:9200 fail_timeout=30s;        Server 192.168.35.132:9200 fail_timeout=30s;        Server 192.168.35.125:9200 fail_timeout=30s;        } server {listen 8080;        server_name 192.168.35.131;        Location/{root/var/www/html/kibana-proxy;        Index index.html index.htm; Proxy_next_upstream Error TimeOut Invalid_header http_500 http_503 http_404;          Proxy_pass Http://kianaca;          Proxy_set_header Host lashou.log.com;          Proxy_set_header X-real-ip $remote _addr;        Proxy_set_header x-forwarded-for $proxy _add_x_forwarded_for;        } error_page 502 503 504/50x.html;        Location =/50x.html {root/var/www/html/kibana-proxy;    }}} #/usr/local/nginx/sbin/nginx-t #/usr/local/nginx/sbin/nginx

8.2 Installing Kibana-authentication-proxy

    #cd /var/www/html/    #git clone https://github.com/wangganyu188/kibana-authentication-proxy.git    #mv kibana-authentication-proxy kibana-proxy    # cd kibana-proxy    #yum install npm    #npm install express    #git submodule init    #npm install    #node app.js    配置 kibana-proxy/config.js    可能有如下参数需要调整:    es_host      #这里是nginx地址    es_port      #nginx的8080    listen_port      #node的监听端口, 9201    listen_host  #node的绑定IP, 可以0.0.0.0    cas_server_url  

8.3 Request Path

    node(9201) <=> nginx(8080) <=> es(9200)

Lagstash + Elasticsearch + kibana 3 + Kafka Log Management System Deployment 02