GRAYLOG2+SYSLOG-NG+MONGODB Building Centralized Management log server--reprint

Source: Internet
Author: User
Tags mongodb server parent directory syslog graylog

Original address: http://blog.chinaunix.net/uid-11065483-id-3654882.html

Because the company needs to monitor the line record of QQ, originally used the structure of the light +panabit+splunk to do record. Panabit use is quite comfortable, but when the day of the Splunk log records more than 500MB, Splunk free version can no longer use, which makes me very depressed. So I began to look for other solutions, and finally I found the Graylog2. Graylog2 with MongoDB can quickly retrieve log information, which is useful for scenarios where a large number of logs are recorded every day and occasionally require important information to be retrieved.
GRAYLOG2 installed after the following:

The effect is OK, using Quickfilter to quickly retrieve the relevant message. "10,000 years too long, seize", we say dry! Here's how to install GRAYLOG2.

One. Source Package download

Click (here) to collapse or open

  1. Cd/usr/src
  2. #mongodb数据库, the primary role here is to store log information
  3. wget http://fastdl.mongodb.org/linux/mongodb-linux-i686-2.2.1.tgz
  4. #graylog2日志服务器
  5. wget http://cloud.github.com/downloads/Graylog2/graylog2-server/graylog2-server-0.9.5.tar.gz
  6. #yaml是一种编程语言,
  7. wget http://pyyaml.org/download/libyaml/yaml-0.1.4.tar.gz
  8. #ruby脚步语言
  9. wget ftp://ftp.ruby-lang.org//pub/ruby/1.9/ruby-1.9.2-p0.tar.gz
  10. #graylog2-web Interface
  11. wget http://cloud.github.com/downloads/Graylog2/graylog2-web-interface/graylog2-web-interface-0.9.5p2.tar.gz
  12. #eventlog, installing syslog-ng requires installing EventLog
  13. wget http://www.balabit.com/downloads/files/syslog-ng/sources/3.2.4/source/eventlog_0.2.12.tar.gz
  14. #syslog-ng Log Server,
  15. wget http://www.balabit.com/downloads/files/syslog-ng/open-source-edition/3.3.5/source/syslog-ng_3.3.5.tar.gz
  16. #JDK, Java Environment
  17. JDK, http://www.oracle.com/technetwork/java/javase/downloads/index.html Open this link to download the appropriate JDK version, Here with Jdk-6u21-linux-i586.bin

You can download it as above, or it can be downloaded to my network, the address is: http://www.kuaipan.cn/file/id_11613978115442246.htm.

Two. Installing dependent packages

Click (here) to collapse or open

    1. Yum-y Install gcc* openssl-devel glib2-devel

The operating system recommends using the CentOS 5.7 X64 version. When I use the CentOS 6 version, when I go to the gem install bundler, there will be a file not found, Google's solution is not many, but I found that can use the find after the root directory, and then copy the file to the parent directory can be resolved. I have obsessive-compulsive disorder, so I recommend using the CentOS 5.7 X64 version.

Three. Install and configure the MongoDB database

1. Installing MongoDB

Click (here) to collapse or open

    1. Useradd MongoDB
    2. mkdir-pv/data/db
    3. Chown-r Mongodb:mongodb/data
    4. Tar xvf/usr/src/mongodb-linux-i686-2.2.1.tgz-c/usr/local/
    5. cd/usr/local/
    6. LN-SV mongodb-linux-i686-2.2.1 MongoDB

The basic operation of the MONGO can be found in: http://blog.csdn.net/shirdrn/article/details/7105539

2. Provide service startup scripts for MongoDB and start

Click (here) to collapse or open

  1. #vim/etc/init.d/mongod #内容如下
  2. ————————————————————————————————
  3. #!/bin/bash
  4. # Chkconfig:-90 11
  5. # DESCRIPTION:MONGODB Server SysV Script
  6. . /etc/rc.d/init.d/functions
  7. if [-f/etc/sysconfig/mongod]; Then
  8. . /etc/sysconfig/mongod
  9. Fi
  10. Mongod=/usr/local/mongodb/bin/mongod
  11. Prog=mongod
  12. Lockfile=/data/db/mongod.lock
  13. Retval=0
  14. options= "--fork--logpath=/data/mongod.log"
  15. Start () {
  16. Echo-n $ "Starting $prog:"
  17. Daemon $mongod $OPTIONS
  18. Retval=$?
  19. Echo
  20. [$RETVAL = 0]
  21. Return $RETVAL
  22. }
  23. Stop () {
  24. Echo-n $ "Stopping $prog:"
  25. Killproc-p ${lockfile} $mongod
  26. Retval=$?
  27. Echo
  28. [$RETVAL = 0] && rm-f ${lockfile}
  29. }
  30. Case "$" in
  31. Start
  32. Start
  33. ;;
  34. Stop
  35. Stop
  36. ;;
  37. Restart
  38. Stop
  39. Start
  40. ;;
  41. Status
  42. Status-p ${lockfile} $mongod
  43. Retval=$?
  44. ;;
  45. *)
  46. echo $ "Usage: $prog {start|stop|restart|status|}"
  47. Exit 1
  48. Esac
  49. Exit $RETVAL


3. Modify Permissions + boot from

Click (here) to collapse or open

    1. chmod A+x/etc/init.d/mongod
    2. Chkconfig--add Mongod
    3. Chkconfig Mongod on
    4. Service Mongod Start


4. Create the required DB instance for GRAYLOG2

Click (here) to collapse or open

    1. /usr/local/mongodb/bin/mongo
    2. >use Admin
    3. >db.adduser (' admin ', ' password ')
    4. >use graylog2
    5. >db.adduser (' Graylog ', ' Redhat ')
    6. >exit

The user name here: "Graylog and Redhat" is behind the GRAYLOG2 connection database that needs to be used. If the subsequent GRAYLOG2 connection MONGO fails, you can use the command: Db.auth ("Graylog", ", Redhat") to add the user to the authentication.

Four. Installing Graylog2-server

1. Installing the JDK Environment

Click (here) to collapse or open

    1. Cd/usr/src
    2. chmod a+x Jdk-6u21-linux-i586.bin
    3. ./jdk-6u21-linux-i586.bin
    4. MV jdk1.6.0_21//usr/local/
    5. cd/usr/local/
    6. LN-SV jdk1.6.0_21 JDK


2. Adding Java Environment variables

Click (here) to collapse or open

    1. #vim/etc/profile
    2. ——————————————————
    3. Java_home=/usr/local/jdk
    4. Path= $JAVA _home/bin: $PATH
    5. Export Java_home PATH
    6. ——————————————————
    7. #source/etc/profile finally don't forget source


3. Installing Graylog-server

Click (here) to collapse or open

    1. Cd/usr/src
    2. TAR-XVF graylog2-server-0.9.5.tar.gz-c/usr/local/
    3. Useradd Graylog
    4. Chown-r graylog:graylog/usr/local/graylog2-server-0.9.5/
    5. Cd/usr/local
    6. LN-SV graylog2-server-0.9.5 graylog2
    7. Cp/usr/local/graylog2/graylog2.conf.example/etc/graylog2.conf


4. Configure Garylog2-server

Click (here) to collapse or open

    1. Vim/etc/graylog2.conf
    2. ——————————————————————
    3. Syslog_listen_port = 515 #将默认的514端口修改为其他未使用端口, because the 514 port is the default port of Syslog-ng, it is necessary to use Syslog-ng 514 port to receive logs from other servers, and to handle the
    4. Syslog_protocol = UDP
    5. Mongodb_user = Graylog #访问mongodb的用户名
    6. Mongodb_password = Redhat #访问mongodb用户名的密码
    7. Mongodb_host = localhost #mogodb的主机
    8. Mongodb_database = Graylog2 #存储日志的数据库
    9. Mongodb_port = 27017 #访问mongodb的端口, default is 27017


5. Modify the Graylog-server service startup script configuration file

Click (here) to collapse or open

    1. Sed-i ' s/java/$JAVA _cmd/'/usr/local/graylog2/bin/graylog2ctl
    2. Sed-i ' 2 Ajava_cmd=/usr/local/jdk/bin/java '/usr/local/graylog2/bin/graylog2ctl


6. Add Graylog2-server as a system service and start

Click (here) to collapse or open

    1. VIM/ETC/INIT.D/GRAYLOG2 #内容如下
    2. —————————————————————————————————
    3. #!/bin/bash
    4. #
    5. # Chkconfig:-83 19
    6. # description:graylog2-server SysV Script
    7. #
    8. Graylog_bin=/usr/local/graylog2/bin
    9. CD $GRAYLOG _bin &&./graylog2ctl


7. Modify Permissions + self-booting

Click (here) to collapse or open

    1. chmod a+x/etc/init.d/graylog2
    2. Chkconfig--add graylog2
    3. Chkconfig graylog2 on
    4. Service GRAYLOG2 Start


Five. Installing Graylog2-web-interface

1. Install the Ruby scripting language environment

Click (here) to collapse or open

    1. Cd/usr/src
    2. Tar xvf yaml-0.1.4.tar.gz
    3. CD yaml-0.1.4
    4. ./configure--prefix=/usr/local/yaml
    5. Make
    6. Make install
    7. Cd/usr/src
    8. Tar xvf ruby-1.9.2-p0.tar.gz
    9. CD ruby-1.9.2-p0
    10. ./configure--prefix=/usr/local/ruby--enable-shared--disable-install-doc--with-opt-dir=/usr/local/yaml
    11. Make
    12. Make install


2. Modify the environment variables in path

Click (here) to collapse or open

    1. Path=......:/usr/local/ruby/bin #在PATH中添加ruby的路径, don't forget to source a bit


3. Installing Graylog2-web-interface

Click (here) to collapse or open

    1. Cd/usr/src
    2. Tar xvf graylog2-web-interface-0.9.5p2.tar.gz-c/usr/local/
    3. cd/usr/local/
    4. Chown-r graylog:graylog/usr/local/graylog2-web-interface-0.9.5p2/
    5. LN-SV Graylog2-web-interface-0.9.5p2/graylog2-web-interface
    6. CD Graylog2-web-interface
    7. Gem Install bundler #需要连网
    8. Bundle Install #需要联网


4. Configure Graylog2-web-interface

Click (here) to collapse or open

    1. Vim/usr/local/graylog2-web-interface/config/mongoid.yml #内容如下
    2. —————————————————————————————————————————————————————————————————
    3. Production
    4. Host:localhost #mongodb的主机
    5. port:27017 #mongodb的端口号
    6. Username:graylog #mongodb的用户名
    7. Password:redhat #mongodb的用户名密码
    8. DATABASE:GRAYLOG2 #mongodb的数据库名


5. Configure Log Cutting

Click (here) to collapse or open

    1. Cat >/etc/logrotate.d/graylog2-web-interface <<eof
    2. /usr/local/graylog2-web-interface/log/*log {
    3. size=512m
    4. Rotate 90
    5. Copytruncate
    6. Delaycompress
    7. Compress
    8. Notifempty
    9. Missingok}
    10. Eof


6. Start Graylog2-web-interface

Click (here) to collapse or open

    1. Cd/usr/local/graylog2-web-interface &&/script/rails server-e Production &

Here I did not learn to do a service script, but to run directly to the machine in the/etc/profile add tmout=0, and then enter the above sentence, it is over. I like the information that keeps scrolling on the screen, it's very technical.

7. Accessing the GRAYLOG2 Web interface
Opening the browser input-->http://server ip:3000<--opens the GRAYLOG2 Web interface. First use basically just add an admin account to be OK.

Six. Install and configure the Syslog-ng service

1. Installing EventLog

Click (here) to collapse or open

    1. cd/usr/src/
    2. Tar xvf eventlog_0.2.12.tar.gz
    3. CD eventlog-0.2.12
    4. ./configure--prefix=/usr/local/eventlog
    5. Make
    6. Make install


2. Installing Syslog-ng

Click (here) to collapse or open

    1. Cd/usr/src
    2. Tar xvf syslog-ng_3.3.5.tar.gz
    3. CD syslog-ng-3.3.5
    4. Export Pkg_config_path=/usr/local/eventlog/lib/pkgconfig
    5. ./configure--prefix=/usr/local/syslog-ng
    6. Make
    7. Make install


3. Configure Syslog-ng

Click (here) to collapse or open

    1. Cat >/usr/local/syslog-ng/etc/syslog-ng.conf <<eof
    2. @version: 3.3
    3. @include "Scl.conf"
    4. SOURCE S_local {
    5. System ();
    6. Internal ();
    7. };
    8. #设置日志的来源为本机udp的514端口
    9. SOURCE S_network {UDP (IP (0.0.0.0) port (514));};
    10. #将日志发送到本机的udp515端口进行处理, Port 515 is the port of Graylog2-server,
    11. Destination d_local {UDP ("127.0.0.1" Port (515));};
    12. Log {
    13. Source (s_local);
    14. Source (s_network);
    15. Destination (d_local);
    16. };
    17. Eof


4. Add As System service

Click (here) to collapse or open

  1. Vim/etc/init.d/syslog-ng #内容如下
  2. ____________
  3. #!/bin/bash
  4. #
  5. # Chkconfig:-60 27
  6. # description:syslog-ng SysV script.
  7. . /etc/rc.d/init.d/functions
  8. Syslog_ng=/usr/local/syslog-ng/sbin/syslog-ng
  9. Prog=syslog-ng
  10. Pidfile=/usr/local/syslog-ng/var/syslog-ng.pid
  11. Lockfile=/usr/local/syslog-ng/var/syslog-ng.lock
  12. Retval=0
  13. STOP_TIMEOUT=${STOP_TIMEOUT-10}
  14. Start () {
  15. Echo-n $ "Starting $prog:"
  16. Daemon--pidfile= $pidfile $syslog _ng $OPTIONS
  17. Retval=$?
  18. Echo
  19. [$RETVAL = 0] && Touch ${lockfile}
  20. Return $RETVAL
  21. }
  22. Stop () {
  23. Echo-n $ "Stopping $prog:"
  24. Killproc-p $pidfile-D $STOP _timeout $syslog _ng
  25. Retval=$?
  26. Echo
  27. [$RETVAL = 0] && rm-f $lockfile $pidfile
  28. }
  29. Case "$" in
  30. Start
  31. Start
  32. ;;
  33. Stop
  34. Stop
  35. ;;
  36. Status
  37. Status-p $pidfile $syslog _ng
  38. Retval=$?
  39. ;;
  40. Restart
  41. Stop
  42. Start
  43. ;;
  44. *)
  45. echo $ "Usage: $prog {start|stop|restart|status}"
  46. retval=2
  47. Esac
  48. Exit $RETVAL


5. Boot from boot

Click (here) to collapse or open

    1. chmod a+x/etc/init.d/syslog-ng
    2. Killall syslogd
    3. Chkconfig--add Syslog-ng
    4. Chkconfig Syslog-ng on
    5. Service Syslog-ng Start


At this point, the GRAYLOG2 configuration is complete. Everyone put the syslog to Graylog2 on the spit on it. If there is a failure or error in the installation of the place welcome message.

Finally, to thank http://ant595.blog.51cto.com/blog/5074217/1081094 's blog, I just after the experiment, and then broadcast. It also indicates my operation.

GRAYLOG2+SYSLOG-NG+MONGODB Building Centralized Management log server--reprint

Related Article

Contact Us

The content source of this page is from Internet, which doesn't represent Alibaba Cloud's opinion; products and services mentioned on that page don't have any relationship with Alibaba Cloud. If the content of the page makes you feel confusing, please write us an email, we will handle the problem within 5 days after receiving your email.

If you find any instances of plagiarism from the community, please send an email to: info-contact@alibabacloud.com and provide relevant evidence. A staff member will contact you within 5 working days.

A Free Trial That Lets You Build Big!

Start building with 50+ products and up to 12 months usage for Elastic Compute Service

  • Sales Support

    1 on 1 presale consultation

  • After-Sales Support

    24/7 Technical Support 6 Free Tickets per Quarter Faster Response

  • Alibaba Cloud offers highly flexible support services tailored to meet your exact needs.