GRAYLOG2+SYSLOG-NG+MONGODB Building Centralized Management log server--reprint

Original address: http://blog.chinaunix.net/uid-11065483-id-3654882.html

Because the company needs to monitor the line record of QQ, originally used the structure of the light +panabit+splunk to do record. Panabit use is quite comfortable, but when the day of the Splunk log records more than 500MB, Splunk free version can no longer use, which makes me very depressed. So I began to look for other solutions, and finally I found the Graylog2. Graylog2 with MongoDB can quickly retrieve log information, which is useful for scenarios where a large number of logs are recorded every day and occasionally require important information to be retrieved.
GRAYLOG2 installed after the following:

The effect is OK, using Quickfilter to quickly retrieve the relevant message. "10,000 years too long, seize", we say dry! Here's how to install GRAYLOG2.

One. Source Package download

Click (here) to collapse or open

2. Cd/usr/src
4. #mongodb数据库, the primary role here is to store log information
6. wget http://fastdl.mongodb.org/linux/mongodb-linux-i686-2.2.1.tgz
8. #graylog2日志服务器
10. wget http://cloud.github.com/downloads/Graylog2/graylog2-server/graylog2-server-0.9.5.tar.gz
12. #yaml是一种编程语言,
14. wget http://pyyaml.org/download/libyaml/yaml-0.1.4.tar.gz
16. #ruby脚步语言
18. wget ftp://ftp.ruby-lang.org//pub/ruby/1.9/ruby-1.9.2-p0.tar.gz
20. #graylog2-web Interface
22. wget http://cloud.github.com/downloads/Graylog2/graylog2-web-interface/graylog2-web-interface-0.9.5p2.tar.gz
24. #eventlog, installing syslog-ng requires installing EventLog
26. wget http://www.balabit.com/downloads/files/syslog-ng/sources/3.2.4/source/eventlog_0.2.12.tar.gz
28. #syslog-ng Log Server,
30. wget http://www.balabit.com/downloads/files/syslog-ng/open-source-edition/3.3.5/source/syslog-ng_3.3.5.tar.gz
32. #JDK, Java Environment
34. JDK, http://www.oracle.com/technetwork/java/javase/downloads/index.html Open this link to download the appropriate JDK version, Here with Jdk-6u21-linux-i586.bin

You can download it as above, or it can be downloaded to my network, the address is: http://www.kuaipan.cn/file/id_11613978115442246.htm.

Two. Installing dependent packages

Click (here) to collapse or open

1. Yum-y Install gcc* openssl-devel glib2-devel

The operating system recommends using the CentOS 5.7 X64 version. When I use the CentOS 6 version, when I go to the gem install bundler, there will be a file not found, Google's solution is not many, but I found that can use the find after the root directory, and then copy the file to the parent directory can be resolved. I have obsessive-compulsive disorder, so I recommend using the CentOS 5.7 X64 version.

Three. Install and configure the MongoDB database

1. Installing MongoDB

Click (here) to collapse or open

1. Useradd MongoDB
2. mkdir-pv/data/db
3. Chown-r Mongodb:mongodb/data
4. Tar xvf/usr/src/mongodb-linux-i686-2.2.1.tgz-c/usr/local/
5. cd/usr/local/
6. LN-SV mongodb-linux-i686-2.2.1 MongoDB

The basic operation of the MONGO can be found in: http://blog.csdn.net/shirdrn/article/details/7105539

2. Provide service startup scripts for MongoDB and start

Click (here) to collapse or open

2. #vim/etc/init.d/mongod #内容如下
4. ————————————————————————————————
6. #!/bin/bash
8. # Chkconfig:-90 11
10. # DESCRIPTION:MONGODB Server SysV Script
14. . /etc/rc.d/init.d/functions
16. if [-f/etc/sysconfig/mongod]; Then
18. . /etc/sysconfig/mongod
20. Fi
22. Mongod=/usr/local/mongodb/bin/mongod
24. Prog=mongod
26. Lockfile=/data/db/mongod.lock
28. Retval=0
30. options= "--fork--logpath=/data/mongod.log"
34. Start () {
36. Echo-n $ "Starting $prog:"
38. Daemon $mongod $OPTIONS
40. Retval=$?
42. Echo
44. [$RETVAL = 0]
46. Return $RETVAL
48. }
50. Stop () {
52. Echo-n $ "Stopping $prog:"
54. Killproc-p ${lockfile} $mongod
56. Retval=$?
58. Echo
60. [$RETVAL = 0] && rm-f ${lockfile}
62. }
64. Case "$" in
66. Start
68. Start
70. ;;
72. Stop
74. Stop
76. ;;
78. Restart
80. Stop
82. Start
84. ;;
86. Status
88. Status-p ${lockfile} $mongod
90. Retval=$?
92. ;;
94. *)
96. echo $ "Usage: $prog {start|stop|restart|status|}"
98. Exit 1
100. Esac
102. Exit $RETVAL

3. Modify Permissions + boot from

Click (here) to collapse or open

1. chmod A+x/etc/init.d/mongod
2. Chkconfig--add Mongod
3. Chkconfig Mongod on
4. Service Mongod Start

4. Create the required DB instance for GRAYLOG2

Click (here) to collapse or open

1. /usr/local/mongodb/bin/mongo
2. >use Admin
3. >db.adduser (' admin ', ' password ')
4. >use graylog2
5. >db.adduser (' Graylog ', ' Redhat ')
6. >exit

The user name here: "Graylog and Redhat" is behind the GRAYLOG2 connection database that needs to be used. If the subsequent GRAYLOG2 connection MONGO fails, you can use the command: Db.auth ("Graylog", ", Redhat") to add the user to the authentication.

Four. Installing Graylog2-server

1. Installing the JDK Environment

Click (here) to collapse or open

1. Cd/usr/src
2. chmod a+x Jdk-6u21-linux-i586.bin
3. ./jdk-6u21-linux-i586.bin
4. MV jdk1.6.0_21//usr/local/
5. cd/usr/local/
6. LN-SV jdk1.6.0_21 JDK

2. Adding Java Environment variables

Click (here) to collapse or open

1. #vim/etc/profile
2. ——————————————————
3. Java_home=/usr/local/jdk
4. Path= $JAVA _home/bin: $PATH
5. Export Java_home PATH
6. ——————————————————
7. #source/etc/profile finally don't forget source

3. Installing Graylog-server

Click (here) to collapse or open

1. Cd/usr/src
2. TAR-XVF graylog2-server-0.9.5.tar.gz-c/usr/local/
3. Useradd Graylog
4. Chown-r graylog:graylog/usr/local/graylog2-server-0.9.5/
5. Cd/usr/local
6. LN-SV graylog2-server-0.9.5 graylog2
7. Cp/usr/local/graylog2/graylog2.conf.example/etc/graylog2.conf

4. Configure Garylog2-server

Click (here) to collapse or open

1. Vim/etc/graylog2.conf
2. ——————————————————————
3. Syslog_listen_port = 515 #将默认的514端口修改为其他未使用端口, because the 514 port is the default port of Syslog-ng, it is necessary to use Syslog-ng 514 port to receive logs from other servers, and to handle the
4. Syslog_protocol = UDP
5. Mongodb_user = Graylog #访问mongodb的用户名
6. Mongodb_password = Redhat #访问mongodb用户名的密码
7. Mongodb_host = localhost #mogodb的主机
8. Mongodb_database = Graylog2 #存储日志的数据库
9. Mongodb_port = 27017 #访问mongodb的端口, default is 27017

5. Modify the Graylog-server service startup script configuration file

Click (here) to collapse or open

1. Sed-i ' s/java/$JAVA _cmd/'/usr/local/graylog2/bin/graylog2ctl
2. Sed-i ' 2 Ajava_cmd=/usr/local/jdk/bin/java '/usr/local/graylog2/bin/graylog2ctl

6. Add Graylog2-server as a system service and start

Click (here) to collapse or open

1. VIM/ETC/INIT.D/GRAYLOG2 #内容如下
2. —————————————————————————————————
3. #!/bin/bash
4. #
5. # Chkconfig:-83 19
6. # description:graylog2-server SysV Script
7. #
9. Graylog_bin=/usr/local/graylog2/bin
10. CD $GRAYLOG _bin &&./graylog2ctl

7. Modify Permissions + self-booting

Click (here) to collapse or open

1. chmod a+x/etc/init.d/graylog2
2. Chkconfig--add graylog2
3. Chkconfig graylog2 on
4. Service GRAYLOG2 Start

Five. Installing Graylog2-web-interface

1. Install the Ruby scripting language environment

Click (here) to collapse or open

1. Cd/usr/src
2. Tar xvf yaml-0.1.4.tar.gz
3. CD yaml-0.1.4
4. ./configure--prefix=/usr/local/yaml
5. Make
6. Make install
7. Cd/usr/src
8. Tar xvf ruby-1.9.2-p0.tar.gz
9. CD ruby-1.9.2-p0
10. ./configure--prefix=/usr/local/ruby--enable-shared--disable-install-doc--with-opt-dir=/usr/local/yaml
11. Make
12. Make install

2. Modify the environment variables in path

Click (here) to collapse or open

1. Path=......:/usr/local/ruby/bin #在PATH中添加ruby的路径, don't forget to source a bit

3. Installing Graylog2-web-interface

Click (here) to collapse or open

1. Cd/usr/src
2. Tar xvf graylog2-web-interface-0.9.5p2.tar.gz-c/usr/local/
3. cd/usr/local/
4. Chown-r graylog:graylog/usr/local/graylog2-web-interface-0.9.5p2/
5. LN-SV Graylog2-web-interface-0.9.5p2/graylog2-web-interface
6. CD Graylog2-web-interface
7. Gem Install bundler #需要连网
8. Bundle Install #需要联网

4. Configure Graylog2-web-interface

Click (here) to collapse or open

1. Vim/usr/local/graylog2-web-interface/config/mongoid.yml #内容如下
2. —————————————————————————————————————————————————————————————————
3. Production
4. Host:localhost #mongodb的主机
5. port:27017 #mongodb的端口号
6. Username:graylog #mongodb的用户名
7. Password:redhat #mongodb的用户名密码
8. DATABASE:GRAYLOG2 #mongodb的数据库名

5. Configure Log Cutting

Click (here) to collapse or open

1. Cat >/etc/logrotate.d/graylog2-web-interface <<eof
2. /usr/local/graylog2-web-interface/log/*log {
3. size=512m
4. Rotate 90
5. Copytruncate
6. Delaycompress
7. Compress
8. Notifempty
9. Missingok}
10. Eof

6. Start Graylog2-web-interface

Click (here) to collapse or open

1. Cd/usr/local/graylog2-web-interface &&/script/rails server-e Production &

Here I did not learn to do a service script, but to run directly to the machine in the/etc/profile add tmout=0, and then enter the above sentence, it is over. I like the information that keeps scrolling on the screen, it's very technical.

7. Accessing the GRAYLOG2 Web interface
Opening the browser input-->http://server ip:3000<--opens the GRAYLOG2 Web interface. First use basically just add an admin account to be OK.

Six. Install and configure the Syslog-ng service

1. Installing EventLog

Click (here) to collapse or open

1. cd/usr/src/
2. Tar xvf eventlog_0.2.12.tar.gz
3. CD eventlog-0.2.12
4. ./configure--prefix=/usr/local/eventlog
5. Make
6. Make install

2. Installing Syslog-ng

Click (here) to collapse or open

1. Cd/usr/src
2. Tar xvf syslog-ng_3.3.5.tar.gz
3. CD syslog-ng-3.3.5
4. Export Pkg_config_path=/usr/local/eventlog/lib/pkgconfig
5. ./configure--prefix=/usr/local/syslog-ng
6. Make
7. Make install

3. Configure Syslog-ng

Click (here) to collapse or open

1. Cat >/usr/local/syslog-ng/etc/syslog-ng.conf <<eof
2. @version: 3.3
3. @include "Scl.conf"
4. SOURCE S_local {
5. System ();
6. Internal ();
7. };
8. #设置日志的来源为本机udp的514端口
9. SOURCE S_network {UDP (IP (0.0.0.0) port (514));};
11. #将日志发送到本机的udp515端口进行处理, Port 515 is the port of Graylog2-server,
12. Destination d_local {UDP ("127.0.0.1" Port (515));};
13. Log {
14. Source (s_local);
15. Source (s_network);
16. Destination (d_local);
17. };
18. Eof

4. Add As System service

Click (here) to collapse or open

2. Vim/etc/init.d/syslog-ng #内容如下
4. ____________
6. #!/bin/bash
8. #
10. # Chkconfig:-60 27
12. # description:syslog-ng SysV script.
14. . /etc/rc.d/init.d/functions
18. Syslog_ng=/usr/local/syslog-ng/sbin/syslog-ng
20. Prog=syslog-ng
22. Pidfile=/usr/local/syslog-ng/var/syslog-ng.pid
24. Lockfile=/usr/local/syslog-ng/var/syslog-ng.lock
26. Retval=0
28. STOP_TIMEOUT=${STOP_TIMEOUT-10}
32. Start () {
34. Echo-n $ "Starting $prog:"
36. Daemon--pidfile= $pidfile $syslog _ng $OPTIONS
38. Retval=$?
40. Echo
42. [$RETVAL = 0] && Touch ${lockfile}
44. Return $RETVAL
46. }
50. Stop () {
52. Echo-n $ "Stopping $prog:"
54. Killproc-p $pidfile-D $STOP _timeout $syslog _ng
56. Retval=$?
58. Echo
60. [$RETVAL = 0] && rm-f $lockfile $pidfile
62. }
66. Case "$" in
68. Start
70. Start
72. ;;
74. Stop
76. Stop
78. ;;
80. Status
82. Status-p $pidfile $syslog _ng
84. Retval=$?
86. ;;
88. Restart
90. Stop
92. Start
94. ;;
96. *)
98. echo $ "Usage: $prog {start|stop|restart|status}"
100. retval=2
102. Esac
104. Exit $RETVAL

5. Boot from boot

Click (here) to collapse or open

1. chmod a+x/etc/init.d/syslog-ng
2. Killall syslogd
3. Chkconfig--add Syslog-ng
4. Chkconfig Syslog-ng on
5. Service Syslog-ng Start

At this point, the GRAYLOG2 configuration is complete. Everyone put the syslog to Graylog2 on the spit on it. If there is a failure or error in the installation of the place welcome message.

Finally, to thank http://ant595.blog.51cto.com/blog/5074217/1081094 's blog, I just after the experiment, and then broadcast. It also indicates my operation.

GRAYLOG2+SYSLOG-NG+MONGODB Building Centralized Management log server--reprint