Several problems in building a log system using Fluency + MongoDB

Source: Internet
Author: User
Tags fluentd

1. the following error occurs during running,

Error = "Unknown output plugin 'mongo '. run'gem search-rd fluent-In in' to find plugins"
Search for mongodb plug-ins
Gem search-rd fluent-plugin-mongo
Expected result
* ** Remote gems ***


Fluent-plugin-mongo (0.6.7)
Author: Masahiro Nakagawa
Homepage: https://github.com/fluent/fluent-plugin-mongo


MongoDB plugin for Fluent event collector
Install this plugin
Gem install fluent-plugin-mongo


2. The following prompt may be displayed during running:
** Notice: C extension not loaded. This is required for optimum MongoDB Ruby driver performance.
You can install the extension as follows:
Gem install bson_ext


If you continue to receive this message after installing, make sure that
Bson_ext gem is in your load path and that the bson_ext and mongo gems are of the same version.
Run the gem install bson_ext command as prompted.

3. flunt. conf

<Source>

Type tail
Path/home/aircom/ mongodb-linux-i686-2.0.6/bin/logs/1001.log
Format/^ *(? <Message>. *) $/
# Format/^ (? <Time> [^] * [^] * [^] *) (? <Host> [^] *) (? <Ident> [a-zA-Z0-9 _ \/\. \-] *) (? :\[(? <Pid> [0-9] +) \])? [^ \:] * \: * (? <Message>. *) $/
# Time_format % B % d % H: % M: % S
Tag mongo. apache
</Source>
<Match mongo. **>
Type mongo
Database apache
Collection fmongo
Host Server202
Port 10001
Flush_interval 10 s
</Match>

4. login primary node run command

PRIMARY> db. fmongo. find (). sort ({"_ id":-1 })
{"_ Id": ObjectId ("5062d3700ce0ea43fc000387"), "message": "Wed Sep 26 18:05:28 [initandlisten] connection accepted from 192.168.19.201: 34114 #3204", "time ": ISODate ("2012-09-26T10: 05: 28Z ")}
{"_ Id": ObjectId ("5062d3700ce0ea43fc000386"), "message": "Wed Sep 26 18:05:28 [conn3201] end connection 192.168.19.201: 34108", "time ": ISODate ("2012-09-26T10: 05: 28Z ")}
{"_ Id": ObjectId ("5062d3700ce0ea43fc000385"), "message": "Wed Sep 26 18:05:28 [initandlisten] connection accepted from 192.168.19.201: 34113 #3203", "time ": ISODate ("2012-09-26T10: 05: 28Z ")}
{"_ Id": ObjectId ("5062d3700ce0ea43fc000384"), "message": "Wed Sep 26 18:05:28 [conn3200] end connection 192.168.19.201: 34107 ",

5. startup logs when fluentd is configured properly

Fluentd-c fluent. conf-vv &
[1] 15847
[Root @ Server202 fluentd-0.10.6] #2012-09-26 16:28:26 + 0800: fluent/supervisor. rb: 153: supervise: starting fluentd-0.10.25
16:28:26 + 0800: fluent/supervisor. rb: 235: read_config: reading config file path = "fluent. conf"
16:28:26 + 0800: fluent/plugin. rb: 85: register_impl: registered buffer plugin 'file'
16:28:26 + 0800: fluent/plugin. rb: 85: register_impl: registered buffer plugin 'memory'
16:28:26 + 0800: fluent/plugin. rb: 85: register_impl: registered input plugin 'exec'
16:28:26 + 0800: fluent/plugin. rb: 85: register_impl: registered input plugin 'forward'
16:28:26 + 0800: fluent/plugin. rb: 85: register_impl: registered input plugin 'gc _ stat'
16:28:26 + 0800: fluent/plugin. rb: 85: register_impl: registered input plugin 'http'
16:28:26 + 0800: fluent/plugin. rb: 85: register_impl: registered input plugin 'object _ space'
16:28:26 + 0800: fluent/plugin. rb: 85: register_impl: registered input plugin 'status'
16:28:26 + 0800: fluent/plugin. rb: 85: register_impl: registered input plugin 'tcp'
16:28:26 + 0800: fluent/plugin. rb: 85: register_impl: registered input plugin 'unix'
16:28:26 + 0800: fluent/plugin. rb: 85: register_impl: registered input plugin 'syslog'
16:28:26 + 0800: fluent/plugin. rb: 85: register_impl: registered input plugin 'tail'
16:28:26 + 0800: fluent/plugin. rb: 85: register_impl: registered output plugin 'copy'
16:28:26 + 0800: fluent/plugin. rb: 85: register_impl: registered output plugin 'exec'
16:28:26 + 0800: fluent/plugin. rb: 85: register_impl: registered output plugin 'exec _ filter'
16:28:26 + 0800: fluent/plugin. rb: 85: register_impl: registered output plugin 'file'
16:28:26 + 0800: fluent/plugin. rb: 85: register_impl: registered output plugin 'forward'
16:28:26 + 0800: fluent/plugin. rb: 85: register_impl: registered output plugin 'null'
16:28:26 + 0800: fluent/plugin. rb: 85: register_impl: registered output plugin 'roundrobin'
16:28:26 + 0800: fluent/plugin. rb: 85: register_impl: registered output plugin 'stdout'
16:28:26 + 0800: fluent/plugin. rb: 85: register_impl: registered output plugin 'tcp'
16:28:26 + 0800: fluent/plugin. rb: 85: register_impl: registered output plugin 'unix'
16:28:26 + 0800: fluent/plugin. rb: 85: register_impl: registered output plugin 'test'
16:28:26 + 0800: fluent/engine. rb: 63: block in configure: adding source type = "forward"
16:28:26 + 0800: fluent/engine. rb: 63: block in configure: adding source type = "http"
16:28:26 + 0800: fluent/engine. rb: 63: block in configure: adding source type = "tail"
16:28:26 + 0800: plugin/in_tail.rb: 49: configure: 'pos _ file path' parameter is not set to a 'tail' source.
16:28:26 + 0800: plugin/in_tail.rb: 50: configure: this parameter is highly recommended to save the position to resume tailing.
16:28:26 + 0800: fluent/engine. rb: 79: block in configure: adding match pattern = "mongo. **" type = "mongo"
16:28:26 + 0800: fluent/plugin. rb: 85: register_impl: registered output plugin 'mongo'
16:28:27 + 0800: plugin/out_0000.rb: 70: configure: Setup mongo configuration: mode = normal
16:28:27 + 0800: fluent/engine. rb: 79: block in configure: adding match pattern = "debug. **" type = "stdout"
16:28:27 + 0800: plugin/in_forward.rb: 60: listen: listening fluent socket on 0.0.0.0: 24224
16:28:27 + 0800: plugin/in_http.rb: 74: start: listening http on 0.0.0.0: 8888
16:28:27 + 0800: plugin/in_tail.rb: 262: initialize: following tail of/home/aircom/mongodb-linux-i686-2.0.6/bin/logs/1001.log
16:28:49 + 0800: fluent/parser. rb: 38: call: pattern not match: Wed Sep 26 16:28:49 [FileAllocator] allocating new datafile/home/aircom/mongodb-linux-i686-2.0.6/bin/db/apache. ns, filling with zeroes...
16:28:49 + 0800: fluent/parser. rb: 38: call: pattern not match: Wed Sep 26 16:28:49 [FileAllocator] creating directory/home/aircom/mongodb-linux-i686-2.0.6/bin/db/_ tmp
16:28:49 + 0800: fluent/parser. rb: 38: call: pattern not match: Wed Sep 26 16:28:49 [FileAllocator] allocating new datafile/home/aircom/mongodb-linux-i686-2.0.6/bin/db/apache.0, filling with zeroes...
16:28:50 + 0800: fluent/parser. rb: 38: call: pattern not match: Wed Sep 26 16:28:50 [FileAllocator] allocating new datafile/home/aircom/mongodb-linux-i686-2.0.6/bin/db/apache.1, filling with zeroes...
16:28:50 + 0800: fluent/parser. rb: 38: call: pattern not match: Wed Sep 26 16:28:50 [conn2616] build index done 0 records 0.001 secs

Related Article

Contact Us

The content source of this page is from Internet, which doesn't represent Alibaba Cloud's opinion; products and services mentioned on that page don't have any relationship with Alibaba Cloud. If the content of the page makes you feel confusing, please write us an email, we will handle the problem within 5 days after receiving your email.

If you find any instances of plagiarism from the community, please send an email to: info-contact@alibabacloud.com and provide relevant evidence. A staff member will contact you within 5 working days.

A Free Trial That Lets You Build Big!

Start building with 50+ products and up to 12 months usage for Elastic Compute Service

  • Sales Support

    1 on 1 presale consultation

  • After-Sales Support

    24/7 Technical Support 6 Free Tickets per Quarter Faster Response

  • Alibaba Cloud offers highly flexible support services tailored to meet your exact needs.