Log4net. NOSQL +elasticsearch Implementing logging

Objective:

two days before finding out how to extend Log4net's log format when finding an open source project Log4net.nosql, it extends The Appender implements the output of the log to Elasticsearch inside. Follow-up, found involved in a lot of projects, so intend to study, recorded here.

A brief introduction to the project, details click on the link to the project homepage to view, and finally provide packaging download:

1. Log4net.nosql

A collection of log4net appenders to NoSQL data stores. Currently only ElasticSearch is supported.

Uses Restsharp to communicate with the ElasticSearch REST Api and contains a custom Layout, which makes use of the Restsha RP JSON serializer to convert the loggingevent into JSON.

2.Restsharp

Simple REST and HTTP API Client for. NET

3.Curl

Curl is an open source file transfer that works using URL syntax in the command line mode tools.

4.Elasticsearch

Elasticsearch is a distributed RESTful search engine built for the cloud.

5.Elasticsearch-head

A Web front end for an Elasticsearch cluster

Deployment notes:

I am here to separate ES and log4net into two machines, because ES needs the JDK, I do not want to install on the development machine, so the ES installed on the XP virtual machine. Log4net.nosql is an extended log4net layout that implements the JSON-formatted log output, and then expands the Appender by calling Restsharp to send the data to ES. The configuration file for Log4net is as follows:

  <Appendername= "Elastic1"type= "log4net." NoSql.Appender.ElasticSearchAppender, Log4net. NOSQL ">    <Hostvalue= "192.168.66.90" />    <Portvalue= "9200" />    <Indexvalue= "Logs" />    <DocumentTypevalue= "Apps" />    <Layouttype= "log4net." Nosql.layout.jsonlayout,log4net. NOSQL " />  </Appender>

Before inserting the log, you need to index the ES first, and NoSQL provides a footstep log4net. Nosql-master\scripts\create_indices.bat:

?

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23

curl -XDELETE localhost:9200/logs echo ""curl -XPOST localhost:9200/logs -d ‘{     "_all" : {"enabled" : true},     "settings" : {         "number_of_shards" : 1     },     "mappings" : {         "apps":{             "_timestamp" : {                 "enabled" : true,                 "store" : true             },             "_ttl" : {                 "enabled" : true,                 "store" : true,                 "default" : "5000"             }         }       } }‘

Unfortunately, there are several problems:

1.window does not have curl running environment by default

Workaround: Download a Curl-7.17.0-win32-nossl file and put the Curl.exe in the script directory

2. Under the command line switch directory to the script directory, run Create_indices.bat error.

Workaround: Through the error message can see the beginning of each line of characters as a command, so change to Curl-xpost localhost:9200/logs-d ' {followed by the line to remove} '

3. Run again or error, the general meaning is "cannot parse."

Workaround: Change the After-D and the last single quotation marks to double quotes.

The following are the steps to organize your deployment:

1.XP, installing the JDK configuration java_home environment variables   

2.XP, run two ES instances, and then run Es-head, such as:

Es path:. \elasticsearch-1.1.1\bin\elasticsearch.bat

Head path:. \elasticsearch-head-master\index.html

3.win7, put the downloaded curl.exe into a nosql script directory, modify the bat file to change localhost to the ES server IP address 192.168.66.90, and then run. This creates an logs index in ES, such as:

4. Use vs to open the Log4net.nosql solution. Modify the log4net. The host address in the NoSql.TestHarness.log4net.config file is 192.168.66.90. Run write log to ES, such as:

There are two problems encountered here:

A. I have two machines set up the agent, by default I do not access the 192.168.66.90 machine, so go to Es query log records. How do I debug to see if the write was successful?

Modify the \log4net. Nosql-master\log4net. Nosql\util\elasticsearchclient.cs 38 lines, output the return value:

   // dont care about response as if it fails should don't throw any exceptions                _restclient.executeasync (Request, response =                    {                        Debug.WriteLine (response). ToString ());                    });

B. After the log is successfully written, ES still cannot query the data.

When you create a new logs index, you specify a time-out of 5 seconds, so the computer has 5 seconds to go, and you can't find the data in ES.

Conclusion:

A few days ago in the garden saw the hacker marathon, inside the word "software development sounded tall, but the reality is very simple, all activities can be divided into two categories: build wheels, building blocks." "It was very penetrating, so I started the building process. Each project is to understand the fur, will take time to carefully study the source code. This simple combination can solve the project's distributed logging and query requirements, with less effort. ES is based on the Luceuce search engine, plus word breaker, there are more application scenarios can be used.

Reference Links:

1.theCurl command uses the http://www.cnblogs.com/wangkangluo1/archive/2012/04/17/2453975.html

2.ElasticSearch First Use summary, learning progress together ha ~http://ruby-china.org/topics/15337

3.ElasticSearch Getting Started note http://www.qwolf.com/?p=1387

4. Distributed search Elasticsearch cluster management tools headhttp://blog.csdn.net/laigood/article/details/8193758

5. Use the Restsharp library to consume restful servicehttp://www.cnblogs.com/shanyou/archive/2012/01/27/restsharp.html