Log4net. NoSql + ElasticSearch implement Logging

Preface:

Two days ago, we found an open-source project log4net. NoSql when looking for how to extend the log format of Log4net. It output logs to ElasticSearch by extending the Appender. I found that there are still many projects involved, so I plan to learn and record them here.

For a brief introduction to the project, click the link to View Details on the project homepage, and finally download the package:

1.Log4net. nosql

A collection of log4net Appenders to NoSQL data stores. Currently only ElasticSearch is supported.

Uses RESTSharp to communicate with the ElasticSearch REST Api and contains a custom Layout, which makes use of the RESTSharp JSON Serializer to convert the LoggingEvent into JSON.

2.RestSharp

Simple REST and http api Client for. NET

3.Curl

Curl is an open source file transmission tool that uses the URL syntax in the command line mode.

4.Elasticsearch

Elasticsearch is a distributed RESTful search engine built for the cloud.

5.Elasticsearch-head

A web front end for an Elasticsearch cluster

Deployment description:

Here, I deploy es and log4net separately to two machines. Because es requires jdk, I don't want to install it on the development machine, so I installed es on the xp Virtual Machine. Log4net. nosql extends the Layout of log4net to implement log output in json format, and then extends the Appender by callingRestSharpSend data to es. The Log4net configuration file is as follows:

  <appender name="Elastic1" type="log4net.NoSql.Appender.ElasticSearchAppender, log4net.NoSql">    <Host value="192.168.66.90" />    <Port value="9200" />    <Index value="logs" />    <DocumentType value="apps" />    <layout type="log4net.NoSql.Layout.JsonLayout,log4net.NoSql" />  </appender>

Before inserting logs, you must create an index in es. nosql provides log4net. NoSql-master \ scripts \ create_indices.bat:

curl -XDELETE localhost:9200/logsecho ""curl -XPOST localhost:9200/logs -d '{"_all" : {"enabled" : true},"settings" : {    "number_of_shards" : 1},"mappings" : {    "apps":{        "_timestamp" : {            "enabled" : true,            "store" : true        },        "_ttl" : {        "enabled" : true,        "store" : true,        "default" : "5000"        }    }  }}'

Unfortunately, there are several problems:

1. No curl runtime environment in window by default

Solution: Download a curl-7.17.0-win32-nosslfile and put curl.exe in the script directory.

2. Switch the directory to the script directory under the command line and run create_indices.bat to report an error.

Solution: the error message shows that the starting character of each line is used as a command, so it is changed to curl-XPOST localhost: 9200/logs-d '{remove the line feed after }'

3. re-run or report an error, which means that ''' cannot be parsed.

Solution: Change the single quotation marks behind-d to double quotation marks.

 

The following describes the deployment steps:

1. Install jdk and configure the JAVA_HOME environment variable in xp.

2. xp: run two es instances and then run es-head, for example:

Es path: .. \ elasticsearch-1.1.1 \ bin \ elasticsearch. bat

Head path: .. \ elasticsearch-head-master \ index.html

 

Two problems are encountered here:

A. I have configured a proxy for both machines. By default, I cannot access machines 192.168.66.90, so no log records can be found in ES. How can I debug and check whether the write operation is successful?

Modify row 38 of \ log4net. NoSql-master \ log4net. NoSql \ Util \ ElasticSearchClient. cs and output the returned value:

   //dont care about response as if it fails should not throw any exceptions                _restClient.ExecuteAsync(request, response =>                    {                        Debug.WriteLine(response.ToString());                    });

B. Data is still not found in elasticsearch After logs are successfully written.

When creating a logs index, the time-out period is set to 5 seconds. Therefore, once the computer gets stuck for 5 seconds, no data can be found in es.

Conclusion:

A few days ago, I saw a hacker marathon in the garden. The phrase "Software Development sounds very tall, but it is very simple. There are two types of activities: Building wheels and building blocks ." I felt very incisive, so I started the building block process this time. Every project has knowledge about the source code and will take time to study the source code. This simple combination can solve the distributed log record and Query Needs in the project, get twice the result with half the effort. Elasticsearch is a luceuce-based search engine. With the word segmentation plug-in, elasticsearch can be used in more application scenarios.

Reference link:

1. curl command using http://www.cnblogs.com/wangkangluo1/archive/2012/04/17/2453975.html

 

2. ElasticSearch's first usage summary, learning and improving together ~ Http://ruby-china.org/topics/15337

3. ElasticSearch entry notes http://www.qwolf.com /? P = 1387

4. distributed search elasticsearch cluster management tool headhttp: // blog.csdn.net/laigood/article/details/8193758

5. Use the RestSharp library to consume Restful Servicehttp: // www.cnblogs.com/shanyou/archive/2012/01/27/RestSharp.html