log4net.redis+logstash+kibana+elasticsearch+redis 實現日誌系統

標籤：style   blog   http   color   java   使用   os   strong   

前端時間寫了個隨筆 log4net.NoSql +Elasticsearch 實現日誌記錄 ，因項目原因需要把日誌根java平台的同事整合採用logstash+kibana+elasticsearch+redis結構實現日誌統計分析，所以需要一個將log4net日誌輸出到redis的組件。沒有找到現成的，就自己動手了。參考了 log4net.NoSql 的代碼。

redis的C#用戶端使用了 ServiceStackRedis，json序列化使用 RestSharp。代碼結構如下：

 

JsonLayout.cs代碼：

  public class JsonLayout : LayoutSkeleton    {        public readonly string HostName;        private readonly ITextTransform _transform;        public string LogType { get; set; }        public string AppName { get; set; }        public JsonLayout()            : base()        {            HostName = Dns.GetHostName();            _transform = TextTransform.Instance;        }        public override string ContentType        {            get { return "application/json"; }        }        public override void ActivateOptions()        {            //nothing to do here        }        public override void Format(TextWriter writer, LoggingEvent loggingEvent)        {            var info = loggingEvent.LocationInformation;            var loggingEventJson = _transform.Serialize(new JsonLogMessage                {                    class_method = info.MethodName,                    class_name = info.ClassName,                    host_ip = HostName,                    line_number = info.LineNumber,                    log_level = loggingEvent.Level.DisplayName,                    log_message = loggingEvent.MessageObject.ToString(),                    exception = BuildExceptionMessage(loggingEvent),                    log_time = loggingEvent.TimeStamp.ToString("yyyy-MM-dd HH:mm:ss"),                    logger_name = loggingEvent.LoggerName,                    run_time = "0", //目前擷取不到                    thread_name = loggingEvent.ThreadName,                    host = HostName,                    log_type = this.LogType,                    app_name = this.AppName,                    path = ""                });            writer.Write(loggingEventJson);        }        private JsonLogException BuildExceptionMessage(LoggingEvent loggingEvent)        {            if (loggingEvent.ExceptionObject == null)                return new JsonLogException { exception_class = "", exception_message = "", exception_stacktrace = "" };            var exception = loggingEvent.ExceptionObject;            return                new JsonLogException                {                    exception_class = exception.Source,                    exception_message = exception.Message,                    exception_stacktrace = exception.StackTrace                };        }    }

View Code

RedisAppender.cs

 public class RedisAppender : AppenderSkeleton    {        private volatile PooledRedisClient _pooledRedisClient;        private readonly object _padlock = new object();        public string Host { get; set; }        public string Port { get; set; }        public string ListId { get; set; }        public string MaxPoolSize { get; set; }        public RedisAppender()        {        }        private PooledRedisClient Client        {            get            {                if (_pooledRedisClient != null) return _pooledRedisClient;                lock (_padlock)                {                    if (_pooledRedisClient == null)                    {                        _pooledRedisClient = new PooledRedisClient(this.Host, int.Parse(this.Port), int.Parse(this.MaxPoolSize));                    }                }                return _pooledRedisClient;            }        }        protected override void Append(LoggingEvent loggingEvent)        {            var sb = new StringBuilder();            var writer = new StringWriter(sb);            base.Layout.Format(writer, loggingEvent);            this.Client.AddItemToListAsync(ListId, writer.ToString());        }    }

View Code

PooledRedisClient.cs

  public class PooledRedisClient    {        private readonly string _baseUri;        private readonly PooledRedisClientManager _clientManager;        /// <summary>        /// 執行個體化串連池用戶端        /// </summary>        /// <param name="host"></param>        /// <param name="port"></param>        /// <param name="maxPoolSize">預設值10</param>        public PooledRedisClient(string host, int port, int maxPoolSize = 10)        {            _baseUri = string.Format("{0}:{1}", host, port);            var config = new RedisClientManagerConfig();            config.MaxReadPoolSize = maxPoolSize;            config.MaxWritePoolSize = maxPoolSize;            _clientManager = new PooledRedisClientManager(new string[] { _baseUri }, new string[] { _baseUri }, config);        }        /// <summary>        /// 非同步記錄        /// </summary>        /// <param name="listId"></param>        /// <param name="log"></param>        public void AddItemToListAsync(string listId, string log)        {            //使用Task任務非同步執行            var task = new Task(() =>            {                try                {                    using (var clinet = _clientManager.GetClient())                    {                        clinet.AddItemToList(listId, log);                    }                }                catch (Exception ex)                {                    Trace.WriteLine(ex.Message);                }            });            task.Start();        }    }

View Code

在RedisClient串連時用了串連池，寫日誌時用Task做了非同步。這種寫法不知道會不會存在問題？！

設定檔

<?xml version="1.0"?><log4net>  <!--Redis日誌記錄-->  <appender name="redisAppender" type="log4net.Redis.Appender.RedisAppender, log4net.Redis">    <Host value="192.168.0.10" />    <Port value="6379" />    <MaxPoolSize value="500" />    <ListId value="logstash" />    <layout type="log4net.Redis.Layout.JsonLayout,log4net.Redis" >      <LogType value="iis_log" />      <AppName value="Demo" />    </layout>  </appender>  <root>    <level value="ALL" />    <appender-ref ref="redisAppender" />  </root></log4net>

View Code

Layout和Appender裡的屬性配置和代碼裡屬性配置名稱一致，Log4net的架構就可以讀取配置資料反射到實體屬性上。

代碼：

http://download.csdn.net/detail/zbl131/7702673

 

參考文章：

1.用Kibana和logstash快速搭建即時日誌查詢、收集與分析系統

http://storysky.blog.51cto.com/628458/1158707/

2.使用ServiceStackRedis連結Redis簡介

http://www.cnblogs.com/daizhj/archive/2011/02/17/1956860.html

3.ServiceStack.Redis的問題與修正

http://blog.csdn.net/susubuhui/article/details/8930417

4.對ServiceStack.Redis的串連池進行容錯移轉改造

http://www.cnblogs.com/smark/archive/2013/05/24/3096488.html