Redis distributed log queue for the Javaweb project architecture

architecture, distributed, log queue, the title itself is looking at bluffing, in fact, is a log collection function, but in the middle of a redis to do Message Queuing.

Why does the preface need Message Queuing?

When there are inconsistencies in the speed or stability of "production" and "consumption" in the system, Message Queuing is required as an abstraction layer to bridge the differences between the two sides.

For example, the common message in our system, send text messages, these functions do not need timely response to the queue, asynchronous processing requests, reduce response time.

How is it implemented?

There are a lot of mature JMS Message Queuing middleware products on the market, but based on the current project architecture and deployment situation, we use Redis as a message queue.

Why use Redis?

The list data structure in Redis, with the characteristics of "double-ended queue", and the ability of Redis to have persistent data, is very safe and reliable for Redis to implement distributed queues.

It is similar to "Queue" in JMS, except that functionality and reliability (transactional) are not strictly JMS. The high performance and "convenient" distributed design (replicas,sharding) of the Redis itself provides a good basis for implementing a "distributed queue".

Provider side

The project uses the third-party Redis plugin Spring-data-redis, not clear how to use the Google or Baidu itself.

Redis.properties:

#redis 配置中心 redis.host=192.168.1.180redis.port=6379redis.password=123456redis.maxIdle=100 redis.maxActive=300 redis.maxWait=1000 redis.testOnBorrow=true redis.timeout=100000

Redis configuration:

    <!--redis configuration --    <beanid="Jedispoolconfig"class="Redis.clients.jedis.JedisPoolConfig" />    <beanid="Jedisconnectionfactory"class="Org.springframework.data.redis.connection.jedis.JedisConnectionFactory">        <propertyname="HostName"value="${redis.host}" />        <propertyname="Port"value="${redis.port}" />        <propertyname="Password"value="${redis.password}" />        <propertyname="Timeout"value="${redis.timeout}" />        <propertyname="Poolconfig"ref="Jedispoolconfig" />        <propertyname="Usepool"value="true" />    </bean>    <beanid="Redistemplate"class="Org.springframework.data.redis.core.StringRedisTemplate">        <propertyname="ConnectionFactory"ref="Jedisconnectionfactory" />    </bean>

Slice log configuration (pseudo code):

/*** System log, slice processing class* Created by Zhang Zhibong* Creation Date January 15, 2018 */@Component@Scope@Aspect Public classSyslogaspect {@Autowired    PrivateRedistemplate<string, string> redistemplate;//Annotations are swagger-based APIs, or they can be self-defined    @Pointcut("@annotation (io.swagger.annotations.ApiOperation)") Public void Logpointcut() {     }@Around("Logpointcut ()") PublicObjectaround(Proceedingjoinpoint point)throwsthrowable {Object result = point.Proceed();//write log messages to Itstyle_log channelRedistemplate.Convertandsend("Itstyle_log","log data, self-processing");returnResult }}

Consumer-side

Redis configuration:

    <!--redis configuration --    <beanid="Jedispoolconfig"class="Redis.clients.jedis.JedisPoolConfig" />    <beanid="Jedisconnectionfactory"class="Org.springframework.data.redis.connection.jedis.JedisConnectionFactory">        <propertyname="HostName"value="${redis.host}" />        <propertyname="Port"value="${redis.port}" />        <propertyname="Password"value="${redis.password}" />        <propertyname="Timeout"value="${redis.timeout}" />        <propertyname="Poolconfig"ref="Jedispoolconfig" />        <propertyname="Usepool"value="true" />    </bean>    <beanid="Redistemplate"class="Org.springframework.data.redis.core.RedisTemplate"  p:connection-factory-ref="Jedisconnectionfactory">          <propertyname="Keyserializer">              <beanclass="Org.springframework.data.redis.serializer.StringRedisSerializer" />          </property>          <propertyname="Hashkeyserializer">              <beanclass="Org.springframework.data.redis.serializer.StringRedisSerializer" />          </property>      </bean>        <!--monitoring implementation class --    <beanid="Listener"class="Com.itstyle.market.common.listener.MessageDelegateListenerImpl"/>    <beanid="Stringredisserializer"class="Org.springframework.data.redis.serializer.StringRedisSerializer" />    <redis:listener-containerconnection-factory="Jedisconnectionfactory">        <!--topic represents the Listening channel, is a regular match is actually the channel you want to subscribe to--        <redis:listenerref="Listener"serializer="Stringredisserializer"method="Handlelog"topic="Itstyle_log"/>    </redis:listener-container> 

Monitoring interface:

publicinterface MessageDelegateListener {    publicvoidhandleLog(Serializable message);}

Monitoring implementation:

publicclassimplements MessageDelegateListener {        @Override        publicvoidhandleLog(Serializable message) {            ifnull){                System.out.println("null");            }else {                //处理日志数据            }        }}

Q&a

• "Question one" why use Redis?
  In fact, there are already some instructions, although there are many stable products, such as may be thought of Kafka, RABBITMQ and ROCKETMQ. However, because the project itself uses Redis as a distributed cache, Redis is selected based on the principle of easy and feasible.

• How does the "Problem II" log data be stored?
  In principle, it is not recommended to store to the relational database, such as MySQL, after all, the number of logs generated is huge, it is recommended to store non-relational database such as Elasticsearch.

• How does the "problem three" slice log collection work?
  The slice log requires the introduction of the Spring-aspects related jar package, and the configuration makes spring adopt the Cglib proxy

Open source project source code (Reference): Https://gitee.com/52itstyle/spring-boot-mail

Redis distributed log queue for the Javaweb project architecture