Redis distributed log queue for the Javaweb project architecture

architecture, distributed, log queue, the title itself is looking at bluffing, in fact, is a log collection function, but in the middle of a redis to do Message Queuing.

Why does the preface need Message Queuing?

When there are inconsistencies in the speed or stability of "production" and "consumption" in the system, Message Queuing is required as an abstraction layer to bridge the differences between the two sides.

For example, the common message in our system, send text messages, these functions do not need timely response to the queue, asynchronous processing requests, reduce response time.

How is it implemented?

There are a lot of mature JMS Message Queuing middleware products on the market, but based on the current project architecture and deployment situation, we use Redis as a message queue.

Why use Redis?

The list data structure in Redis, with the characteristics of "double-ended queue", and the ability of Redis to have persistent data, is very safe and reliable for Redis to implement distributed queues.

It is similar to "Queue" in JMS, except that functionality and reliability (transactional) are not strictly JMS. The high performance and "convenient" distributed design (replicas,sharding) of the Redis itself provides a good basis for implementing a "distributed queue".

Provider side

The project uses the third-party Redis plugin Spring-data-redis, not clear how to use the Google or Baidu itself.

Redis.properties:

#redis 配置中心 redis.host=192.168.1.180redis.port=6379redis.password=123456redis.maxIdle=100 redis.maxActive=300 redis.maxWait=1000 redis.testOnBorrow=true redis.timeout=100000

Redis configuration:

    <!-- redis 配置 -->    <bean id="jedisPoolConfig" class="redis.clients.jedis.JedisPoolConfig" />    <bean id="jedisConnectionFactory"        class="org.springframework.data.redis.connection.jedis.JedisConnectionFactory">        <property name="hostName" value="${redis.host}" />        <property name="port" value="${redis.port}" />        <property name="password" value="${redis.password}" />        <property name="timeout" value="${redis.timeout}" />        <property name="poolConfig" ref="jedisPoolConfig" />        <property name="usePool" value="true" />    </bean>    <bean id="redisTemplate"  class="org.springframework.data.redis.core.StringRedisTemplate">        <property name="connectionFactory" ref="jedisConnectionFactory" />    </bean>

Slice log configuration (pseudo code):

/** * 系统日志，切面处理类 * 创建者 小柒2012 * 创建时间 2018年1月15日 */@Component@Scope@Aspectpublic class SysLogAspect {    @Autowired    private RedisTemplate<String, String> redisTemplate;    //注解是基于swagger的API，也可以自行定义    @Pointcut("@annotation(io.swagger.annotations.ApiOperation)")    public void logPointCut() {     }    @Around("logPointCut()")    public Object around(ProceedingJoinPoint point) throws Throwable {        Object result = point.proceed();        //把日志消息写入itstyle_log频道        redisTemplate.convertAndSend("itstyle_log","日志数据，自行处理");        return result;    }}

Consumer-side

Redis configuration:

    <!--redis configuration-<bean id= "Jedispoolconfig" class= "Redis.clients.jedis.JedisPoolConfig"/> <bean        Id= "Jedisconnectionfactory" class= "Org.springframework.data.redis.connection.jedis.JedisConnectionFactory" >        <property name= "HostName" value= "${redis.host}"/> <property name= "port" value= "${redis.port}"/> <property name= "Password" value= "${redis.password}"/> <property name= "Timeout" value= "${redis.timeo UT} "/> <property name=" poolconfig "ref=" Jedispoolconfig "/> <property name=" Usepool "Value=" Tru                      E "/> </bean> <bean id=" redistemplate "class=" Org.springframework.data.redis.core.RedisTemplate "              p:connection-factory-ref= "Jedisconnectionfactory" > <property name= "Keyserializer" >          <bean class= "Org.springframework.data.redis.serializer.StringRedisSerializer"/> </property> <property Name= "Hashkeyserializer" > <bean class= "Org.springframework.data.redis.serializer.StringRedisSerializer"/&G          T </property> </bean> <!--monitoring implementation class--<bean id= "listener" class= "com.itstyle.market.common.li Stener. Messagedelegatelistenerimpl "/> <bean id=" Stringredisserializer "class=" Org.springframework.data.redis.serializer.StringRedisSerializer "/> <redis:listener-container connection-factory= "Jedisconnectionfactory" > <!--topic represents the Listening channel, is a regular match is actually the channel you want to subscribe to and <redis:li Stener ref= "Listener" serializer= "Stringredisserializer" method= "Handlelog" topic= "Itstyle_log"/> </redis:  Listener-container>

Monitoring interface:

public interface MessageDelegateListener {    public void handleLog(Serializable message);}

Monitoring implementation:

public class MessageDelegateListenerImpl implements MessageDelegateListener {        @Override        public void handleLog(Serializable message) {            if(message == null){                System.out.println("null");            }else {                //处理日志数据            }        }}

Q&a

• "Question one" why use Redis?
  In fact, there are already some instructions, although there are many stable products, such as may be thought of Kafka, RABBITMQ and ROCKETMQ. However, because the project itself uses Redis as a distributed cache, Redis is selected based on the principle of easy and feasible.

• How does the "Problem II" log data be stored?
  In principle, it is not recommended to store to the relational database, such as MySQL, after all, the number of logs generated is huge, it is recommended to store non-relational database such as Elasticsearch.

• How does the "problem three" slice log collection work?
  The slice log requires the introduction of the Spring-aspects related jar package, and the configuration enables spring to adopt the Cglib proxy <aop:aspectj-autoproxy proxy-target-class= "true"/>.

Open source project source code (Reference): Https://gitee.com/52itstyle/spring-boot-mail

Xiao Qi

Source: https://blog.52itstyle.com

Sharing is happy, but also witnessed the personal growth process, the article is mostly work experience summary and usually learn to accumulate, based on their own cognitive deficiencies are unavoidable, also please correct, common progress.

Redis distributed log queue for the Javaweb project architecture