Message Service Based on Netty and RabbitMQ, nettyrabbitmq message

Message Service Based on Netty and RabbitMQ, nettyrabbitmq message

As a high-performance Asynchronous Network Development Framework, Netty can be used as a development framework for various services.

Some time ago, a project involved real-time data collection of hardware devices. Netty was used as the collection service implementation framework, and RabbitMQ was also used as the collection service and communication message queues of various modules, the overall service architecture is shown below:

Extract the code of the Business Code and the actual protocol parsing part to obtain the above simple design diagram. The code is open-source on GitHub, which briefly introduces several key technical points involved in the NettyMQServer collection service:

1. device TCP Message Parsing:

TCP communication is used between NettyMQServer and the Device of the collection Device. For TCP Message Parsing, you can use LengthFieldBasedFrameDecoder (message header and message body) to effectively solve the TCP Message "sticky packet" problem.

The message package parsing diagram is as follows:

 lengthFieldOffset   =  0 lengthFieldLength   =  2 lengthAdjustment    = -2 (= the length of the Length field) initialBytesToStrip =  0 BEFORE DECODE (14 bytes)         AFTER DECODE (14 bytes) +--------+----------------+      +--------+----------------+ | Length | Actual Content |----->| Length | Actual Content | | 0x000E | "HELLO, WORLD" |      | 0x000E | "HELLO, WORLD" | +--------+----------------+      +--------+----------------+

In the code, the message length is stored in 4 bytes and decoded using LengthFieldBasedFrameDecoder (65535,0, 4, netty obtains the message length from the first 4 bytes of the received data, and then obtains a TCP Message packet.

2. send messages to devices:

First, keep the TCP connection when creating the connection:

static final ChannelGroup channels = new DefaultChannelGroup(            GlobalEventExecutor.INSTANCE);    @Override    public void channelActive(ChannelHandlerContext ctx) throws Exception {        // A closed channel will be removed from ChannelGroup automatically        channels.add(ctx.channel());    }

When a Channel is Active (the connection is created), use ChannelGroup to save the Channel connection. When sending messages to a device, you can traverse the Channel group and find the corresponding Channel, send messages to the Channel:

for (io.netty.channel.Channel c : EchoServerHandler.channels) {                            ByteBuf msg = Unpooled.copiedBuffer(message.getBytes());                            c.writeAndFlush(msg);                        }

All connected devices are sent here. When the connection is disconnected, ChannelGroup automatically removes the connection and does not need to be manually managed.

3. Heartbeat Detection

When a Device fails to collect data due to power failure or other reasons, the Netty server needs to know whether the Device is working properly. You can use Netty's IdleStateHandler. The sample code is as follows:

// 3 minutes for read idlech.pipeline().addLast(new IdleStateHandler(3*60,0,0));ch.pipeline().addLast(new HeartBeatHandler());/** * Handler implementation for heart beating. */public class HeartBeatHandler extends ChannelInboundHandlerAdapter{    @Override    public void userEventTriggered(ChannelHandlerContext ctx, Object evt)            throws Exception {        if (evt instanceof IdleStateEvent) {            IdleStateEvent event = (IdleStateEvent) evt;            if (event.state() == IdleState.READER_IDLE) {                // Read timeout                System.out.println("READER_IDLE: read timeout from "+ctx.channel().remoteAddress());                //ctx.disconnect(); //Channel disconnect            }        }    }}

If no data is read for three minutes, a READER_IDLE event is triggered.

4. receive and send RabbitMQ messages

The NettyMQServer uses Spring AMQP for message sending. You only need to configure it in the configuration file for ease of use.

Spring AMQP can also be used for receiving messages from NettyMQServer. However, due to lack of familiarity with Spring-related configurations, RabbitMQ Client Java API is used for more flexible use of MQ:

                    Connection connection = connnectionFactory.newConnection();                    Channel channel = connection.createChannel();                    channel.exchangeDeclare(exchangeName, "direct", true, false, null);                    channel.queueDeclare(queueName, true, false, false, null);                    channel.queueBind(queueName, exchangeName, routeKey);                    // process the message one by one                    channel.basicQos(1);                    QueueingConsumer queueingConsumer = new QueueingConsumer(channel);                    // auto-ack is false                    channel.basicConsume(queueName, false, queueingConsumer);                    while (true) {                        QueueingConsumer.Delivery delivery = queueingConsumer.nextDelivery();                        String message = new String(delivery.getBody());                        log.debug("Mq Receiver get message");                        // Send the message to all connected clients                        // If you want to send to a specified client, just add                        // your own logic and ack manually                        // Be aware that ChannelGroup is thread safe                        log.info(String.format("Conneted client number: %d",EchoServerHandler.channels.size()));                        for (io.netty.channel.Channel c : EchoServerHandler.channels) {                            ByteBuf msg = Unpooled.copiedBuffer(message.getBytes());                            c.writeAndFlush(msg);                        }                        // manually ack to MQ server the message is consumed.                        channel.basicAck(delivery.getEnvelope().getDeliveryTag(), false);

                    }

The above Code reads data from a Queue. To effectively process data and prevent abnormal data loss, manual Ack is used.

RabbitMQ usage: http://www.cnblogs.com/luxiaoxun/p/3918054.html

 

Code hosted on GitHub: https://github.com/luxiaoxun/NettyMqServer

 

Refer:

Http://netty.io/

Http://netty.io/4.0/api/io/netty/handler/codec/LengthFieldBasedFrameDecoder.html

Http://netty.io/4.0/api/io/netty/handler/timeout/IdleStateHandler.html