The solution for UDP to receive millions of data and udp to receive data

The solution for UDP to receive millions of data and udp to receive data
Small order

Shortly after arriving at the new company, the Service receives A task: A sender sends some information through udp, and then the Service receives the information and saves it to A table A in the database, the stored data is processed in A series. After processing, it is accumulated to another table B, and then the data of the processed Table A is cleared. At present, the number of senders is relatively small and will soon increase to 100.

Solution

I use netty5 for udp network communication. I save the received data to BlockingQueue, read the data in BlockingQueue, and get 100 pieces of data to the hbase database.

Some code

• Initialize netty

 

Int DEFAULT_PORT = 6000; EventLoopGroup group = new NioEventLoopGroup (); try {Bootstrap bootstrap = new Bootstrap (); bootstrap. group (group ). channel (niodomainramchannel. class ). option (ChannelOption. SO_BROADCAST, true ). handler (new UdpServerHandler (); Channel channel = bootstrap. bind (DEFAULT_PORT ). sync (). channel (); channel. closeFuture (). await (); LOGGER.info ("netty initialization successful! ");} Catch (InterruptedException e) {e. printStackTrace ();} finally {group. shutdownGracefully ();}

 

　　

 

• Receive udp data

 

Public BlockingQueue <Map <String, Object> queue =
New inclublockingqueue <Map <String, Object> (990000); protected void messageReceived (ChannelHandlerContext ctx, DatagramPacket msg) throws Exception {// because Netty encapsulates UDP, therefore, we receive the DatagramPacket object. String result = msg. content (). toString (CharsetUtil. UTF_8); Map <String, Object> getMap = new HashMap <String, Object> (); // process data
Queue. put (getMap );
Ctx. writeAndFlush (new DatagramPacket (
Unpooled. copiedBuffer ("Result:", CharsetUtil. UTF_8), msg. sender ()));}

 

• Read data and store hbase

Public void getDate () {LOGGER.info ("start to fetch data"); List <Map <String, Object> jsonList = new ArrayList <Map <String, Object> (); while (true) {Map <String, Object> takeMap = null; try {takeMap = queue. take (); if (takeMap = null) {continue;} jsonList. add (takeMap); if (jsonList. size () == 100) {String httpJson = HbaseUtil. toHttpJson (vo. getTableName (), jsonList); LOGGER.info (httpJson); List <HbaseDataEntity> hbaseDatas = ParseJson. getData (httpJson); HbaseAPI. insertDataList (hbaseDatas); jsonList. clear (); LOGGER.info ("100 items saved in hbase") ;}} catch (Exception e) {jsonList. clear (); continue ;}}}

Pitfalls

 

• BlockingQueue must be set to the maximum value of int, which may cause memory overflow;
• When retrieving data from BlockingQueue, take () must be blocked, which will lead to endless loops, accounting for of CPU;
• The hbase library is congested when connected. If the connection fails, it will be blocked all the time.