The solution for UDP to receive millions of data and udp to receive data

Source: Internet
Author: User

The solution for UDP to receive millions of data and udp to receive data
Small order

Shortly after arriving at the new company, the Service receives A task: A sender sends some information through udp, and then the Service receives the information and saves it to A table A in the database, the stored data is processed in A series. After processing, it is accumulated to another table B, and then the data of the processed Table A is cleared. At present, the number of senders is relatively small and will soon increase to 100.

Solution

I use netty5 for udp network communication. I save the received data to BlockingQueue, read the data in BlockingQueue, and get 100 pieces of data to the hbase database.

Some code
  • Initialize netty

 

Int DEFAULT_PORT = 6000; EventLoopGroup group = new NioEventLoopGroup (); try {Bootstrap bootstrap = new Bootstrap (); bootstrap. group (group ). channel (niodomainramchannel. class ). option (ChannelOption. SO_BROADCAST, true ). handler (new UdpServerHandler (); Channel channel = bootstrap. bind (DEFAULT_PORT ). sync (). channel (); channel. closeFuture (). await (); LOGGER.info ("netty initialization successful! ");} Catch (InterruptedException e) {e. printStackTrace ();} finally {group. shutdownGracefully ();}

 

  

 

  • Receive udp data

 

Public BlockingQueue <Map <String, Object> queue =
New inclublockingqueue <Map <String, Object> (990000); protected void messageReceived (ChannelHandlerContext ctx, DatagramPacket msg) throws Exception {// because Netty encapsulates UDP, therefore, we receive the DatagramPacket object. String result = msg. content (). toString (CharsetUtil. UTF_8); Map <String, Object> getMap = new HashMap <String, Object> (); // process data
Queue. put (getMap );
Ctx. writeAndFlush (new DatagramPacket (
Unpooled. copiedBuffer ("Result:", CharsetUtil. UTF_8), msg. sender ()));}

 

  • Read data and store hbase
Public void getDate () {LOGGER.info ("start to fetch data"); List <Map <String, Object> jsonList = new ArrayList <Map <String, Object> (); while (true) {Map <String, Object> takeMap = null; try {takeMap = queue. take (); if (takeMap = null) {continue;} jsonList. add (takeMap); if (jsonList. size () == 100) {String httpJson = HbaseUtil. toHttpJson (vo. getTableName (), jsonList); LOGGER.info (httpJson); List <HbaseDataEntity> hbaseDatas = ParseJson. getData (httpJson); HbaseAPI. insertDataList (hbaseDatas); jsonList. clear (); LOGGER.info ("100 items saved in hbase") ;}} catch (Exception e) {jsonList. clear (); continue ;}}}
Pitfalls

 

  • BlockingQueue must be set to the maximum value of int, which may cause memory overflow;
  • When retrieving data from BlockingQueue, take () must be blocked, which will lead to endless loops, accounting for of CPU;
  • The hbase library is congested when connected. If the connection fails, it will be blocked all the time.

 

Contact Us

The content source of this page is from Internet, which doesn't represent Alibaba Cloud's opinion; products and services mentioned on that page don't have any relationship with Alibaba Cloud. If the content of the page makes you feel confusing, please write us an email, we will handle the problem within 5 days after receiving your email.

If you find any instances of plagiarism from the community, please send an email to: info-contact@alibabacloud.com and provide relevant evidence. A staff member will contact you within 5 working days.

A Free Trial That Lets You Build Big!

Start building with 50+ products and up to 12 months usage for Elastic Compute Service

  • Sales Support

    1 on 1 presale consultation

  • After-Sales Support

    24/7 Technical Support 6 Free Tickets per Quarter Faster Response

  • Alibaba Cloud offers highly flexible support services tailored to meet your exact needs.