??
"Your agent occupies a lot of system ports, killing many of our business systems and causing great losses to us. We ask your leaders to apologize next week."-- One of our customers.
How is it possible? We do not believe that our agent only occupies one port!
The fact is beyond words. After verification, it is indeed because our agent occupies many system ports. I read the logs and basically occupied the available system ports!
Why? The Mina framework is private!
Our agent uses the NIO communication framework Mina, but it is not used well, resulting in this almost destructive disaster.
Let's take a look at the code first.
/*** Send messages asynchronously * @ Param agent * @ Param Request */Public void sendmessagetoagent (Agent agent, hyrequest request) {iosession session = NULL; ioconnector conne= NULL; long starttime = system. currenttimemillis (); try {// create a non-blocking client program connector = new niosocketconnector (); // set the connection timeout time connector. setconnecttimeoutmillis (connecttimeoutmillis); objectserializationcodecfactory objscodec = new objectserializationcodecfactory (); objscodec. setdecodermaxobjectsize (defaultdecoder); objscodec. setencodermaxobjectsize (defaultdecoder); protocolcodecfilter codecfilter = new protocolcodecfilter (objscodec); // data conversion, encoding setting connector. getfilterchain (). addlast ("codec", codecfilter); // message connector. sethandler (clienthandler); socketaddress = new inetsocketaddress (agent. getipaddr (), agent. getagentport (); connectfuture future = connector. connect (socketaddress); future. awaituninterruptibly (); Session = Future. getsession (); string JSON = mapper. writevalueasstring (request); Session. write (JSON); long endtime = system. currenttimemillis (); logerr. debug ("send-time:" + (endtime-starttime);} catch (exception e) {logerr. error ("Host:" + agent. getipaddr () + ", agentport:" + agent. getagentport () + ", connection exception... "+ E. getmessage (); clienthandler. handlerconnecterror (agent, request );}}
Public class minaclienthandler extends iohandleradapter {// log private logger log = logger. getlogger (getclass (); Private minaresponseprocesser; objectmapper mapper = NULL; @ overridepublic void messagereceived (iosession session, object message) throws exception {string MSG = message. tostring (); log.info ("Receive message from" + session. getremoteaddress (). tostring () + ", message:" + message); If (null = Mapper) {mapper = new objectmapper ();} // The request message is converted to the hyresponse object hyresponse response = mapper. readvalue (MSG, hyresponse. class); string remoteip = (inetsocketaddress) session. getremoteaddress ()). getaddress (). gethostaddress (); response. setremoteip (remoteip); hyrequest request = minaresponseprocesser. processresponse (response); If (request = NULL) {// close the current sessionclosesessionbyserver (Session, response);} else {session. write (mapper. writevalueasstring (request ));}}}
The above logic is that when you want to send a message, create a new connector and obtain a session to send the message and then return it directly, process the received response data in messagereceived of the minaclienthandler class and process the service. If you do not need to send the request again, close the current session.
In fact, the problem at the beginning of this article is caused here.
After our agent occupied a large number of ports, our engineers quickly located the problem and fixed it quickly. However, the fix was not ideal, but the code after the fix.
/*** Send messages asynchronously * @ Param agent * @ Param Request */Public void sendmessagetoagent (Agent agent, hyrequest request) {iosession session = NULL; ioconnector conne= NULL; long starttime = system. currenttimemillis (); try {// create a non-blocking client program connector = new niosocketconnector (); // set the connection timeout time connector. setconnecttimeoutmillis (connecttimeoutmillis); objectserializationcodecfactory objscodec = new objectserializationc Odecfactory (); objscodec. setdecodermaxobjectsize (defaultdecoder); objscodec. setencodermaxobjectsize (defaultdecoder); protocolcodecfilter codecfilter = new protocolcodecfilter (objscodec); // data conversion, encoding setting connector. getfilterchain (). addlast ("codec", codecfilter); // message connector. sethandler (clienthandler); socketaddress = new inetsocketaddress (agent. getipaddr (), agent. getagentport (); connectfuture Future = connector. connect (socketaddress); future. awaituninterruptibly (); Session = Future. getsession (); string JSON = mapper. writevalueasstring (request); Session. write (JSON); // wait for the session to be disconnected. getclosefuture (). awaituninterruptibly (); long endtime = system. currenttimemillis (); logerr. debug ("send-time:" + (endtime-starttime); // connector. dispose ();} catch (exception e) {logerr. error ("Host:" + agent. getip ADDR () + ", agentport:" + agent. getagentport () + ", connection exception... "+ E. getmessage (); clienthandler. handlerconnecterror (agent, request);} finally {If (null! = Session) {session. Close (true); Session = NULL;} If (null! = Connector) {connector. Dispose ();}}}
Only one place is changed. After the message is sent, a waiting for disconnection statement and finally statement block are added to close the session and connector.
Although the program will not occupy a large number of system ports, it will cause another problem-when a message queue needs to asynchronously call the preceding statement to send messages, the original asynchronous (direct return after sending, equivalent to fast concurrent sending) is converted to pseudo asynchronous (after sending the message and waiting for the message to be returned for processing, equivalent to processing the messages in the queue in sequence ).
The above modification is not the expected result, but at least the problem of occupying a large number of ports is fixed.
With the idea of completely fixing this problem, I 'd like to take a closer look at the Mina source code.
NIO framework-mina source code analysis (1): Background