Direct memory overflow analysis due to high concurrency under Netty4 underlying bugs

Accident record:

10-point game open service, quickly break through the 2300+ single-zone simultaneously online

18:15, operation Reaction player can't get in, the person who is ready to eat is caught back to troubleshoot

found that because the direct memory is occupied, has been in full GC, and recovery is not lost, so completely do not process the player request, notify the operation to restart the server, temporary resolution.

2. Consider whether to change the number of RPC connections to 8, the time-out to change the length of the resulting, try to reduce the number, time-out to 2 hours, found that direct memory increases over time.

3. Dump the memory data and find that the Netty bottom occupancy ratio is much higher than the normal level.

• Output Buffer Channeloutboundbuffer

，recycler accounted for a large proportion, asked the degree Niang, there is no useful information to find Google, with the keyword "channeloutboundbuffer out of the Direct memory full GC"

I found an article that was basically the same as the situation I met,

Direct memory exhausted after sending many notificationsrecycler could cache infinite resources
netty.4.0.24.finalversion has a bug,The reason is that if the data in the buffer exceeds max_capacity, if (size==maxcapacity) {...} Will never be set up, so the memory footprint will increase indefinitely.

Update Netty 4.0.25 final version to continue observation.

Direct memory overflow analysis due to high concurrency under Netty4 underlying bugs