Netstat statistics of TCP connections vs. PROC?PID?FD Analysis of the inconsistent number of socket type FD

New Blog Address:

http://hengyunabc.github.io/netstat-difference-proc-fd-socket-stat/

Recently, an online application, found that the number of sockets slowly increased, and do not recycle, after the warning line, is automatically restarted by the operation and maintenance monitoring.

First, observing the JVM history on Zabbix, found that jvm-perm space had no data for the last two weeks, guessing that the program switched from JDK7 to JDK8. After asking the developer, the program has not been restarted for a long time and has only recently been republished. During this time, the Java operating environment on the line has been upgraded from JDK7 to JDK8.

Because there is no perm space in the JDK8, replaced by Metaspace.

Netstat

To the online server, use Netstat to count the number of connection in the process.

-l

found that compared to Zabbix on the number of statistics socket less than 100, netstat statistics only more than 100, and Zabbix on the monitoring data more than 300.

So to/proc/$pid/fd the number of FD for the socket type:

cd /proc/$pid-l

Discovery data is consistent with data on Zabbix.

Netstat is how to count the source code of the download netstat

Http://unix.stackexchange.com/questions/21503/source-code-of-netstat

source net-tools

From the Netstat code, it is possible to read the data inside the/PROC/NET/TCP to get the statistical information.

Java and C version of the simple Netstat implementation

Java version of

Http://www.cs.earlham.edu/~jeremiah/LinuxSocket.java

C Version of:

Http://www.netmite.com/android/mydroid/system/core/toolbox/netstat.c

Track Netstat with Starce

-antp

It can be found that Netstat has read a lot of the data under/proc. So it is generally possible to know that the netstat is the/proc/pid/fd below the data and/proc/net/the following data, in contrast to obtain statistical results.

Which sockets are not counted by netstat?

Again on the Internet to find the next, found here that the socket if created, no bind or connect, will not be netstat statistics.

Http://serverfault.com/questions/153983/sockets-found-by-lsof-but-not-by-netstat

In fact, that is, if the socket is created and not used, it will only be available under/PROC/PID/FD, and will not have the relevant data under/proc/net/.

After a simple test, it's true:

int socket = socket(PF_INET,SOCK_STREAM,0//不使用

In addition, even if the socket is used, if the implementation of shutdown, just in the beginning, with Netstat can be counted to the state of the socket is fin_wait1. After a while, netstat statistics are not the information of the socket, but under the/proc/pid/fd, still can be found.

In the middle, I wrote a program, the/PROC/PID/FD under the Inode and/proc/net/below the data comparison, found that some sockets do not appear in the Inode/proc/net/.

View with Lsof

To view the socket inode with lsof:

Trigger GC, Recycle socket

Then try to trigger the GC to see if the socket will be recycled:

jmap -histo:live <pid>

As a result, the sockets are found to be recycled.

Then look at the Finalize method of Abstractplainsocketimpl:

    /**     * Cleans up if the user forgets to close it.     */    protectedvoidfinalizethrows IOException {        close();    }

You can see that the socket was close when it was in the GC.
Write a program to test under:

publicclass TestServer {    publicstaticvoidmainthrows IOException, InterruptedException {        for(int010; ++i){            new10000);            System.err.println(socket);        }        System.in.read();    }}

First, to see the/PROC/PID/FD, you can find the relevant socket FD, and then trigger GC, you can find that the socket was recycled.

Other Dongdong Anon_inode:[eventpoll]

ls -al /proc/pid/fd

You can see that there are output like this:

-anon_inode:[eventpoll]

This type of inode is created by Epoll.

Again, the selector implementation in Java under Linux is epoll with a pipe to implement the event notification function. So in the NIO program, there will be anon_inode:[eventpoll] and pipe type FD.

Why Tail-f/proc/$pid/FD/1 cannot read to stdout data

Http://unix.stackexchange.com/questions/152773/why-cant-i-tail-f-proc-pid-fd-1

Summarize

The reason is that after the JDK upgrade, the GC works differently and the FULLGC executes longer, resulting in some idle sockets not being recycled.

This article is chaotic, documenting some of the tools and techniques.

Netstat statistics of TCP connections vs. PROC?PID?FD Analysis of the inconsistent number of socket type FD