One MongoDB socket Exception

Today re-deployment of a project, the project will be launched to visit MongoDB to get some data, a pretty simple project, the previous release is no problem, this launch is directly Socekt Exception:

Nested exception is org.springframework.data.mongodb.UncategorizedMongoDbException:socket exception [Send_error] for X.x.x.x:xx; Nested exception is com.mongodb.MongoException:socket exception [Send_error] for X.X.X.X:XX

The first reaction is that the intranet network is not very good, the network fluctuations, after the investigation of the matter.

Solution: Try a few more times ...

Obviously, this approach doesn't solve the problem (it won't be written here if it's settled). At the same time, ping the host of the Mong instance, returned within 0.3MS, there is no packet loss.

Then went to MONGO over there to see, there is no problem, the operation is normal, there is no error in log. and MONGO the maximum number of connections configured here is 2W, this connection is now used less than 9000 ah.

This thing is more strange, never met before, since is Socekt exception can only continue to forget the network direction to think.

First look at the number of connections in each state in the TCP connection:

Admin@linux:~> Netstat-an | awk '/^tcp/{++s[$NF]} END {for (a in S) print A, S[a]} '
time_wait 3615
close_wait '
fin_wait1 '
Fin_wai T2
established 1983
syn_recv 7
CLOSING
30 last_ack

This command, I also search, the above shows a service is running on the machine above the results, and the problem that the station, established more than 6,000, this is obviously unscientific, on the business level, it should be less pressure on the point.

It depends on which machine this extra connection is built with, continue:

Netstat-an |grep ESTA |awk ' {print$5 ' \ n '} ' |awk ' BEGIN {fs= ': '} {count[$1]++}end{for (A in COUNT) print A, Count[a]} ' |so Rt-k 2-NR

Print out all and the machine to establish a connection between the IP and the number of connections, a look is sure to be with the MONGO machine built more connections, accounted for the majority of the established connection.

It depends on which process is out of the question, continue:

NETSTAT-ANP |grep ESTA |awk ' {print$7} ' |awk ' {count[$1]++}end{for (A in COUNT) print A, Count[a]} ' |sort-k 2-nr

This way to see which processes occupy more established connections, after getting the process number, it is easy to find the corresponding service, the result of another service takes up a lot of established connection.

NETSTAT-ANP | grep {pid}| grep ESTA |awk ' {print$5 ' \ n '} ' |awk ' BEGIN {fs= ': '} {count[$1]++}end{for (A in COUNT) print A, count[a]} '

The following PID, in turn, can look at the process and which machines established the connection, and how many connections were established.

This service actually occupies 5,000 MONGO connection, too exaggerated. It's probably the end of the connection, which is the culprit.

Wait, our MONGO is not a maximum of 2w connections. Even if this surprisingly take up so much, it's not going to make anything else. First 5000 This error correction let service run up again, check, this is the configuration of the problem, connections-per-host this parameter configured to 500, changed to a reasonable value such as 100, all return to normal, the original service can rise up.

The next two questions are analyzed: Connections-per-host What does this parameter mean? MONGO the maximum number of connections did not take effect

First, the second one, this is relatively simple.

The main reason is that even if the maximum number of connections in the MONGO configuration is 2w, the maximum number of open files per process in the system (open files) is only 65535, so MONGO's maximum number of connections is actually only 65535. The number of connections with the problem of the service took up 5,000, the rest is occupied by other services, so the redeployment of the service can no longer get new connections.

This place is the only more depressing is MONGO this side log in also no error.

Say the first question. This parameter is interpreted in the official API as follows: The maximum number of connections allowed per host for this Mongo instance. The maximum number of connections allowed per host to be created for this MONGO instance. Depending on the usage, the individual is interpreted as allowing each process to be more appropriate for the maximum number of connections created for this MONGO instance. If the maximum number of connections per machine is allowed, then the maximum number of connections between the machine and the MONGO should be the value of the set, regardless of how many services are deployed on the same machine, but this is not the case with the MONGO.

In addition, there is no mechanism to recover the MONGO connection pool, according to personal experience, the number of connections is basically only increased (even when the connection is not needed), up to the set of Connections-per-host.

The above analysis is an individual opinion, may not be accurate, please correct me.