Golang MgO MONGO Connection pool settings: Must be manually added Maxpoolsize

This is a creation in Article, where the information may have evolved or changed.

Panda TV's gift system uses the Golang of the MONGO library MgO, in the middle of some pits, summed up to avoid everyone to step on the pit

Golang's MgO library illustrates the use of the open connection multiplexing, but the observation experiment found that this does not realize the control of the connection at all, the connection multiplexing is actually only in the current operation (session. Close) takes effect, and ultimately requires the programmer to limit the connection itself.

Nonsense not much to say, start on the code

Globalmgosession, err: = MgO. Dial (Host) func (M *mongobasedao) Get (tablename string, id string, result interface{}) interface{} {    session: = Global Mgosession.clone ()    defer session. Close ()     Collection: = Session. DB (Globalmgodbname). C (tablename)    ERR: = collection. Findid (Bson. Objectidhex (ID)). One (Result)     if err! = Nil {        Logkit. Logger.error ("Mongo_base method:get" + Err. Error ())    }    return result}

　

Golang main entrance is started, we will create a global session, and then each time using the clone session information and connection, for this request, after use call session. Close () releases the connection.

//Clone works just like Copy, but also reuses the same socket as the original//Sessio  N, in case it had already reserved one due to its consistency//guarantees. This behavior ensures is writes performed in the old session//is necessarily observed when using the new session, as L  Ong as it was a//strong or monotonic session. That's said, it also means that long operations//could cause other goroutines using the original session to Wait.func (s *ses sion) Clone () *session {s.m.lock () Scopy: = Copysession (S, True) S.m.unlock () return scopy}//Close Termina  TES the session. It's a runtime error to use a session//after it had been Closed.func (S *session) Close () {s.m.lock () if S.cluster _! = Nil {DEBUGF ("Closing session%p", s) s.unsetsocket ()//frees the current thread occupied by the socket to be nil s.cluster_. Release () S.cluster_ = nil} s.m.unlock ()} 

clone the method note that the original session will be reused the socket connection, but the concurrent request a large, the other process is too late to release the connection, the current process will do?

Func (S *session) Acquiresocket (slaveok bool) (*mongosocket, error) {//read-only lock to check for previously reserve    D socket. S.m.rlock ()//If There is a slave socket reserved and its use was acceptable, take it as long//as there isn ' t a ma    Ster socket which would is preferred by the Read preference mode. If s.slavesocket! = Nil && s.slaveok && slaveok && (s.mastersocket = Nil | | s.consistency! = Prim Arypreferred && s.consistency! = monotonic) {socket: = S.slavesocket socket. Acquire () S.m.runlock () Logkit.        Logger.info ("Sgp_test 1 acquiresocket slave is ok!") return socket, nil} if s.mastersocket! = Nil {socket: = S.mastersocket socket. Acquire () S.m.runlock () Logkit.        Logger.info ("Sgp_test 1 acquiresocket Master is ok!")  return socket, nil} s.m.runlock ()//No go. We may have a new socket and change the session with request a,//So try again but WITH an exclusive-lock now. S.m.lock () defer s.m.unlock () if s.slavesocket! = Nil && s.slaveok && slaveok && (s.masterso Cket = = Nil | | S.consistency! = primarypreferred && s.consistency! = monotonic) {s.slavesocket.acquire () Logkit.        Logger.info ("Sgp_test 2 acquiresocket Slave is ok!") Return s.slavesocket, nil} if s.mastersocket! = nil {s.mastersocket.acquire () Logkit.        Logger.info ("Sgp_test 2 acquiresocket Master is ok!")  Return s.mastersocket, nil}//Still not good.    We need a new socket. Sock, err: = S.cluster (). Acquiresocket (s.consistency, Slaveok && s.slaveok, S.synctimeout, S.socktimeout, S.queryconfig.op.servertags , S.poollimit) ... logkit.    Logger.info ("Sgp_test 3 acquiresocket cluster acquiresocket is ok!") Return sock, nil}

Debug in the source code, the result log explains everything:

Mar 09:46:40 dev02.com[12607]:  [INFO] sgp_test 1  acquiresocket Master is ok! Mar 09:46:40 dev02.com[12607]:  [INFO] sgp_test 1  acquiresocket Master is ok! Mar 09:46:41 dev02.com[12607]:  [INFO] sgp_test 1 acquiresocket slave is ok! Mar 09:46:41 dev02.com[12607]:  [INFO] sgp_test 3   acquiresocket cluster Acquiresocket is ok! Mar 09:46:41 dev02.com[12607]:  [INFO] sgp_test 3   acquiresocket cluster Acquiresocket is ok! Mar 09:46:41 dev02.com[12607]:  [INFO] sgp_test 3   acquiresocket cluster Acquiresocket is ok!

Constant creation of connection Acquiresocket

 $  netstat-nat|grep-i 27017|wc-l

400

If each session does not call close, it will reach the terror of 4096 and block other requests, so clone or copy session must be defer close off

Enabling the Maxpoollimit parameter will limit the total connection size, and the current coprocessor will sleep until the connection can be created, causing a lock problem with high concurrency, resulting in several more connections being created

Src/gopkg.in/mgo.v2/cluster.go s, abended, err: = Server. Acquiresocket (Poollimit, sockettimeout) If Err = = Errpoollimit {if!warnedlimit {warned Limit = True Logkit. Logger.error ("Sgp_test warning:per-server connection limit reached." + Err.            Error ()) log ("Warning:per-server connection limit reached.") } time. Sleep (* time.  Millisecond) Continue} session.go://Setpoollimit sets the maximum number of sockets in  Server//Before this session would block waiting for a socket to be available.  The default limit is 4096. This limit must is set to cover more than any expected workload of the//application. It is a bad practice and an unsupported use case to use the//database driver to define the concurrency limit of a appl Ication. Prevent//Such concurrency "at the door" instead, by properly restricting the amount//of used resources and number OF Goroutines before they is created. Func (S *session) setpoollimit (limit int) {S.m.lock () S.poollimit = Limit S.m.unlock ()}

Connection pooling Setup Method:

1. Increase in configuration

[host]:[port]?maxpoolsize=10

2, in the code:

Dao. Globalmgosession.setpoollimit (10)

Again do the pressure test:

 $  netstat-nat|grep-i 27017|wc-l

15

Conclusion:

After each clone session, the session is called at the end of the operation. Close will unset socket, set nil, so the socket is reused, only in the current session scope to take effect, so the non-global session can not be shared, each request arrival will create a socket connection until the maximum value of 4096, And MONGO the limit of the number of connections is generally 10,000, that is, a port you can only start a process to ensure that the connection does not explode, too many connections client efficiency is not high, the server side is more expensive memory and CPU, so need to enable the custom connection pool, Enabling connection pooling also requires that it is tragic if there are poomaxlimit or dead loops that do not release a socket connection.

MgO does not realize the pre-creation of the socket from the bottom and the connection pool reuse of the entire life cycle, it needs to optimize itself.