Solve slow query problems with MONGODB client Samus code Research

Recent projects need to use MONGODB, so downloaded the source of MongoDB on the internet, according to the sample written test code, but found a very strange problem: the speed of inserting records faster than the speed of data acquisition, and the most important problem is that the speed of data acquisition is unacceptable.
Test scenario: Main Document storage personnel basic information, sub-document storage student class contract data collection, this set can reach hundreds of, sub-document two storage contract payment records collection, the collection size generally no more than 50. According to the Personnel ID query personnel documents, the size after serialization is less than 180K, but consumes more than 400ms of time.
My main problem is that the inability to receive a 180K record requires more than 400MS, which has no advantage over the traditional RDBMS, and MongoDB is a memory mapping mechanism, unreasonable performance is so poor, and the network on its performance test data far better than my test results.
Exclusion method One: Is it because of the reasons for having sub-documents?
Find a document without any contract records query, found that the results are still, there is no obvious improvement;
Exclusion Mode Two: No index created?
Create index on search column ID, the result is still;
Exclusion method Three: is not the number of documents too large?
More than 10,000 lines is just a small number, no reason, MongoDB management tens of millions of documents are no problem, at the time or decided to try, will record all delete, insert a record and then query, the result is still;
Exclusion Mode four: is not due to the problem of client serialization?
Since I am storing a custom object, not the default document, I decided to try to store the document,document directly in two fields and get the speed or 180ms.
Exclusion Mode Five: Is it because the client machine is 32 bits while the MongoDB service is 64?
Put the program on a 64-bit machine to test, the problem remains.
Exclusion Method VI: Is it due to network transmission problems?
It makes no sense. The test client and server are all on the same LAN, but try to execute the client program directly on the MongoDB servers.
The above six ways have been tried, did not solve, finally decided to turn to the old generation, after all, is the use of MongoDB, give me two advice on the fix:
How to exclude seven: Look at the MongoDB data file to see if it is already large?
After viewing, the total size is 64M, which is more than the 32-bit file limit of 2G, can be basically ignored;
Exclusion mode eight: connection string.
servers=ip:27017; connecttimeout=30000; connectionlifetime=300000; minimumpoolsize=8; maximumpoolsize=256; Pooled=true

The first impression I had when I saw this reference string was that my writing was not the same as it (string connectionString = "" ; ), and then I found two important parameters:
1:connectionlifetime=300000, in the literal sense, refers to the life cycle of the connection, and its numerical setting is so large that it clearly means that the connection is not closed immediately, which differs from the approach of SQL Server;
2:pooled=true, from the literal point of view, should be the concept of a connection pool.

Analysis: From the above connection parameters, I understand the connection, is the client and the server connection between, it needs to shut down immediately after use, that is, the client and the server does not have a TCP connection. But I do not have a good understanding of the role of connection pooling. The connection pool actually stores many connection that have established a TCP connection with the server, maintains the connection to the server during its lifetime, and becomes a failed connection waiting to be recycled after the life cycle.
Re-modify the connection string to test, the problem is resolved, only the first request, because the need to create a TCP connection, performance will be affected, the subsequent request, because there is a connection pool exists, performance has been multiplied.
Finally look at the next Samus source code, you can see how it uses the connection pool.
First look at a MongoDB help Class I wrote: There are general operations such as creating MONGO objects.

public class Mongodbfactory2<t>: IDisposable where T:class
{
public string connectionString = "mongodb://10.1.55.172";
public string connectionString = configurationmanager.appsettings["MongoDB"];
public string databaseName = "MyDatabase";
Mongo Mongo;
Mongodatabase mongodatabase;
Public mongocollection<t> mongocollection;
Public MongodbFactory2 ()
{
MONGO = Getmongo ();
Mongodatabase = MONGO. Getdatabase (DatabaseName) as mongodatabase;
Mongocollection = mongodatabase.getcollection<t> () as mongocollection<t>;
Mongo. Connect ();
}
public void Dispose ()
{
This.mongo.Disconnect ();
}
<summary>
Configure MONGO to map the class T to the collection
</summary>
Private Mongo Getmongo ()
{
var config = new Mongoconfigurationbuilder ();
Config. Mapping (Mapping =
{
Mapping. DefaultProfile (profile =
{
Profile. Subclassesare (t = t.issubclassof (typeof (T)));
});
Mapping. Map<t> ();
});
Config. ConnectionString (ConnectionString);
return new Mongo (config. Buildconfiguration ());
}

From the code above you can see that there is a sentence: MONGO. Connect (), my first impression is to create a client-to-server connection, in fact, there is a connection pool, this operation does not create a remote connection each time, and in some cases only directly from the connection pool to return the available connection objects.
From source code analysis is how connection pooling is used, and how connections are created.
The Connect function of the 1:mongo class: The _connection object needs to be traced.

<summary>
Connects to server.
</summary>
<returns></returns>
<exception cref = "Mongodb.mongoconnectionexception" >thrown when connection fails.</exception>
public void Connect ()
{
_connection. Open ();
}

2: Look again at this sentence: return new Mongo (config. Buildconfiguration ());

<summary>
Initializes a new instance of the <see cref = "Mongo"/> class.
</summary>
<param name = "Configuration" >the MONGO configuration.</param>
Public Mongo (mongoconfiguration configuration) {
if (configuration = = null)
throw new ArgumentNullException ("Configuration");
Configuration. Validateandseal ();
_configuration = Configuration;
_connection = connectionfactoryfactory.getconnection (configuration. ConnectionString);
}

The last sentence of the above code has a _connection generation process.
3: Can trace to the final generation connection function, finally see Builder. Pooled this parameter, the value of which is the parameter in the connection string.

<summary>
Creates the factory.
</summary>
<param name= "connectionString" >the connection string.</param>
<returns></returns>
private static Iconnectionfactory Createfactory (string connectionString) {
var builder = new Mongoconnectionstringbuilder (connectionString);
if (builder. Pooled)
return new Pooledconnectionfactory (connectionString);
return new Simpleconnectionfactory (connectionString);
}

4: See how pooledconnectionfactory is creating a connection: The function is to put the available connections into the connection pool, and ultimately the functions that actually create the connection are createrawconnection ()

<summary>
Ensures the size of the minimal pool.
</summary>
private void Ensureminimalpoolsize ()
{
Lock (_syncobject)
while (Poolsize < builder.minimumpoolsize)
_freeconnections.enqueue (Createrawconnection ());
}

5: True remote connection section.

View Code

<summary>
Creates the raw connection.
</summary>
<returns></returns>
Protected Rawconnection createrawconnection ()
{
var endPoint = Getnextendpoint ();
Try
{
return new Rawconnection (EndPoint, builder.connectiontimeout);
}catch (SocketException exception) {
throw new Mongoconnectionexception ("Failed to connect to server" + EndPoint, ConnectionString, EndPoint, exception);
}
}
Private readonly TcpClient _client = new TcpClient ();
Private readonly list<string> _authenticateddatabases = new list<string> ();
private bool _isdisposed;
<summary>
Initializes a new instance of the <see cref= "Rawconnection"/> class.
</summary>
<param name= "EndPoint" >the end point.</param>
<param name= "ConnectionTimeout" >the connection timeout.</param>
Public rawconnection (Mongoserverendpoint Endpoint,timespan connectiontimeout)
{
if (EndPoint = = null)
throw new ArgumentNullException ("EndPoint");
EndPoint = EndPoint;
CreationTime = Datetime.utcnow;
_client. Nodelay = true;
_client. ReceiveTimeout = (int) connectiontimeout.totalmilliseconds;
_client. Sendtimeout = (int) connectiontimeout.totalmilliseconds;
Todo:custom exception?
_client. Connect (Endpoint.host, Endpoint.port);
}

Then we look at how the life cycle of the connection is implemented: The primary logic is in pooledconnectionfactory, and if the connection is found to have expired, the connection is placed in an unavailable queue and removed from the idle connection.

View Code

<summary>
Checks the free connections alive.
</summary>
private void Checkfreeconnectionsalive ()
{
Lock (_syncobject)
{
var freeconnections = _freeconnections.toarray ();
_freeconnections.clear ();
foreach (Var freeconnection in freeconnections)
if (IsAlive (freeconnection))
_freeconnections.enqueue (freeconnection);
Else
_invalidconnections.add (freeconnection);
}
}
<summary>
Determines whether the specified connection is alive.
</summary>
<param name= "Connection" >the connection.</param>
<returns>
<c>true</c> If the specified connection is alive; otherwise, <c>false</c>
</returns>
private bool IsAlive (rawconnection connection)
{
if (connection = = null)
throw new ArgumentNullException ("Connection");
if (!connection. isconnected)
return false;
if (connection. Isinvalid)
return false;
if (builder.connectionlifetime! = TimeSpan.Zero)
if (connection. Creationtime.add (Builder.connectionlifetime) < DateTime.Now)
return false;
return true;
}

Finally, let's look at my top MongoDB help class The following method: That is, the release of the connection, and here is not the release of the connection between the client and the server, but to remove the connection from the busy queue, re-return to the available queue:

public void Dispose ()
{
This.mongo.Disconnect ();
}

And look at MONGO. Disconnect ()

<summary>
Disconnects this instance.
</summary>
<returns></returns>
public bool Disconnect ()
{
_connection. Close ();
Return _connection. isconnected;
}

Continue down to the following core content:

View Code

<summary>
Returns the connection.
</summary>
<param name = "Connection" >the connection.</param>
public override void Close (Rawconnection connection)
{
if (connection = = null)
throw new ArgumentNullException ("Connection");
if (! IsAlive (Connection))
{
Lock (_syncobject)
{
_usedconnections.remove (connection);
_invalidconnections.add (connection);
}
Return
}
Lock (_syncobject)
{
_usedconnections.remove (connection);
_freeconnections.enqueue (connection);
Monitor.pulse (_syncobject);
}
}

Summary: After all the different attempts, finally solved the MongoDB query slow reason, not mongodb itself problem, also non-network, non-data problem, but is not the correct use of client connection, not easy ah, thank the old generation of guidance.

References:

MongoDB Learning Notes

Http://www.360doc.com/content/16/0720/17/35239163_577069265.shtml

Monogodb Find method calls JavaScript where

On the source of Samus under MongoDB

Mongodb vs. SQL statement comparison

Optimistic concurrency control in MongoDB

Mongodb Insert Save Difference

Solve slow query problems with MONGODB client Samus code Research