Release a TokyoTyrant-Based C # Open-source client project

The main function code of the client is open here. The purpose of open source is to hope that more people will study TC and TT. At the same time, you can download the C # source code to continue optimization and improve performance, looking for bugs at the same time, I must have limited energy and capabilities, while Discuz! NT Enterprise Edition has too many features (I will write more articles to introduce them in time), but I cannot help myself, haha :)

Well, for ease of use, the following describes the project files in the source code:

The source package contains three items:
1. Discuz. EntLib. TokyoTyrant Core Function Code (currently the namespace is named after the product)
2. TTSample is mainly used to load test data and compare the speed at which SQL Server databases can be created and queried.
3. Example of using the core function code in TTSampleConsole (this article will introduce the main functions)

The class diagram in Discuz. EntLib. TokyoTyrant is as follows:

The client has the following features:
Support TcpClient connection pool
Support UTF-8 Encoding
You can set the number of initial links, link expiration time, maximum idle time, and maximum working time.

The following describes how to use it:
1. initialize the link pool:
Copy codeThe Code is as follows:
Pool = TcpClientIOPool. GetInstance ("dnt_online"); // link pool name (DNT online table)
Pool. SetServers (new string [] {"10.0.4.66: 11211 "});
Pool. InitConnections = 8;
Pool. MinConnections = 8;
Pool. MaxConnections = 8;
Pool. maxidle= 30000;
Pool. MaxBusy = 50000;
Pool. MaintenanceSleep = 300000;
Pool. TC pclienttimeout = 3000;
Pool. TcpClientConnectTimeout = 30000;
Pool. Initialize ();


2. CRUD operation:
Create a record (with DISCUZ! The NT online table field is used as an example ):

Copy codeThe Code is as follows:
IDictionary <string, string> columns = new System. Collections. Generic. Dictionary <string, string> ();
Columns. Add ("olid", I. ToString ());
Columns. Add ("userid", I. ToString ());
Columns. Add ("ip", "10.0.7." + I );
Columns. Add ("username", "user" + I );
Columns. Add ("nickname", "user" + I );
Columns. Add ("password ","");
Columns. Add ("groupid", "5 ");
Columns. Add ("olimg ","");
Columns. Add ("adminid", "0 ");
Columns. Add ("invisible", "0 ");
Columns. Add ("action", "0 ");
Columns. Add ("lastactivity", "1 ");
Columns. Add ("lastposttime", DateTime. Now. ToString ());
Columns. Add ("lastpostpmtime", DateTime. Now. ToString ());
Columns. Add ("lastsearchtime", DateTime. Now. ToString ());
Columns. Add ("lastupdatetime", DateTime. Now. ToString ());
Columns. Add ("forumid", "0 ");
Columns. Add ("forumname ","");
Columns. Add ("titleid", "0 ");
Columns. Add ("title ","");
Columns. Add ("verifycode ","");
Columns. Add ("newpms", "0 ");
Columns. Add ("newnotices", "0 ");
TokyoTyrantService. PutColumns (TTPool. GetInstance (), I. ToString (), columns, true); // true indicates that if there is a record in tc, the record is overwritten. If no record exists, the record is created.



Query operation:
First, a query (condition) object is created. For example, the online user information of the query field olid = 1 is defined as follows:
New Query (). NumberEquals ("olid", 1)

Put it in the QueryRecords method of TokyoTyrantService (note that the link pool is bound), as follows:

Var qrecords = TokyoTyrantService. QueryRecords (TTPool. GetInstance (), new Query (). NumberEquals ("olid", 1 ));
// Traverse the current result set
Foreach (var k in qrecords. Keys)
{
Var column = qrecords [k];
... Data Binding operation
}

A more complex query is as follows (query forumid = 16 and userid <1000, and sort the first three records in descending order of The userid field ):

Qrecords = TokyoTyrantService. QueryRecords (pool, new Query (). NumberGreaterThanOrEqual ("forumid", 16 ).
NumberLessThan ("userid", 1000). OrderBy ("userid", QueryOrder. NUMDESC). LimitTo (3, 0 ));
The comparison operator here can refer to the enumeration type in the source code as follows:

Copy codeThe Code is as follows:
Public enum QueryOperation
{
STREQ = 0, // # query condition: it indicates the text content of the operation object is exactly the same (=)
STRINC = 1, // # query condition: indicates the content containing the operation object text (LIKE '% text % ')
STRBW = 2, // # query condition: start with the text column of the operation object (LIKE 'text % ')
STREW = 3, // # query condition: it indicates the end of the text row and column of the operation object (LIKE '% text ')
STRAND = 4, // # query conditions: indicates all the fields separated by commas (,) in the text column of the operation object (name LIKE '% text (I) %' AND name LIKE '% text (ii) % ')
STROR = 5, // # query conditions: indicates a part of the text segment of the operation object that contains commas (,) (name LIKE '% text (I) %' OR name LIKE '% text (ii) % ')
STROREQ = 6, // # query condition: indicates that some of the parts separated by commas in the text segment of the operation object are exactly the same (name = 'text (I) 'OR name = 'text (ii ')
STRRX = 7, // # query condition: The table matches the regular expression.
NUMEQ = 8, // # query condition: the value is equal to the value of the operation object (=)
NUMGT = 9, // # query condition: the value is greater than the value of the operation object (>)
NUMGE = 10, // # query condition: The table is greater than or equal to the value of the operation object (> =)
NUMLT = 11, // # query condition: smaller than the value of the operation object (<)
NUMLE = 12, // # query condition: the value is less than or equal to the value of the operation object (<=)
NUMBT = 13, // # query condition: the value is in the middle of the two values separated by commas (,) in the text segment of the operation object (between 100 and 200)
NUMOREQ = 14, // # query condition: the value is in the middle of the two values separated by commas (,) in the text segment of the operation object (between 100 and 200)
NEGATE = 1 <24, // # query condition: Negative flag
NOIDX = 1 <25 // # query condition: Non-index flag
}

Query the specified Primary Key (for example, olid in this example, the most efficient)
Copy codeThe Code is as follows:
Var qrecords = TokyoTyrantService. GetColumns (pool, new string [] {"1", "2", "3 "});
Foreach (string key in qrecords. Keys)
{
Var column = qrecords [key];
}

Update operation:
Because the TCT structure of TC does not provide the function of directly updating a field in the record, you can only retrieve all the fields of the relevant record and then update all the fields (this approach is inefficient, but some fields can be updated in MONGODB ). Therefore, you need to combine the syntax in the query and creation operations to find the corresponding record, and then use the PutColumns method to update the record. The format is as follows:
Copy codeThe Code is as follows:
Var qrecords = TokyoTyrantService. QueryRecords (TTPool. GetInstance (), new Query (). NumberEquals ("olid", 1 ));
Foreach (var k in qrecords. Keys)
{
Var column = qrecords [k];
... Data Binding operation
TokyoTyrantService. putColumns (TTPool. getInstance (), column ["olid"], columns, true); // column ["olid"] is the primary key, similar to the primary key in the database. It is used as the query condition and the fastest speed.
}

Delete operation
This operation can be performed in two ways. One is to query the records that meet the conditions and then delete (delete in sequence ), one is to directly Delete the primary key to be deleted (the efficiency is higher than the former ). First (you can query fields that are not required, and delete the primary keys of the corresponding results)

Copy codeThe Code is as follows:
Var qrecords = TokyoTyrantService. QueryRecords (TTPool. GetInstance (), new Query (). NumberEquals ("userid", 1 ));
Foreach (var k in qrecords. Keys)
{
Var column = qrecords [k];
... Data Binding operation
TokyoTyrantService. Delete (TTPool. GetInstance (), column ["olid"]); // column ["olid"] is the primary key, similar to the primary key in the database
}


(Delete key value records with an olid of 1, 2, 3, or 4. Only records with the primary key as the condition can be deleted ):
TokyoTyrantService. DeleteMultiple (pool, new string [] {"1", "2", "3", "4 "});

Create an index
TC supports the following types of field indexes (numeric and numeric ):
Copy codeThe Code is as follows:
/// <Summary>
/// Index type
/// </Summary>
Public enum IndexOption: int
{
LEXICAL = 0, // # Text Index
DECIMAL = 1, // # numeric index
TOKEN = 2, // # mark the inverted index.
QGRAM = 3, // # QGram inverted index.
OPT = 9998, // #9998, optimize the index
VOID = 9999, // #9999, remove the index.
KEEP = 1 <24 // #16777216, KEEP existing indexes.
}


For example, the frequently used field index settings in online tables are as follows:
Copy codeThe Code is as follows:
TokyoTyrantService. SetIndex (pool, "olid", IndexOption. DECIMAL );
TokyoTyrantService. SetIndex (pool, "userid", IndexOption. DECIMAL );
TokyoTyrantService. SetIndex (pool, "password", IndexOption. LEXICAL );
TokyoTyrantService. SetIndex (pool, "ip", IndexOption. LEXICAL );
TokyoTyrantService. SetIndex (pool, "forumid", IndexOption. DECIMAL );
TokyoTyrantService. SetIndex (pool, "lastupdatetime", IndexOption. DECIMAL );

3. Other Common Operations

Copy codeThe Code is as follows:
LimitTo (int max, int skip): similar to the LIMIT Method in MYSQL, where max is like TOP in mssql, and skip indicates how many records are skipped (similar to the Skip method in LINQ)
Vanish (TcpClientIOPool pool); clear all records
QueryRecordsCount (TcpClientIOPool pool, Query query) // Query the number of records under a specified condition
GetRecordCount (TcpClientIOPool pool) // returns the total number of records in the current table
GetDatabaseSize (TcpClientIOPool pool); // gets database (table) Information
IteratorNext (TcpClientIOPool pool) // an iterator used to traverse all records


4. Because it is compatible with Memcached, it provides method support (key-value pairs)

Copy codeThe Code is as follows:
Put (TcpClientIOPool pool, string key, string value, bool overwrite) // This operation method will not get the information returned by the server like Put
PutFast (TcpClientIOPool pool, string key, string value) // stores key-value pairs quickly (no longer obtains the server-side response information). If the key value already exists, it will be overwritten.
PutMultiple (TcpClientIOPool pool, IDictionary <string, string> items) // Add multiple values at a time
Delete (TcpClientIOPool pool, string key) // Delete the record with the specified key
DeleteMultiple (TcpClientIOPool pool, string [] keys) // deletes records of a specified key group
Get (TcpClientIOPool pool, string key) // gets the record of the specified key (single record)
GetSize (TcpClientIOPool pool, string key) // gets the size of the specified key
GetColumns (TcpClientIOPool pool, string [] keys) // obtain records of a specified key group (multiple Records)

5. Sort

Public enum QueryOrder
{
STRASC = 0, // # sorting type: indicates the ascending order of the text content in the text field in the dictionary
STRDESC = 1, // # sorting type: indicates the descending order of the text content in the text field in the dictionary
NUMASC = 2, // # sorting type: indicates the ascending order of the value size.
NUMDESC = 3 // # sorting type: indicates the descending order of the value size.
}

Usage (such as descending order and taking the first 16 records ):

Qrecords = TokyoTyrantService. QueryRecords (pool, new Query (). OrderBy ("userid", QueryOrder. NUMDESC). LimitTo (16, 0 ));
Note: Try to avoid sorting large datasets (such as million records), which is time-consuming. Therefore, try to specify the query conditions before OrderBy to reduce the size of the query result set.

Other Instructions:
The startup parameters of TT (Here we take the TCT type as an example ):
Note: There are some tests on the speed of TC + TT, MONGODB, and Redis on the Internet, so here I think it is necessary to introduce the startup parameters of TT, because this will be related to the final test results.
Because both use the MMAP mode, and TC + TT must use the following parameters to use MMAP:

Xmsiz: Specify the TCHDB extended MMAP memory size. The default value is 67108864, that is, 64 M. If the database file exceeds 64 M, only the front branch is mapped to the memory, therefore, the write performance will decrease.
Bnum: specifies the number of bucket arrays. We recommend that you set bnum to 0.5 ~ The hash distribution of keys is more even, reducing the time complexity of binary search in the bucket.

For example, if there are million records, run the following command to start ttserver:
Ttserver-host 10.0.4.66-port 11211-thnum 1024-dmn-pid/ttserver. pid-log/ttserver. log-le-ulog/ttserver/-ulim 256 m-sid 1-rts/ttserver. rts/ttserver/database. tct # bnum = 1000000 # rcnum = 1000000 # xmsiz = 1073741824 (Note: 1073741824 = 1G)

Of course, TTServer has startup configurations (repeated) for different databases (six types of TC Support) with corresponding parameters ), this results in a big difference between the query result and the result of data insertion. For more information, see this link.

Below I will explain the test results of TC + TT (only the TCT file type is used, and the other 5 types are much faster than this type) and MONGODB:
Machine is a common desktop: 1.5 GB memory + gCPU, 64-bit centos machine, GB hard disk.

Mongodb (centos 64bit ):
Insert 1000000 records, time consumed: 250377 milliseconds
For 1000000 records, query 10000 records, time consumed: 8100 milliseconds (occasionally 7500 milliseconds) (query the "_ id" primary key speed is up to 6995 milliseconds)
Query 1000000 records for 100000 records, time consumed: 77101 milliseconds

Ttcache (centos 64bit, using the above startup parameters ):
It takes 1000000 milliseconds to create 589472 data records
It takes 1000000 milliseconds to query 10000 data records
It takes 1000000 milliseconds to query 100000 data records

Note: The query conditions change dynamically to simulate the actual production environment.
It is found that MONGODB is inserted at least twice faster than TTCACHE (MONGODB is also inserted in WINDOWS), but the speed of 10000 queries is about 40%-50% slower. The query and insert operations Connect the server every time you perform an operation. When the operation ends, the current link is placed in the Link pool, instead of enabling a long link for batch operations. The client used by TTSERVER is the tool in this article, and MONGODB uses MongoDB. Driver.
The following figure shows the MSSQL database operation result:
Batch create 1000000 data records, which takes 9020196 milliseconds
Batch query of 10000 data records, which takes 106040 milliseconds
Batch query of 100000 data records, which takes 773867 milliseconds

I want to perform this type of test, or do not use any WINDOWS (try to use MONGODB to insert data quickly under the WINDOW) or other operating systems. Instead, use LINUX (64-bit as much as possible ). Of course, the memory should be as large as possible, because although TC + TT is already very memory-saving (it must be in line with Japan's national conditions and work more with fewer resources), if you want to increase the query and insertion speed, we recommend that you test the memory above 4 GB.
Try. MONGODB has high requirements on memory (including CPU ).

Because mongodb is very fast to insert, and a large number of new files can be created in the database to store new data (unlike using a data file using TCT ), therefore, the performance of data insertion with a higher level is still stable. It seems that it is feasible to regard it as a distribution solution for massive data storage. Of course, I am currently considering an architecture that combines MongoDb and TC/TT to implement read/write splitting (that is, using TC as the read database slavedb, with high concurrency and query speed ), mongoDb is used as the write database masterdb (fast update and insertion speed ). The distributed MongoDb data files correspond to the files in the front-end TC in sequence (use the C # code to implement data synchronization and logical calls between the two), so as to combine the results of their respective advantages. Of course, this is just an idea, and it is getting farther and farther away from the text content.

Well, today's content will be selected here.
Download link: http://tokyotyrantclient.codeplex.com/