CYQ. Data V5 distributed cache Redis application development and implementation algorithm principle introduction, cyq. dataredis

CYQ. Data V5 distributed cache Redis application development and implementation algorithm principle introduction, cyq. dataredis
Preface:

Since the CYQ. Data framework provides large functions such as database read/write splitting, distributed cache MemCache, and automatic cache, it has entered the stage of frequent detail polishing and optimization.

We can see from the following update list that the function is updated 100 times in three months:

305: When the view name is duplicated, The MDataTable code is simplified, and the ReadFromDbDataReader of the MDataTable is canceled (all read using the CreateFrom (sdr) method. () 306: the column structure obtained through Reader is optimized. (This method is unreliable. You need to modify the DataType, Size, Scale, DalType, and other parameters of the metadata.) 307: added the reload Set (key, value, state) for the MDataTable and MAction Set methods. During Cyclic value assignment, you can assign a value to state 2 () 308 when batch update is generated: added the intimate function: Custom parameterized statement @ symbol, which is automatically replaced with the corresponding symbol in each database? Or: The symbol is compatible with multiple databases. () 309: enhanced MDataTable binding to Winform and WFP () 310: fixed the automatic cache problem of MProc ExeMDataTableList) 311: the GetMapTable function of DBTool adds the compatibility between dashes and spaces in the table name () 312: CYQ. data. projectTool upgraded version to V2.0 (English environment supported) () 313: XHtmlAction vigorously adjusted and upgraded (detailed changes) () 314: MDataRow: the Bind functions of SetToAll and MDataTable support XHtmlAction object () 315: The Dtd file is converted into a resource file in V5, and automatically decompress the file to improve the user experience (to reduce the file size, I almost got blind when I deleted the comment.) () 316: processing the MDataTable The Bug caused by the GetChange method and the initial status of the data generated by CreateFrom are set to 1 () 317: the processing of the radio tag by XHtmlAction. () 318: added the clearflag label of html in XHtmlAction [value: 0 (clear InnerXml) or 1 (delete node)] (used to process the label when the node is not processed) 319: XHtmlAction processes img, select, input checkbox, and other html nodes. () 320: XHtmlActon rewrite Load method (optimized loading, automatic identification, and processing of the escape symbol) () 321: AppConfig reduces an Xml-related configuration item (UseFileLoadXml) () 322: MDataTable corrected the Select method (corrected to reference) () 323: added Lock () to the GetTables method of DBTool 324: fixed the problem of caching when a failure occurred) 325: AppConfig is added. runPath attribute, get the folder where the framework is running () 326: Processing Configuration tool ProjectTool upgrade () 327: Fixed MDataCell second value assignment to binary data () 328: Adjusted MDataRow: the row status of CreateFrom (external data) starts from 1; LoadFr Om (external data) status is related to its own value () 329: Corrected automatic cache () 330: oracle fixed the 1st page paging problem [when the sorting condition is a string] () 331: AppConfig added the NoCacheTables attribute, allowing specified tables not to be cached (when automatic cache is enabled) () 332: XHtmlAction added the processing of & symbols in the Xml document () 333: XHtmlAction optimized the SetForeachEventHandler event () 334: added support for MSSQL User-Defined table types in the setm m method of MProc () 335: StaticTool: improved the conversion performance of ChangeType method () 336: the ToList <T> method of MDataTable adds a judgment condition to prevent inheritance Or MBase remote entities use Emit () 337: JsonHelper: optimized to Improve the Performance of a large number of ToString () 338: AutoCache: when there are more than 0.1 million data records, the index value is not automatically cached () 339: MDataRow: the index value is corrected (The problem occurs when the field name is two symbols and the number of fields is greater than 10) 340: Internal SQL statement optimization () 341: MAction: Select method (optimize the code of the total number of query records, and use the automatic cache function to avoid re-calculation on pages) () 342: appConfig. cache. ignoreCacheColumns: You can specify that some columns in the table do not update the Cache during update operations () 343: JsonHelper added support for array detection () 344: JsonHelper supports binary and Base64 Conversion Change () 345: cache processing after DBTool. CreateTable or DBTool. DropTable ). 346: MAction and MProc cancel the SetAopOn and SetAopOff methods, which are the SetAopState methods (simplified method, and more States can be processed at the same time, including disabling automatic cache) () 347: the Select method of MDataTable is enhanced (comparison of floating point numbers) () 348: the cache time of AutoCache is changed to the time configured by DefaultCacheTime (you can configure the automatic cache time by yourself) 349: MProc's ExeMDataTableList method adds support for batch statements in Oracle () 350: optimize statements in Oracle table structure () 351: MDataTable's Merge method Amendment () 352: method for adding a heavy load to AddGlobalThread of ThreadBreak () 353: CacheManage Provides the PreLoadDBSchemaToCache method () 354: JsonSplit processing IsJson judgment problem () 355: MDataTable AcceptChanges (Update method, processing when AppConfig is configured. DB. problems caused by DeleteField) () 356: DBTool. error generated when the GetColumns method is used to process the "\ r \ nwhere" scenario () 357: added support for select a as B alias in the Text Database (NoSqlCommand) 358: MAction's problem of not clearing old values when processing multiple Fill operations () 359: ORM (OrmBase and SimpleOrmBase) added SetAopState method () 360: AutoCache: Processing MAction Fill side Cache reference (changed to clone to avoid multiple Fill pointing to the same cache) () 361: added Description attribute for MDataTable. () 362: DBTool's GetColumns added support for table ing () 363: corrected the ResetTable method of the text database (the original table was not cleared) () 364: modify and remove the internal MD5 (the encryption algorithm in win2008 causes an exception by default) () 365: remove the constraints of the ing table (support for more external ing) () 366: when the read/write splitting is corrected, insert... select statement processing to database sharding () 367: SqlCreate converts Oracle date conditions. () 368: SqlCreate added GUID type detection () 369: OrmBase, SimpleOrmBase delayed loading initialization () 370: MAction In the Insert statement, for Oracle, mysql and other placement (obtain the maximum value) Transactions () 371: the default InsertOp option of MAction during Insert is changed to: ID, originally (Fill) () 372: JsonHelper. toJson adds support for List <MDataTable> and List <DataTable> () 373: DBBase processes the DataBase name returned by Oracle. () 374: minor detail Optimization for Oracle Loading Method () 375: StaticTool converts guids in ChangeType () 376: SqlCompatible adds a pair (+ |), Left and Right functions () 377: Oracle's ODP. NET parameter added BindByName set to true () 378: MDataRowCollection AddNew method, processing Winform (when the DataGrid is bound) When blank rows and data rows are clicked back and forth to add blank data. () 379: MAction SetPara added overload method () 380: When the where condition of MAction Update is Error, the RecordsAffected value is changed from 0 to-2; () 381: MDataTable fixed the returned values of batch Update () 382: added IsIgnoreDeleteField internal attribute () 383: XHtmlBase corrected Xml loading () 384: sqlValue adjusted the names of two names (GUID and ISNULL) () 385: MDataTable corrected the number judgment problem where the Select condition is <= () 386: autoCache (added the Escape attribute in JsonHelper and added the ToJson overload in MDataTable) not processed \ Escape replacement of n () 387: MDataTable ToJson for null data, default output xx: null value () 388: Oracle (DBTool. getTables) added filter to the view () 389: JsonHelper fixed the issue of entity nesting and added support for arrays () 390: MDataTable AcceptChange: Batch update without a primary key () 391: MDataTable adds the GetIndex method (counts the index of rows that meet the condition) () 392: noSqlAction Text Database fixed the problem that the last data record could not be deleted () 393: added index values or assigned values to MDictionary. () 394: The OnForeach parameter of XHtmlAction and RSS is changed from Dictionary to MDictionary () 395: JsonHelper corrected the output and restoration of arrays () 396: JsonHelper corrected the Json Nesting Problem. () 397: MDataTable optimized batch update. () 398: ToTable () Adjustment of MDataRow and MDataColumn () Adaptation (New Smart Tips) () 399: MySql Stored Procedure Out value. () 400: the MySql batch method solved the problem that the auto-increment ID of the Bit type and empty table was set to 1 () 401: JsonHelper, NoSqlAction small optimization adjustment) 402: The AcceptChanges of the MDataTable adds the Truncate attribute () 403: The GetJosnValue (json written incorrectly) of JsonHelper is changed to GetValue () 404: concurrent processing of last data deleted by NoSqlAction Text Database () 405: DBTool. getColumns fixed the problem of using the table structure for group by statements without where () 406: AppConfig added the SetConn method (Added link cache at the same time) () 407: sqlCreateForPager handles the order by aa of pages. bb does not contain asc () 408: NoSqlAction (fixed the 404th issue that cannot be inserted in batches after deletion) () 409: the AcceptChanges of MDataTable is used to process duplicate batches (when no transaction object is generated externally) () 410: keywords compatible with multiple statements of SqlCompatible (Case Insensitive) 411: added table field Description for Description of MDataTable () 412: StaticTool optimized GetDbName details () 413: added support for Redis distributed cache (configured AppConfig. cache. redisServers) () 414: Add backup node support for Redis and MemCache (configure AppConfig. cache. redisserversbk, AppConfig. cache. memCacheServersBak)

In fact, more time is spent on the ASP. NET Aries business development framework, and all the above and below are reconstructed.

A few days ago, I decided to integrate Redis.

Let's share the following experiences:

First thought:

At first, I refused to call a third-party client dynamically (too many associated dll files ).

I plan to support Redis recently. It's a bit of a compromise. I just need to load it dynamically:

Consider introducing StackExchange. Redis or ServiceStack. Redis?

Looking at these DLL, It's too heavyweight, and it's hard to reflect the method!

Intermediate thinking paused for a while...

Lightweight: Bettle. Redis

When searching for Redis API materials, I did not intend to find this open-source lightweight Bettle. Redis.

It was only 46 KB after the source code was compiled.

Just a few moments later, I found the following problems:

1: although it is 46 K, the code references the other two three dll (too many dependencies ):

2: The method used does not conform to the usage habits. A command type corresponds to a class.

3: horizontal scaling of clusters is not supported (Consistent Hash is not supported ).

4: the code is written in versions earlier than. NET 4.0. (The CYQ. Data framework supports 2.0 and the code is changed to a pain point)

Therefore, the above reason is probably that it is not popularized, and it is also the reason why it is not selected for integration.

However, it is open to the source code, which is a little inspiration and reference for me.

Redis API Literacy:

In the process of deciding to support Redis, it took a lot of time to scan the Redis documentation:

 

For more command details, see: http://doc.redisfans.com

From such a bunch of commands, find the basic commands: Get, Set, Exists, Expire, Info, and no Add.

Most of other commands can be implemented using basic commands, which is ignored.

After a lot of concentrated thinking in a short period of time, I decided to implement it myself:

How to reposition:

The framework has already integrated MemCache, while Redis and MemCache are similar.

Some common things can be reused:

1: hash algorithm.

2: consistent Hash (horizontal scaling ).

3: SocketPool.

4: ServerPool.

5: serialization (compression)

The rest is the interaction and usage of Socket and Redis.

The following are the protocol specifications of Redis, which were discovered only after I implemented Redis-related functions:

Protocol Specification redis allows the client to connect in TCP mode, default port 6379. All transmitted data ends with \ r \ n. Request format * <number of arguments> \ r \ n $ <number of bytes of argument 1> \ r \ n <argument data> \ r \ n example: * 1 \ r \ n $4 \ r \ nINFO \ r \ n response format 1: simple string, non-binary secure string, usually status reply. +, For example, + OK \ r \ n 2: error message. -Start with-ERR unknown command 'mush '\ r \ n3: integer. : Start, for example: 1 \ r \ n4: large block reply value, up to 512 M. $ Starts with + Data Length. Example: $4 \ r \ mush \ r \ n5: Multiple replies. *, For example, * 2 \ r \ n $3 \ r \ nfoo \ r \ n $3 \ r \ nbar \ r \ n

Tossing:

Bettle. Redis has the source code. You can check the implementation, so you didn't find the Protocol Specification:

After several hours of introduction, code adjustment, and testing.

When the result is success, NetworkStream reports an exception when the Set data is too large: This stream does not support the Seek operation.

Is there a size limit on the Redis Set?: I tried Bettle. Redis and found that it was normal, dream B.

After code debugging, it is found that the Socket implementation (Socket. Send) of Bettle is different from that of the Socket pool (NetworkStream. Write.

Bettle. Redis constructs all the protocols for one-time Socket. Send (byte []).

Why is the default cache pool of NetworkStream too small? :MemCache is used to Set a large amount of data. It is found that NetworkStream has not thrown an exception and dream B.

I doubt the Redis protocol? :Transform the code, split the protocol, first send: $ length, and then send data, it turns out that it is normal, speechless asked heaven!

After one day of hard work, four classes are added to the Cache directory, and algorithm optimization is performed to clear useless code.

After Redis is supported, we found that the cyq. data. dll size has not changed, and the result has exceeded expectations. It is very good!

The final source code structure is:

 

Complete source code has been submitted in: https://github.com/cyq1162/cyqdata

Redis usage:

AppConfig. Cache. RedisServers = "127.0.0.1: 6379,127.0 .0.1: 1121"; // Configuration Enabled, AppConfig. Cache. redisserversbk = "127.0.0.1: 6379"; // backup configuration. CacheManage cache = CacheManage. redisInstance; // operation object cache. add ("obj", cache. cacheTable); // Add DataTable MDataTable obj = cache. get <MDataTable> ("obj"); Console. writeLine (obj. rows. count); Dictionary <string, string> dic = new Dictionary <string, string> (); dic. add ("fall by", "http://www.cnblogs.com/cyq1162"); cache. add ("dic", dic); // Add the field Dictionary <string, string> dicObj = cache. get <Dictionary <string, string> ("dic"); Console. writeLine (dicObj ["passing by autumn"]); cache. remove ("dic"); // Remove Dic bool hasKey = cache. contains ("dic"); // checks whether a Console exists. writeLine (hasKey); Console. read ();

Result:

 

Improved Storage types:

Because Redis Get only supports strings, in order to support any type, I must improve the algorithm:

1: Archive: when the target is an object = (k will be compressed)

2: 1st bytes of data: archive data type.

3: When obtaining data: accurately restores the data type based on the 1st bytes.

(Aaa is Set through the command line, while a0 is Set through the code, so the type of \ x02 is missing)

Therefore, the Framework relies on Set and Get to support any type of access files!

Improvements to distributed algorithms: 1. Horizontal node Scaling:

Internal implementationConsistent Hash Algorithm, Thus saving a lot of work:

A simple description is: Generate N hash values for ip1, and generate N hash values for ip2,... and then sort them. (finally, we can see who is new to the hash value of the key)

The following figure is used:

2: transfer of node faults:

During the test, I filled in an abnormal host and found that the read and write operations of the key assigned to the abnormal host did not respond:

(By default, I thought it would be automatically transferred to an adjacent host)

Default algorithm:

1: There is no automatic switch between adjacent hosts [think about the code: Active switching may lead to an avalanche effect (the accumulated pressure may put all servers down )].

2: There is a retry Connection Mechanism (try once every 2 minutes ).

Improved Algorithm: added the configuration of a backup machine (AppConfig. Cache. redisserversbk)

1: According to Hash, each host points to a backup machine. 2: when the host is abnormal, the proxy server of the backup machine is used for 15 minutes (that is, the host is checked once every 15 minutes. If the host is normal, the host service is restored ).

3: When the host recovers, it restores data from the backup machine and clears the data of the Backup Machine (Not Implemented)

Since N servers may be mounted at the same time, the backup server may archive information on multiple hosts.

Therefore, there are three algorithms:

1: No data is needed (the host can be cached again) 2: when the host is requested (check whether it has been mounted, if yes, read itself (if not) = read backup machine (with the removal Command issued at the same time) (if there is data) = return (write to the host at the same time) 3: When the host is requested (check whether it has been mounted, if the thread is enabled (read all the keys of the backup machine and check whether the Hash matches itself. If yes, read and write data from the backup and clear the data of the backup machine at the same time)

Summary:

So far, CYQ. Data has supported Redis, and it is more practical to use memCache and improved algorithms in distributed algorithms!

Of course, the details still need to be polished, and the code can be modified more concisely and beautifully.

In today's era of distribution, correct judgment and good use of distributed frameworks are a manifestation of capabilities.

Someone in the Group sent this message:

In fact, the previous problems can be ignored, because the final solution is to deploy Redis from Windows to Linux.

When the QPS is the maximum, I heard that there are more than 70 thousand (more than 30 thousand of the two Web sites, most of which are requests caused by ticket flushing)

Redis is not as good as Linux in Windows.

However, there is no pressure to adjust the architecture design scheme.

Finally, I found that the root cause of the problem lies in the lack of an architect with sufficient knowledge and thinking in people:. NET.

Don't be overwhelmed when you encounter any problems. Stick to the. NET camp, young man!