Redis Practice Notes

I recently practiced redis In the project. I have encountered and solved several problems in the process and recorded them.    
  Why redis
Our project is an improvement to the original cache system. The application scenarios are Forum posts, replies, top posts, and operation logs. The original system will be replaced.AlgorithmThe cold data in the memory cache is gradually swapped out from the memory, and the memory objects are serialized into XML files for persistence to the disk. The memory cache aims at the access speed and shares the access pressure on the backend databases; the XML File Cache is designed to avoid avalanche, that is, when the system restarts, a large number of identical requests will impact the backend database because the cache is not fully filled; when the project was initially taken over, the boss of the company was told that the XML File Cache must be retained. Where is the problem with the original system?
Understanding the bottleneck of the original system is the first step. The main problems of the original system are: [1] The operation granularity is too large. operations such as shujia 1 will cause serialization and deserialization of the entire dataset data object. [2] Because dataset itself is a complex data structure, on the one hand, it will occupy more memory, on the one hand, serialization and deserialization will be more time-consuming than simple objects [3] dataset serializes XML files. xml files may have concurrent reads and writes. This is also a frequent fault.   In this way, we can sort out the key points of the design: [1] provides memory cache service [2] A persistent cache is required. After the system is restarted, it can be loaded from the persistent cache to the memory to avoid avalanche. [3] solve the problem of Memory Object lock and file lock [4] Since data is read from a relational DB, the associations between data have been established.     The starting point of thinking about the new solution is from the data operation granularity. If we follow the previous solution, the Operation granularity is the DataSet object of the entire post, reply, related post, and operation log, it is too costly to modify only one or two fields. Therefore, we decided to reduce the granularity of data operations. First, we split the Business Objects in dataset. This is the first step, this is what we should do at the very least. After the split, the object container no longer uses dataset. We call this granularity ①. After the split, the granularity is reduced again, the goal is to control the granularity to the field level. The judgment here is based on the business logic. For example, if an operation log reads and writes an object at a time, it does not have much benefit to put its Read and Write granularity at the field level; however, such as the number of visits to the primary post plus 1, it is very suitable for the field-level granularity. We call this granularity level ②. memcache can solve the memory cache issue well, however, the following problems exist: 1. it is easy to support granularity ①, but it is a bit difficult to construct and maintain the logic of the object, and there are too many interactions with memcache; 2. on this basis, the persistent cache must be implemented. What if it is nosql? After comparison, redis can replace the original memory cache and XML File Cache: [1] can be used as memory cache [2] Data Persistence [3] support for multiple data structures and operation granularity ②   Learning redis My Learning reference for redis is mainly focused on:

• Official Website http://redis.io/
• Redis command http://redis.io/commands
• Redis protocol http://redis.io/topics/protocol.
• Nosqlfan redis Data Summary topic http://blog.nosqlfan.com/html/3537.html

After getting familiar with the commands, I carefully read the redis protocol and summarized it in [Erlang 0019] redis protocol interpretation and implementation (. net & Erlang), through the Protocol learning actually has a psychological Bottom Line: as long as the function can be achieved through the protocol, we can implement in the client. nosqlfan collects a lot of excellent redis data. I have reserved some time to read the aboveArticle.   Servicestack. redis practices Redis C # client I chose servicestack. redis, compared with booksleeve redis-sharp and other solutions, provides a complete set of mechanisms for converting strong type objects from redis data structures. Let's take a look at an example to learn about servicestack. how redis organizes data. The object class we use is defined as follows:

  Public   class  User 
{< br>  Public  User () 
{< br>  This . blogids =  New  List  long  (); 
}< br>  Public   long  ID { Get ;  set ;}< br>  Public   string  name { Get ;  set ;}< br>  Public  List  long  blogids { Get ;  set ;}< BR >}

Use the followingCodeSegment, we store two pieces of data to redis:

 
Using(VaRRedisusers = redisclient. gettypedclient <user> ())
{
VaRAyende =NewUser {id = redisusers. getnextsequence (), name ="Oren eini"};
VaRMythz =NewUser {id = redisusers. getnextsequence (), name ="Demis Bellot"};
Redisusers. Store (ayende );
Redisusers. Store (mythz );
}
 

Let's take a look at the results in redis:

Redis127.0.0.1:6379[1]> Keys *
1)"SEQ: User"
2)"IDS: User"
3)"URN: User: 1"
4)"URN: User: 2"
 

Let's check the data types one by one:

SEQ: User	String	Maintain the ID auto-incrementing sequence of the current type of users, and use it as the unique ID of the object
IDS: User	Set	List of all object IDs of users of the same type
URN: User: 1	String	User object

SEQ: the user maintains the ID sequence of the user type.Redisusers. getnextsequence ()

Public LongGetnextsequence (IntIncrby)
{
ReturnIncrementvalueby (sequencekey, incrby );
}
Public LongIncrementvalue (StringKey)
{
ReturnClient. incr (key );
}
 

HereSequencekey is"SEQ: User", Then, we can save an object to redis to see what the other two keys are:

  Public T store (T entity)
{
 VaR Urnkey = entity. createurn ();
 This . Setentry (urnkey, entity );

 Return Entity;
}
 //  Entity. createurn (); the result is "urn: User: 1"  
          Public   Void Setentry ( String Key, T value)
{
 If (Key = Null )
 Throw   New Argumentnullexception ( "  Key  " );

Client. Set (Key, serializevalue (value ));
Client. registertypeid (value );
}

 Internal   Void Registertypeid <t> (T value)
{
 VaR Typeidssetkey = gettypeidssetkey <t> ();
 VaR Id = value. GETID (). tostring ();

If ( This . Pipeline! = Null )
{
 VaR Registeredtypeidswithinpipeline = getregisteredtypeidswithinpipeline (typeidssetkey );
Registeredtypeidswithinpipeline. Add (ID );
}
 Else 
{
 This . Additemtoset (typeidssetkey, ID );
}
}

The typedidssetkey here is"IDS: User"

  IDS: the user is equivalent to an index that contains all the IDs of users of the same type. because such group information is maintained, it is easy to implement the getall <user> () function;   In redis-CLI, query get urn: User: 1. the returned value is in JSON format: "{\" ID \ ": 1, \" Name \": \ "Oren eini \", \ "blogids \": [1]} "servicestack. redis implements a set of serialization functions, fastest JSON serializer. net released supports poco (plain old CLR object) serialization. in practical applications, because the data we use comes from relational databases and includes associations, we do not need such an object organization method; we only need to express one-to-many relationship in the relational data in redis; here I expanded and modified the redisclient implementation, because the redisclient itself has passed Partial Mode Split into several files, so it is easy to set the changed code in the same code file. for specific business object storage, the primary post and reply posts are modified at the field level. Therefore, it is designed as a hash structure. The read and write of other sub-objects are measured in objects and designed as poco for persistence;

Problems with pipeline Pipeline   Using pipelines can reduce the number of round-trips from the client to redis, but servicestack is used. redis encounters such a problem. For example, if you want to store a list <log> All, the Code cannot be written as follows:

 
% Method 1
Logs. foreach (n =>
{
Pipeline. queuecommand (r =>
{
(Redisclient) R). Store <oplog> (N, N. getobjectid (), N. geturnkey ());
(Redisclient) R). expire (N. geturnkey (), datalifetime );
});
});
 

It should be written as follows:

 
% Method 2
Logs. foreach (n =>
{

Pipeline. queuecommand (r => (redisclient) R). Store <log> (N, N. ID, N. geturnkey ()));
Pipeline. queuecommand (r => (redisclient) R). expire (N. geturnkey (), datalifetime ));

});
 

Why? The addcurrentqueuecompletableoperation method of redisqueuedoperation will execute currentqueuedoperation = NULL; if you follow the first write method, the callback function will be lost, resulting in the absence of timely extraction of returned values, in subsequent operations, the backlog of result information is obtained first, and an exception occurs. The second method avoids this problem.

 
Protected Virtual VoidAddcurrentqueuedoperation ()
{
This. Queuedcommands. Add (currentqueuedoperation );
Currentqueuedoperation =Null;
}
 

Redis tools The redis-cli client of redis is not very easy to use, and neither the backspace key nor the arrow can be used normally. This does affect the efficiency. You still need to find a suitable tool. I like it very much: Redisconsole Https://github.com/ptrofimov/RedisConsole/downloads This tool is very useful for learning, but once the data size increases, the list on the left will be messy, and it will be suspended when you click it. Therefore, it is recommended to use it only in the learning phase;

In an environment without a window, it is also a good choice to start an Erlang client.   Which of the following is the Web-based management tool I chose? Redis admin UI This servicestack. redis supporting project, modify the Web. config can be used, address in this: http://www.servicestack.net/mythz_blog? P = 381

Redis books and materials:[1]Big Data Glossary-O 'Reilly media [2] The little redis bookByKarl Seguin [3] redis Cookbook (O 'Reilly media, 2011) [4] professional nosql (wrox, 2011)

 I just saw an article on nosqlfan: <is memcached really outdated?> It must be transferred:Original article: http://blog.nosqlfan.com/html/3729.html

Redis has been booming over the past two years, and redis is often referred to as a challenger to memcached on the desktop. Comparison between redis and memcached is everywhere. However, does redis surpass memcached in terms of functionality, performance, and memory usage efficiency?

The following is an answer from the redis author on stackoverflow. The corresponding question is "is memcached A dinghuur in comparison to redis?". (Compared with redis, is memcached really outdated ?)

• You shoshould not care too much about performances. redis is faster per core with small values, but memcached is able to use multiple cores with a single executable and TCP port without help from the client. also memcached is faster with big values in the order of 100 K. redis recently improved a lot about big values (unstable Branch) but still memcached is faster in this use case. the point here is: nor one or the other will likely going to be your bottleneck for the query-per-second they can deliver.
• There is no need to worry too much about performance because the performance of both is high enough. Because redis only uses a single core, while memcached can use multiple cores, redis has higher performance than memcached in storing small data on average. Memcached has higher performance than redis in data of more than kb. Although redis has recently optimized its performance in storing big data, it is inferior to memcached. The conclusion is that no matter which one you use, the number of requests processed per second will not become a bottleneck. (For example, the bottleneck may be on the NIC)
• You shoshould care about memory usage. For simple key-value pairs memcached is more memory efficient. If you use redis hashes, redis is more memory efficient. Depends on the use case.
• For memory usage efficiency and simple key-value storage, memcached memory usage is higher. If redis uses the hash structure for key-value storage, due to its combined compression, the memory usage is higher than that of memcached. Of course, this is related to your application scenarios and data features.
• You shoshould care about persistence and replication, two features only available in redis. Even if your goal is to build a cache it helps that after an upgrade or a reboot your data are still there.
• If you have requirements on data persistence and data synchronization, we recommend that you choose redis because memcached does not have these two features. Even if you just want to upgrade or restart the system and the cache data will not be lost, it is wise to choose redis.
• You shoshould care about the kind of operations you need. in redis there are a lot of complex operations, even just considering the caching Use Case, you often can do a lot more in a single operation, without requiring data to be processed client side (a lot of I/O is sometimes needed ). this operations are often as fast as plain get and set. so if you don't need just get/set but more complex things redis can help a lot (think at timeline caching ).
• Of course, you have to talk about your specific application requirements. Compared with memcached, redis has more data structures and supports richer data operations. Generally, in memcached, you need to get the data to the client for similar modifications and then set it back. This greatly increases the number of network I/O operations and data volumes. In redis, these complex operations are generally as efficient as general get/set operations. Therefore, if you need the cache to support more complex structures and operations, redis will be a good choice.

 

 
Note: The practical version of redis 2.4.9
 

  16:30:30 update

Rdbtools is a parser for redis 'dump. RDB files. The parser generates events similar to an xml sax Parser, and is very efficient memory wise.

In addition, rdbtools provides utilities:

1. Generate a memory report of your data processing SS all databases and keys
2. Convert dump files to JSON
3. Compare two dump files using standard diff tools

Https://github.com/sripathikrishnan/redis-rdb-tools