- Common commands: sadd/spop/smembers/sunion, etc.;
- Application Scenario: Redis set provides functionality that is similar to a list, especially if set is automatic, and when you need to store a list of data and do not want duplicate data, set is a good choice. and set provides an important interface for determining whether a member is within a set set, which is not provided by the list;
- Implementation: The internal implementation of set is a value is always null HashMap, in fact, by calculating the hash of the way to quickly row weight, which is also set can provide to determine whether a member is within the set of reasons.
- Common commands: Zadd/zrange/zrem/zcard, etc.;
- Scenario: Redis Sorted Set's use scenario is similar to set, except that set is not automatically ordered, and sorted set can sort members by providing an additional priority (score) parameter for the user, and is inserted in an orderly, automatic sort. When you need an ordered and not duplicated list of collections, you can choose to sorted set data structures, such as Twitter's public timeline can be stored as score by publication time, which is automatically sorted by time.
- Implementation: Redis sorted set of the internal use of HashMap and jump Table (skiplist) to ensure the storage and order of data, HashMap is placed in the member to the score map, and the jump tables are stored all members, The sorting is based on the score in the HashMap, the structure of the jump table can obtain the high search efficiency, and it is simpler to realize.
2, the memory management mechanism is different
In Redis, not all data is stored in memory. This is one of the biggest differences compared with memcached. When physical memory is exhausted, Redis can swap out some of the value of a long time to disk. Redis only caches all key information, and if Redis finds that memory usage exceeds a certain threshold, the operation of swap will be triggered, Redis according to "swappability = Age*log (size_in_memory)" calculates which key corresponding value needs to be swap to disk. The value corresponding to these keys is then persisted to disk and purged in memory. This feature allows Redis to maintain data that exceeds the memory size of its machine itself. Of course, the memory of the machine itself must be able to maintain all key, after all, the data is not swap operation. At the same time, because Redis will swap the data in memory to disk, the main thread of the service and the child thread that carries out the swap share this part of the memory, so if you update the data that requires swap, REDIS will block the operation until the child thread completes the swap operation before it can be modified. When reading data from Redis, if the value of the read key is not in memory, then Redis will need to load the corresponding data from the swap file and return it to the requester. There is a problem with the I/O thread pool. By default, Redis will be blocked, that is, all the swap files will be loaded before the corresponding. This strategy is relatively small in number of clients and is appropriate for bulk operations. But if you apply Redis to a large web site application, this is obviously not a big concurrency scenario. So Redis run we set the size of the I/O thread pool to perform concurrent operations on the read requests that need to load the corresponding data from the swap file, reducing the blocking time.
For memory-based database systems such as Redis and memcached, the efficiency of memory management is a key factor affecting system performance. The Malloc/free function in traditional C language is the most commonly used method of allocating and releasing memory, but this method has a great flaw: first of all, the mismatch of malloc and free for developers is likely to cause memory leaks, followed by frequent calls can cause a lot of memory fragmentation can not be recycled. Reduced memory utilization; Finally, as a system call, the system overhead is much greater than the normal function call. Therefore, in order to improve the efficiency of memory management, efficient memory management scheme will not directly use the Malloc/free call. Redis and memcached both use their own design of memory management mechanism, but the implementation of the method is very different, the following will be the memory management mechanism for each of the introduction.
Memcached uses the slab allocation mechanism to manage memory by default, and its main idea is to split the allocated memory into a specific length block to store the corresponding length of the Key-value data record in a predetermined size to completely resolve the memory fragmentation problem. The slab allocation mechanism is designed only for storing external data, which means that all key-value data is stored in the slab allocation system, while memcached other memory requests are applied through the normal malloc/free. Because the number and frequency of these requests determines that they do not affect the performance of the entire system slab allocation principle is fairly straightforward. As the figure shows, it first requests a chunk of memory from the operating system and divides it into block chunk of various sizes and blocks the same size into group slab Class. Among them, chunk is the smallest unit used to store key-value data. The size of each slab class can be controlled by setting up growth factor when the memcached is started. Assuming that the value of growth factor in the graph is 1.25, if the first group of Chunk is 88 bytes, the second chunk size is 112 bytes, and so on.
When the memcached receives the data sent by the client, it first selects the most appropriate slab Class based on the size of the data received, and then queries memcached the Slab A list of idle chunk within class can find a chunk that can be used to store data. When a database expires or is discarded, the chunk that the record occupies can be recycled and added back to the free list. From the above process we can see that memcached memory management system is high efficiency, and does not cause memory fragmentation, but its biggest drawback is that it can lead to space waste. Because each chunk allocates a specific length of memory space, variable length data does not fully utilize the space. As the figure shows, 100 bytes of data are cached into 128-byte chunk, and the remaining 28 bytes are wasted.
Redis memory management mainly through the source zmalloc.h and zmalloc.c two files to achieve. Redis in order to facilitate the management of memory, after allocating a piece of memory, it will be the size of the memory block to the head of memory. As shown in the figure, Real_ptr is the pointer returned after Redis calls malloc. Redis the size of the memory block to the head, size occupies a known amount of memory, is the length of the size_t type, and then returns RET_PTR. When memory needs to be freed, Ret_ptr is passed to the memory management program. By Ret_ptr, the program can easily calculate the real_ptr value, and then pass real_ptr to free to release the memory.
Redis by defining an array to record all memory allocations, the length of this array is zmalloc_max_alloc_stat. Each element of the array represents the number of memory blocks allocated by the current program, and the size of the memory block is the subscript for that element. In the source code, this array is zmalloc_allocations. ZMALLOC_ALLOCATIONS[16] represents the number of memory blocks that have been allocated lengths of 16bytes. ZMALLOC.C has a static variable used_memory used to record the total amount of memory currently allocated. Therefore, in general, Redis adopts the packaging of mallc/free, compared to the memcached of memory management method, it is much simpler.
3. Support of data persistence
Redis is a memory-based storage system, but it itself supports the persistence of memory data and provides two major persistence strategies: RDB Snapshots and aof logs. The memcached does not support data persistence operations.
Redis supports the persistence mechanism of saving snapshots of the current data into a data file, that is, a rdb snapshot. But how does a database that continues to write generate snapshots? Redis with the copy on write mechanism of the fork command. When the snapshot is generated, fork the current process out of a subprocess, and then loops through all the data in the subprocess, writing the data to the Rdb file. We can configure the timing of the RDB snapshot generation with the Redis save instruction, for example, configure 10 minutes to generate snapshots, or you can configure 1000 writes to generate snapshots, or multiple rules to implement them. These rules are defined in the Redis configuration file, and you can also set the rules at Redis run time using the Redis config set command without restarting the Redis.
Redis's Rdb file won't break, because the write operation is done in a new process, when a new Rdb file is generated, the Redis-generated subprocess writes the data to a temporary file and then renames the temporary file to the Rdb file through an atomic rename system call, In this way, Redis RDB files are always available when a failure occurs at any time. At the same time, Redis's Rdb file is also a link in the internal implementation of Redis master-slave synchronization. Rdb has his shortcoming, is once the database has the problem, then our RDB file saves the data is not brand-new, from the last Rdb file generation to Redis downtime this period of time data all lost. This can be tolerated in some businesses.
The full name of the AOF log is append only file, which is an append-write log file. Unlike the binlog of a general database, the AoF file is an identifiable plain text, and its content is a Redis standard command. Only commands that cause data changes are appended to the AoF file. Each command to modify the data generates a log, aof file will become larger, so Redis also provides a function, called aof rewrite. Its function is to regenerate a copy of the AoF file, a record in the new aof file only once, rather than as an old file, may record multiple operations on the same value. The generation process is similar to RDB, and is also a fork process that traverses data directly and writes new aof temporary files. In the process of writing a new file, all write logs will still be written to the old aof file and recorded in the memory buffer. When the redo operation completes, the log in all buffers is written to the temporary file at once. The Atomic Rename command is then invoked to replace the old aof file with the new AoF file.
AoF is a write-file operation, which is designed to write the action log to disk, so it will also encounter the write process described above. After aof call write write in Redis, the Appendfsync option is used to control the time that the call Fsync writes to disk, and the security intensity of the three settings below Appendfsync is gradually stronger.
- Appendfsync No when the Appendfsync is set to No, Redis does not actively call Fsync to sync the AOF log contents to disk, so it all depends on the debugging of the operating system. For most Linux operating systems, a fsync is performed every 30 seconds to write data from the buffer to disk.
- Appendfsync Everysec When the Appendfsync is set to Everysec, Redis makes a fsync call every second and writes the data from the buffer to disk. But when the fsync call of this time is longer than 1 seconds. Redis will take a delayed fsync strategy and wait another second. That is, in two seconds after the Fsync, this time the Fsync no matter how long it will be carried out. This is because the file descriptor is blocked at Fsync, so the current write operation is blocked. So the conclusion is that, in most cases, Redis will perform a fsync every second. In the worst case, a fsync operation is performed in two seconds. This operation is referred to as group commit in most database systems, that is, data that combines multiple writes and writes the log to disk one at a time.
- Appednfsync always when the appendfsync is set to always, each write operation invokes a Fsync, at which point the data is the safest and, of course, its performance is affected by the execution of Fsync each time.
For general business requirements, it is recommended to use a RDB approach for persistence, because RDB costs are much lower than aof logs and it is recommended that you use the AOF log for applications that cannot tolerate data loss.
4, cluster management of the different
Memcached is a full memory data buffering system, while Redis supports data persistence, but full memory is the essence of high performance after all. As a memory-based storage system, the size of the machine's physical memory is the maximum amount of data the system can hold. If the amount of data to be processed exceeds the physical memory size of a single machine, a distributed cluster needs to be built to extend the storage capacity.
The memcached itself does not support distributed, so distributed memcached can only be implemented on the client side through a distributed algorithm such as a consistent hash. The following figure shows the memcached distributed storage implementation architecture. Before the client sends data to the memcached cluster, it first calculates the target node of the data through the built-in distributed algorithm, and then the data is sent directly to the node for storage. However, when the client queries the data, it also calculates the node where the query data resides, and then sends a query request directly to the node to get the data.
Compared to memcached can only use the client to implement distributed storage, Redis is more inclined to build distributed storage on the server side. The latest version of Redis has already supported distributed storage capabilities. Redis cluster is a Redis advanced version that implements a distributed, single point of failure that does not have a central node and has a linear, scalable function. The following figure gives the distributed storage architecture of Redis cluster, in which the nodes communicate with each other through binary protocol, and the node communicates with the client through the ASCII protocol. On the placement strategy of the data, Redis cluster divides the value field of the entire key into 4,096 hash slots, and one or more hash slots can be stored on each node, meaning that the maximum number of nodes supported by the current Redis cluster is 4096. The distributed algorithm used by Redis cluster is also simple: CRC16 (key)% Hash_slots_number.
To ensure data availability under a single point of failure, Redis cluster introduces the master node and the slave node. In Redis cluster, each master node will have two corresponding slave nodes for redundancy. In this way, the downtime of any two nodes in the entire cluster will not cause data to be unavailable. When the master node exits, the cluster automatically selects a slave node to become the new master node.
