A nosql approach to everyone loves Redis: (4) Primary exploration of Redis master-slave replication architecture

introduction of Master-slave replication Architecture

Through the previous introduction, we are all on a single use of Redis for related practices, from this chapter, we will initially explore the Redis cluster, and the most classic cluster architecture is the master-slave replication architecture. So, first of all, let's see if God's horse is a master-slave replication architecture.

1.1 Read-write separation from relational database

With the continuous development of the website business, the number of users is increasing, the amount of data is increasing exponentially, and the number of access to the database increases linearly. Especially during the peak of user access, the increase of concurrent access, the load pressure of the database will increase, if the schema is not robust enough, the database server is likely to be under high concurrent access load pressure down, resulting in the failure of the data access service, resulting in the disruption of the site's business, the company and users caused double losses. So, there is a way to solve this problem, so that the database is no longer because of high load pressure to become the bottleneck of the site? There must be some answer.

At present, most of the mainstream relational database provides master-slave hot standby function, through the configuration of two (or more) database master-slave relationship, one database server can synchronize data updates to another server. Web site can take advantage of this function of the database, to achieve the database read and write separation, thereby improving the load pressure of the database .

Using the database read-write separation, the Web server when writing data, access to the primary database (master), the primary database through the master-slave replication mechanism to synchronize data updates to the slave database (Slave), so that when the Web server read data, you can obtain data from the database. This scheme makes it easy for Web applications to read data in a large number of read operations, and the primary database will only tolerate a small amount of write operations, and it can also be a double benefit solution for data hot backup.

1.2 mysql-based data replication process

We have just learned that the read and write separation of relational database can realize the master-slave architecture of the database, so what is the most important data replication in the master-slave architecture? MySQL as one of the most popular relational databases, by understanding the MySQL data replication process, will make our knowledge of Redis master-slave replication will help.

From the perspective of the overall, there are the following three steps to gather:

(1) Master changes the record to binary log (these are called binary log events, binary logs event);

(2) Slave copies the binary log event of master to its trunk log (relay log);

PS: It can be seen that an I/O thread (I/O threads) in the slave server is constantly listening for updates to the master's binary logs (binary log): If it does not sleep wait for master to generate new log events If there is a new log event, it will be copied to the relay log (Relay log) in the slave server.

(3) Slave redo the event in the relay log (Relay log) to reflect the changes on master to its own database.

PS: As you can see, there is a SQL thread (SQL thread) in the slave server that is reading events from the log and re-doing the events in it to update the slave data so that it is consistent with the data in master. As long as the thread is consistent with the I/O thread, the trunk log is typically located in the OS cache, so the overhead of the trunk log is minimal.

After a brief introduction, we initially learned what is the master-slave replication and how the data is replicated in the relational database. In this, we do not question in Redis how to achieve data replication? Don't worry, we first to practice, first of the master-slave copy to get a perceptual knowledge, and then from the perceptual awareness to understand the rational understanding. So,let ' s start doing.

second, simulation of master-slave replication architecture Practice on a single machine2.1 Copying two services to a specified disk folder

(1) Copy the Redis Service folder downloaded in the first article two copies, and give two folders named: Redismasterservice and Redisslaveservice;

(2) Copy the two folders to the folder specified in Windows, and I copy it to the e:\ to facilitate the subsequent start-up test;

2.2 Modifying the master and slave configuration files separately

(1) Modify the configuration file of the Master service (redis.conf)

Here, mainly to modify the master service to bind the IP address (that is, the master server IP address), I am here because it is on the local, so directly set to 127.0.0.1. By searching for "bind" in redis.conf, you can see that the default configuration already has a string of #bind 127.0.0.1, and all we have to do is uncomment the sentence.

PS: It is recommended to use an editor similar to EditPlus, notepad++, etc. to open the redis.conf configuration file, so that the lookup and editing are more straightforward. If you don't have any of these, then you can use Visual Studio to open to edit (if you don't even have a VS, I can only hehe, you use Notepad to edit it, click). Well, I have no editplus and notepad++, re-installed the system will not be installed these editors, you see through.

(2) Modify the configuration file for the slave service (redis.conf)

① Modify the port number of the slave binding: because both Master and slave are on a single machine, you need to modify the port number to differentiate between two redis services. If not modified, the default port number bit 6379, the modification method is also very simple, search "port" keyword, the port 6379 is changed to Port 6380. Note that as long as it is not 6379, you can change, 6,380 is the port number I set here.

② Modify the IP address of the slave binding, here and master consistent, all changed to bind 127.0.0.1 can;

③ modify the corresponding relationship between master and slave configuration: Search "slaveof" keyword, will find such a sentence: # slaveof <masterip> <masterport>. All we have to do is to cancel the comment and change <masterip> and <masterport> to the IP address and port number of the primary server, where our master service is native, so instead slaveof 127.0.0.1 6379.

2.3 Starting the master and slave services separately

(1) Start the Master service: through CMD jump to the Master folder, use the redis-server.exe redis.conf command to start the master service, Here we need to specify redis.conf because we have just edited and modified the redis.conf and need to reload the configuration file;

(2) Start slave service: Operation step together ibid, is also redis-server.exe redis.conf, attention do not forget to add redis.conf this sentence;

(3) When the slave service is started, we can see that there are some different log messages in CMD. Here, let's take a brief look:

① first, the following log information appears in the command line of the slave service:

Here, you can see that by editing the configuration file, the slave service will actively connect to the master service after it has been started (mainly by sending the Sync command to request a synchronous connection). The intermediate master sends a ping command to detect the slave's surviving state (the existing copy continues, the invalidation terminates the back step), and then waits for master to send back its memory snapshot file (here you first understand the memory snapshot file as a data backup file or log). As can be seen here, slave has accepted the master's bytes data, and stored the data in memory, and finally successfully completed synchronization with master.

② Second, the following log information appears in the command line of the Master service:

Here, Master detects that a slave sends the sync command to determine if there is a memory snapshot (or an updated memory snapshot) in master, does not start a memory snapshot (mostly), then waits for it to end and sends the snapshot file to slave. At this point, the master end of the sync communication is terminated. The slave will save the data snapshot file to local, after receiving the completion, empty the memory table, re-read the memory snapshot file from master, form a State loop.

2.4 Simple Data read-write test at the command line

  PS: This is mainly done by opening two cmd windows to simulate two clients to operate Redis

(1) write a key/value data pair in the master service

① first, open a new cmd, and in cmd jump to the Master folder, use the redis-cli.exe-p 6379 command to enter the Masterredis client command mode

② then, to make sure the test was successful, we emptied the data from the current master and emptied all the existing data using the FLUSHDB command (Redis periodically writes data from memory to file to persist data)

③ Finally, a simple set command to the key value of Testkey,value to Edisonchou is stored in master;

(2) Read the value data of the key that was just written at Master in slave

① first, open a new cmd, and in cmd jump to the Slave folder, use the redis-cli.exe-p 6380 command to enter the Slaveredis client command mode

② second, a simple get command is queried for the existence of key testkey data

As you can see, our slave successfully obtained data changes from master and saved them in our own data file, allowing us to access the slave and get the data written on master.

(3) Take a look at the cmd window for master and Redis Services, is there some more log information: Master writes 1 changes to the data snapshot file and sends it to slave. Slave also deposited the change into its own data file and saved it, making the data for {Testkey,edisonchou} available in two Redis services, or successfully replicating the Key/value pair in master in slave.

2.5 Simple data reading and writing test in the program

(1) Create a new C # console project and create a new Lib folder in the project folder to hold the. NET driver for Redis (Servicestack.redis DLL);

(2) write the following code:

 
 1         public static IRedisClientsManager redisClientManager = new PooledRedisClientManager(new string[] 
 2         {
 3             "127.0.0.1:6379","127.0.0.1:6380"
 4         });
 5 
 6         public static IRedisClient redisClient = redisClientManager.GetClient();
 7 
 8         static void Main(string[] args)
 9         {
10             if(redisClient != null)
11             {
12                 redisClient.Set<string>("UserName","EdisonChou");
13                 string userName = redisClient.Get<string>("UserName");
14 
15                 Console.WriteLine("Hello,My name is {0}.Nice to meet you!", userName);
16             }
17             Console.ReadKey();
18         } 

The following results can be obtained from the debug run:

(3) View the storage situation through the command line client of master and slave:

In ①master:get UserName

In ②slave:get UserName

third, look back to Redis master-slave replication modeltwo ways to store 3.1 Redis

Redis is an in-memory database that supports persistence , and how is it possible to persist? The answer is that Redis often needs to synchronize in-memory data to hard disks to ensure persistence. So, how does Redis store data?

Redis supports two persistence modes:

(1) snapshotting: Data snapshot, this is the default way. This is done by making a backup of the data and storing the data in a binary file (here's a comparison of the MySQL replication process that was introduced at the beginning of this article). The default file name for this binary file is Dump.rdb. We can also automate snapshot persistence through configuration settings, such as: We can configure Redis to take snapshots automatically if more than M key keys are modified within n seconds.

However, although the snapshot mode is perfect, there is a flaw in the throw: because the snapshot mode is done at a certain interval, if Redis accidentally goes down, it will lose all the changes after the last snapshot . As a result of the perfectionist's push, the author has added the AoF way, which is the way we are going to introduce.

(2) Append-only file: abbreviated to AOF, meaning to only add files. This method saves the action command to the log file (named Appendonly.aof by default) while writing the memory data, and the database state can be restored through the log file when the Redis encounters an unexpected condition. However, because of this, in a system with tens of thousands of concurrent changes, the command log is a very large amount of data (log files will be larger), management maintenance costs are very high, recovery rebuild time is very long, this will lead to the loss of aof high availability of the original intention ( AoF's intentions are data reliability and high availability . What's more, Redis is a memory data structure model, and all of the advantages are based on the efficient atomic operation of complex memory structures, so that aof is a very uncoordinated part.

Just now we're talking about Redis default storage mode, so what if we're going to turn on aof mode? You only need to change the AOF mode from No to Yes in the redis.conf configuration file.

3.2 Redis's data synchronization process

First, the Redis replication feature is built on the previous understanding of the memory snapshot-based persistence strategy, which means that no matter what your persistence strategy chooses, there is always a memory snapshot of the Redis replication function.

(1) The slave end in the configuration file added slave of <masterip> <masterport> instructions, so slave read the configuration file at startup and send sync to master command, Then wait for master to send back its memory snapshot file;

(2) First of all, first of all: Whether it is the first connection or reconnect, Master will start a background process, the data snapshot is saved to a data file (for example: Dump.rdb), and master records all the data modified commands and cached in the data file . Here, immediately after the first step, master receives the sync command from slave, it first sends a ping command to slave to detect the surviving state of the slave (mainly see if the slave is invalid, the failure continues, and the subsequent operation fails.) Then, (when the background process completes the data cache operation), Master sends the data file in real-time to slave.

PS: The above process is repeated for slave and master disconnection, regardless of cause.

(3) When the slave receives the data file from master, it is saved locally, and is loaded into memory after receiving completion, which completes a data copy. After that, master writes the data file and sends it to each slave as soon as the data is updated, and slave listens for updates from master and reloads it to form a loop of data synchronization.

(4) If the slave failure results in downtime, the return to normal will automatically reconnect, master received the slave connection, the full data file sent to slave, if Mater also received multiple slave sent synchronization request, Master will only start a process in the background to save the data file and then send it to all slave to make sure the slave is normal. However, with the big data volume, the full master snapshot is re-acquired, while the slave recovery time is very slow and the main library is stressed on the other hand.

Iv. Summary of learning

In this article, I first briefly introduce the basic concepts of the master-slave replication architecture, and then single-machine emulation on Windows to implement the master-slave replication architecture of Redis and test it with data read-write commands and verify that the data is replicated successfully. Finally, the master-slave replication model of Redis knows how Redis is replicating data, from basic theory to basic practice to basic theory, for master-slave replication is not unfamiliar (at least mixed face cooked). It's 2:30 in the night, so wash up and sleep.

Finally, if you park friends think my article is good or useful to you, trouble points " recommended ", let me more power to write down, thank you!

Reference Documents

(1) Linux China, "the largest Redis cluster: The secret of the Sina Redis cluster", http://www.linuxeden.com/html/itnews/20131010/144377.html

(2) Tianqi, "Redis Replication in Extensible cluster Building", Http://www.infoq.com/cn/articles/tq-redis-copy-build-scalable-cluster

(3) Stephen Liu, Redis Learning Handbook (key operating manual), http://www.cnblogs.com/stephen-liu74/archive/2012/03/26/2356951.html

(4) zfl092005, "Redis master-slave configuration and master-slave switching", http://blog.csdn.net/zfl092005/article/details/17523945

(5) Cool Breeze, "Building high-performance Database Cache Redis Master-slave Replication", http://database.51cto.com/art/201407/444555.htm

(6) Wang Chengwei Podcast, "Redis Open Class for data optimization", http://bbs.itcast.cn/thread-26525-1-1.html

(7) Beauty Dance, redis.conf description of Redis configuration file, http://blog.sina.com.cn/s/blog_636415010101970j.html

Accessories Download

(1) Redisdemo.masterandslave:http://pan.baidu.com/s/1dd1nigt

Article from: http://www.cnblogs.com/edisonchou/p/3883763.html

A nosql approach to everyone loves Redis: (4) Primary exploration of Redis master-slave replication architecture