Master-slave replication configuration for redis

Master-slave replication configuration for redis

I. Principles

Redis's master-slave replication function is very powerful. One master can have multiple slave instances, and one slave can have multiple slave instances. In this way, a powerful multi-level server cluster architecture is formed. The following are some features of redis master-slave replication:
1. The master can have multiple slave instances.
2. In addition to connecting multiple slave instances to the same master, slave can also be connected to other slave instances to form a graph structure.
3. Master-slave replication does not block the master. That is to say, when one or more slave and master synchronize data for the first time, the master can continue to process the requests sent by the client. On the contrary, slave blocks requests that cannot process the client when synchronizing data for the first time.
4. Master-slave replication can be used to improve the scalability of the system. We can use multiple slave for client read requests. For example, sort operations can be processed using slave. It can also be used for simple data redundancy.
5. You can disable data persistence on the master. You only need to comment out all the Save configurations in the master configuration file, and then configure data persistence only on the slave.

The following describes the process of master-slave replication.
After the slave server is configured, slave establishes a connection with the master and sends the sync command. The master will start a background process to save database snapshots to files, whether it is the first synchronization established connection or the re-connection after the connection is disconnected, at the same time, the master process starts to collect new write commands and cache them. After the background process completes file writing, the master sends the file to slave. slave saves the file to the disk and then loads it To the memory to restore the database snapshot to slave. Then the master will forward the cache command to slave. In addition, the write commands received by the master will be sent to slave through the established connection. The command for synchronizing data from Master to slave is in the same protocol format as the command sent from client. When the master and slave are disconnected, slave can automatically establish a new connection. If the master receives synchronous connection commands from multiple slave instances at the same time, only one process is started to write the database image and then sent to all slave instances.

Ii. Configuration

Next I will demonstrate how to perform redis data master-slave replication on multiple servers. I suppose there are two servers, one is the Linux operating system (LAN IP: 192.168.1.4, master server), and the other is the Linux operating system (LAN IP: 192.168.1.5, slave server)

It is easy to configure the slave server. You only need to add the following configuration in the configuration file (redis. conf ).
Bind 192.168.1.5 (slave server, which defaults to 127.0.0.1. Please modify the IP address of the cost machine. Otherwise, the client cannot access it)

Slaveof 192.168.1.4 6379 (mapped to the master server)

If you configure the master-slave relationship on a machine, you also need to modify the default port number of the slave server, which is also modified in redis. conf.

For more information about redis configurations, see my other article.

Iii. Test

After the master machine is started and data is written to the master machine, start the slave machine and you can find the server Load balancer:

 

A Sync request is sent from the master, and it supports automatic reconnection, that is, when the master is offline, it will be in the waiting state.

On the master:

 

The dump files of the two machines are of the same size:

192.168.1.5 (slave ):

192.168.1.4 (master ):

 

From the redis source code, we can find that the RDB file is implemented using the lzf compression algorithm. By default, the lzf compression algorithm is enabled.

In this way, you can use other client programs or Web platforms to read data from the slave Disk Database, truly achieving read/write splitting.

Posted @ Weafer reading (576) Comments (0) Edit

 

Install and configure redis

Redis is a high-performance Key-value database. The emergence of redis largely compensates for the shortage of keyValue storage such as memcached, and can play a very good complementary role in relational databases. It provides python, Ruby, Erlang, and PHP clients for ease of use. The problem is that this project is still quite new and may not be stable enough, and there are no actual instances for large-scale system applications. In addition, the lack of batch get in MC is also a big problem. The network overhead of batch get is different from that of multiple get.

Performance test results:

The set operation is performed 110000 times per second, and the get operation is performed 81000 times per second. The server configuration is as follows:

Linux 2.6, Xeon x3320 2.5 GHz.

The stackoverflow website uses redis as the cache server.

Installation Process:

Redis is an advanced key-value database. It is similar to memcached, but data can be persistent and supports a wide range of data types. There are string, linked list, set and ordered set. Allows you to compute, merge, and merge sets (difference) on the server side. It also supports multiple sorting functions. So redis can also be seen as a data structure server.

All redis data is stored in the memory and then asynchronously stored to the disk (this is called the "semi-persistent mode "); you can also write every data change to an append only file (AOF) (this is called "Full persistence mode ").

1. download the latest version

Wget http://redis.googlecode.com/files/redis-2.0.0-rc4.tar.gz

Ii. Decompression

Tar redis-2.0.0-rc4.tar.gz

3. Install the C/C ++ compilation component (optional)

Apt-Get install build-essential

Iv. Compilation

CD redis-2.0.0-rc4
Make

After the make command is executed, an executable file is generated under the current directory, including redis-server, redis-CLI, redis-benchmark, and redis-Stat. Their functions are as follows:

• Redis-server: Daemon Startup Program of the redis Server
• Redis-CLI: redis command line operation tool. Of course, you can also use Telnet to operate based on its plain text protocol.
• Redis-benchmark: redis performance testing tool to test the read/write performance of redis in your system and your configuration
• Redis-stat: redis status detection tool that can detect redis's current status parameters and latency

These commands will be explained later. Of course they are copied from the Internet...

5. modify the configuration file

/Etc/sysctl. conf

Add

VM. overcommit_memory = 1

Refresh configuration to make it take effect

Sysctl VM. overcommit_memory = 1

Additional information:

** If the memory is insufficient, you need to set the kernel parameters:
Echo 1>/proc/sys/Vm/overcommit_memory

The kernel parameters are described as follows:

The overcommit_memory file specifies the kernel's memory allocation policy. The value can be 0, 1, or 2.
0 indicates that the kernel will check whether there is enough available memory for use by the process. If there is enough available memory, the memory application will be allowed; otherwise, the memory application will fail, and return the error to the application process.
1 indicates that the kernel allows all physical memory allocation regardless of the current memory status.
2. indicates that the kernel is allowed to allocate more memory than the total physical memory and swap space.

** Edit the redis. conf configuration file (/etc/redis. conf) and make appropriate adjustments as needed, for example:
Daemonize yes # convert to daemon; otherwise, a monitoring information line is output every 5 seconds at startup.
Save 60 1000 # reduce the number of changes, which can be specified as needed
Maxmemory 256000000 # allocate MB of memory

After we successfully install redis, we can directly execute redis-server to run redis. At this time, it runs according to the default configuration (the default configuration is not even the background operation ). If we want redis to run as required, we need to modify the configuration file. The redis configuration file is the redis operated by the second CP above. CONF file, which is copied to the/usr/local/redis/etc/directory. Modify it to configure our server. How to modify it? The following figure shows the meaning of parameters to be configured for redis. conf:

• Daemonize: whether to run in daemon mode
• Pidfile: PID File Location
• Port: the port number of the listener.
• Timeout: Request timeout
• Loglevel: log information level
• Logfile: Location of the log file
• Databases: number of databases Enabled
• Save **: the frequency at which snapshots are saved. The first "*" indicates the duration and the third "indicates the number of write operations performed. Snapshots are automatically saved when a certain number of write operations are performed within a certain period of time. You can set multiple conditions.
• Rdbcompression: whether to use Compression
• Dbfilename: Data snapshot file name (only file name, excluding directory)
• Dir: directory for storing data snapshots (this is the Directory)
• Appendonly: whether to enable appendonlylog. If it is enabled, a log is recorded for each write operation, which improves data risk resistance but affects efficiency.
• Appendfsync: How to synchronize appendonlylog to the disk (three options are force-call fsync for each write, enable fsync once per second, and do not call fsync to wait for the system to synchronize itself)

The following is a slightly modified configuration file:

daemonize yespidfile /usr/local/redis/var/redis.pidport 6379timeout 300loglevel debuglogfile /usr/local/redis/var/redis.logdatabases 16save 900 1save 300 10save 60 10000rdbcompression yesdbfilename dump.rdbdir /usr/local/redis/var/appendonly noappendfsync alwaysglueoutputbuf yesshareobjects noshareobjectspoolsize 1024

　　

Write the above content as redis. conf and save it to the/usr/local/redis/etc/directory.

Then run the following command in the command line:

1	/usr/local/redis/bin/redis-server /usr/local/redis/etc/redis.conf

You can start the redis service in the background.

1	telnet 127.0.0.1 6379

Connect to your redis service.

6. Start and verify the service

Start the server

./Redis-Server
Or
$ Redis-server/etc/redis. conf
Check whether startup is successful
$ PS-Ef | grep redis
Or
./Redis-cli Ping
Pong

7. Start the command line client and assign values

Redis-cli set mykey somevalue

./Redis-cli get mykey

8. Close the service

$ Redis-cli Shutdown

# Disable the redis-server on the specified port

$ Redis-cli-P 6380 Shutdown

9. The client can also be connected through Telnet.

[[Email protected] conf] # telnet 127.0.0.1 6379
Trying 127.0.0.1...
Connected to dbcache (127.0.0.1 ).
Escape Character is '^]'.
Set Foo 3
Bar
+ OK
Get foo
$3
Bar
^]
Telnet> quit
Connection closed.

Posted @ Weafer reading (159) Comments (0) Edit

 

MySQL replication, master-slave and dual-master configurations

MySQL replication is a master-slave synchronization solution for multiple MySQL databases. It features Asynchronization and is widely used in various scenarios that require higher performance and reliability for MySQL. Another technique corresponding to this is the synchronized MySQL cluster, but because of its complexity, there are fewer users.

MySQL officially provides the scenario for using replication:

Replication principles 

MySQL replication is an asynchronous replication process, from a MySQL node (called the master node) to another MySQL node (called the slave ). The entire replication process between the master and slave is mainly completed by three threads. Two threads (SQL thread and I/O thread) are on the slave side, another thread (I/O thread) is on the master side.

To implement MySQL replication, you must first enable the binary log on the master end, the whole replication process is actually because slave obtains the log from the master end and then executes the operations recorded in the log in full order on itself.

It seems that the replication principle of MySQL is very simple. Here is a summary:
* Only one master can be set for each slave.
* After the master executes the SQL statement, it records the binary log file (bin-log ).
* Connect to the primary database, obtain the BINLOG from the primary database, store it in the local relay-log, and execute the SQL statement from the last remembered position. If an error occurs, the synchronization is stopped.

From the perspective of these replication principles, we can draw these inferences:
* Databases between the master and slave databases are not synchronized in real time. Even if the network connection is normal, the master and slave databases may be instantly inconsistent.
* If the master-slave network is disconnected, the slave will be synchronized in batches after the network is normal.
* If you modify the slave data, it is very dangerous to stop synchronization because an error occurs during the execution of the main bin-log. Therefore, you should be very careful when modifying the data from the top.
* One derivative configuration is dual-master, which is a master-slave configuration. It works well as long as the modifications made by both parties do not conflict with each other.
* If you need multiple masters, you can use the ring configuration so that any node modifications can be synchronized to all nodes.

Master/Slave settings 

Because the principle is relatively simple, replication is supported from MySQL 3 and can work on all platforms. Multiple MySQL nodes can even be different platforms, versions, and local networks. The replication configuration includes the user and my. ini (My. CnF in Linux.

First, create a user for slave on the master MySQL node:

Grant replication slave, replication client on *. * To 'slave '@ '192. 168.1.10' identified by 'slave ';

In fact, to support master-slave dynamic synchronization or manual switching, this user is generally created on all master-slave nodes. Then the MySQL configuration is completed. You need to modify the my. CNF or my. ini file. Add the following content in the section "mysqld:

Server-id = 1
Auto-increment = 2
Auto-increment-offset = 1
Log-bin
BINLOG-do-DB = mstest
Binlog_format = mixed

Master-host = 192.168.1.62
Master-user = slave
Master-Password = slave
Replicate-do-DB = mstest

In the above two settings, the first one is set as the master, and the last one is set as the slave. That is to say, you can add one segment on two MySQL nodes. BINLOG-do-DB and replicate-do-DB are the databases that need to be synchronized, auto-increment and auto-increment-offset are set to support dual-master (refer to the next section). You can also leave them unspecified when only the master and slave nodes are used.

Dual-Master Settings 

From the original theory, MySQL also supports dual-Master Settings, that is, two MySQL nodes are mutually active and standby. However, in theory, dual-master can work well as long as data does not conflict with each other, however, in actual situations, data conflicts may occur. For example, before synchronization is completed, both parties modify the same record. Therefore, in practice, it is best not to allow both sides to modify at the same time. That is, the logic still works in the master-slave mode. However, the dual-master setting still makes sense, because after this is done, switching between the master and slave will become very simple. After a fault occurs, if a dual-master is configured, it is easy to directly switch between the master and slave databases.
When setting the dual master node, you only need to copy the preceding settings to write the configuration files of the two MySQL nodes, but you need to modify the corresponding server-id, auto-increment-offset and master-host. Auto-increment-offset is used to ensure that the dual-master node does not conflict with the ID when adding a table at the same time. Therefore, set auto-increment-offset to different values on the two nodes. Also, do not forget to create users for each other on both nodes.Load Balancing at the application layerThis article only introduces the repilication configuration of MySQL. As shown in the figure above, with replication, you also need to use the application layer (or middleware) for load balancing, in this way, we can maximize the advantages of MySQL replication, which will be discussed later.

Master-slave replication configuration for redis