Redis Data Storage Solution

1. Background
1.1 redis Introduction

Official Website: http://redis.io/, redisis the abbreviation of remote dictionary server.

Redis is an open-source log-type and key-value database written in ansi c language that supports Network, memory-based persistence, and provides APIs in multiple languages. From January 1, March 15, 2010, the development of redis was hosted by VMware. It is similar to memcached, but data can be persistent and supports a wide range of data types. It keeps key-value databases simple and fast, while absorbing the advantages of some relational databases. So that it is located between the relational database and the key-value database. Redis not only saves strings data, but also stores lists (ordered) and sets (unordered) data. It also supports advanced functions such as sorting, this ensures the atomicity of operations when implementing functions such as incr and setnx. In addition, it also supports master-slave replication and other functions. Redis can be viewed as a data structure server.

All redis data is stored in the memory and then asynchronously stored to the disk (this is called the "semi-persistent mode "); you can also write every data change to an append only file (AOF) (this is called "Full persistence mode ").
1.2 bottleneck

Currently, in the kabbu xiyou project, data storage uses MySQL. The master game database is divided into 20 databases distributed on five machines, and each database has a data volume of about 10 Gb, each machine uses RAID5 to accelerate disk access. When the number of concurrent online users reaches, the I/O pressure on the DB disk is high, resulting in game cards and the number of online users cannot go up. Redis reads and writes data directly to the memory, which can effectively solve this bottleneck.
1.3 solution Overview

After some data is compressed and imported to redis, the total data volume is about 15 GB (converted to redis data type data volume), one master, one slave model, two groups in total. One machine has 32 GB memory and 20 instances, with a total of 40 instances. It is estimated that each instance can store up to 2 GB of data (not so much now ). Multiple instances facilitate persistence.

The master database does not use aof or snapshots, and only schedules a task at every night and transfers the snapshot to a remote location. Enable aof from the slave database and implement it in 1 second.

A time point (crontab implementation) is inserted to the master database every five minutes to facilitate record time points from the database aof for point restoration.
2. install and configure
2.1 hardware configuration

Dell r410 32 GB memory; two CPUs, 4 cores each; GB hard disk
2.2 Installation

A) install the tcmalloc Library

Source code

Wget http://download.savannah.gnu.org/releases/libunwind/libunwind-0.99-alpha.tar.gz
Tar zxf libunwind-0.99-alpha.tar.gz
CD libunwind-0.99-alpha/
Cflags =-FPIC./configure
Make cflags =-FPIC
Make cflags =-FPIC install

B) install Google-perftools:

Source code

Wget http://google-perftools.googlecode.com/files/google-perftools-1.8.2.tar.gz

Tar zxf google-perftools-1.7.tar.gz

CD google-perftools-1.7/

./Configure

Make

Make install

Echo "/usr/local/lib">/etc/lD. So. conf. d/usr_local_lib.conf

/Sbin/ldconfig

C) install redis

Source code

Wget http://redis.googlecode.com/files/redis-2.2.14.tar.gz

Tar xzvf redis-2.2.14.tar.gz

CD redis-2.2.14

Make use_tcmalloc = Yes

// Make will generate redis-server redis-cli redis-benchmark redis-check-dump in the src directory.

Copy the configuration file redis. conf to the/usr/local/redis/bin directory.

// Create a unified directory for convenient management

Mkdir-P/usr/local/redis/bin

Mkdir-P/usr/local/redis/etc

Mkdir-P/usr/local/redis/var

Mkdir-P/data/redis-Data

# Mkdir-P/data/redis-data/redis60 {01,02, 03,04, 05,06, 07,08, 09,10}

Check whether tcmalloc is effective

# Lsof-N | grep tcmalloc

The following information takes effect:

Redis-Ser 13768 root mem Reg 1616491 788696/usr/local/lib/libtcmalloc. so.0.1.0

// After installation, the libtcmalloc may not be registered and the above information is not found. Add one to the mysqld_safe file:

Export ld_preload =/usr/local/lib/libtcmalloc. so''

// Restart MySQL again (exercise caution when performing online operations !!!) (With tmalloc, the memory allocation efficiency is higher)
2.3 Configuration

// Currently, a total of 40 instances are planned to be created, with 20 instances per machine. The following uses instance 01 as an example:

// Configuration file name: redis6001.conf ~ Redis6040.conf

Source code

Daemonize Yes

Pidfile/var/run/redis6001.pid

Port 6001

Timeout 300

Loglevel debug

Logfile/usr/local/redis/var/debug6001.log

Syslog-enabled Yes

Databases 16

Rdbcompression Yes

Dbfilename redis6001.rdb

DIR/data/redis-data/redis6001

Slave-serve-stale-data Yes

Requirepass my # redis

Appendonly No

No-appendfsync-on-Rewrite No

VM-enabled No

Hhash-max-zipmap-entries 512

Hash-max-zipmap-value 64

List-max-ziplist-entries 512

List-max-ziplist-value 64

Set-max-intset-entries 512

Activerehashing Yes
3. Master-slave synchronous backup and related scripts
3.1 master-slave synchronization target

The main objective of Master/Slave is data persistence, disaster recovery, and redundancy. The master database is not implemented, reducing consumption.
3.2 Master/Slave Configuration

The slave database installation configuration is consistent with that of the master database. The master database does not need to be modified (Allow access from the slave database). The slave database requires two additional configurations: slaveof + Aof.

Slaveof 192.168.3.180 6301

Masterauth my # redis

Appendonly Yes

Appendfilename appendonly01.aof

Appendfsync everysec

// Configuration file name: slave6001.conf ~ Slave6040.conf

Source code

Daemonize Yes

Pidfile/var/run/slave6001.pid

Port 6001

Timeout 300

Loglevel debug

Logfile/usr/local/redis/var/debug6001.log

Syslog-enabled Yes

Databases 16

Rdbcompression Yes

Dbfilename slave6001.rdb

DIR/data/redis-data/slave6001

Slaveof 192.168.0.139 6001

Masterauth my # redis

Slave-serve-stale-data Yes

Requirepass my # redis

Appendonly Yes

Appendfilename slave6001.aof

Appendfsync everysec

No-appendfsync-on-Rewrite No

VM-enabled No

VM-Swap-file/tmp/redis. Swap

VM-max-memory 0

VM-page-size 32

VM-pages 134217728

VM-max-threads 4

Hhash-max-zipmap-entries 512

Hash-max-zipmap-value 64

List-max-ziplist-entries 512

List-max-ziplist-value 64

Set-max-intset-entries 512

Activerehashing Yes
3.3 backup

Backup policy: the master database serializes a snapshot (bgsave) to each instance at every night. The aof file is packaged and backed up (TAR) from the database at each night ), after packaging and backup, compress the aof file (bgrewriteaof ). The starting point of each day is the data after rewriteaof the previous night.

The master database takes a snapshot (bgsave) for each instance at every night ). Master database backup script (redis_bgsave.sh)

#! /Bin/bash

# Date = 'date + % Y % m % d _ % H % m % s'

Redispassword = My # redis

For port in 'seq 6001 6020'

Do

/Usr/local/redis/bin/redis-cli-H 127.0.0.1-p $ Port-A $ redispassword bgsave

Sleep 10

Done

Date>/usr/local/redis/var/bgsave. Log

Copy the AOF of each instance to another directory and package it at each night. The compressed package must have a remote backup, and then compress the aof (bgrewriteaof ).

Back up the aof and bgrewriteaof script from the database (redis_backup.sh: for a single instance)

#! /Bin/sh

# FD: file dir

# RD: runing dir

# The first parameter is the redis Instance name

If [$ #-ne 1]; then

Echo "Usage: $0 redis_name"

Exit

Fi

Curdate = 'date + % Y % m % d'

Curhour = 'date + % Y % m % d _ % H'

Curtime = 'date + % Y % m % d _ % H % m % s'

Redispassword = My # redis

Redisname = $1

Port = 'echo $ redisname | cut-c6-9'

Logfile =/data/logs/redisbak/redis_allbak _ $ {curdate}. Log

If ["$ {redisname}" = ""]; then

Echo "redis name Error !"

Exit 1

Else

If [! -D "/data/redis-data/$ {redisname}"]; then

Echo "redis name Error !"

Exit 1

Fi

Fi

Ddir =/data/backup/redis/$ curhour

Mkdir-p $ {ddir}

Rdir =/data/redis-data/$ redisname

CD $ {rdir}

Tar-zcf $ ddir/zookeeper redisname1__zookeeper curtime=.tar.gz $ redisname. aof

If [$? ! = 0]; then

Echo "tar error $ redisname. aof" >>$ logfile

# Exit 1

Fi

Sleep 5

/Usr/local/redis/bin/redis-cli-H 127.0.0.1-p $ Port-A $ redispassword bgrewriteaof

Sleep 5

### Delete old backup data dir ###

#/Bin/Rm-RF 'date-D-7day + "% Y % m % d "'

Find/data/backup/redis/-mtime + 7 | xargs Rm-RF

Echo "backup $ redisname OK at $ curtime !" >>$ Logfile

Back up all instances from the slave database (/data/SH/redis_allbak.sh)

#! /Bin/sh

Curdate = 'date + % Y % m % d'

Logfile =/data/logs/redisbak/redis_allbak _ $ {curdate}. Log

For port in 'seq 6001 6020'

Do

/Data/SH/redis_backup.sh slave $ {port} & Echo "Slave $ {port} OK 'date + % Y % m % d _ % H % m % s'">> $ logfile 2> & 1 | echo "Slave $ {port} backup error" >>$ logfile 2> & 1

Done
4. Operation precautions

If the master database crashes, the master database program cannot be started directly. If the master database program is enabled directly, the aof file from the slave database will be flushed out. This will only cause the backup to be restored to 12 of the previous night.

When a program is running, the network cannot be restarted (the program reads and writes through the port of the network interface ). Network interruption may cause data loss during the interruption.

Make sure that all the configuration files are correct before starting (especially if the file names are not the same). Otherwise, the original file will be washed out, which may cause irreparable loss.

5. Disaster Recovery
5.1 master database faults, quickly restoring to the nearest state

Description: when the master database is down (the redis program is down/the machine is down) and the database is normal and recovered to the time when the master database is down: manually taking a snapshot from the slave database, copy the snapshot to the corresponding directory of the master database, start, OK.

Take a snapshot from the slave database, transfer the snapshot file to another directory, copy the snapshot file directory to the corresponding directory of the master database, and start the master database. OK!

(/Data/SH/redis_bgsave_cp.sh)

#! /Bin/bash

Redispassword = My # redis

For port in 'seq 6001 6020'

Do

/Usr/local/redis/bin/redis-cli-H 127.0.0.1-p $ Port-A $ redispassword bgsave

Sleep 5

Done

Sleep 15

For port in 'seq 6001 6020'

Do

Sdir =/data/redis-data/Slave $ {port }/

Ddir =/data/redis_recovery/redis-data/

Mkdir-p $ ddir/redis $ {port }/

CD $ sdir

CP-RF slave $ {port}. RDB $ ddir/redis $ {port}. RDB

# Sleep 1

Done

Rename the original data directory in the master database.

Copy the snapshot file from the slave database to the master database.

Start the master database.
5.2 restore to on the current day

Pay attention to data backup (current status aof + normal exit snapshot )!

Stop redis.

Decompress aof (/data/SH/redis_untar.sh)

#! /Bin/bash

Day = 20111102

Sdir =/data/backup/redis/20111102_00/

CD $ sdir

For port in 'seq 6001 6020'

Do

Tar-zxf slave $ {port} $ day * .tar.gz

Sleep 2

Done

Cut aof (/data/SH/redis_sed.sh)

#! /Bin/bash

Ddir =/data/redis_recovery/

Tag = "tag1111000001200 ″

For port in 'seq 6001 6020'

Do

Sdir =/data/backup/redis/20111102_00/

Saof =$ {sdir}/Slave $ {port}. aof

Line = 'sed-n "/$ tag/=" $ saof'

Num = $ [$ line + 3]

Mkdir-p $ {ddir}/Slave $ {port }/

Sed "$ {num}, \ $ D" $ saof >$ {ddir}/Slave $ {port}. aof

Done

Rename the original data directory.

Rename the cut aof directory to the data directory of the configuration file. (Comment out the master-slave synchronization configuration items ).

Start slave database.

Create a snapshot, copy the snapshot to the master database, and start the master database (same as 5.1 ).
5.3 restore to the status of two days or a few days ago

Back up data before aof without bgrewriteaof from the database every night. You can back up data at that night and restore the aof file without bfrewriteaof. The method is the same as 5.2.
6. Monitoring
6.1 Nagios monitoring

#/Usr/local/redis/bin/redis-cli-H 127.0.0.1-P 6301-a my # redis info

Used_memory_human: 1.49g

Master_link_status: down up/down

Db0: Keys = 2079581, expires = 0

// Monitor the number of processes (the number of redis instances on each machine)

Define Service {

Use local-Service

Host_name 119.147.163.xxx

Servicegroups procs

Service_description redis_20

Check_command check_nrpe! Check_procs \! 20:20 \! 20:25 \! Redis-Server

}

// Monitor the master-slave synchronization status (obtain the values of master_link_status and keys from the slave database)

// Status = down direct severe warning, keys difference 10 warning, keys difference 100 severe warning

Define Service {

Use local-Service

# Hostgroup_name kabuxiyou

Host_name 119.147.161.xxx

Servicegroups redis

Service_description redis_239

Check_command check_nrpe! Check_redis_slave \! 192.168.0.139 \! 6379 \! 127.0.0.1 \! 6380

}

// Monitor the memory usage of the instance, with a warning of GB and a severe warning of 2 GB

Define Service {

Use local-Service

# Hostgroup_name kabuxiyou

Host_name 119.147.161.xxx

Servicegroups redis

Service_description redis_mem_22

Check_command check_nrpe! Check_redis_mem \! 127.0.0.1 \! 6381

}
7. Materials
7.1 references

// Redis manual (zh_v1.0133

Http://www.infoq.com/cn/articles/tq-why-choose-redis

// Why use redis and its product positioning

Http://www.infoq.com/cn/articles/tq-redis-memory-usage-optimization-storage

// Redis memory usage optimization and storage

Http://www.infoq.com/cn/articles/tq-redis-copy-build-scalable-cluster

// Redis replication and Scalable Cluster Creation

Http://blog.prosight.me/index.php/2011/08/802

// Seven Collection types of application scenarios in redis
Version 7.2

Latest stable version:

Wget http://redis.googlecode.com/files/redis-2.2.14.tar.gz // 2011.10.1

Wget http://redis.googlecode.com/files/redis-2.4.2.tar.gz // 2011.11.15

Wget http://redis.googlecode.com/files/redis-2.4.10.tar.gz // 2012.04.16
8. apsaradb for redis Distribution
8.1 Server distribution

Server distribution: Instance name: Port:

... ... ...
8.2 rules for conversion from MySQL to redis

1. Key naming rules

The key is in the form of letters and numbers to prevent multiple applications from overwriting each other when using the same key. The key in the main project adopts GJ # [uid] example: gj12345678

2. Field naming rules

The key in the main project contains multiple fields, which indicates that the value range of the fields currently reserved for each application by multiple applications is 100. (in actual use, the number of fields should not exceed 10. Otherwise, hash is of little significance. ).

Current reserved field division:

100 ~ 199 is related to achievements (currently, the history and current fields in the achieve_user table of MySQL 150 and 151 are used)

200 ~ 299 is related to monsters (currently, two more fields corresponding to MySQL's spirit_store table correspond to MySQL's spiritmaster and spirit tables)

3. Instance Division rules

It is estimated that the gamer data will be divided into 40 instances and allocated to different instances based on the player's uid modulo (whether to split the database or not is unknown)

4. Other rules for using redis

At present, the manor also has some data using redis, but the data is placed in a separate redis instance (119.147.164.24)

Use different library and key value Key naming rules for different applications as [Application name]-[uid] example: GrowTimes-12345678/StolenList-12345678
9. Test Data

// The test data is related to the stored data and the hardware configuration of the machine. The specific environment may be different.
9.1 close startup

Start: 1.6g aof file, 132 m rdb, start time 107 s, Generate debug. Log 128 M, use memory 1.71G.

2G data, start for about 2 minutes, master-slave synchronization for about 2 minutes (Intranet ).

Close: shutdown is very short, generally 1 ~ 2 S. Save is required before shutdown. The save time is related to the data volume, generally several seconds.
9.2 data size

Simple set (set a 1), 30 thousand treaty 1 m aof, 300 k rdb, 0.12 million Treaty 4 m aof, 1 m RDB.

Set a record. What does aof record:

Redis> set yuwei wangjiancheng

* 3

$3

Set

$5

Yuwei

$3

Wangjiancheng

9.3 backup Packaging

Plain text aof:

Tar-zcf * .tar.gz * 1.9g/20 s/16 m/

Tar-JCF * .tar.bz2*1.9g/150 s/5.3 m/

Currently, apsaradb for redis is used online. Data Import is in the binary stream format (AOF files are data files and non-text files). Compression has been performed before the import, so the compression ratio is very low.
9.4 other tests

The data directory in the instance configuration file does not exist and the instance cannot be started.

Aof writes and packs (TAR) without causing data corruption. Tar only reads files.

When redis is running, the data directory cannot be deleted, bgsave, aof cannot be written, and slave database can be synchronized; bgsave cannot be created, and aof cannot be written.

Bgsave is executed in the background. Therefore, a problem occurs when bgsave and shutdown are written together. bgsave has been shutdown since it has not been executed. Use save instead.

After the master database instance is restarted, the slave database will be "fully synchronized" again to save the same data as the master database.

Transient network disconnection does not cause loss of Master/Slave Data, and can be resumed after the network recovers.

Instance a snapshot file, put instance B data directory to start (file name changed to instance B data file), OK!

After rewriteaof/bgrewriteaof is executed, reidis re-creates aof and aof re-arranges the order based on the memory data.
10. Connect the client to redis
10.1 PHP

Install phpredis:

Phpredis package address (N versions have been tried. Only one of them can be installed in our custom system environment)

Http://ftp.nluug.nl/pub/NetBSD/packages/distfiles/php-redis/nicolasff-phpredis-2.1.3-0-g43bc590.tar.gz

CD/Dist/src

Tar xzf ../Dist/nicolasff-phpredis-2.1.3-0-g43bc590.tar.gz

CD nicolasff-phpredis-43bc590

/Usr/local/PHP/bin/phpize

# If PHP is an installed RPM package, you need to install PHP-devel

./Configure-with-PHP-Config =/usr/local/PHP/bin/PHP-config

Make & make install

Configure PHP. ini

VI/usr/local/PHP/lib/PHP. ini: Extension = "redis. So"

Restart FastCGI to take effect/root/fastcgi_restart # PHP-M // check whether redis exists and whether the installation is successful.

Redis Extension Test

Test code:

<? PHP $ redis = new redis (); $ redis-> connect ('127. 0.0.1 ', 127); $ redis-> set ('test', 'Hello world! '); Echo $ redis-> get ('test ');

?>

Return Hello world! Success

# Password

$ Redis-> auth ('My # redis ');

# Error:

Configure: Error: invalid value of Canonical build

#./Configure

./Configure-with-PHP-Config =/usr/local/PHP/bin/PHP-config

10.2 C ++

Antirez-hiredis-3cc6a7f.tar.gz 340 KB

This is the official library, which contains an example

Run this command with parameters./hiredis-test

The example is test. C.