Redis experience and data types used in the Python O & M Project

Redis experience and data types used in the Python O & M Project

I would like to lament that learning things must be learned and used! I have been using redis for a few years. Today I only want to use a set as a python list. Recently I have made several projects mixed with redis, and encountered some problems and methods for improving performance in development. I will share this and learn together.

The following describes the data types of the Redis set.

Sets are a set. The concept of a set is a combination of non-repeated values. Using the Sets data structure provided by Redis, you can store some aggregated data. For example, in a Weibo application, you can store all the followers of a user in one collection, store all its fans in a set. Redis also provides operations such as intersection, union, and difference set for the set, which can be easily implemented, such as common concerns, common preferences, and friends of two degrees, you can also use different commands to choose whether to return the result to the client or save the result to a new set. The above is the application of Sina Weibo.

Create a set and add data.

 
 

  
  
[root@66 ~]# redis-cli   
  
  
redis 127.0.0.1:6379>    
  
  
redis 127.0.0.1:6379>    
  
  
redis 127.0.0.1:6379> sadd xiaorui aaa   
  
  
(integer) 1   
  
  
redis 127.0.0.1:6379> sadd xiaorui bbb   
  
  
(integer) 1   
  
  
redis 127.0.0.1:6379> sadd xiaorui ccc   
  
  
(integer) 1   
  
  
redis 127.0.0.1:6379>    
  
  
redis 127.0.0.1:6379> SMEMBERS xiaorui   
  
  
1) "aaa" 
  
  
2) "ccc" 
  
  
3) "bbb" 
  
  
redis 127.0.0.1:6379>    
  
  
redis 127.0.0.1:6379> 
 
 

The set cannot write duplicate content.

 
 

  
  
redis 127.0.0.1:6379> sadd xiaorui fuck_shencan   
  
  
(integer) 1   
  
  
redis 127.0.0.1:6379> sadd xiaorui fuck_shencan   
  
  
(integer) 0   
  
  
redis 127.0.0.1:6379> 
 
 

View the collection size

 
 

  
  
redis 127.0.0.1:6379> SCARD xiaorui   
  
  
(integer) 3   
  
  
redis 127.0.0.1:6379> 
 
 

Delete

 
 

  
  
redis 127.0.0.1:6379> SREM xiaorui aaa   
  
  
(integer) 1   
  
  
redis 127.0.0.1:6379> SMEMBERS xiaorui   
  
  
1) "ccc" 
  
  
2) "bbb" 
  
  
redis 127.0.0.1:6379> 
 
 

Intersection of Two Sets

 
 

  
  
redis 127.0.0.1:6379> SADD key1 a   
  
  
(integer) 1   
  
  
redis 127.0.0.1:6379> SADD key1 b   
  
  
(integer) 1   
  
  
redis 127.0.0.1:6379> SADD key1 c   
  
  
(integer) 1   
  
  
redis 127.0.0.1:6379> SADD key2 c   
  
  
(integer) 1   
  
  
redis 127.0.0.1:6379> SADD key2 d   
  
  
(integer) 1   
  
  
redis 127.0.0.1:6379> SADD key2 e   
  
  
(integer) 1   
  
  
redis 127.0.0.1:6379> SINTER key1 key2   
  
  
1) "c" 
  
  
redis 127.0.0.1:6379> 
 
 

You can use a set as a redis list queue. Note that the set Member mode cannot have duplicate values. If your values are not repeated, you may also use the set as a queue.

 
 

  
  
redis 127.0.0.1:6379> sadd myset one   
  
  
(integer) 1   
  
  
redis 127.0.0.1:6379> sadd myset two   
  
  
(integer) 1   
  
  
redis 127.0.0.1:6379> sadd myset three   
  
  
(integer) 1   
  
  
redis 127.0.0.1:6379> SPOP myset   
  
  
"one" 
  
  
redis 127.0.0.1:6379> SMEMBERS myset   
  
  
1) "three" 
  
  
2) "two" 
  
  
redis 127.0.0.1:6379>  
 
 

Two days ago, I told my friend that my monitoring platform's memory was so bad that he suddenly said that redis's memory consumption must be very high... Nima, brother only uses his large queue. Here we will talk about the queue strength of redis. For example, million queue data records occupy 73 MB of memory left and right. About million data records have around MB of memory.
 

It is best to reduce the number of tasks blocked by redis. If more than five threads go to brpop, redis's cpu usage will be pushed to around 80%, which will seriously affect access to other processes, if you are sure that the task is not always at all times, it is best to set the Access frequency and time interval under the control of your program.
 

When using python to process redis, it is best to use a pool to significantly save speed and resources.
 

 
 

  
  
>>> pool = redis.ConnectionPool(host='localhost', port=6379, db=0)  
  
  
>>> r = redis.Redis(connection_pool=pool) 
 
 

The new version of redis supports pipelines, pipline! Some friends don't quite understand the advantages of the pipeline here. Although pyhton uses a connection pool when connecting to redis, it only performs keepalive for the connection. However, every command push, the pyhton command interacts with each other. After the pipline pipeline is blocked, he will merge all the commands into a pipeline and push them to the redis server. This saves a lot of trouble. This is especially suitable for high concurrency.

Gevent can be used to solve the pub sub communication performance problem of redis. You can directly import the gevent monkey.

 
 

  
  
import gevent.monkey   
  
  
gevent.monkey.patch_all()   
  
  
#http://rfyiamcool.blog.51cto.com/1030776/1435539    
  
  
import os   
  
  
import sys   
  
  
import fcntl   
  
  
import gevent   
  
  
from gevent.socket import wait_read   
  
  
     
  
  
from redis import Redis   
  
  
     
  
  
PID = os.getpid()   
  
  
     
  
  
red = Redis('localhost')   
  
  
     
  
  
def echo_stdin():   
  
  
    # make stdin non-blocking   
  
  
    fcntl.fcntl(sys.stdin, fcntl.F_SETFL, os.O_NONBLOCK)   
  
  
    red.publish('echo', "[%i] joined" % (PID,))   
  
  
    while True:   
  
  
        wait_read(sys.stdin.fileno())   
  
  
        l = sys.stdin.readline().strip()   
  
  
        s = "[%i] %s" % (PID, l)   
  
  
        # save to log   
  
  
        red.rpush('echo_log', s)   
  
  
        # publish message   
  
  
        red.publish('echo', s)   
  
  
        if l == 'quit':   
  
  
            break 
  
  
     
  
  
def handler():   
  
  
    pubsub = red.pubsub()   
  
  
    # first subscribe, then print log (no race condition this way)   
  
  
    pubsub.subscribe('echo')   
  
  
    # print log   
  
  
    for line in red.lrange('echo_log', 0, -1):   
  
  
        print '.', line   
  
  
    # print channel   
  
  
    for msg in pubsub.listen():   
  
  
        print '>', msg['data']   
  
  
     
  
  
gevent.spawn(handler)   
  
  
gevent.spawn(echo_stdin).join() 
 
 

Of course, General set get sadd hset can also be used with redis. However, there is no advantage, because redis only enables one process to read and write data. We need to call the process by reusing the connections in the program and finally retrieving data, you might as well let him work honestly. Don't make a multi-thread, let him go. I did a stress test here. After using a gevent in python2.7, there was no significant growth in reading and writing in batches.

 
 

  
  
>>> import geventredis   
  
  
>>> redis_client = geventredis.connect('127.0.0.1', 6379)   
  
  
>>> redis_client.set('foo', 'bar')   
  
  
'OK' 
  
  
>>> for msg in redis_client.monitor():   
  
  
       print msg 
 
 

Blog: http://rfyiamcool.blog.51cto.com/1030776/1435539