Connection between Python and Redis

This article describes how to connect Python to Redis. Redis is a high-performance, memory-based database, for more information, see The python redis module used when I wrote the zabbix storm job monitoring script today. it has been useful before, but I don't know much about it, today, I read the relevant api and source code and saw the ConnectionPool implementation. here I will briefly describe it.
Before ConnectionPool, if you need to connect to redis, I use the StrictRedis class. the detailed explanation of this class can be seen in the source code:

Redis. StrictRedis Implementation of the Redis protocol. This abstract class provides a Python interface to all Redis commands and
Implementation of the Redis protocol. Connection and Pipeline derive from this, implementing how the commands are sent and have ed to the Redis server

Usage:

 r=redis.StrictRedis(host=xxxx, port=xxxx, db=xxxx) r.xxxx()

With the ConnectionPool class, you can use the following method:

pool = redis.ConnectionPool(host=xxx, port=xxx, db=xxxx)r = redis.Redis(connection_pool=pool)

Redis is a subclass of StrictRedis.
A simple analysis is as follows:
In the _ init _ method of the StrictRedis class, you can initialize the connection_pool parameter, which corresponds to a ConnectionPool object:

class StrictRedis(object):........  def __init__(self, host='localhost', port=6379,         db=0, password=None, socket_timeout=None,         socket_connect_timeout=None,         socket_keepalive=None, socket_keepalive_options=None,         connection_pool=None, unix_socket_path=None,         encoding='utf-8', encoding_errors='strict',         charset=None, errors=None,         decode_responses=False, retry_on_timeout=False,         ssl=False, ssl_keyfile=None, ssl_certfile=None,         ssl_cert_reqs=None, ssl_ca_certs=None):     if not connection_pool:       ..........       connection_pool = ConnectionPool(**kwargs)     self.connection_pool = connection_pool

The execute_command method is called when the StrictRedis instance executes a specific command. here we can see that the specific implementation is to obtain a specific connection from the connection pool, then execute the command, and release the connection after completion:

# Command execution and protocol parsing def execute_command (self, * args, ** options): "Execute a command and return a parsed response" pool = self. connection_pool command_name = args [0] connection = pool. get_connection (command_name, ** options) # Call ConnectionPool. get_connection method to get a connection try: connection. send_command (* args) # run the command, which is Connection. send_command return self. parse_response (connection, command_name, ** options) except T (ConnectionError, TimeoutError) as e: connection. disconnect () if not connection. retry_on_timeout and isinstance (e, TimeoutError): raise connection. send_command (* args) return self. parse_response (connection, command_name, ** options) finally: pool. release (connection) # Call ConnectionPool. release connection

Let's take a look at the ConnectionPool class:

Class ConnectionPool (object ):........... def _ init _ (self, connection_class = Connection, max_connections = None, ** connection_kwargs ): # The constructor max_connections = max_connections or 2 *** 31 if not isinstance (max_connections, (int, long) or max_connections <0: # judge whether the input max_connections is valid raise ValueError ('"max_connections" must be a positive integer') self. connection_class = connection_class # set the corresponding Self. connection_kwargs = connection_kwargs self. max_connections = max_connections self. reset () # reset operation def reset (self): self when ConnectionPool is initialized. pid = OS. getpid () self. _ created_connections = 0 # counter of the created connection self. _ available_connections = [] # declare an empty array to store available connections self. _ in_use_connections = set () # declares an empty set to store the used connection self. _ check_lock = threading. lock ()....... def get_connection (self, command_name, * Keys, ** options): # obtain the connection method "Get a connection from the pool" self. _ checkpid () try: connection = self. _ available_connections.pop () # obtain and delete the element representing the connection. when connectiong is obtained for the first time, because _ available_connections is an empty array, the make_connection method will be called directly to handle T IndexError: connection = self. make_connection () self. _ in_use_connections.add (connection) # add the element return connection def make_connection (self) to the set that represents the connection in use: # In _ available_c When the onnections array is null, obtain the connection call method "Create a new connection" if self. _ created_connections> = self. max_connections: # determines whether the created connection has reached the maximum limit. max_connections can initialize raise ConnectionError ("Too many ONS") self through the parameter. _ created_connections + = 1 # add the value representing the created connection + 1 return self. connection_class (** self. connection_kwargs) # returns a valid Connection. the default value is Connection (** self. connection_kwargs) def release (self, connection): # release the connection. the link is not broken, but it exists in the connection pool. "Releases the connection back to the pool" self. _ checkpid () if connection. pid! = Self. pid: return self. _ in_use_connections.remove (connection) # Delete the element self from the set. _ available_connections.append (connection) # add it to the _ available_connections array def disconnect (self): # disconnect the connection from all connection pools "Disconnects all connections in the pool" all_conns = chain (self. _ available_connections, self. _ in_use_connections) for connection in all_conns: connection. disconnect ()

Execute_command finally calls the Connection. send_command method, closes the Connection as the Connection. disconnect method, and implements the Connection class:

Class Connection (object): "Manages TCP communication to and from a Redis server" def _ del _ (self): # Call disconnect to release the Connection try: self. disconnect () failed t Exception: pass

The core connection establishment method is implemented through the socket module:

Def _ connect (self): err = None for res in socket. getaddrinfo (self. host, self. port, 0, socket. SOCK_STREAM): family, socktype, proto, canonname, socket_address = res sock = None try: sock = socket. socket (family, socktype, proto) # TCP_NODELAY sock. setsockopt (socket. IPPROTO_TCP, socket. TCP_NODELAY, 1) # TCP_KEEPALIVE if self. socket_keepalive: # socket_keepalive = False by default in the constructor. Therefore, sock is a transient connection by default. setsockopt (socket. SOL_SOCKET, socket. SO_KEEPALIVE, 1) for k, v in iteritems (self. socket_keepalive_options): sock. setsockopt (socket. SOL_TCP, k, v) # set the socket_connect_timeout before we connect sock. settimeout (self. socket_connect_timeout) # socket_connect_timeout = None by default in the constructor, that is, the connection mode is blocking # connect sock. connect (socket_address) # set the socket_timeout now that we're re connected sock. settimeout (self. socket_timeout) # socket_timeout = None return sock into T socket by default in the constructor. error as _: err = _ if sock is not None: sock. close ().....

How to close the link:

Def disconnect (self): "Disconnects from the Redis server" self. _ parser. on_disconnect () if self. _ sock is None: return try: self. _ sock. shutdown (socket. SHUT_RDWR) # First shut down and then close self. _ sock. close () handle T socket. error: pass self. _ sock = None

The following can be summarized:
1) by default, a ConnectionPool instance is created every time a Redis instance is created, and a connection is obtained from this connection pool every time redis is accessed, after the operation is complete, the connection is put back into the connection pool (the connection is not released). you can construct a unified ConnectionPool. when creating a Redis instance, you can pass in the ConnectionPool, in the future, the connection will be obtained from the specified ConnectionPool, and the ConnectionPool will not be created again.
2) keepalive and timeout are not set by default. the established connection is a short connection in blocking mode.
3) without considering the underlying tcp, connections in the connection pool will be centrally destroyed in ConnectionPool. disconnect.