Python與Redis的串連教程

今天在寫zabbix storm job監控指令碼的時候用到了python的redis模組，之前也有用過，但是沒有過多的瞭解，今天看了下相關的api和源碼,看到有ConnectionPool的實現，這裡簡單說下。
在ConnectionPool之前，如果需要串連redis，我都是用StrictRedis這個類，在源碼中可以看到這個類的具體解釋：

redis.StrictRedis Implementation of the Redis protocol.This abstract class provides a Python interface to all Redis commands and an
implementation of the Redis protocol.Connection and Pipeline derive from this, implementing how the commands are sent and received to the Redis server

使用的方法：

 r=redis.StrictRedis(host=xxxx, port=xxxx, db=xxxx) r.xxxx()

有了ConnectionPool這個類之後，可以使用如下方法

pool = redis.ConnectionPool(host=xxx, port=xxx, db=xxxx)r = redis.Redis(connection_pool=pool)

這裡Redis是StrictRedis的子類
簡單分析如下：
在StrictRedis類的__init__方法中，可以初始化connection_pool這個參數，其對應的是一個ConnectionPool的對象：

class StrictRedis(object):........  def __init__(self, host='localhost', port=6379,         db=0, password=None, socket_timeout=None,         socket_connect_timeout=None,         socket_keepalive=None, socket_keepalive_options=None,         connection_pool=None, unix_socket_path=None,         encoding='utf-8', encoding_errors='strict',         charset=None, errors=None,         decode_responses=False, retry_on_timeout=False,         ssl=False, ssl_keyfile=None, ssl_certfile=None,         ssl_cert_reqs=None, ssl_ca_certs=None):     if not connection_pool:       ..........       connection_pool = ConnectionPool(**kwargs)     self.connection_pool = connection_pool

在StrictRedis的執行個體執行具體的命令時會調用execute_command方法，這裡可以看到具體實現是從串連池中擷取一個具體的串連，然後執行命令，完成後釋放串連：

  # COMMAND EXECUTION AND PROTOCOL PARSING  def execute_command(self, *args, **options):    "Execute a command and return a parsed response"    pool = self.connection_pool    command_name = args[0]    connection = pool.get_connection(command_name, **options) #調用ConnectionPool.get_connection方法擷取一個串連    try:      connection.send_command(*args) #命令執行，這裡為Connection.send_command      return self.parse_response(connection, command_name, **options)    except (ConnectionError, TimeoutError) as e:      connection.disconnect()      if not connection.retry_on_timeout and isinstance(e, TimeoutError):        raise      connection.send_command(*args)       return self.parse_response(connection, command_name, **options)    finally:      pool.release(connection) #調用ConnectionPool.release釋放串連

在來看看ConnectionPool類：

class ConnectionPool(object):     ...........  def __init__(self, connection_class=Connection, max_connections=None,         **connection_kwargs):  #類初始化時調用建構函式    max_connections = max_connections or 2 ** 31    if not isinstance(max_connections, (int, long)) or max_connections < 0: #判斷輸入的max_connections是否合法      raise ValueError('"max_connections" must be a positive integer')    self.connection_class = connection_class #設定對應的參數    self.connection_kwargs = connection_kwargs    self.max_connections = max_connections    self.reset() #初始化ConnectionPool 時的reset操作  def reset(self):    self.pid = os.getpid()    self._created_connections = 0 #已經建立的串連的計數器    self._available_connections = []  #聲明一個空的數組，用來存放可用的串連    self._in_use_connections = set() #聲明一個空的集合，用來存放已經在用的串連    self._check_lock = threading.Lock().......  def get_connection(self, command_name, *keys, **options): #在串連池中擷取串連的方法    "Get a connection from the pool"    self._checkpid()    try:      connection = self._available_connections.pop() #擷取並刪除代表串連的元素，在第一次擷取connectiong時，因為_available_connections是一個空的數組，      會直接調用make_connection方法    except IndexError:      connection = self.make_connection()    self._in_use_connections.add(connection)  #向代表正在使用的串連的集合中添加元素    return connection    def make_connection(self): #在_available_connections數組為空白時擷取串連調用的方法    "Create a new connection"    if self._created_connections >= self.max_connections:  #判斷建立的串連是否已經達到最大限制，max_connections可以通過參數初始化      raise ConnectionError("Too many connections")    self._created_connections += 1  #把代表已經建立的串連的數值+1    return self.connection_class(**self.connection_kwargs)   #返回有效串連，預設為Connection(**self.connection_kwargs)  def release(self, connection): #釋放串連，連結並沒有斷開，只是存在連結池中    "Releases the connection back to the pool"    self._checkpid()    if connection.pid != self.pid:      return    self._in_use_connections.remove(connection)  #從集合中刪除元素    self._available_connections.append(connection) #並添加到_available_connections 的數組中  def disconnect(self): #斷開所有串連池中的連結    "Disconnects all connections in the pool"    all_conns = chain(self._available_connections,             self._in_use_connections)    for connection in all_conns:      connection.disconnect()

execute_command最終調用的是Connection.send_command方法，關閉連結為 Connection.disconnect方法，而Connection類的實現：

class Connection(object):  "Manages TCP communication to and from a Redis server"  def __del__(self):  #對象刪除時的操作，調用disconnect釋放串連    try:      self.disconnect()    except Exception:      pass

核心的連結建立方法是通過socket模組實現：

 def _connect(self):    err = None    for res in socket.getaddrinfo(self.host, self.port, 0,                   socket.SOCK_STREAM):      family, socktype, proto, canonname, socket_address = res      sock = None      try:        sock = socket.socket(family, socktype, proto)        # TCP_NODELAY        sock.setsockopt(socket.IPPROTO_TCP, socket.TCP_NODELAY, 1)        # TCP_KEEPALIVE        if self.socket_keepalive:  #建構函式中預設 socket_keepalive=False，因此這裡預設為短串連          sock.setsockopt(socket.SOL_SOCKET, socket.SO_KEEPALIVE, 1)          for k, v in iteritems(self.socket_keepalive_options):            sock.setsockopt(socket.SOL_TCP, k, v)        # set the socket_connect_timeout before we connect        sock.settimeout(self.socket_connect_timeout) #建構函式中預設socket_connect_timeout=None，即串連為blocking的模式        # connect        sock.connect(socket_address)        # set the socket_timeout now that we're connected        sock.settimeout(self.socket_timeout) #建構函式中預設socket_timeout=None        return sock      except socket.error as _:        err = _        if sock is not None:          sock.close().....

關閉連結的方法：

  def disconnect(self):    "Disconnects from the Redis server"    self._parser.on_disconnect()    if self._sock is None:      return    try:      self._sock.shutdown(socket.SHUT_RDWR) #先shutdown再close      self._sock.close()    except socket.error:      pass    self._sock = None

可以小結如下
1）預設情況下每建立一個Redis執行個體都會構造出一個ConnectionPool執行個體，每一次訪問redis都會從這個串連池得到一個串連，操作完成後會把該串連放回串連池（串連並沒有釋放）,可以構造一個統一的ConnectionPool，在建立Redis執行個體時，可以將該ConnectionPool傳入，那麼後續的操作會從給定的ConnectionPool獲得串連，不會再重複建立ConnectionPool。
2）預設情況下沒有設定keepalive和timeout，建立的串連是blocking模式的短串連。
3）不考慮底層tcp的情況下，串連池中的串連會在ConnectionPool.disconnect中統一銷毀。

