Example of a throttling in a python distributed environment

Example of a throttling in a python distributed environment

Throttling is used in the project. Due to some implementation methods, a simple server Throttling is opened.

The difference between server-side throttling and client-side Throttling is:

1) throttling on the server

API requests are throttled by the number of requests per unit time in exchange for high availability through lossy methods.

For example, in our scenario, a service receives a request and, after processing, bulk the data to Elasticsearch for index storage. bulk index is a resource-consuming operation, in case of a surge in request traffic, Elasticsearch may be overwhelmed (queue blocking and memory surge). Therefore, you need to limit the peak traffic.

2) Client throttling

The limit is the number of client accesses.

For example, the thread pool is a natural throttling. Max_connection limits the number of concurrent connections. If the number of connections is too large, it will be placed in a Buffer Queue. If the number of connections is too large, the queue will be discarded.

This document describes the server throttling.

Advantages of my current throttling:

1) Simple
2) Management

Disadvantages:

1) smooth Throttling is not allowed

For example, you can try out the token bucket algorithm and the missing Bucket Algorithm (I think these two algorithms are essentially one thing) to achieve smooth throttling. What is smooth throttling? For example, we need to limit the number of workers to no more than 1000 in five seconds. Smooth throttling can be achieved, with 200 records per second and no more than 1000 records per five seconds. This is very balanced. non-smooth Throttling is possible, 1000 accesses were made in the first second, and all accesses were restricted in the next four seconds. • 2) inflexible

Throttling is only implemented in seconds.

Two scenarios are supported:

1) for single-process multi-thread scenarios (using thread-safe Queue as a global variable)

In this scenario, only one instance is deployed, and the instance is throttled. Rarely used in a production environment.

2) For multi-process distributed scenarios (using redis for global variables)

Multi-instance deployment is generally used in the production environment.

In this scenario, we need to control the overall traffic. For example, if the user service deploys three instances to expose the query interface, the interface-level traffic limit is required, that is, the peak value allowed for the query interface as a whole, instead of worrying about which instance the load is.

This can be done through nginx.

Let's talk about the implementation of the throttling.

1. Interface BaseRateLimiter

According to my ideas, first define an interface, which can also be called an abstract class.

During initialization, you must configure the rate and the speed limit of the throttling.

Provide an abstract method, acquire (), call this method, and return whether to limit traffic.

class BaseRateLimiter(object):  __metaclass__ = abc.ABCMeta  @abc.abstractmethod  def __init__(self, rate):    self.rate = rate  @abc.abstractmethod  def acquire(self, count):    return

2. throttling of ThreadingRateLimiter in a single-process multi-thread scenario

Inherits the BaseRateLimiter abstract class and uses the thread-safe Queue as the global variable to eliminate the impact of the race state.

A process in the background clears the queue every second;

When a request comes, call the acquire function and the queue is incr once. If the request exceeds the speed limit, the system returns the limit. Otherwise, access is allowed.

class ThreadingRateLimiter(BaseRateLimiter):  def __init__(self, rate):    BaseRateLimiter.__init__(self, rate)    self.queue = Queue.Queue()    threading.Thread(target=self._clear_queue).start()  def acquire(self, count=1):    self.queue.put(1, block=False)    return self.queue.qsize() < self.rate  def _clear_queue(self):    while 1:      time.sleep(1)      self.queue.queue.clear()

2. throttling DistributeRateLimiter in Distributed scenarios

Inherits the abstract class BaseRateLimiter and uses external storage as shared variables. The access mode of external storage is cache.

class DistributeRateLimiter(BaseRateLimiter):  def __init__(self, rate, cache):    BaseRateLimiter.__init__(self, rate)    self.cache = cache  def acquire(self, count=1, expire=3, key=None, callback=None):    try:      if isinstance(self.cache, Cache):        return self.cache.fetchToken(rate=self.rate, count=count, expire=expire, key=key)    except Exception, ex:      return True

We implemented the Cache class for decoupling and flexibility. Provides an abstract method getToken ()

If you use redis, you will inherit the Cache abstract class to obtain the token through redis.

If mysql is used, you can inherit the Cache abstract class to obtain the token through mysql.

Cache abstract class

class Cache(object):  __metaclass__ = abc.ABCMeta  @abc.abstractmethod  def __init__(self):    self.key = "DEFAULT"    self.namespace = "RATELIMITER"  @abc.abstractmethod  def fetchToken(self, rate, key=None):    return

A redis implementation redistmencache is provided.

Create a key every second and count the request incr. When the Count value of this second exceeds the speed limit rate, the token cannot be obtained, that is, the traffic limit.

The key created per second times out and expire. Ensure that the key does not continuously occupy the storage space.

There is no difficulty. Here we use redis transactions to ensure that incr and expire can be executed successfully at the same time.

class RedisTokenCache(Cache):  def __init__(self, host, port, db=0, password=None, max_connections=None):    Cache.__init__(self)    self.redis = redis.Redis(      connection_pool=        redis.ConnectionPool(          host=host, port=port, db=db,          password=password,          max_connections=max_connections        ))  def fetchToken(self, rate=100, count=1, expire=3, key=None):    date = datetime.now().strftime("%Y-%m-%d %H:%M:%S")    key = ":".join([self.namespace, key if key else self.key, date])    try:      current = self.redis.get(key)      if int(current if current else "0") > rate:        raise Exception("to many requests in current second: %s" % date)      else:        with self.redis.pipeline() as p:          p.multi()          p.incr(key, count)          p.expire(key, int(expire if expire else "3"))          p.execute()          return True    except Exception, ex:      return False

Code testing in multi-threaded scenarios

Limiter = ThreadingRateLimiter (rate = 10000) def job (): while 1: if not limiter. acquire (): print 'throttle' else: print 'normal' threads = [threading. thread (target = job) for I in range (10)] for thread in threads: thread. start ()

Test code in Distributed scenarios

Token_cache = redisponencache (host = '10. 93.84.53 ', port = 6379, password = 'bigdata123') limiter = DistributeRateLimiter (rate = 10000, cache = token_cache) r = redis. redis (connection_pool = redis. connectionPool (host = '10. 93.84.53 ', port = 6379, password = 'bigdata123') def job (): while 1: if not limiter. acquire (): print 'throttle' else: print 'normal' threads = [multiprocessing. process (target = job) for I in range (10)] for thread in threads: thread. start ()

You can run it on your own.

Note:

The speed limit here is second-level, for example, limit 400 requests per second. It is possible that the first 100 ms of the second will be displayed, and 400 requests will be sent, and the last ms will all be restricted. That is, the traffic cannot be smoothly throttled.

However, if your backend logic has a buffer such as a queue or a thread pool, this smoothing will not actually affect you much.

The above is all the content of this article. I hope it will be helpful for your learning and support for helping customers.