Use Python's Treq on Twisted for HTTP stress testing, pythontreq

Use Python's Treq on Twisted for HTTP stress testing, pythontreq

API-related work is very challenging. Maintaining System Stability and robustness during peak hours is one of them. This is why we do a lot of stress testing in Mailgun.

For so long, we have tried many methods, from simple apachetings to complicated custom test sets. However, this post describes how to use python to perform "Fast and rough" but flexible stress testing.
When writing an HTTP client using python, we like to use the Requests library. This is also recommended to our API users. Requests is very powerful, but there is a drawback that it is a modular each thread calls something, it is difficult or impossible to use it to quickly generate thousands of levels of Requests.
Introduction to Treq on Twisted

To solve this problem, we introduced Treq (Github library ). Treq is an HTTP client library affected by Requests, but it runs on Twisted. It has a typical powerful capability of Twisted: It is asynchronous and highly concurrent when processing network I/O.

Treq is not limited to stress testing: it is a good tool for writing highly concurrent HTTP clients, such as web page capturing. Treq is elegant, easy to use, and powerful. This is an example:

 >>> from treq import get   >>> def done(response): ...   print response.code ...   reactor.stop()   >>> get("http://www.github.com").addCallback(done)   >>> from twisted.internet import reactor 200

Simple Test script
The following is a simple script that uses Treq to bomb a single URL with a request with maximum energy.

 #!/usr/bin/env python from twisted.internet import epollreactor epollreactor.install()   from twisted.internet import reactor, task from twisted.web.client import HTTPConnectionPool import treq import random from datetime import datetime   req_generated = 0 req_made = 0 req_done = 0   cooperator = task.Cooperator()   pool = HTTPConnectionPool(reactor)   def counter():   '''This function gets called once a second and prints the progress at one   second intervals.   '''   print("Requests: {} generated; {} made; {} done".format(       req_generated, req_made, req_done))   # reset the counters and reschedule ourselves   req_generated = req_made = req_done = 0   reactor.callLater(1, counter)   def body_received(body):   global req_done   req_done += 1   def request_done(response):   global req_made   deferred = treq.json_content(response)   req_made += 1   deferred.addCallback(body_received)   deferred.addErrback(lambda x: None) # ignore errors   return deferred   def request():   deferred = treq.post('http://api.host/v2/loadtest/messages',              auth=('api', 'api-key'),              data={'from': 'Loadtest <test@example.com>',                 'to': 'to@example.org',                'subject': "test"},             pool=pool)   deferred.addCallback(request_done)   return deferred   def requests_generator():   global req_generated   while True:     deferred = request()     req_generated += 1     # do not yield deferred here so cooperator won't pause until     # response is received     yield None   if __name__ == '__main__':   # make cooperator work on spawning requests   cooperator.cooperate(requests_generator())     # run the counter that will be reporting sending speed once a second   reactor.callLater(1, counter)     # run the reactor   reactor.run()

Output result:

 2013-04-25 09:30 Requests: 327 generated; 153 sent; 153 received 2013-04-25 09:30 Requests: 306 generated; 156 sent; 156 received 2013-04-25 09:30 Requests: 318 generated; 184 sent; 154 received

The number of the "Generated" class indicates a request that has been prepared by the Twisted reactor but has not yet been sent. This script ignores all error handling for simplicity. To add timeout information for it, you can leave it to the reader as an exercise.

This script can be used as a starting point. You can improve it to customize the processing logic of a specific application through expansion. We recommend that you use collections. Counter to replace the ugly global variable during the improvement process. This script runs on a single thread. to squeeze the largest number of requests from a single machine, you can use techniques similar to mulitprocessing.

May you enjoy stress testing!