How to deploy Python applications on the server without interruption

How to deploy Python applications on the server without interruption

When you start to deploy an application, you can restart my_app or all services as an administrator to upgrade the product to the current version. Everything works fine at the beginning, but eventually you will find that once the application is started, trying to connect during the restart will get a lot of HTTP 503 errors.

Finally, you may find that Gunicorn and uWSGI can reload your application without closing the socket. In this way, when your application is started, the network request is only delayed a little. As long as your application does not take a long time to start, it will work well. Unfortunately, many existing applications may take one minute to start, which is too long for the link waiting on the socket.

Gunicorn uses kill-HUP $ PID to shut down all worker processes and then start them to reload them. However, slow initialization may lead to problems. UWSGI uses chained overload, which starts only one worker process at a time. I need support for Tornado, which is not very suitable for uWSGI currently.

Use Server Load balancer

A common technology is to remove a single server from the Server Load balancer, upgrade/restart the application, and load it back. We are using a Server Load balancer, but to schedule the entire process, we need to coordinate the use of HAProxy to manage sockets When configuring nodes. Our current deployment scheme is to deploy at the same time to all nodes, rather than one by one, with a considerable change. During waiting for LBs to remove a node from the pool, you can use the 404 ing status page to cheat healthcheck. This is a little more time than I want. For each server, the interval between two healthcheck failures is 5 seconds, which includes the time for the web process to recover after the upgrade is completed.

Gunicorn heavy load ++

Gunicorn automatically restarts the failed web process, so it may kill each process and sleep until all the sub-processes are executed. This is very effective, but if the number of application launches changes significantly, we will either wait too long for the restart, or wait for a short period of time and bear the risk of some failures.

Because Gunicorn contains a Python hook pointing to the application, it is entirely possible to write a short piece of code to notify the restart process when the worker process is ready. Gunicorn does not contain the required hooks, but it is easy to make changes. It requires some modifications before the new version is released.

Now, restarting processes has taken advantage of the fact that a single soket has multiple processes that accept connections. Restarting the service will only slightly reduce the service capability (1/N), but we can continue to process the traffic without waiting for a long time.

This process is generally like this
 

for child_pid of gunicorn-master: kill child_pid wait for app startup

In my first version, shell and nc are used to listen to UDP packets started by the application. Although it is a little more difficult to integrate our process manager into a shell environment than I expected, it works well.

When the restart script is called, The Gunicorn PID should be carried, that is, the $ PID of masterrestart. sh.
 

echo 'Killing children of ' $1; children=$(pgrep -P $1)for child in $childrendo echo 'Killing' $child kill $child response=$(timeout 60 nc -w 0 -ul 4012) if [ "$response" != '200 OK' ]; then  echo 'BROKEN'  exit 1; fidone

The post_worker_init script is concatenated to notify the restart script when the app is running.
 

Import socketimport time def post_worker_init (worker): _ send_udp ('2017 OK \ n') def _ send_udp (message): udp_ip = "127.0.0.1" udp_port = 200 sock = socket. socket (socket. AF_INET, # Internet socket. SOCK_DGRAM) # UDP sock. sendto (message, (udp_ip, udp_port) If we have such a WSGI (Python Web Server Gateway Interface) Application: from werkzeug. wrappers import Request, Response @ Request. applicationdef application (request): r Esp = Response ('Hello World! ') If request. path ='/_ status': resp. status = '2017 OK 'else: resp. status = '2017 Not Found' return resp

We can even check the/_ status page to verify whether the application is running.
 

def post_worker_init(worker): env = {  'REQUEST_METHOD': 'GET',  'PATH_INFO': '/_status', } def start_response(*args, **kwargs):  _send_udp(args[0])  worker.wsgi(env, start_response)

Do not try to run too many applications in this health check. If your post_worker_init has an error for whatever reason, the working process will exit and stop the application from starting. This can be a problem when you check for database links that may be invalid. Even if your application can work, it cannot be started again.

Now, with one-minute application startup, we implement rolling restart without stopping the application or dropping any links!