Solve the problem of Apscheduler repeated operation in multi-process

Transferred from: http://blog.csdn.net/raptor/article/details/69218271

Problem

In a Python Web application, you need to perform some tasks on a regular basis, so use the Apscheduler library. Because it is used flask This web framework, so the use of Flask-apscheduler this plug-in (essentially the same as the direct use of Apscheduler, there is no distinction).

There is no problem with direct test runs in development, but there is a recurring problem with Gunicorn deployment:

Each task is executed several times at the same time.

Note that the number of repetitions is precisely the number of worker processes configured in Gunicorn, and it is clear that each worker process has initiated a copy of the scheduler.

Solve

There are several scenarios that can be thought of:

• Use --preload startup Gunicorn to make sure Scheduler is only created once in loader
• Also create a separate scheduled task project that runs in a single process
• Use global locks to ensure that scheduler runs only once

In practice, only a third scheme is better.

Preload's question:

Although this can be done using scheduler to create code only once, the problem is that it only executes once, and after redeployment, if you restart Gunicorn with kill-hup, it does not restart and even the entire project is not updated. This is a side effect of preload, unless you rewrite the deployment script to completely restart the app.

Issues with individual processes:

It's also because of the hassle of deployment and the need for a set of deployment scenarios, though Docker would be convenient, but still disliked, and there were many unnecessary things to maintain for two projects at the same time.

Global locking is a good solution, but the problem is finding a suitable lock.

Python comes with a multi-process multithreaded locking scheme that requires a shared variable to maintain, but because worker processes are initiated by Gunicorn's main process and are not easily maintained, a system-level lock is required.

On the StackOverflow saw someone using a socket port to do the lock implementation, but I also do not like this waste of a valuable port resources. But it gave me an inspiration:

You can use the file lock!

So with this solution:

Import atexitimport fcntlfrom flask_apscheduler import apschedulerdef init (APP):    f = open ("Scheduler.lock", "WB") C1/>try:        fcntl.flock (F, fcntl. LOCK_EX | Fcntl. LOCK_NB)        Scheduler = Apscheduler ()        Scheduler.init_app (APP)        Scheduler.start ()    except:        Pass    def unlock ():        Fcntl.flock (F, fcntl. Lock_un)        f.close ()    atexit.register (unlock)

　　

Principle

The init function is called for the initialization of the Flask project, which is the initialization part of the Scheduler module.

First open (or create) a Scheduler.lock file, plus a non-blocking mutex. After success, create the scheduler and start.

If the Add file lock fails, the scheduler has been created, and the section to create the scheduler is skipped.

Finally, an exit event is registered, and if the flask item exits, the lock for the Scheduler.lock file is unlocked and closed.

Solve the problem of Apscheduler repeated operation in multi-process