Python implements simple queues and cross-process lock instances based on mysql, pythonmysql

Python implements simple queues and cross-process lock instances based on mysql, pythonmysql

In the process of multi-process application development, it is inevitable that multiple processes access the same resource (critical resource). At this time, we must add a global lock, to Achieve Synchronous access to resources (that is, only one process can access resources at a time ).

For example:

Suppose we use mysql to implement a task queue. The implementation process is as follows:

1. Create a Job table in Mysql to store queue tasks., As follows:

create table jobs(  id auto_increment not null primary key,  message text not null,  job_status not null default 0);

Message is used to store task information. job_status is used to identify the task status. Assume that there are only two statuses: 0: in the queue, 1: Out of the queue.
 
2. There is a producer process that places new data into the job table.To queue:

insert into jobs(message) values('msg1');

3. Assume that there are multiple consumer processes that fetch queuing information from the job table.To do the following:

Select * from jobs where job_status = 0 order by id asc limit 1; update jobs set job_status = 1 where id = ?; -- Id is the id of the obtained record.

4. If there is no cross-process lock, the two consumer processes may receive duplicate messages at the same time, resulting in one message being consumed multiple times. We don't want to see this, so we need to implement a cross-process lock.

========================================================= ======================================

When talking about cross-process lock implementation, we mainly have several implementation methods:

(1) semaphores
(2) file lock fcntl
(3) socket (port number binding)
(4) signal
These methods have their own advantages and disadvantages. In general, the first two methods may be a little more. I will not detail them here. You can refer to the materials.
 
When querying data, we found that mysql has a lock implementation, which is suitable for scenarios where performance requirements are not very high. Large concurrent distributed access may cause bottlenecks.
 
Python is used to implement a demo, as shown below:
 
File Name: glock. py

#!/usr/bin/env python2.7 # # -*- coding:utf-8 -*- # #  Desc  : # import logging, time import MySQLdb class Glock:   def __init__(self, db):     self.db = db   def _execute(self, sql):     cursor = self.db.cursor()     try:       ret = None       cursor.execute(sql)       if cursor.rowcount != 1:         logging.error("Multiple rows returned in mysql lock function.")         ret = None       else:         ret = cursor.fetchone()       cursor.close()       return ret     except Exception, ex:       logging.error("Execute sql \"%s\" failed! Exception: %s", sql, str(ex))       cursor.close()       return None   def lock(self, lockstr, timeout):     sql = "SELECT GET_LOCK('%s', %s)" % (lockstr, timeout)     ret = self._execute(sql)      if ret[0] == 0:       logging.debug("Another client has previously locked '%s'.", lockstr)       return False     elif ret[0] == 1:       logging.debug("The lock '%s' was obtained successfully.", lockstr)       return True     else:       logging.error("Error occurred!")       return None   def unlock(self, lockstr):     sql = "SELECT RELEASE_LOCK('%s')" % (lockstr)     ret = self._execute(sql)     if ret[0] == 0:       logging.debug("The lock '%s' the lock is not released(the lock was not established by this thread).", lockstr)       return False     elif ret[0] == 1:       logging.debug("The lock '%s' the lock was released.", lockstr)       return True     else:       logging.error("The lock '%s' did not exist.", lockstr)       return None #Init logging def init_logging():   sh = logging.StreamHandler()   logger = logging.getLogger()   logger.setLevel(logging.DEBUG)   formatter = logging.Formatter('%(asctime)s -%(module)s:%(filename)s-L%(lineno)d-%(levelname)s: %(message)s')   sh.setFormatter(formatter)   logger.addHandler(sh)   logging.info("Current log level is : %s",logging.getLevelName(logger.getEffectiveLevel())) def main():   init_logging()   db = MySQLdb.connect(host='localhost', user='root', passwd='')   lock_name = 'queue'    l = Glock(db)    ret = l.lock(lock_name, 10)   if ret != True:     logging.error("Can't get lock! exit!")     quit()   time.sleep(10)   logging.info("You can do some synchronization work across processes!")   ##TODO   ## you can do something in here ##   l.unlock(lock_name) if __name__ == "__main__":   main() 

In the main function:

In l. lock (lock_name, 10), 10 indicates that the timeout time is 10 seconds. If the lock cannot be obtained within 10 seconds, the system returns and performs the subsequent operations.
 
In this demo, the logic for consumers to retrieve messages from the job table can be put here where TODO is marked. That is, the split line is above.

2. Assume that there are multiple consumer processes that fetch the queuing information from the job table. The operation to do is as follows:

Select * from jobs where job_status = 0 order by id asc limit 1; update jobs set job_status = 1 where id = ?; -- Id is the id of the obtained record.

In this way, multiple processes can be synchronized when accessing critical resources to ensure data consistency.
 
Start Two glock. py during the test. The result is as follows:

[@tj-10-47 test]# ./glock.py  2014-03-14 17:08:40,277 -glock:glock.py-L70-INFO: Current log level is : DEBUG 2014-03-14 17:08:40,299 -glock:glock.py-L43-DEBUG: The lock 'queue' was obtained successfully. 2014-03-14 17:08:50,299 -glock:glock.py-L81-INFO: You can do some synchronization work across processes! 2014-03-14 17:08:50,299 -glock:glock.py-L56-DEBUG: The lock 'queue' the lock was released. 

We can see that the first glock. py was unlocked at 17:08:50, and the following glock. py obtained the lock at 17:08:50, which proves that this is completely feasible.

[@tj-10-47 test]# ./glock.py 2014-03-14 17:08:46,873 -glock:glock.py-L70-INFO: Current log level is : DEBUG2014-03-14 17:08:50,299 -glock:glock.py-L43-DEBUG: The lock 'queue' was obtained successfully.2014-03-14 17:09:00,299 -glock:glock.py-L81-INFO: You can do some synchronization work across processes!2014-03-14 17:09:00,300 -glock:glock.py-L56-DEBUG: The lock 'queue' the lock was released.[@tj-10-47 test]#