Test methods for accessing the same data's competitive conditions in Python _python

When you have multiple processes or threads accessing the same data, the competition condition is a threat. This article explores how to test the competitive conditions after they are found.

Incrmnt

You work for a fiery new company called "Incrmnt", which does only one thing and does a better job.

You show a global counter and a plus sign, the user can click the plus sign, the counter plus one. It's too easy, and it's easy to get addicted to. There is no doubt that this is the next big thing.

Investors are scrambling to get into the boardroom, but you have a big problem.

Competitive conditions

In your beta, Abraham and Belinda are so excited that everyone points the plus button 100 times. Your server log shows 200 requests, but the counter is displayed as 173. Obviously, there are some requests that have not been added.

First of all, "incrmnt into a piece of crap" news, you check the code (all the code used in this article can be found on the GitHub).

# incrmnt.py
Import DB
 
def increment ():
  count = Db.get_count ()
 
  New_count = count + 1
  db.set_count ( New_count) return
 
  New_count

Your Web server uses multiple processes to process traffic requests, so this function can be executed simultaneously in different threads. If you don't have a good time, it will happen:

# thread 1 and Thread 2 execute at the same time in different processes
# for the purpose of the display, place here side in the
vertical # to show what code to execute at each point in time
# thread 1 (thread 1)         # thread 2 (Threads 2)
def increment ():
                  def increment ():
  # get_count returns 0
  count = Db.get_count ()
                    # get_count returns 0 again
                    count = Db.get_count ()
  New_count = count + 1
  # set_count called with 1
  db.set_count (new_count )
                    New_count = count + 1
                    # Set_count called with 1 again
                    Db.set_count (New_count)

So, despite the two increase in counting, it ended up with only a 1 increase.

You know you can modify this code to become thread safe, but before you do that, you want to write a test to prove the existence of competition.

Reproduce competition

Ideally, the test should reproduce the above scenario as much as possible. The key factors of competition are:

? Two get_count calls must be executed before two Set_count calls, thus making the count in two threads have the same value.

Set_count calls, and when they do, it doesn't matter, as long as they're all in the get_count call.

For simplicity's sake, let's try to reproduce this nesting situation. Here the entire thread 2 executes after the first get_count call to Thread 1:

# thread 1             # thread 2
def increment ():
  # get_count returns 0
  count = Db.get_count ()
                  def increment (): c6/># get_count returns 0 again
                    count = Db.get_count ()
 
                    # set_count called with 1
                    new_count = count + 1
                    db . Set_count (New_count)
  # Set_count called with 1 again
  New_count = count + 1
  db.set_count (new_count)

Before_after is a library that provides the tools to help reproduce this situation. It can insert arbitrary code before or after a function.

Before_after relies on mock libraries, which are used to supplement some functionality. If you are unfamiliar with mocks, I recommend reading some excellent documentation. A particularly important part of the document is Where to Patch.

We expect thread 1 to execute all thread 2 after calling Get_count, and then resume thread 1.

Our test code is as follows:

# test_incrmnt.py
 
Import unittest
 
import before_after import
 
db
import incrmnt
 
class Testincrmnt (UnitTest. TestCase):
  def setUp (self):
    db.reset_db ()
 
  def test_increment_race (self):
    # After a call to Get_count, Call Increment
    and Before_after.after (' Incrmnt.db.get_count ', incrmnt.increment):
      # Start off the race with A call to increment
      incrmnt.increment ()
 
    count = Db.get_count ()
    self.assertequal (count, 2)

After the first get_count call, we use the Before_after context manager after to insert another increment call.

By default, Before_after only calls once after functions. This is useful in this particular case, because otherwise the stack will overflow (increment call Get_count,get_coun T also call increment,increment and call get_count ...). ）。

The test failed because the count equals 1, not 2. Now we have a failed test that reproduces the competitive conditions and fixes it together.

Prevent competition

We're going to use a simple locking mechanism to slow down the competition. This is clearly not an ideal solution, and a better solution is to use atomic updates for data storage-but this approach can better demonstrate Before_after's role in testing multithreaded applications.

To add a new function to the incrmnt.py:

# incrmnt.py
 
def locking_increment (): With
  Db.get_lock (): Return
    increment ()

It guarantees that only one thread at a time reads and writes the count. If a thread attempts to acquire a lock and the lock is persisted by another thread, a Couldnotlock exception is thrown.

Now we add such a test:

# test_incrmnt.py
 
def test_locking_increment_race (self):
  def erroring_locking_increment ():
    # trying to Get a lock while the other thread has it'll cause a
    # Couldnotlock Exception-catch it here or the test'll fail
   with self.assertraises (db. Couldnotlock):
      incrmnt.locking_increment ()
 
  with Before_after.after (' Incrmnt.db.get_count ', erroring_ locking_increment):
    incrmnt.locking_increment ()
 
  count = Db.get_count ()
  self.assertequal (count, 1)

Now at the same time, there is only one thread that can increase the count.

Slowing down competition

We also have a problem here, through this way, if two requests conflict, one will not be registered. To alleviate this problem, we can get increment to reconnect to the server (there's a neat way to use something like Funcy retry):

# incrmnt.py
 
def retrying_locking_increment ():
  @retry (tries=5, errors=db. Couldnotlock
  def _increment (): Return
    locking_increment ()
 
  _increment ()

When we need a larger operation than this approach, we can transfer increment as an atomic update or transaction to our database to take responsibility for being away from our applications.

Summarize

Incrmnt now there is no competition, people can happily click a whole day without worrying about not being counted.

This is a simple example, but before_after can be used for more complex competitive conditions to ensure that your function handles all situations correctly. Being able to test and reproduce competitive conditions in a single threaded environment is a key to making you more certain that you are dealing with competitive conditions correctly.