A high-performance counter (Counter) instance is implemented in Django _python

Counter (Counter) is a very common functional component, which takes the unread message number as an example, and introduces the basic points of implementing a high-performance counter in Django.

The beginning of the story:. Count ()

Suppose you have a notification model class that saves mainly all of the station notifications:

Copy Code code as follows:

Class Notification (models. Model):
"" "a simplified notification class with three fields:

-' user_id ': User ID for the message owner
-' has_readed ': Indicates whether the message is read
"""

USER_ID = models. Integerfield (Db_index=true)
has_readed = models. Booleanfield (Default=false)

Of course, at first you will get the number of unread messages for a user through such a query:

Copy Code code as follows:

# Gets the number of unread messages for the user with ID 3074
Notification.objects.filter (user_id=3074, Has_readed=false). Count ()

When your notification table is smaller, there is no problem in this way, but slowly, as the volume of business expands. There are billions of data in the message table. Many lazy users have thousands of unread messages.

At this point, you need to implement a counter that counts the number of unread messages per user, so that we can get the number of unread messages in real time just by executing a simple primary key query (or better).

Better scenario: Set up counters

first, let's create a new table to store the number of unread messages per user.

Copy Code code as follows:

Class Usernotificationscount (models. Model):
"" "This model holds the number of unread messages per user." "

USER_ID = models. Integerfield (Primary_key=true)
Unread_count = models. Integerfield (default=0)

def __str__ (self):
Return ' <usernotificationscount%s:%s> '% (self.user_id, Self.unread_count)

We provide each registered user with a corresponding Usernotificationscount record to save the number of unread messages. Each time you get the number of unread messages, you only need UserNotificationsCount.objects.get (pk=user_id). Unread_count on it.

Next, the point is, how do we know when we should update our counters? What shortcuts does Django provide in this area?

Challenge: Update your counter in real time

In order for our counter to work properly, we have to update it in real time, which includes:

1. When a new unread message comes over, for the counter +1
2. When the message is deleted unexpectedly, if the associated message is unread, the counter-1
3. When reading a new message, for the counter-1
Let's deal with the situation in one way or another.

Before throwing a solution, we need to introduce a feature in Django: Signals, Signals is a Django-provided event notification mechanism that allows you to listen to certain custom or preset events and invoke the implementation-defined method when these events occur.

For example, Django.db.models.signals.pre_save & Django.db.models.signals.post_save represents an event that triggers before and after a model calls the Save method. It is functionally similar to the triggers provided by the database.

More about signals can refer to the official documentation, let's look at what the signals can do to our counter benefits.

1. When a new message comes over, for the counter +1

This situation should be best handled, using Django signals, just a few lines of code, and we can implement the counter update in this case:

Copy Code code as follows:

From django.db.models.signals import Post_save, Post_delete

def incr_notifications_counter (sender, instance, created, **kwargs):
# Only when this instance is newly created, and has_readed is the default false update
If not (created and not instance.has_readed):
Return

# Call the Update_unread_count method to update the counter +1
Notificationcontroller (instance.user_id). Update_unread_count (1)

# Monitor the post_save signal of notification model
Post_save.connect (Incr_notifications_counter, sender=notification)

This way, whenever you create a new notification by using a method such as Notification.create or. Save (), our Notificationcontroller is notified of the counter +1.

Note, however, that since our counter is based on Django signals, if you have a place in your code that uses raw SQL and does not add new notifications through the Django Orm method, our counters are not notified, so it is best to standardize all new notification creation methods, For example, use the same API.

2. When the message is deleted unexpectedly, if the associated message is unread, the counter-1

With the first experience, this situation is relatively simple to deal with, only need to monitor the notification post_delete signal on it, the following is an example code:

Copy Code code as follows:

def decr_notifications_counter (sender, instance, **kwargs):
# when deleted messages have not been read obsolete, counter-1
If not instance.has_readed:
Notificationcontroller (instance.user_id). Update_unread_count (-1)

Post_delete.connect (Decr_notifications_counter, sender=notification)

At this point, the notification deletion event can also update our counter normally.

3. When reading a new message, for the counter-1

Next, when the user reads an unread message, we also need to update our unread message counter. You might say, what's so hard about it? All I have to do is update my counter manually in the way I read the messages.

Like this:

Copy Code code as follows:

Class Notificationcontroller (object):

... ...

def mark_as_readed (self, notification_id):
notification = Notification.objects.get (pk=notification_id)
# There is no need to repeat a notice that has been read
If notication.has_readed:
Return

notification.has_readed = True
Notification.save ()
# Here to update our counter, well, I feel great
Self.update_unread_count (-1)

With some simple tests, you can feel that your counter works very well, but there is a very fatal problem with this implementation, and there is no way to handle concurrent requests properly.

For example, you have an unread message object with an ID of 100, at which time there are two requests to mark this notification as read:

Copy Code code as follows:

# Because of two concurrent requests, suppose that the two methods are called almost simultaneously
Notificationcontroller (user_id). mark_as_readed (100)
Notificationcontroller (user_id). mark_as_readed (100)

Obviously, these two methods will successfully mark this notification as read, because in the case of concurrency, if notification.has_readed such checks do not work, so our counter will be wrong-12 times, but in fact we read only one request.

So, how should such a problem be solved?

Basically, there is only one way to resolve data conflicts generated by concurrent requests: lock, and introduce two simpler solutions:

Using the Select for Update database query

Select ... for update is a database-level specifically used to solve the concurrency of data and then modified after the scene, the mainstream relational database such as MySQL, PostgreSQL support this feature, the new version of the Django ORM even provides a shortcut of this functionality directly. For more information about it, you can search for an introductory document for the database you use.

With select for update, our code might become like this:

Copy Code code as follows:

From django.db Import Transaction

Class Notificationcontroller (object):

... ...

    def mark_as_readed (self, notification_id):
        # Manually let the select for update and UPDATE statements occur in a complete transaction
        with transaction.commit_on _success ():
            # using Select_for_ Update to ensure that concurrent requests are processed with only one request, and other requests
            # Wait for lock release
            notification = Notification.objects.select_for _update (). Get (pk=notification_id)
            There is no need to repeatedly mark an already read notification
            if notication.has_readed :
                return

notification.has_readed = True
Notification.save ()
# Here to update our counter, well, I feel great
Self.update_unread_count (-1)

In addition to using the ' Select For Update ' feature, there is a simpler way to solve this problem.

Using update to make atomic modifications

In fact, a simpler approach, as long as we change our database into a single update can solve the problem of concurrent situations:

Copy Code code as follows:

def mark_as_readed (self, notification_id):
Affected_rows = Notification.objects.filter (pk=notification_id, has_readed=false) \
. Update (Has_readed=true)
# Affected_rows will return the number of entries modified by the UPDATE statement
Self.update_unread_count (Affected_rows)

In this way, the concurrent markup read operation can also correctly affect our counter.

Performance?

We have previously described how to implement an unread message counter that can be updated correctly, and we may be using the UPDATE statement directly to modify our counters, like this:

Copy Code code as follows:

From Django.db.models import F

def update_unread_count (self, count)
# Use the UPDATE statement to update our counters
UserNotificationsCount.objects.filter (pk=self.user_id) \
. Update (Unread_count=f (' unread_count ') + count)

However, in a production environment, such a process is likely to cause serious performance problems, because if our counters are updated frequently, a large amount of update can cause a lot of pressure on the database. So in order to achieve a high-performance counter, we need to save the changes, and then batch write to the database.

Using Redis's sorted set, we can do this very easily.

Use sorted set to cache counter changes

Redis is a very useful memory database, where the sorted set is a data type it provides: An ordered set, with which we can cache all counter changes very simply, and then write back to the database in batches.

Copy Code code as follows:

Rk_notifications_counter = ' Ss_pending_counter_changes '

def update_unread_count (self, count):
"" "Modified Update_unread_count Method" "
Redisdb.zincrby (Rk_notifications_counter, str (self.user_id), count)

# at the same time we also need to modify to get the user unread message number method, so that it gets the redis of those who have not been written back
# buffer data to the database. Here, the code is omitted.

Through the above code, we have the counter update buffered in the Redis inside, we also need a script to the buffer inside the data back to write to the database.

By customizing the Django command, we can do this very easily:

Copy Code code as follows:

# file:management/commands/notification_update_counter.py

#-*-Coding:utf-8-*-
From django.core.management.base import Basecommand
From Django.db.models import F

# Fix Import Prob
From Notification.models import Usernotificationscount
From notification.utils import Rk_notifications_counter
From Base_redis import Redisdb

Import logging
Logger = Logging.getlogger (' stdout ')

Class Command (Basecommand):
Help = ' Update usernotificationscounter objects, Write changes from Redis to database '

def handle (self, *args, **options):
# First, get all modified user IDs for buffers by Zrange command
For user_id in Redisdb.zrange (rk_notifications_counter, 0,-1):
# It's worth noting that, in order to ensure the atomic nature of the operation, we used the REDISDB pipeline
Pipe = Redisdb.pipeline ()
Pipe.zscore (Rk_notifications_counter, user_id)
Pipe.zrem (Rk_notifications_counter, user_id)
Count, _ = Pipe.execute ()
count = Int (count)
If not count:
Continue

Logger.info (' Updating unread count user%s:count%s '% (user_id, Count)
UserNotificationsCount.objects.filter (pk=obj.pk) \
. Update (Unread_count=f (' unread_count ') + count)

After that, the changes in the buffer can be written back to the database in batches via a command such as Python manage.py notification_update_counter. We can also configure this command into crontab to define execution.

Summarize

The article here, a simple "high-performance" unread message counter is completed. Said so much, in fact, the main point of knowledge is all these:

1. Use Django signals to get model's new/delete operation update
2. Using the database's select for update to correctly handle concurrent database operations
3. Use Redis's sorted set to cache counter modification operations
I hope to be of some help to you. :)