A tutorial on the counter implementation of message notifications in Python's Django framework

The beginning of the story:. Count ()
Suppose you have a notification model class that saves mostly all of the site notifications:

Class Notification (models. Model): "" "  a simplified notification class with three fields:  -' user_id ': User ID of the message owner  -' has_readed ': Indicates whether the message was read  " "" USER_ID = models. Integerfield (db_index=true)  has_readed = models. Booleanfield (Default=false)

As a matter of course, at first you'll get the number of unread messages for a user with this query:

# Gets the number of unread messages for the user with ID 3074 Notification.objects.filter (user_id=3074, Has_readed=false). Count ()

When your notification table is smaller, there is no problem in this way, but slowly, as the volume of business expands. There are billions of data in the message table. Many lazy users have thousands of unread messages.

At this point, you need to implement a counter, so that the counter to count the number of unread messages per user, so that compared to the previous count (), we only need to execute a simple primary key query (or better) to get real-time unread message number.

Better scenario: Set up a counter
First, let's set up a new table to store the number of unread messages per user.

Class Usernotificationscount (models. Model): "" "  this model holds the number of unread messages per user" ""  user_id = models. Integerfield (primary_key=true)  Unread_count = models. Integerfield (default=0)  def __str__ (self):    return '
 
  
   
  % (self.user_id, Self.unread_count)
 
  

We provide each registered user with a corresponding Usernotificationscount record to save his unread message count. Each time you get the number of unread messages, you only need UserNotificationsCount.objects.get (pk=user_id). Unread_count.

Next, the point of the question comes, how do we know when we should update our counters? What shortcuts does Django offer in this regard?

Challenge: Update your counters in real time

In order for our counters to work properly, we must update it in real time, which includes:

• When there is a new unread message coming up, for the counter +1
• When the message is deleted unexpectedly, if the associated message is unread, the counter-1
• When you have finished reading a new message, for counter-1

Let's resolve these situations one by one.

Before we throw the solution, we need to introduce a feature in Django: Signals, signals is an event notification mechanism that Django provides that allows you to listen to certain custom or pre-defined events, and invoke implementation-defined methods when these events occur.

For example, Django.db.models.signals.pre_save & Django.db.models.signals.post_save represents an event that is triggered before and after a model calls the Save method. It is functionally similar to the trigger provided by database.

For more signals about the official documentation, let's look at what the signals can do for our counters.

1. When a new message comes in, for counter +1

This should be the best thing to do, and with Django's signals, we can implement counter updates in this case with just a few lines of code:

From django.db.models.signals import Post_save, Post_deletedef incr_notifications_counter (sender, instance, created, * *kwargs):  # Only if this instance is newly created, and has_readed is the default of false to update  if not (created and not instance.has_readed):    Return  # Call the Update_unread_count method to update the counter +1  Notificationcontroller (instance.user_id). Update_unread_count (1 ) # Monitor Notification model Post_save signal post_save.connect (Incr_notifications_counter, sender=notification)

This way, whenever you create a new notification using a method such as Notification.create or. Save (), our notificationcontroller will be notified as counter +1.

Note, however, that since our counters are based on Django signals, if you have a place in your code that uses raw SQL and does not add new notifications via the Django Orm method, our counters are not notified, so it's best to standardize all new notification-building methods, Use the same API, for example.

2. When the message is deleted abnormally, if the associated message is unread, the counter-1

With the first experience, this situation is relatively simple to deal with, only need to monitor the notification post_delete signal can be, the following is an example code:

def decr_notifications_counter (sender, instance, **kwargs):  # When the deleted message has not been read out, counter-1  if not instance.has_readed :    Notificationcontroller (instance.user_id). Update_unread_count ( -1) post_delete.connect (decr_notifications_ Counter, sender=notification)

At this point, notification's Delete event can also update our counters properly.

3. When reading a new message, for counter-1

Next, when the user reads an unread message, we also need to update our unread message counter. You might say, what's so hard about this? I just need to update my counter manually in the way I read the message, okay?

such as this:

Class Notificationcontroller (object):  ...  def mark_as_readed (self, notification_id):    notification = Notification.objects.get (pk=notification_id)    # It is not necessary to repeatedly mark an already read notification    if notication.has_readed:      return    notification.has_readed = True    Notification.save ()    # Update our counters here, yes, I feel wonderful    self.update_unread_count (-1)

With some simple tests, you can feel that your counter works very well, but there is a very deadly problem with this approach, and there is no way to handle concurrent requests properly.

For example, you have an unread message object with an ID of 100, and at this time there are two requests to mark this notification as read:

# Because of two concurrent requests, it is assumed that both methods are almost simultaneously called Notificationcontroller (user_id). mark_as_readed (user_id) Notificationcontroller. mark_as_readed (100)

Obviously, these two times will successfully mark this notification as read, because in the case of concurrency, if notification.has_readed such a check does not work properly, so our counter will be wrong-12 times, but in fact we read only one request.

So, how should such a problem be solved?

Basically, there is only one way to resolve the data conflicts that occur with concurrent requests: Lock up and introduce two simpler solutions:

Querying with the Select for Update database

Select ... for update is a scenario that is specifically designed to address concurrent fetch data at the database level, and the main relational database, such as MySQL, PostgreSQL, supports this feature, and the new version of Django ORM even provides the shortcut of this function directly. For more information about it, you can search the documentation for the database you are using.

With the Select for update, our code might turn out like this:

From django.db import Transactionclass Notificationcontroller (object):  ...  def mark_as_readed (self, notification_id):    # Manually let the select for UPDATE and UPDATE statements take place in a complete transaction with    Transaction.commit_on_success ():      # Use Select_for_update to ensure that concurrent requests are processed at the same time, other requests      # waits for lock release      notification = Notification.objects.select_for_update (). Get (pk=notification_id)      # There is no need to repeat the notification if a read has been      Notication.has_ readed:        return      notification.has_readed = True      notification.save ()      # Update our counters here, yes, I feel great.      Self.update_unread_count (-1)

In addition to the ability to use "Select for Update", there is a relatively simple way to solve this problem.

Using update to implement atomic modification

In fact, the simpler way, just change our database to a single update can solve the problem of concurrency:

def mark_as_readed (self, notification_id):    affected_rows = Notification.objects.filter (pk=notification_id, Has_ Readed=false) \                      . Update (Has_readed=true)    # Affected_rows will return the number of entries modified by the UPDATE statement    Self.update_unread_count ( Affected_rows)

In this way, concurrent tag-read operations can also correctly affect our counters.

Performance?
Before we described how to implement an unread message counter that could be updated correctly, we might modify our counters directly using the UPDATE statement, just like this:

From django.db.models import Fdef update_unread_count (self, Count)  # Update our counters  with the UPDATE statement UserNotificationsCount.objects.filter (pk=self.user_id) \                 . Update (unread_count=f (' unread_count ') + count)

But in a production environment, this approach is likely to cause serious performance problems, because if our counters are updated frequently, massive updates can put a lot of pressure on the database. So in order to achieve a high performance counter, we need to save the changes and write them to the database in bulk.

With Redis's sorted set, we can do this very easily.

Use sorted set to cache counter changes

Redis is a very useful memory database in which the sorted set is one of the data types it provides: an ordered set, using it, we can cache all the counter changes very simply, and then write back to the database in bulk.

Rk_notifications_counter = ' Ss_pending_counter_changes ' def update_unread_count (self, count): "" "  modified Update_ Unread_count method "" "  Redisdb.zincrby (Rk_notifications_counter, str (self.user_id), Count) # We also need to modify the method of getting the number of unread messages for the user. Gets the buffer data in Redis that are not written back to the database. In this case, the code is omitted.

With the above code, we buffer the update of the counter in Redis, and we also need a script to write the data in the buffer back to the database.

By customizing the command of Django, we can do this very easily:

# file:management/commands/notification_update_counter.py#-*-coding:utf-8-*-from django.core.management.base Import Basecommandfrom django.db.models Import F # Fix import probfrom notification.models Import Usernotificationscountfrom notification.utils Import rk_notifications_counterfrom base_redis import Redisdbimport Logginglogger = Logging.getlogger (' stdout ') class Command (Basecommand): Help = ' Update Usernotificationscounter objects , Write changes from Redis to database ' Def handle (self, *args, **options): # First, get the buffer through the Zrange command all modified user IDs for U ser_id in Redisdb.zrange (rk_notifications_counter, 0,-1): # Here it is worth noting that, in order to ensure the atomicity of the operation, we used the REDISDB pipeline pipe = Redi      Sdb.pipeline () Pipe.zscore (Rk_notifications_counter, user_id) Pipe.zrem (Rk_notifications_counter, user_id) Count, _ = Pipe.execute () count = Int (count) if not count:continue logger.info (' Updating unread cou NT user%s:count%s '% (user_id, count)) UsernotificationscounT.objects.filter (pk=obj.pk) \. Update (unread_count=f (' unread_count ') + count) 

After that, the changes in the buffer can be batched back to the database via a command such as Python manage.py notification_update_counter. We can also configure this command to crontab to define execution.

Summarize
The article here, a simple "high-performance" unread message counter is completed. Speaking so much, in fact, the main point of knowledge is all these:

Use Django's signals to get new/deleted action updates for model
Use the database's select for update to correctly handle concurrent database operations
Use Redis's sorted set to cache counter modification operations
Hope to be of help to you. :)

