Python Django framework implements the Message notification counter tutorial, pythondjango

Python Django framework implements the Message notification counter tutorial, pythondjango

The beginning of the story:. count ()
Assume that you have a Notification Model class that stores all intra-site notifications:

Class Notification (models. model): "A Simplified Notification class with three fields:-'user _ id': user id of the message owner-'has _ readed ': indicates whether the message has been read "" user_id = models. integerField (db_index = True) has_readed = models. booleanField (default = False)

Naturally, at the beginning, you will obtain the number of unread messages of a user through such a query:

# Notification. objects. filter (user_id = 3074, has_readed = False). count ()

When your Notification table is small, there is no problem with this method. However, as the business volume grows. The message table contains hundreds of millions of data records. Many lazy users have thousands of unread messages.

At this time, you need to implement a counter to count the number of unread messages for each user, so that compared with the previous count (), we only need to execute a simple primary key query (or better) to get the number of unread messages in real time.

Better solution: Create a counter
First, let's create a new table to store the number of unread messages for each user.

Class UserNotificationsCount (models. model): "This Model stores the number of unread messages of each user" user_id = models. integerField (primary_key = True) unread_count = models. integerField (default = 0) def _ str _ (self): return '<UserNotificationsCount % s: % s>' % (self. user_id, self. unread_count)

We provide a corresponding UserNotificationsCount record for each registered user to save the number of unread messages. Each time you get the number of unread messages, you only need UserNotificationsCount. objects. get (pk = user_id). unread_count.

Next, the question arises. How do we know when to update our counters? Does Django provide any shortcuts in this regard?

Challenge: update your counters in real time

In order for our counter to work normally, we must update it in real time, including:

• When a new unread message comes in, it is a counter + 1
• When a message is deleted abnormally, if the associated message is unread, it is counter-1.
• When a new message is read, it is counter-1.

Let's solve these problems one by one.

Before throwing a solution, we need to introduce a function in Django: Signals, which is an event notification mechanism provided by django, it allows you to listen to custom or preset events and call the implemented methods when these events occur.

For example, django. db. models. signals. pre_save & django. db. models. signals. post_save indicates the events that will be triggered before and after a Model calls the save method. It is similar to the triggers provided by the Database in terms of function.

For more information about Signals, refer to the official documentation. Let's take a look at what benefits Signals can bring to our counters.

1. When a new message comes in, it is a counter + 1

This should be the best case. Using Django's Signals, we can update the counter in this case with just a few lines of code:

From django. db. models. signals import post_save, post_deletedef incr_icationications_counter (sender, instance, created, ** kwargs): # only when this instance is newly created, in addition, if not (created and not instance. has_readed): return # Call the update_unread_count method to update the counter + 1 icationicationcontroller (instance. user_id ). update_unread_count (1) # Listen to the post_save signal of the Notification Model post_save.connect (incr_icationications_counter, sender = Notification)

In this way, every time you create a new Notification using methods such as Notification. create or. save (), our icationicationcontroller will receive a Notification, which is counter + 1.

However, please note that our counter is based on Django's signals. If you use the original SQL in your code and do not use the Django ORM method to add new notifications, our counters will not be notified. Therefore, it is best to standardize all new notification creation methods, such as using the same API.

2. When a message is deleted abnormally, if the associated message is unread, it is counter-1.

With the first experience, it is easy to handle this situation. You only need to monitor the post_delete signal of Notification. The following is an example code:

Def decr_icationications_counter (sender, instance, ** kwargs): # When the deleted message is not read, the counter-1 if not instance. has_readed: icationicationcontroller (instance. user_id ). update_unread_count (-1) post_delete.connect (decr_icationications_counter, sender = Notification)

At this point, the deletion event of Notification can also update our counter.

3. When reading a new message, the counter is-1.

Next, when you read an unread message, we also need to update our unread message counter. You might say, what's the problem? I only need to update my counter manually in my message reading method?

For example:

Class icationicationcontroller (object ):...... def mark_as_readed (self, icationication_id): notification = Notification. objects. get (pk = icationication_id) # There is no need to mark a read notification if notication again. has_readed: return notification. has_readed = True notification. save () # update our counter here. Well, I feel good about self. update_unread_count (-1)

Through some simple tests, you may think that your counter works very well, but such an implementation method has a very fatal problem, this method cannot process concurrent requests normally.

For example, if you have an unread message object with id 100, you must mark the notification as read when two requests are sent at the same time:

# For two concurrent requests, assume that these two methods are almost simultaneously called icationicationcontroller (user_id). mark_as_readed (100) icationicationcontroller (user_id). mark_as_readed (100)

Obviously, both methods successfully mark this notification as read, because in the case of concurrency, if notification. the check such as has_readed cannot work normally, so our counter will be mistakenly-1 twice, but in fact we read only one request.

So how should we solve this problem?

Basically, there is only one way to solve the data conflicts generated by concurrent requests: Lock and introduce two simple solutions:

Query by using select for update database

Select... for update is a scenario in which data can be modified after data is retrieved concurrently at the database level. Mainstream relational databases such as mysql and postgresql support this function, the new version of Django ORM even directly provides the feature 'shortcut. For more information about the database, you can search for the database introduction documents you are using.

After select for update is used, our code may become like this:

From django. db import transactionclass icationicationcontroller (object ):...... def mark_as_readed (self, icationication_id): # manually let the select for update and update statements occur in a complete transaction with transaction. commit_on_success (): # Use select_for_update to ensure that only one concurrent request is being processed. Other requests # Wait for the lock to release notification = Notification. objects. select_for_update (). get (pk = icationication_id) # There is no need to mark a read notification if notication again. has_readed: return notification. has_readed = True notification. save () # update our counter here. Well, I feel good about self. update_unread_count (-1)

In addition to the ''select for Update' feature, there is also a simple solution to this problem.

Atomic modification using update

In fact, a simpler way is to change our database to a single update to solve the concurrency problem:

Def mark_as_readed (self, icationication_id): affected_rows = Notification. objects. filter (pk = icationication_id, has_readed = False )\. update (has_readed = True) # affected_rows returns the number of modified entries in the update statement self. update_unread_count (affected_rows)

In this way, the mark of concurrent reads can also correctly affect our counters.

High performance?
We previously introduced how to implement an unread message counter that can be correctly updated. We may directly use the UPDATE statement to modify our counter, as shown below:

From django. db. models import Fdef update_unread_count (self, count) # Use the Update statement to Update our counter UserNotificationsCount. objects. filter (pk = self. user_id )\. update (unread_count = F ('unread _ count') + count)

However, in the production environment, such a processing method may cause serious performance problems, because if our counters are frequently updated, massive Update will put a lot of pressure on the database. Therefore, to implement a high-performance counter, we need to save the changes and write them to the database in batches.

Using redis sorted set, we can do this very easily.

Use sorted set to cache counter changes

Redis is a very useful memory database. The sorted set is a data type provided by it: an ordered set. With it, we can easily cache all counter changes, then write the data back to the database in batches.

RK_NOTIFICATIONS_COUNTER = 'ss _ pending_counter_changes 'def update_unread_count (self, count): "update_unread_count method modified" redisdb. zincrby (rk_icationications_counter, str (self. user_id), count) # At the same time, we also need to modify the method to get the number of unread messages of the user, so that it gets the buffer data in redis that is not written back # to the database. The code is omitted here.

Through the above Code, we buffer the update of the counter in redis. We also need a script to regularly write the data in the buffer to the database.

With custom django command, we can easily do this:

# File: management/commands/icationication_update_counter.py #-*-coding: UTF-8-*-from django. core. management. base import BaseCommandfrom django. db. models import F # Fix import probfrom notification. models import UserNotificationsCountfrom notification. utils import RK_NOTIFICATIONS_COUNTERfrom base_redis import redisdbimport logginglogger = logging. getLogger ('stdout') class Command (BaseCommand): help = 'Update UserNotificationsCounter objects, Write changes from redis to database' def handle (self, * args, ** options ): # First, use the zrange command to obtain all the modified user IDs in the buffer zone for user_id in redisdb. zrange (rk_icationications_counter, 0,-1): # It is worth noting that to ensure the atomicity of operations, we use the redisdb pipeline pipe = redisdb. pipeline () pipe. zscore (RK_NOTIFICATIONS_COUNTER, user_id) pipe. zrem (rk_icationications_counter, user_id) count, _ = pipe.exe cute () count = int (count) if not count: continue logger.info ('updating unread count user % s: count % s' % (user_id, count) UserNotificationsCount. objects. filter (pk = obj. pk )\. update (unread_count = F ('unread _ count') + count)

Then, you can use commands such as python manage. py icationication_update_counter to batch write the changes in the buffer to the database. We can also configure this command to crontab to define the execution.

Summary
Here, a simple "high-performance" unread message counter is complete. The main knowledge points are as follows:

Use Django's signals to obtain the update of the Model's new/delete operations.
Use select for update of the database to correctly handle concurrent database operations
Use redis sorted set to cache counter modification operations
Hope to help you. :)