Nosql-redis Application Scenarios

Introduction of the problem

DB (Oracle, MySQL, PostgreSQL, etc.) +memcached this architectural pattern is very common in our production environment, typically we load hot data into the cache via Memcached, and the application layer first requests the data to Memcached. If there is data in the cache, return to the application layer directly, but as the volume of business data increases and the volume of traffic continues to grow, we also encounter many problems:

• How data is guaranteed to be consistent between DB and memcached.

• memcached data hit rate low or down machine, the application direct access to the DB, to form an avalanche effect, the database pressure instantaneous explosion, directly caused the database response slow, or crash off.

• Cache sync problem across room.

Redis

• In many nosql we generally use Redis to replace memecached for the following reasons:

• Redis supports more data types (strings, map, list, sets, sorted sets, etc.)

• Redis supports replication functionality.

• Redis supports data persistence, which keeps the in-memory data on disk and can be loaded again for use when restarting.

• Redis supports sharding technology to easily distribute data across multiple Redis instances for quick and easy scaling.

• Redis uses a request allocation method for memory allocation, which is more efficient in memory usage.

The most commonly used data types for Redis are the following:

• String

• Hash

• List

• Set

• Sorted Set

• Pub/sub

• Transactions

Data type Application

1. String

Common commands:get, set, INCR, Decr mget, and so on.

Application Scenarios: String is the most commonly used type of data, and normal Key/value storage can be categorized as such. It is possible to fully implement the current Memcached functionality and be more efficient. You can also enjoy Redis's timed persistence, operation logs, and replication functions.

In addition to providing operations like get, set, INCR, DECR, and so on, Redis provides the following Memcached:

• Get string length

• Append content to a string

• Set and get a section of a string

• Set and get one of the strings (bit)

• Bulk set the contents of a series of strings

2. Hash

Common commands:Hget,hset,hgetall and so on.

scenario: in memcached, we often package structured information into HashMap, which is stored as a string value after the client is serialized, such as the user's nickname, age, gender, integral, and so on, when one of these items needs to be modified. It is usually necessary to remove all values after deserialization, modify the value of an item, and then serialize the store back. This not only increases the overhead, but also does not apply to some scenarios where concurrent operations are possible (for example, two concurrent operations need to modify the integral). The hash structure of Redis allows you to modify only one item property value just as you would update a property in a database.

For example, we want to store a user information object data that contains the following information:

The user ID is the key to find, the stored value user object contains the name, age, birthday and other information, if the ordinary key/value structure to store, mainly has the following 2 kinds of storage methods:

• The disadvantage of using the user ID as a lookup key to encapsulate other information as a serialized object is to increase the cost of serialization/deserialization and to retrieve the entire object when one of the information needs to be modified, and the modification operation requires concurrency protection. Introduce complex problems such as CAs.

  {"ID": "xxxxxx", "Username,age,birthday"}

• The second method is how many members of this user information object will be saved into the number of key-value, with the user id+ the name of the corresponding property as a unique identifier to obtain the value of the corresponding property, although the cost of serialization and concurrency is omitted, but the user ID is repeated storage, if there is a large number of such data, The memory waste is still very considerable.

  {"Xxxidusername": "xxxxx", "Xxxidage": "xxxx", "Xxxidbirthday": "XXXXX"}

So the hash provided by Redis is a good solution to this problem, and the Redis hash is actually the internal stored value as a hashmap, and provides direct access to the map member's interface:

{"ID": "xxxxxx": "userinfo": "\" username\ ": \" xxxxname\ ", \" age\ ": \" xxxx\ ", \" birthday\ ": \" Xxxxxx\ ""}, i.e., Key is still the user ID, value is a map, the map key is a member of the property name, value is the property value, so that the data can be modified and accessed directly through its internal map key (Redis called internal map key field), that is, through the The key (User ID) + field (attribute tag) can manipulate the corresponding attribute data, without the need to store data repeatedly, and without the problem of serialization and concurrency modification control. A good solution to the problem.

3. List

Common commands:lpush,rpush,lpop,rpop,lrange and so on.

Application Scenarios: Redis list has a lot of applications and is one of the most important data structures of redis, such as Twitter watchlist, fan list, etc. can be implemented using Redis's list structure.

Lists are linked lists, and people who believe that they have a knowledge of data structures should be able to understand their structure. With the lists structure, we can easily achieve the latest message ranking and other functions. Another application of lists is Message Queuing,

The lists push operation can be used to present the task in lists, and then the worker thread then takes the task out of execution with the pop operation. Redis also provides an API for manipulating a section of lists, where you can directly query and delete elements from a section of lists.

4. Set

Common commands:sadd,spop,smembers,sunion and so on.

Application Scenarios: The functionality provided by Redis set externally is a list-like feature, except that set is automatically weight-saving, and set is a good choice when you need to store a list of data and you don't want duplicate data. and set provides an important interface to determine whether a member is within a set set, which is not available in list.

The concept of a sets collection is a combination of a bunch of distinct values. Using the sets data structure provided by Redis, you can store some aggregated information, such as in a microblog application, where you can have a collection of all the followers of a user and a collection of all their fans. Redis also provides for the collection of intersection, set, difference sets and other operations, can be very convenient to achieve such as common concern, common preferences, two-degree friends and other functions, to all of the above collection operations, you can also use different commands to choose whether to return the results to the client or save set into a new collection.

5. Sorted Set

Common commands:Zadd,zrange,zrem,zcard and so on.

Usage Scenarios: The usage scenario for Redis sorted set is similar to set, except that the set is not automatically ordered, and the sorted set can be ordered by the user with an additional priority (score) parameter, and is inserted in an orderly, automatic sort. When you need an ordered and non-repeating collection list, you can choose sorted set data structures, such as the public Timeline of Twitter, which can be stored as score in the publication time, which is automatically sorted by time.

Also can use sorted sets to do with the weight of the queue, such as the normal message score is 1, the important message of the score is 2, and then the worker can choose to press score reverse order to get work tasks. Let important tasks take precedence.

6. Pub/sub

Pub/sub literally is the release (Publish) and Subscription (Subscribe), in Redis, you can set a key value for message publishing and message subscription, when a key value on a message published, all subscribed to its client will receive the corresponding message. The most obvious use of this function is to use it as a real-time messaging system, such as regular live chat, group chat, and other functions.

Practical Application Scenarios

1. Display the latest project list

The following statement is often used to show the latest items, and with more data, the query will undoubtedly become slower.

SELECT * FROM foo WHERE ... ORDER BY time DESC LIMIT 10

In a web app, queries such as "list up-to-date replies" are common, which often leads to extensibility issues. This is frustrating, because the project was created in this order, but it had to be sorted in order to output it.

A similar problem can be solved with redis. For example, one of our web apps wants to list the latest 20 reviews posted by users. We have a "show all" link on the side of the latest comment, and you can get more comments when you click on it.

We assume that each comment in the database has a unique incrementing ID field.

We can use pagination to make page and comment pages, use the Redis template, and each time a new comment is published, we'll add its ID to a redis list: Lpush latest.comments <ID>

We crop the list to a specified length, so Redis only needs to save the latest 5,000 comments:

LTRIM latest.comments 0 5000

Every time we need to get the scope of the latest review project, we can get the range ID list from Redis, and then take the ID list into the DB to fetch the data directly, this ID is usually the UNIQ key or primary key in our RDBMS, so since We omit the time to sort the RDBMS and use the Consat method directly

Our system does not "refresh" the cache in the traditional way, and the information in the Redis instance is always consistent. The SQL database (or other type of database on the hard disk) is only triggered when the user needs to get "very far" data, and the home page or the first comment page will not bother the database on the hard disk.

2. Delete and filter

We can use Lrem to delete comments. If the deletion is very small, the other option is to skip the entry of the comment directly and report that the comment no longer exists.

There are times when you want to attach different filters to different lists. If the number of filters is limited, you can simply use a different Redis list for each of the different filters. After all, there are only 5,000 items per list, but Redis is able to use very little memory to handle millions of items.

3. Leaderboard Related

Another common requirement is that data from a variety of databases is not stored in memory, so the performance of the database is not as good as the ability to sort by points and update them in real-time, almost every second.

Typically, for example, the leaderboard for online games, such as a Facebook game, based on the score you usually want:

-List Top 100 high-score contestants

-List A user's current global rankings

These operations are a piece of cake for redis, and even if you have millions of users, there will be millions of new points per minute.

The pattern is this, each time we get a new score, we use this code:

ZADD leaderboard  <score>  <username>

You may replace username with a userid, depending on how you designed it.

Getting the top 100 high-score users is simple:

ZREVRANGE leaderboard 0 99

The global rankings for users are similar, and only need to:

ZRANK leaderboard <username>

4, according to the user vote and time sorting

A common variant pattern of the leaderboard is, like Reddit or hacker news, that the news is sorted according to a score similar to the following formula:

score = points / time^alpha

So the user's vote will be the corresponding to dig out the news, but the time will follow a certain index to bury the news. Here's our pattern, of course the algorithm is up to you.

The pattern is this, starting with looking at items that might be up-to-date, such as 1000 of the news on the first page are candidates, so let's just ignore the others, which is easy to implement.

Each time a new news post comes up, we add the ID to the list and use Lpush + LTRIM to ensure that only the latest 1000 items are removed.

There is a background task to get this list and continue to calculate the final score for each of the 1000 news articles. The results are populated by the Zadd command in the new Order, and the old news is cleared. The key idea here is that the sort work is done by the background task.

5. Processing Overdue Items

Another common sort of item is sorting by time. We use Unix time as a score.

The pattern is as follows:

-Each time a new item is added to our non-Redis database, we add it to the sorted collection. Then we use the time attribute, Current_time and time_to_live.

-Another background task using Zrange ... Scores queries the sorted collection and takes out the latest 10 items. If the Unix time is found to have expired, delete the entry in the database.

6. Counting

Redis is a good counter, thanks to Incrby and other similar commands.

I believe that you have tried many times to add new counters to your database to get statistics or display new information, but eventually you have to discard them because of write sensitivity.

Okay, now using redis doesn't have to worry anymore. With atomic increment (atomic increment), you can safely add a variety of counts, reset with Getset, or let them expire.

For example, this action:

INCR user:<id> EXPIRE

You can figure out the number of page views that have recently been paused for up to 60 seconds between pages, and when the count reaches like 20 o'clock, you can show some banner hints, or anything else you want to show.

7. Specific projects within a specific time period

Another is difficult for other databases, but the easy part of Redis is to count how many specific users have visited a particular resource during a particular period of time. For example, I want to know some specific registered users or IP addresses, how many of them have visited an article.

Every time I get a new Page view I just need to do this:

SADD page:day1:<page_id> <user_id>

Of course, you might want to replace day1 with Unix time, such as timing ()-(Times ()%3600*24) and so on.

Want to know the number of specific users? You only need to use:

SCARD page:day1:<page_id>

Need to test if a particular user has access to this page?

SISMEMBER page:day1:<page_id>

8, real-time analysis of the situation is happening, for data statistics and prevention of spam, etc.

We've only done a few examples, but if you look at Redis's command set and combine it, you'll get a lot of real-time analytics, effective and very labor-efficient. Using the Redis Primitives command makes it easier to implement spam filtering systems or other real-time tracking systems.

9, Pub/sub

Redis's pub/sub is very, very simple, stable and fast to run. Supports pattern matching and enables real-time subscription and cancellation of channels.

10. Queue

You should have noticed that Redis commands such as List push and list pop are easy to perform queue operations, but can do more than that: for example, Redis also has a variant command of list pop, which blocks the queue when the list is empty.

Message Queuing (Messaging) is used extensively in modern Internet applications. Message Queuing is used not only for communication between components within the system, but also for interaction between the system and other services. The use of Message Queuing can increase the scalability, flexibility, and user experience of the system. A system that is not message-queuing-based, whose speed depends on the speed of the slowest component in the system (note: short-board effect). Message Queuing allows the components in the system to be decoupled so that the system is no longer constrained by the slowest components, and the components can run asynchronously to perform their work faster.

In addition, when the server is in high concurrency, such as writing log files frequently. You can use Message Queuing to implement asynchronous processing. This enables high-performance concurrent operations.

Nosql-redis Application Scenarios