Redis, MongoDB paging/ranking performance comparison at large offsets

The topic is not very accurate, because the database does not provide paging , ranking and other functions, providing only data access, page ranking these are our database-based practical cases only. However, whether it is Redis or MongoDB, there are usually some common ways to do paging and ranking. This article introduces some test data to show you the performance difference between Redis and MongoDB (and traditional relational databases).

Pagination

First, let's do a paging in MongoDB where the sample data is not as follows:

Db.scores.find (); {Lid:objectid ("4fe506dabb2bfa742d000001"), Score:1, Name: ' User_1 '} {Lid:objectid ("4fe506dabb2bfa742d000001"), Score:2, Name: ' User_2 '} {Lid:objectid ("4fe506dabb2bfa742d000001"), Score:3, Name: ' User_3 '} {Lid:objectid ("4fe506dabb2bfa742d000001"), Score:4, Name: ' User_4 '}

Where the lid field is used to distinguish between different latitude, mainly used in the screening, in the test collection, a total of five different lid values, each corresponding 1,200,000 data, altogether 6,000,000 data. The indexes are on lid and score. (The following query can be used to index)

Then we perform the following performance tests:

Collection = mongo::connection.new.db (' Test '). Collection (' scores ') benchmark.bmbm do |x|   x.report ("Mongo small")  do    100.times do |i|       collection.find ({: lid => lids.sample}, {:fields => {:_ Id => false, :score => true, :user => true}}). Sort ({: Score  => -1}). Limit (a). Skip (I * 20) .to_a    end  end   x.report ("Mongo medium")  do    100.times do |i|       collection.find ({: lid => lids.sample}, {:fields => {:_ Id => false, :score => true, :user => true}}). Sort ({: Score  => -1}). Limit (a). Skip (i * 1000) .to_a    end  end   x.report ("Mongo larGE ")  do    100.times do |i|       Collection.find ({: Lid => lids.sample}, {:fields => {:_id => false,  :score => true, :user => true}}). Sort ({: score => -1}). Limit ( ). Skip (i * 10000). To_a    end  endend

Above, the number of skip bars is smaller, medium-sized and very large in the case of testing. And the limit specifies that the data obtained is the same as 20. The test results in these three cases were: 0.6 seconds, 17 seconds, 173 seconds.

As we can see, for MongoDB, the size of skip seriously affects performance, should be strictly avoid the special large skip operation.

Redis = redis.new (:d River => :hiredis) benchmark.bmbm do |x|   X.report ("Redis small")  do    100.times do |i|       start = i * 20      redis.zrevrange (Lids.sample,  start, start + 20, :with_scores => true)     end   end  x.report ("Redis medium")  do    100.times do  |i|      start = i * 1000       redis.zrevrange (lids.sample, start, start + 20, :with_scores =>  true)     end  end  x.report ("Redis large")  do     100.times do |i|      start = i *  10000    &nbsP; redis.zrevrange (lids.sample, start, start + 20, :with_scores =>  True)     end  end

The value of skip here is the same as in MongoDB above, so how does Redis perform? The test results in these three cases were: 0.028 seconds, 0.025 seconds, 0.028 seconds.

A mongodb-like data structure is stored in PostgreSQL and the same test is done, with the result being nearly as close to MongoDB. The specific results are as follows:

MONGO Small0.6
MONGO Medium -
MONGO Large173
Redis Small0.028
Redis Medium0.025
Redis Large0.028
PG Small1
PG Medium122
PG Large650

Ranking

The ranking function is similar to the paging function, but the ranking is done by calculating the number of bars that are greater than a certain value.

Like what:

Sqlselect Count (*) from scores where lid = $ and score > $2//mongodb.scores.find ({lid:lid, score: {$gt: Score}}). Co UNT ()

Because the rankings and pagination implementations are similar in principle, the results are virtually the same. The test results are as follows:

MONGO Top Rank 1.155847
MONGO average 22.291007

Redis Top Rank 0.169442
Redis average 0.162205

PG Top Rank 0.714144
PG Average 21.771570

Conclusion

The above is a comparison, then what is the point of this article?

First, in MongoDB, try to avoid larger skip operations, such as paging, if you can know the need to get the data of the previous score, then you may be able to use the following method to obtain the data you want, rather than through a large skip operation.

Db.scores.find ({lid:lid, score: {$lt: Last_score}}). Sort ({score:-1}). Limit (20)

Also, if you need to perform a larger skip operation or a larger number of counts, consider using Redis's sorted sets.

Postscript

This article on the microblog has aroused some technical friends of the discussion, for the comparison of the question here to do a description.

We know that Redis is a memory database, and MongoDB is not, so there are friends who question whether the comparison here is just memory versus disk. In fact, this argument is justified, the above test data from the original author's article, and its article does not mention whether MongoDB is in memory. According to my own experimental results, when the data are all in memory, there is really no significant difference in MongoDB performance as mentioned in this article. However, as the skip becomes larger, the operating time is significantly longer, while Redis's sorted sets is relatively stable.

We also welcome more experimental comparisons of data and analysis of the principles of the discussion. Thank you all.

Redis, MongoDB paging/ranking performance comparison at large offsets