Performance Comparison of nosql databases: MongoDB vs redis vs Tokyo tyrant

 

 

* MongoDB vs redis vs Tokyo tyrantWe are going to perform a simple test on reading and writing MongoDB, redis, and Tokyo tyrant. To perform a fair test, we need to understand the implementation mechanism behind them. Below are some comparisons: Comparison of storage implementation:* Memory file image (memory-file mapping) redis, MongoDB * file + cache Tokyo tyrant * Memory: redis, Tokyo tyrant Key/value index format:* B + tree: MongoDB, Tokyo tyrant * hash table: redis, Tokyo tyrant * Fixed Length: Tokyo tyrant from the above comparison, we can see that redis and MongoDB are based on system memory image files, when data hits the memory, the read/write operation performance should be very strong. Of course, if the data is scattered, it cannot hit the memory, the overhead of Memory Page switching will be terrible. The difference between MongoDB and redis is that data is stored in multiple files, A new data space file is created whenever the previous one is full. Because MongoDB is the main object to be compared, and it uses B + tree for storage, TT also uses B + tree engine for comparison. So what should we test? We can naturally know that, although the memory image file read and write operations will be fast (to what extent), when the memory is full, what should we do? File size limit:32bit: MongoDB <= 2g tt no limits if u./configure -- enable-off64bit: MongoDB and TT are unrestricted. Note: redis is always limited by the memory size. To perform a relatively fair test: first, use the VM to limit the memory usage, because MongoDB and Redi actually read and write data in the memory (using the memorymap file ), therefore, when the database size exceeds the memory size, the performance is particularly important. Therefore, a virtual machine is used to set a small memory size to quickly observe the performance when the database size exceeds the memory size. The VM memory is set to 256 MB. The actual memory usage is about 9.10 mb. the CPU is 2 cores and the unbuntu server is. Test records:Key: 512 random string value: About 5 k random string data size per record: About 5 K plan to insert 100000 data records: 5.5 K * 1000 = 5.5 m * 100 = 550 m data volume about 550 m. Note: The key starts to be tested with a random string of 1 K. However, when the test MongoDB reports key too large to index, the size of the key is reduced to 512 bytes. When no data exists: MongoDB size: 64 M: (db.0, db.1 ,..) data File 16 M: (database. NS) Name Space index file. TC size: 133 K btree. tcb256 fixed. tcf517k hash. tchredis size: virtualmemfile: 41 m redis-3546.vmDB: 0m note: the initial size of the redis file is basically equal to the size of your set memory and Memory Page, you can adjust it yourself. Redis saves data by means of the timed disk storage policy, which can be set by itself. Generally, the redis database must <= memory. To make the redis database larger than the memory, you must enable the vm_enabled option in the configuration (it seems useless. When the data inserted exceeds the memory, will be killed by the unbuntu background protection process. If the maximum memory usage is set, the new value cannot be inserted after the number of existing data records reaches the memory limit ). Key/value function: redis: read/write key/value, value can have various structures, but value is not indexed. MongoDB: Organized by collection. If the key is not specified, it will be generated by the system as objectid (the "_ id" field is specified). The value is structured and the fields in the value can be indexed. Tokyotyrant: read/write key/value. The table data engine supports structured value and field indexes. Other data engines do not support this function. B + tree can use key indexes. Benchmark Testing Machine:The virtual machine runs on 2 CPU 2.26g Intel Core 2 Duo, the memory is 2 GB Virtual Machine: CPU 2 core memory 256 m Operating System: unbuntu server 9.10 32bit using software version: * MongoDB: mongodb-linux-i686-2010-02-26 * tokyotyrant: tt1.1.40; tc1.4.42 * redis: 2010-03-01 (git src) Startup: redis-server. /redis. conf (the maximum memory size is 210 MB: maxmemory 210000000, Vm-enable = Yes, Vm-max-memory 20000000, Vm-pages 1342177 ). /ttserver-port 1974/data/DB/TT. TCB bin/mongod-port 1974 -- dbpath/data/DB/Mongo MongoDBAdd 0.1 million pieces of data in the test as described above: the memory occupied 48564 of the virtual memory and 3432 of the physical memory at the beginning. After 2000 pieces of data are inserted, the virtual memory reached 143 MB, the physical memory is 33 MB, and the memory grows rapidly. At last, the virtual memory is stable at 1048 MB, while the physical memory is at 160m-211m. The Minimum CPU usage is 6%, and the maximum CPU usage is 30%. The average CPU usage is between 8% and 10%. From the test, when dB space is allocated each time, all the insert operations are frozen. The worst insert operation takes more than one minute (a space allocation file occurs at this time, it takes about 17-18 seconds to insert 2000 data records. At last, MongoDB's total data file size reaches 0.1 million MB, and then tests MongoDB to read records (in the form of non-hit: The Key is randomly generated, so most of the records are not stored in the database: the virtual memory is stable at 1048 m, and the physical memory is occupied at 90 m-94 m. CPU: The Minimum CPU usage is 8%, and the maximum CPU usage is 45%-10%. It takes about 3-4 seconds to read 2000 records. It takes 6 seconds to read the first record. RedisIn the same test, add 0.1 million pieces of data: memory. I forgot to read it at the beginning. Generally, the starting virtual memory occupies 112 MB, and the physical memory is 82 MB, at the time of 40 thousand records, the VM occupies 196 MB, the physical memory occupies 163 MB, the VM occupies 237 MB, and the physical memory occupies 204 MB. CPU: The Minimum CPU usage is 3%. The maximum CPU usage is 15%, which is usually between 7% and 11%. When redis writes data to the disk, it slows down (2000 records take 21 seconds). It usually takes about 18-19 seconds to store 2000 records. However, when maxmemory is not set, the server is suspended after more than 60 thousand pieces of data are written. When the maximum memory usage (200 MB) is set, the memory limit is reached and cannot be written (48136 data records have been written. The size of the redis file when writing 48136 data (including VM files): 277 m, of which 41 m is Vm and 236 m is database. Test redis to read 0.1 million records (in the form of non-hit: The Key is mostly not stored in the database) memory: 237 MB of virtual memory, and 204 mcpu of physical memory: reading 26% records from 43% to 2000 takes about 3-4 seconds. Tokyo tyrantAdd 0.1 million pieces of data in the test as described above: use the default configuration parameter to run tt B + Tree Memory: VM: 76928 physical memory: 1232 at the initial time, with a small increase in memory during the insertion process, when 40 thousand records are inserted, the virtual memory is only 99540, the physical memory is 23 MB, the final virtual memory is 117 MB, and the physical memory is 37 MB. The CPU usage is always stable at 2%. before inserting 50 thousand records, it takes about 2000 to 20 seconds to insert 80 thousand records on average. It takes 20-22 seconds to insert 2000 records before records, in the next 20 thousand records, the average insertion time of 2000 records gradually increased and oscillated, 28 seconds, and finally to 42 seconds (is the index node of B + tree full in memory? You may need to adjust the parameter ?). The TT database has only one file size: 589 MB, and then test TT to read 0.1 million records (in the form of non-hit: Most of the keys are not in the database). The memory is stable at: vm110m; the physical memory is 36 MB. CPU: the minimum value is 2%, and the maximum value is 6%. It usually takes 7-8 seconds to read 4% records, and 6 or 9 seconds to read. Summary:Data written to MongoDB and redis is not directly written to the disk. Therefore, when the system is restarted, all data that is not stored on the disk will be lost. TT actually has a memory buffer, but it is much smaller than the former. The above tests are not complete, but they are just the beginning. For example, there is no small data test (using numbers as the key, 100 bytes value), and there is no big data test (about 20 KB ); performance in hit situations is not tested; concurrent read/write performance is not tested, it is reported that MongoDB's concurrent read/write efficiency is not particularly outstanding, and MongoDB features a powerful query language, its syntax is a bit similar to the Object-Oriented Query Language. It can almost implement a vast majority of functions similar to the single-Table query of relational databases, and implement the automatic sharding management of storage nodes and other supporting functions; and because MongoDB is distributed in multiple files, when the data volume is large, the performance of the memory is distributed in enough files; the replication test after the synchronization log is enabled .... for TT, we need to test other TT data engines and how to optimize various TT data engines? In the practical application of Mixi, TC/TT stores more than 20 million pieces of data and supports tens of thousands of concurrent connections. It is a tested project. While ensuring high concurrent read/write performance, TC has a reliable data persistence mechanism and supports hashtable similar to the relational database table structure and simple conditions, paging and sorting operations, is a great nosql database. The main drawback of TC is that when the data volume reaches hundreds of millions, the performance of Concurrent Data writing will be greatly reduced (read is not affected). nosql: If only it was that easy mentioned, they found that when 0.16 billion 2-20 KB data is inserted in TC, the write performance began to drop sharply. Redis is essentially a key-value type memory database, similar to memcached. The entire database is loaded into the memory for operations. The database data is regularly flushed to the hard disk for storage through asynchronous operations. Because it is a pure memory operation, redis has excellent performance. The biggest charm of redis is that it supports saving the data structure of the list linked list and set, and also supports various operations on the list, for example, push and POP data from both ends of the list, get the list range, sort, etc. Set supports the Union set intersection operation of various sets. In addition, the maximum value of a single value is 1 GB, unlike memcached, which can only store 1 MB of data, redis can be used to implement many useful functions. For example, it uses its list as a FIFO two-way linked list to implement a lightweight, high-performance Message Queue Service, the set can be used as a high-performance tag system. In addition, redis can also set the expire time for the key-value stored, so it can also be used as a functional enhanced version of memcached. For test procedures and detailed records, see the appendix testbench.tgz.zip.

 

Refs: * Why http://www.eb163.com/club/viewthread.php? Tid = 2470 * http://timyang.net/data/mcb-tt-redis/* http://www.javaeye.com/topic/524977* http://bjclark.me/2009/08/04/nosql-if-only-it-was-that-easy/

 

Source: http://www.cnblogs.com/riceball/archive/2010/03/05/1679041.html