-Linear hash ets/dets/mnesia all use the linear hash algorithm http://en.wikipedia.org/wiki/Linear_hashing redis dict implementation is similar to the linear hash, progressive rehash, ensure that the operation is O (1 ). However, in addition to executing a bucket rehash each time, the rehash process is accelerated every MS of execution. Although the rehash process is progressive, when the key space is too large and the LRU expires at the same time, the malloc of the large array of buckets can be used for a while on the refcm card. One case I encountered: redis on the current network uses the master-slave automatic switch mode, and the automatic switch is automatically switched for some time without any reason. Check that the key space is kW +, and a large number of evict are involved during switchover. bluckets requires malloc a * 2 memory, that is, 10 m * 24*2 = m memory, and the memory remains full, with LRU replacement, you need to clear such a large part, resulting in the redis instance stopping response for several seconds, resulting in switching. From the perspective of this case and memory utilization, try to ensure that the keyspace is not too large when using redis. -ETS Erlang built-in database challenges 7000 wqps http://blog.yufeng.info/archives/876 ETS implementation is simple, just a memory dictionary. Using the read/write lock, the TPS is very high in read-only scenarios. I tested the dictionary reading and writing in my old t420 notebook at 400 w/s in a single core scenario. According to the test data, the read operation of ETS is similar to that of the global memory dictionary, which is highly efficient. Because of the global lock, the write performance is inevitably limited and the higher the concurrency, the worse the performance. We recommend that you perform table sharding for frequently written ets. -Dets implements storage with a size limit of 2 GB for a single table. You can use a cache, but the default cache is 0, that is, the disk is read/write by default. As mentioned above, dets is based on linear Hash Storage. The hash method is not very disk-friendly, not file block cache-friendly. The cache is just a row-level index without block-level indexes. In general, there is still a gap between dets and the truly complete storage engine. It is of little value to use independently, so it is basically used based on its mnesia cluster version.
Since all operations saved med by dets are disk operations, it is important to realize that a single look-up operation involves a series of disk seek and read operations. for this reason, the dets functions are much slower than the corresponding ETS functions, although dets exports a similar interface.
Dets organizes data as a linear hash list and the hash list grows gracefully as more data is inserted into the table. space management on the file is saved med by what is called a buddy system. the current implementation keeps the entire Buddy System in Ram, which implies that if the table gets heavily fragmented, quite some memory can be used up. the only way to defragment a table is to close it and then open it again with the repair option set to force.
-Mnesia is a powerful Distributed Database Based on ETS/dets. The disc mnesia table size is limited by dets, but fragmentation can be used. Frag is similar to partitioned tables. Replace dets with leveldb (1/4 startup time, 1/2 conflict, 1/3 memory usage) mnesia backend plugin framework and a leveldb-based plugin: Roland Karlsson, Malcolm matalkahttps: // www.youtube.com/watch? V = xbcgriozgbq Whatsapp: disc_copies tablespartitioned islands and fragmented tablesall operations run async_dirtyuse key hashing to collapse all OPS per keyto a single process http://stackoverflow.com/questions/23180484/why-big-companies-use-mnesia-instead-of-using-riak-or-couchdb
First of all, mnesia has no 2 gigabyte limit. it is limited on a 32bit architecture, but hardly any are present anymore for real work. and on 64bit, you are not limited to 2 gigabyte. I have seen databases on the order of several hundred gigabytes. the only problem is the initial start-up time for those.
Mnesia is built to handle:
- Very low latency K/V lookup, not necessarily linearizible.
- Proper transactions with linearizible changes (C in the CAP theorem). These are allowed to run at a much worse latency as they are expected to be relatively rare.
- On-line schema change
- Interval Val even if nodes fail in a cluster (where cluster is smallish, say 10-50 machines at most)
The design is such that you avoid a separate process since data is in the Erlang system already. You have qlc for datalog-like queries. And you have the ability to store any Erlang term.
Mnesia fares well if the above is what you need. Its limits are:
- You can't get a machine with more than 2 terabytes of memory. And loading 2 teras from scratch is going to be slow.
- Since it is a CP system and not an AP system, the loss of nodes requires manual intervention. you may not need transactions as well. you might also want to be able to seamlessly add more nodes to the system and so on. for this, Riak is a better choice.
- It uses Optimistic Locking which gives trouble if your processes tries to access the same row in a transaction.
Mnesia Implementation Analysis
