A Free Trial That Lets You Build Big!
Start building with 50+ products and up to 12 months usage for Elastic Compute Service
I believe many people have seen the hot American drama "Silicon Valley", which describes the future technology is that can be compressed data on the search, without the need to extract data beforehand. In reality, we are developing this technology. Based on this core technology, we have released the storage Engine product terarkdb, which has very high technical barriers. Our goal is to go beyond Facebook's rocksdb,google Leveldb,mongodb Wiredtiger to make the world's best-performing storage engine.
The TERARKDB is a storage engine with very high performance and data compression rates. This is similar to Facebook's rocksdb, but with more functionality than ROCKSDB, here are the features of TERARKDB:
TERARKDB has a wide range of applications in the Internet as well as in traditional industries. Because TERARKDB is optimized for read operations, it is more suitable for reading less and writing large volumes of read-in scenarios.
The TERARKDB approach is quite flexible and can be used as a standalone library to accommodate customer-specific scenarios. Download packages and Docker are available to facilitate user downloads. Currently supports linux,windows and Mac OS OS.
TERARKDB, as a storage engine, has its own native interface, while providing a compatible LevelDB interface that can be adapted to all systems and applications that use LevelDB, such as SSDB for most Redis interfaces. In addition, the widely used ROCKSDB interface is a superset of the LevelDB interface, so most systems and applications that use ROCKSDB can easily be adapted to TERARKDB.
Terark official provides terarkdb to MongoDB adaptation, to MySQL and other distributed database system adaptation also during the tense development process, the stable version of the Mongoterark product has been scheduled to be released in the near future.TERARKDB Performance Test Report
This section is from Terark official website to view the original contentDirectory
|Cpu||Intel (R) Xeon (r) CPU e5-2630 v3 @ 2.40GHz (2 x 8 physical cores)|
|Memory||Up to GB of DDR4 RAM|
|Ssds||Intel? SSD 520 Series (480GB, 2.5in SATA 6gb/s, 25nm, MLC)|
Amazon Movie Data (million reviews) with an average length of approximately 1K per piece of dataRaw data format
Meta data (column name)
product/productId: B00006HAXWreview/userId: A1RSDE90N6RSZFreview/profileName: Joseph M. Kotowreview/helpfulness: 9/9review/score: 5.0review/time: 1042502400review/summary: Pittsburgh - Home of the OLDIESreview/text: I have all of the doo wop DVD‘s and this one is as good or better than the1st ones. Remember once these performers are gone, we‘ll never get to see them again.Rhino did an excellent job and if you like or love doo wop and Rock n Roll you‘ll LOVEthis DVD !!
1.4.Benchmark Source Code
moviesThe total size of the dataset is about
9GB , and the number of records is approximately
Benchmark source code See GitHub Warehouse1.5.Compression Ratio
All read operations are random queries on a single record. All write operations are also randomly inserted or updated on a single record.
In this case our memory is large enough to load all the data into memory while the TERARKDB does not require a proprietary cache, but other databases require a proprietary cache (primarily used to cache the extracted data for block compression), and we set the private cache setting to 3GB for these databases.
At the same time this test we do not limit the operating system memory usage (total memory 64GB), the amount of data is much smaller than memory, the operating system can cache all the data.
We can see that TERARKDB is better than other databases in this case:
When the amount of data is not fully loaded into memory, we need to store the data on a physical disk (we use SSDs as storage media here).
In this case, the advantages of TERARKDB are more obvious:
Because TERARKDB is much higher than the data in other databases, this image uses logarithmic coordinates to make it easier to see the order of magnitude (see the vertical axis)2.2.Random Write
The environment for random write tests and random read tests is similar:
Similar to the environment for random read tests:
Test results on SSDs more realistically reflect the impact of disk I/O on performance:
Similarly, due to the magnitude difference, we look at the data by logarithmic coordinates:2.4 Read Latency Test
The data set in this test is still 9G movie review data, only the Read Query delay is tested, no Write operation is in the test.
Because the TERARKDB compression rate is very high, the system memory 3G can be loaded with all the data (actually compressed data only 2.1G, but the test program itself to account for about 750M of memory), so the following 3 sets of comparisons, the terarkdb are in the 3G memory under the conditions of testing. For Rocksdb and Wiredtiger, we tested them in 8g,4g and 3G memory respectively. In all tests, we used 8 threads.2.4.1. Data slightly larger than memory
X, Y%) indicates a query with a delay of less than
Xmicroseconds for the total number of query
TERARKDB used very advanced and complex technology, and also applied for 4 patents. Its core technology is fundamentally different from other database products such as B + Tree, LSM tree, and block compression technology. The benefit is that the compression ratio and performance are greatly improved, not simple time-space interchange. This article briefly introduces a few technical points, more technical details please go to terark.com to view the document.Not "space-for-time" or "time-to-space" existing technology
The existing mainstream database is also using compression technology, but they are mainly on
时间与空间的折衷 : The compression method is the use of universal compression technology
按块/页(block/page)压缩 (block size is usually 4k~32k, the compression rate known as the TOKUDB block size is 2m~4m).
When compression is enabled, it follows that
访问速度下降 this is because:
When writing, many records are packaged together to compress into blocks, increasing the size of the block, the compression algorithm can get a larger context, and
提高压缩率 Conversely, reducing the block size decreases the compression ratio.
Read, and even
读取很短的数据，也需要先把整个块解压 then read the extracted data. Thus, the larger the block size, the greater the number of records contained within the same block, the more unnecessary decompression is done to read a single piece of data, and the worse the performance will be. Conversely, the smaller the block size, the better the performance.
Once the compression is enabled, in order to alleviate the above problems, the traditional database generally need relatively large
专用缓存，用来缓存解压后的数据 , so that can be large
提高热数据的访问性能 , but also caused by
双缓存 the space occupancy problem, one is
操作系统缓存中的压缩数据 , two
专用缓存中解压后的数据 . There is also a very serious problem: after all, the
专用缓存 cache, when the cache misses, still need to extract the entire block, this is
慢Query a source of the problem; Another source of slow query is when the operating system cache misses ...
The Btree index of a traditional database also occupies a larger space, because the compression rate typically used by Btree
前缀压缩 is very low.
All these lead to the existing traditional database on
空间占用 on is a problem that can not be solved completely, only to make such a compromise.
For data compression (which can be considered as the compression of value in Key-value), TERARKDB mainly uses its own research and development of the database-specific
全局压缩 technology, compression rate is higher, and there is no concept of block compression, there is no problem of double caching. This compression technology can press Rowid/recordid Direct
读取单条数据 , if this
读取单条数据 is regarded as an decompression, then, according to
RowID 顺序 decompression, the decompression speed is generally 500MB per second (single-threaded), up to about 7gb/s;
RowID 随机 when decompressed, the decompression speed is generally 300MB per second (single thread), up to about 3gb/s.
For index compression, Terark mainly uses
Succinct technology, the compression rate is higher than the existing technology, and, in addition to compression, the
不用解压就可以高效地执行搜索 index can be supported
正则表达式搜索 (without having to iterate through matching regular expressions). This technology-based
Succinct Index also has additional support for
反向搜索 : Forward is to get RowID from key, reverse search is to get key from RowID, so that key does not need to store one copy (traditional Btree index does not have this function). This provides a technical fulcrum for TERARKDB to support multiple indexes on the same Table.
Succinct technology has been around for a long time, but since performance issues have not been widely used, Terark succinct technology has been specifically optimized at the CPU instruction level, significantly improving succinct performance.
It is the use of these new technologies, the compression rate and access speed of TERARKDB is greatly improved, and the function is very rich.TERARKDB Database Schema
The TERARKDB database contains multiple segment, which can be divided into writing segment,writable frozen segment, as well as ReadOnly segment, according to segment status. The data is written to writing segment first, and the data in this segment can be updated and retrieved directly. When the data is written to a certain size, the writing segment becomes writable frozen segment, and begins to be compressed by the background thread. When the background compression is complete, readonly segment is generated and writable frozen segment is deleted. In addition, the physical deletion of data, segment merging, and so on, are also performed in a background thread. Eventually, most of the data will be in the ReadOnly segment, resulting in very high compression rates and access performance.Automata Technology and succinct technology
With Terark at the same time in the engineering succinct technology and the famous Berkeley Amplab Laboratory, Spark was born in this laboratory. Terark has its own advantages in algorithms, data structures and engineering techniques.
There are a lot of applications of automata technology in TERARKDB, the self-motive is a state transfer diagram, which is used to express data, along the edges of the graph, to access the nodes according to certain rules, so that the required data can be extracted. Using traditional techniques to store this graph, memory consumption is large, and Terark uses succinct technology to compress this state transition diagram. The essence of succinct technology is to use bitmap to represent data structures, and memory usage is greatly reduced while maintaining fast access performance. On the other hand, because it is based on automata, it is possible to natively support regular expression retrieval.Conclusion
Welcome to download the use of Terark products. Future Terark plans to port the core engine to more distributed systems for more scenarios, such as Elastic Search,spark, mobile phones and embedded devices. Terark at this stage of the plan is to find more research and development and business cooperation, the product to the market as soon as possible. We are currently hiring and interested friends can contact us directly. You can also visit the official website for more information.
Performance reporting and technical analysis of the TERARKDB database
Start building with 50+ products and up to 12 months usage for Elastic Compute Service