Qdbm support for large files

Recently, the application's data volume is getting bigger and bigger, and the access volume is getting bigger and bigger, so MySQL databases are overwhelmed. We want to store part of the data in the key-value type file database.

 

Because of its high speed requirements, qdbm, which is said to be fast, was selected at the beginning. After testing, the interface is indeed very simple. But there are two problems,

 

The first is backup. Qdbm comes with some management tools, including exportdb/importdb, but cannot operate on running data files. At present, you can only copy one copy and then repair. It doesn't feel good.

 

The second problem is troublesome. during the test, when the data file exceeds GB, data cannot be written. The author of qdbm mentioned in this document:

As for example file systems, it is impossible to handle a file whose size is more than 2 GB. to deal with this problem, qdbm provides a directory database containing multiple database files. due to this feature, it is possible to handle a database whose total size is up to 1 TB in theory.

It seems that it can also be solved, but I have no energy to study it...

 

In the current solution, Berkeley dB is used, which is slower but fast enough. In addition, the db_dump tool of Berkeley dB supports data export during runtime well. The next problem is that I/O is a little large. You have to think about it. The data size is too large...

 

God, change all my hard disks to SSD, or just give me 10000 servers!