Research Report on the Use of Memory Database fastdb

Compared with traditional disk databases, a memory database fully loads data to the memory to manage data in the memory, the improvements have been made in data synchronization, data transmission, transaction processing, and parallel operations, allowing the memory database to process data much faster than the disk database, it can effectively solve the high response requirements of the system in the billing system, such as credit control and real-time accounting.

The project decided to use the memory database mechanism in the credit control module to replace the billing center mode for real-time credit concessions to trigger credit control.

The following describes FastDB:

[Open Source Code link: http://sourceforge.net/projects/fastdb]

1. Function Induction

1)Memory databases have some basic functions of traditional databases:

A:Data management. The memory database mechanism supports permanent data management, including database definition, storage, maintenance, and other functions.

B:Data operations. In-memory databases support data addition, deletion, modification, query, data integrity verification, and other basic functions.

C:Transaction Management, memory databases support scheduling, and some concurrent operations between processes and threads.

D. Data Recovery and backup mechanism. In-memory databases support online backup and automatic recovery after system crash.

2)Features of FastDB

A:Query Optimization: compared with the traditional database management system, when all the data is stored in the memory, the query speed is very fast, however, FastDB further uses indexes (hash, T-tree), reverse pointers, and query parallelization for query optimization.

B:It provides a flexible and convenient application language interface that allows you to easily write query and other statements.

2. Working Principle

FastDB is an efficient memory database system. The database files on disks are mapped to the virtual memory space occupied by each application using the database, in this way, data transmission in the data file and buffer pool is canceled. Then, the entire file data is read into the memory, and a high-performance lock tool is used to implement concurrent execution between threads in read-only mode, a single thread in change mode, and multiple threads in read-only mode. FastDB uses bitmap to allocate memory. The minimum unit is the allocation of quantum (16 bytes ). This greatly improves the data reference locality (Object Data is allocated to the continuous memory area as much as possible), minimizes the number of modification pages and reduces the transaction commit time. The transaction commit protocol is based on a shadow root page algorithm and performs atomic update operations on the database, resulting in high recovery efficiency, the T-tree structure can be used for data storage (T-tree is similar to AVL-tree, but multiple values are stored in each node of T-tree in sequence ), the query performance of a large number of similar and repetitive data is quite high. Hash Storage can also be used, which is the best way to locate records in a table with a keyword segment (query with an equal sign ).

Algorithm overview of the Shadow root page: each object in the FastDB database has a unique identifier (OID) and is used as the subscript of an array (object index). The element value indicates a handle to the object, there are two indexes (current index and shadow index) in the FastDB database. when an object is modified for the first time, it creates a copy and the object handle in the current index is changed to a copy, the Shadow index still contains a handle pointing to the original version of the object. All changes occur on the copy, and FastDB marks the Index containing the modified object handle on a special bitmap page of the object index.

When a transaction is committed, FastDB first checks the size of the object index. If it increases, it will re-allocate memory for the Shadow Copy of the object index, then, the memory occupied by the "old object" is released. After the modified bitmap pages are flushed to the disk, FastDB changes the index indicator of the current object in the database header, to switch the role of the object index. FastDB
Copy all the modified handles from the new object index to the previously shadow and now the current object index. At this time, both indexes are synchronized. (Details of memory block mobile index management need to be further considered)

3. Deployment Method

1)The application compilation environment requires that any FastDB application must contain the header file: fastdb. h;

Then you can choose to call the library file (static library (libfastdb_r.a) and shared library (libfastdb_r.so/libfastdb_r.so.2) after FastDB compilation to call );

Finally, FastDB provides many compilation option interfaces that you can set as needed, such as fault tolerance support, diskless mode, and lock detection and cleanup mechanisms.

2)In theory, the minimum size of database files loaded with memory is 1 MB, and the maximum size is the memory and disk capacity (FastDB ).
The overall optimization design is based on the real database system stored in the physical memory of the machine, but it still supports applications in databases that exceed the physical memory, but the efficiency is not very high)

4. Interface call Method

1)FastDB provides subsql interactive tools for users to back up, query, and monitor data. (Follow the prompts)

2)C ++ Interface

A:Open or create a database:

DbDatabase db (parameter );

Db. open (parameter );

Modes include dbReadOnly, dbAllAccess, dbConcurrentRead, and dbConcurrentUpdate.

B:Data Types supported by FastDB:

Type description

Boolean (true, false)

Int1 a byte signed integer (-128 .. 127)

Int2 two-byte signed integer (-32768 .. 32767)

Int4 four-byte signed integer (-2147483648 .. 2147483647)

Int8 eight-byte signed integer (-2 ** 63 .. 2 ** 63-1)

Real4 four-byte ANSI
Floating Point Type

Real8 eight-byte dual-precision floating point

Char const * Non-interrupted integer

Dbreference <t> to class t
Pointer.

Dbarray <t> element type is t
Dynamic Array.

C:Fastdb Table Interface Description

C ++ needs to define the table structure in the form of a class, and then map them to the fields of the table one by one. If the class has a method, the macro: class_descriptor (name, field_list) should be used for description, there are also methods for macro type_descriptor (field_list) (For details, refer to the help Documentation)

Finally, you only need to register the class: Register (Class Name );

D:Cursor

Two cursor modes are available: dbcursorviewonly and dbcursorforupdate.

Define example dbcursor <Class Name> instance (dbcursorforupdate );

Provides interfaces for modifying, deleting, and querying databases.

Instance.Udate ()

Instance.Remove ()/RemoveAllSelected
()/RemoveAll
()

Instance.Select ()/Select (dbquery & q);

E: dbquery Q;

Q = "id =" idvalue, "and datetime =", currtime;

F:Insert

The data insert method fastdb provides the overload

DbReference <T> insert (T const & record );

Therefore, it is easy to use.

For detailed usage, refer to the manual. Hoho...

5. Transformation Plan

You do not need to modify the existing mechanism for the moment. If you do not want to form a comprehensive bill table in the memory database (from the four memory tables in the original bill Center (tired Bill table, periodic expense schedule, and discount result table, to reduce redundancy, You need to transform the existing transaction commit mechanism. After the transaction is committed, the program can control the visibility of data changes to other processes.

6. I have two concerns.

1)FastDB is suitable for applications that take the lead in the Read mode. In the large-scale data group processing, the efficiency and accuracy of changing the transaction processing can be further investigated.

2)Although FastDB is an excellent open-source memory database, it has not or is rarely tested in a professional way. Complex memory management processes in the system may easily generate excessive memory fragments, resulting in system instability.