In-memory database

In contrast to disk, memory data reads and writes more than a few orders of magnitude, saving data in memory compared to accessing from disk can greatly improve the performance of the application. At the same time, the memory database abandoned the traditional way of disk data management, based on all the data in memory to redesign the architecture, and in the data cache, fast algorithm, parallel operation has also made corresponding improvements, so the data processing speed than the traditional database processing speed is much faster, generally more than 10 times times. The most important feature of a memory database is its "primary copy" or "working version" of resident memory, which means that the active transaction only deals with the memory copy of the real-time memory database. Obviously, it requires a larger amount of memory, but not all the time the entire database is stored in RAM, that is, the in-memory database system or the I/O processing.

In exchange for real-time data processing at the expense of memory resources, the memory database and disk database are the relational database products that every enterprise in today's information society needs, and the disk database solves the problem of mass storage and data analysis, while the memory database solves the problems of real-time processing and high concurrency. The existence of the two is mutually reinforcing, the memory database transaction real-time processing performance is much stronger than the disk database. But in contrast, his data security has not reached the point where the disk database is a par.
The memory database takes physical memory as the first storage medium for the data, and the disk as a backup. With the development of telecom business, the requirement of real-time and flexible modification of the system is very high, and in this case, the requirement of the memory database is also more and more high. The practice of disk databases is to store data in memory for processing, which is not guaranteed for manageability and data security reliability. The memory database is an improvement on this weakness.

In fact, the memory database is not a fashionable technology, it appeared in the late 60, but because of the market demand for reasons in the late 90 began to develop. As a new generation of databases, altibase products have gone to a hybrid database, its version Altibase 4.0 has a set of disk database, the user once purchased the Altibase memory database, there is no need to purchase disk database. It puts the thermal data (frequently used, high-access, frequently-accessed data) in the in-memory database, and puts historic data in the disk database to further reduce investment for users.
For an in-memory database, part of the same database can be stored on disk, while the other part is stored in memory. The user can choose to store the data in a memory table to provide immediate data access. This data can be stored in a disk table if the access time is not urgent or if the data is too large to occupy in memory.

For example, when a mobile phone user starts to pull the phone, if the hybrid data management engine based on memory database technology is applied, the service options are retrieved through the memory table and the user identity is immediately authenticated, and the call list and billing list are archived to the disk table. Thus, the balance between speed and resource use is achieved.

In-memory database technology, a very important feature is that in-memory data can be implemented in full transaction, it is only the data in the form of arrays and other forms in memory completely different. Also, the memory database is irrelevant to the application, and it is obvious that the architecture is reasonable. The memory engine can implement query and archive functionality using exactly the same database, while the memory table and disk table also use the exact same access method. The choice of storage is completely transparent to the application developer.

For in-memory databases, data is managed in memory, not just as a database cache. Unlike other databases that cache disk data blocks into main memory, the memory engine of the database uses data structures and algorithms specifically designed for random access to memory, which avoids the problem of frequently destroying cache database performance by using sort commands. Through the in-memory database, the disk I/O is reduced, and the traditional database with disk I/O is not comparable to the processing speed.

Therefore, the application of the memory database technology can greatly improve the speed of the database, which provides a strong support for the application of database, such as telecom and finance, which need high-speed response.

The memory database has much higher performance than disk database due to the fact that most of the data is put into memory, which is very fit for the real-time requirement of telecom enterprise Operation support System.

Competition in the telecommunications industry is unfolding in all directions, and this competition will inevitably bring new value chain models and new billing methods, which are a challenge to the current telecom operations support system. For example, the billing of multiple businesses will no longer be a single charge per time or communication distance, but may be based on time, content, usage, and other parameters combined billing. To meet these challenges, telecom companies have introduced in-memory databases to improve the real-time, accuracy, and flexibility of back-office data management.

Although the memory database is not the traditional disk database concept, but the memory database is essentially a database, it also has the basic functions of the general database:

Management of permanent data, including definition, storage and maintenance of database;

Complete various data operations, such as query processing, access, integrity check;

Transaction management, including scheduling and concurrency control;

Control and safety inspection of access;

Has a reliability recovery mechanism for the database.

The memory database has its advantages over invoking memory processing by means of program development. First, the memory database is a product of the database management software, greatly shorten the development cycle; Secondly, the memory database has the open platform and the interface, the program development and the transplant is more flexible and convenient, also facilitates the maintenance and two times development; Thirdly, the data in memory can be easily queried by using the unified SQL language; Finally, the security and integrity of the data can be guaranteed in the database. These advantages are beneficial for rapid deployment and simplified maintenance.

But the memory database also has its unavoidable shortcomings, such as: not easy to recover, the data in the memory database is not always permanent, in order to ensure real-time, it is not necessarily consistent and absolutely correct, some short, some temporarily inconsistent or not absolutely correct.

Telecom enterprises have been the main users of memory database, in recent years, with the rapid development of computer hardware technology, the increase in memory capacity, the price fell and the computer into the 64-bit era of the operating system can support a larger address, for the implementation of the memory database provides the possibility. At present, the application of the memory database in the telecom industry is becoming more and more mature, there are more than 90G telecom system cases, can automatically expand the memory space, do not need to restart the database, provide ESOL custom stored procedures, support multi-threading, high development efficiency, easy to transplant programs and so on.

The following two examples describe the application of a memory database.

? loading of telecom billing data
The two-time price and real-time debit of Telecom are the two necessary functions in the accounting system.
The so-called two-time price is relative to the first price.
The price is calculated according to the national standard tariff, for example: the global access to local calls for 0.4 yuan per minute, after the completion of a price, will be based on the user's package for another calculation. Beijing Global users to answer the phone for 4 minutes as an example, once the price is completed, this statement is priced at 1.6 yuan, if the user participated in the 10 Yuan monthly answer package, then two times the price, the cost of this call is 0 yuan.
Once the price is used for the settlement between the major operators, and two times the price is for the user's personal.

Real-time debit is the number of users from 1th per month to date all the cost of the cumulative, that is, users can now pass 10086 to check until the day before the real-time charges. The accrued value can help the user to control the high charge or for the user to inquire the consumption information instantly.

Two times the price and the real-time accounting process involves user information, user plans and other user-related messages, the telecommunications support system must be loaded at the beginning of the price of this data. Slightly larger provincial operators of this data will be more than 10 million, the billing model is also due to the combination of packages, product combinations and different preferential rules become quite complex, loading this part of the data on the system is a large amount of overhead, which makes the current billing processing speed is relatively slow, And it's hard to update the data in real time. The introduction of the memory database solves this problem to some extent.

In the process of billing two times the largest amount of data is the detailed data, this part of the data is not placed in the memory database, each processing a single word file or to reach the set number of committed records directly operating the disk database, does not affect system performance. The most urgent thing is to put the user data, packages, business plans and billing package corresponding relationship data, billing model data and user accumulated data into the in-memory database, this part of the data query operation is much more than the data new and update operations frequently. In addition to this data, there are, of course, other data that the application needs to be loaded into the in-memory database.

In the use of memory database, the user through the Sales department or customer query real-time phone calls can be real-time, than at present can only provide a query to the day before the real-time charges in the business has a qualitative leap. Because the system in processing this part of the data query process is the same as before, but the system eliminates the previous memory data and disk database data synchronization link, so you can do real-time query. For the same, the system in the past after the exhaustion of the account to follow a certain period to refresh the data, there is a time lag, can not be fully real-timely.

After using the memory database, the data can be obtained directly from the real-time credit accumulation table in the memory database, and the real-time warning and shutdown is realized. Two times the price and the use of memory in the account of the database, fraud prevention, revenue security system also has considerable benefits, so as to fully guarantee the vital interests of operators.

In addition, after adopting the memory database, the overall increase of the system price, the processing speed of the account, greatly alleviate the pressure of accessing the disk database, improve the efficiency of data query, modification, deletion, but also for the integration of post-payment and prepayment to provide the possibility.

Synchronization of telecom billing data
The data in the telecom business data and billing system is always changing, which involves the synchronization of data and disk database data in the memory database (for the sake of clarity, the disk database is illustrated in Oracle DB for example). Data synchronization consists of two parts: from the memory database to the Oracle DB data synchronization and from the Oracle DB to the in-memory database.

1. Oracle db-to-memory database synchronization

This part of the data synchronization in the form of delta tables, business systems or CRM new or updated data will be generated into the Delta table of Oracle, the billing daemon first to these delta tables query data. If you can find the data in these delta tables to update the data into the memory database corresponding table, if not found, directly from the memory database directly query, so as to ensure the integrity and real-time performance of the data. Because the amount of data in the delta tables is generally small, this part of the operation does not affect the performance of the system.

2. In-Memory database to Oracle DB synchronization

Because Oracle's billing backend batches and accrued data are almost all loaded into the in-memory database, the data tables for the Oracle database will be used primarily for data backups of the in-memory database.

Users of the latest real-time phone calls and other information are stored in the memory database, real-time phone calls will be directly connected to the in-memory database query, to ensure that users get the latest cost information. The trust also queries data directly from the in-memory database, so there is no real-time requirement for this part of Oracle data. At this point, the memory database to Oracle synchronization can be generated by the application files, scheduled to synchronize the backup to the Oracle database, or the use of Oracle stored procedures in the system relative idle time period for data import.

In general, due to the rapid development of the market and technology, the telecommunications business is expanding, its operation and management is constantly optimized, some traditional support system architecture has gradually failed to meet the growing business requirements and customer needs, introduce some new technology to solve our production problems are inevitable. For example, the use of memory database to replace the previous shared memory technology, so that the original in memory of non-standard things, including interface, format and management are standardized.

The memory database is only a representative of a variety of new technologies, as long as the liberation of thought, choose properly, can completely in the case of small investment to overcome the bottleneck in the system, with minimal cost to obtain the maximum return.

General database You see more, Oracle, DB2, SQL Server, Sybase, Informix and the recent comparison of the fire MySQL, and pqllite, of course, can not forget the open source PostgreSQL. Typically, these databases can take on important business, but there is still a slight shortage in terms of high performance requirements. In the billing system, if the user information is often changed, the delay will have a relatively large impact, and can even affect the normal operation of the billing system.

I was exposed to the only memory database that CICA used in the Mobile Billing Center audit system. As the audit system needs to implement synchronization user status information and ordering information, and then the resulting single audit, if the response is slow, it will produce incorrect results. Initially there is no audit system, the standard of billing is basically the SP sent over, but the user often found that they do not actually use or have canceled the business, their bills are still charged, so the move is determined to SP's words single audit, with their own data as the standard, The means to completely cut off SP charges.

If you want to take the user status information and ordering information to be synchronized from a number of systems, and the dialogue sheet for audit, the intermediate processing time requirements are more stringent (users may check their own charges in a short time), the system response time should be as short as possible.

The general database is at a disadvantage in this regard. The letter to three rx8420 as a database host, the 31 provinces users of the information according to the number of share to three hosts, each province has at least one inbound process, for the user more than the use of multiple processes for storage. The source of data acquisition is mainly through the boss and the billing of the first-level system.

Since the data is stored in memory, the data structure stored is different from the common database, and in order to ensure the security of the data, there is a mirror of the memory data on the disk, which synchronizes the memory data to disk at a certain time, and can recover the data from the disk when the host fails. When a host fails, an alternate host is taken over by Ha. But for data operations logs and rollbacks there is no Oracle to do well, only provides a simple recovery mechanism.

In the billing system first to the SP sent a single audit, the main standard is the user status and ordering information. For example, the user has been in a shutdown state for the last 7 days, and if a new ordering information appears on the SP's list, it will be treated as a wrong order. Mobile in this way in and SP's game to get active. After the audit system on-line users of the SP complaints significantly reduced.

Link one: The similarities and differences between the memory database and the traditional database

The traditional database system is a relational database, and the purpose of developing such a database is to deal with permanent and stable data. The relational database emphasizes to maintain the integrity and consistency of the data, but it is difficult to take into account the data and its processing timing constraints, can not meet the needs of industrial production management real-time applications, because real-time transactions require the system to accurately forecast the running time of the transaction.

In the case of disk databases, the actual average execution time of transactions differs greatly from the worst-case execution time due to disk access, internal and external data transfer, buffer management, queued waits, and latency of locks, if the entire database or its primary "work" part is put into memory so that each transaction does not have I/O during execution , it provides strong support for the system to estimate and arrange the running time of the transaction accurately, and it also lays a foundation for realizing the timing limit of the transaction. This is the main reason for the memory database to appear.

The data that is processed by the in-memory database is usually "ephemeral", that is, it has a certain amount of time, and new data is generated when obsolete, and the current decision derivation becomes invalid. So, in real-time application, the memory database is used to deal with the data of business logic processing. The traditional database is designed to deal with permanent and stable data, its performance target is high system throughput and low cost, processing the real-time data of the relatively little consideration. In practical application, the traditional database is used to store the data with low real-time requirement.

In practical applications, these two databases are often used in conjunction, rather than in a memory database instead of a traditional database.

Link two: Several Memory database products

Oracle TimesTen

Oracle TimesTen is a memory-optimized relational database acquired by Oracle from TimesTen, providing instant responsiveness and very high throughput for applications in real-time enterprises and industries such as telecommunications, capital markets, and defense. Oracle TimesTen can be deployed as a cache or embedded database in the application layer, using a standard SQL interface to manipulate data stores that are completely in physical memory.

Altibase

Altibase is a software solution that provides high performance and high availability in a transaction-first environment. It provides high performance, fault tolerance, and transaction management capabilities, especially for communications, online banking, securities trading, real-time applications, and embedded systems. Altibase can maximize the potential of the database service system and enhance the processing power of the data server. The altibase supports either a client/server architecture or an embedded architecture. The client/server architecture is ideal for general applications. The embedded architecture embeds the application into the database server and is suitable for real-time systems with high-aging requirements.

eXtremeDB

The eXtremeDB real-time database is a database designed specifically for real-time and embedded system data management by McObject, with only 50K to 130K overhead and a microsecond speed. The eXtremeDB resides completely in main memory and does not use the file system (including memory disks). eXtremeDB uses a new disk fusion technology that extends memory to disk, uses disk as virtual memory, and real-time performance at a microsecond level, while data management can reach 20G at 32BIT.

In-memory database