SQLite profiling (9): dynamic memory allocation

This article is based on http://sqlite.org/malloc.html.
SQLite uses dynamic memory allocation to obtain the memory required for various objects (such as database connections and SQL preprocessing statements), create memory cache for database files, and save query results. We have made a lot of efforts to make the dynamic memory allocation subsystem of SQLite reliable, predictable, robust, and efficient. This article describes the dynamic memory allocation of SQLite. Software developers can obtain the best performance when using SQLite.
1. Features
The SQLite kernel and its memory allocation subsystem provide the following features:
(1) Robust processing of memory allocation failures. If a memory allocation request fails (malloc () or realloc () returns NULL), SQLite releases the unrelated cache page and re-allocates the request. If it fails, SQLite returns sqlite_nomem to the application.
(2) No Memory leakage. The application is responsible for destroying any allocated objects. For example, the application must end each pre-processing SQL statement using sqlite3_finalize () and use sqlite3_close to close each database connection. As long as the application works together, SQLite will never leak the memory even if the memory allocation fails or the system fails.
(3) memory usage restrictions. Sqlite3_soft_heap_limit64 () allows applications to set SQLite memory usage restrictions. SQLite will reuse the memory from the cache, instead of allocating new memory to meet the configured limits.
(4) Zero allocation option. The application can provide SQLite with several buffer blocks of memory at startup. SQLite will use these buffers as its need for all memory allocation and will no longer call the system's malloc () and free ().
(5) The application provides a memory distributor. At startup, the application can provide SQLite with an optional memory distributor to replace the system's malloc () and free ().
(6) provide a formal guarantee to prevent memory allocation failure or heap memory fragments. SQLite can be configured to ensure that no memory allocation failure or memory fragmentation occurs. This feature is critical to embedded systems with long-term running and high reliability. A memory allocation error on such systems may cause the entire system to fail.
(7) memory usage statistics. The application can count the memory used by SQLite and check whether the memory usage is close to or exceeds the design limit.
(8) Call the distributor as few as possible. The system's malloc () and free () implementations are inefficient in many systems. SQLite uses malloc () and free () as few as possible to reduce the entire processing time.
(9) open access. You can use the sqlite3_malloc (), sqlite3_realloc (), and sqlite3_free () interfaces to access the underlying memory distributor used by SQLite.
2. Test
The testing infrastructure verifies that SQLite does not mistakenly use dynamically allocated memory by using a special detection memory distributor. Check that the memory distributor is activated by using the sqlite_memdebug option during compilation. It is slower than the default memory distributor, so it is not recommended to be used in the product. But when activating it during testing, you can perform the following checks:
(1) border check. The detection memory distributor places the Sentinel value at the end of each memory allocation to verify that SQLite does not have any routine to write the allocation boundary.
(2) memory usage after release. When each memory is released, each byte is filled with useless bit modes. This ensures that there will be no traces of past use after the inner is released.
(3) Release of memory obtained from non-malloc. Each memory allocation from the detection memory distributor contains a sentinel value to verify that each allocation is released from the previous malloc.
(4) uninitialized memory. Check that the memory distributor initializes the allocation of each memory in some useless bit modes to ensure that the user cannot make any assumptions about the content of the allocated memory.
Whether or not the detection memory distributor is used, SQLite will track how much memory has been taken out. There are hundreds of test scripts used to test SQLite. At the end of each script, all objects are destroyed, and a test is conducted to ensure that all memory is released. This is the method for detecting memory leaks. Note that memory leakage detection is performed in large batches at any time, including testing and building the product. Every time a developer runs any single test script, the memory leakage detection will be activated. Therefore, the memory leakage introduced during the development process can be quickly detected and repaired.
SQLite uses a special memory distributor overlayer that can simulate memory failures to test out-of-memory (OOM) errors. The overwriting layer is inserted between the memory distributor and the SQLite kernel. It directly transmits the memory allocation request to the underlying distributor and transmits the result back to the requester. The overwriting layer can be set to cause the nth memory allocation failure. To run the OOM test, the overwriting layer first sets the first allocation failure and runs some test scripts to verify that the allocation is correctly captured and processed. Set the second allocation failure in the overwriting layer and run the test again. The failure point continues to notify a allocation until the entire test process is completed and no memory allocation error occurs. This complete test process runs twice. The first time the overwriting layer sets only the nth allocation failure, the second time the overwriting layer sets the nth allocation failure, and the subsequent allocation failure. Note that the memory leakage detection logic will continue to work even if the OOM overlayer is encountered to verify that SQLite will not leak the memory even if it encounters a memory allocation error. The OOM layer can work with any underlying memory distributor, including checking the memory distributor (in this case, we can verify that OOM errors do not introduce other memory usage errors ).
The detection logic of the memory distributor and memory leak is applied to the entire SQLite test suite. Among them, the Tcl test suite provides 99% statement test coverage, and the th3 test provides 100% branch test coverage to ensure no memory leakage. Therefore, the dynamic memory distributor of SQLite can work correctly in various scenarios.
3. Configuration
3.1 available underlying memory distributor
SQLite Source Code contains several different memory distributor modules, which can be selected during compilation or as a limited extension at startup.
(1) default memory distributor
By default, SQLite uses the malloc (), realloc (), and free () routines in the C standard library to allocate memory. In implementation, these routines are also packaged in a thin layer to provide a memsize () function to return an existing allocated size. Memsize () can also precisely track the number of bytes of memory not returned. It can determine the number of bytes of memory not returned when a allocation is released. By default, memsize () in the memory distributor is used to allocate an additional 8 bytes to each malloc () request as the header and save the allocated size to the 8-byte header.
In most applications, we recommend that you use the default memory distributor. If you do not have the mandatory requirements for the optional memory distributor, use the default one.
(2) debug the memory distributor
If SQLite uses the sqlite_memdebug compile-time option to compile, a memory distributor with a different, heavily packaged system malloc (), realloc (), and free () is used. The heavy wrapper allocates an additional 100 bytes of space for each allocation request to place the Sentinel value at the end of the allocation. When an allocation is released, check the Sentinel values to ensure that the SQLite kernel does not exceed the two ends of the buffer. When the system library comes from glibc, the heavy wrapper will also use the GNU backtrace () function to check the stack and record the ancestor functions called by malloc. When a test suite is running, the heavy wrapper also records the name of the current test case. These two features are very useful for tracking memory leaks.
Heavy wrapper is only used for SQLite testing, analysis, and debugging. It has significant performance and memory overhead and is generally not used in the final product.
(3) Zero-allocation memory distributor
When SQLite is compiled using the sqlite_enable_memsys5 option, it contains an optional memory distributor that does not use malloc. SQLite developers call it "memsys5 ". Memsys5 is disabled by default even if it is included in the version. The application must call the following SQLite interface at startup:
Sqlite3_config (sqlite_config_heap, pbuf, szbuf, mnreq );
Pbuf points to a large continuous memory block, which SQLite uses to meet all memory allocation needs. Pbuf can also point to a static array or a piece of memory obtained from other application-specific mechanisms. Szbuf is the number of bytes in pbuf memory. Mnreq is the minimum number of bytes allocated at a time. Any call to sqlite3_malloc (n) will be rounded up to mnreq when n is less than mnreq. Nmreq must be a power of 2. Later we will see that the mnreq parameter is crucial for reducing the N value and the Minimum Memory requirement in the Robson proof.
Memsys5 splitters are designed for embedded systems and are not allowed to be used on workstations without any restrictions. Szbuf usually ranges from several hundred kb to dozens of MB, depending on system requirements and memory budget. The algorithms used by memsys5 can be summarized as "2 power, first hit ". The size of all memory allocation requests is assigned to the power of 2. During allocation, the first idle memory block in pbuf is used. The partner system is used to merge adjacent idle memory blocks. If used properly, this algorithm can provide mathematical assurance to avoid Memory fragments and internal crashes (refer to the description below ).
(4) experimental memory distributor
We can see from the name "memsys5" that there may be other optional memory splitters, indeed. The default memory distributor is called "memsys1" and the debugging memory distributor is called "memsys2 ". If SQLite is compiled using sqlite_enable_memsys3, another zero-allocation Memory Allocator, memsys3, is included in the version. Similar to memsys5, it must be activated by calling sqlite3_config (sqlite_config_heap. Memsys3 uses a memory buffer to meet all memory allocation needs. The difference between memsys5 and memsys5 is that it uses different memory allocation algorithms, which seem to work well in practice, but cannot provide mathematical assurance to avoid Memory fragments and memory crashes. Memsys3 is the predecessor of memsys5. SQLite developers believe that memsys5 is better than memsys3. All applications that require no memory allocation should use memsys5 instead of memsys3. Memsys3 may be removed in future SQLite versions.
In SQLite 3.6.1, memsys4 code is still in the source code tree and has been removed from 3.6.5. Memsys4 tries to use MMAP () to get the memory and use madvise () to release unused pages to the operating system so that they can be used by other processes. Now memsys4 has been abandoned. Memsys6 uses the system malloc () and free () to obtain the required memory, which is a aggregator. Memsys6 only calls malloc () to obtain large-capacity memory, and then splits the large-capacity memory into multiple small-memory segments to meet the needs of the SQLite kernel. Memsys6 is used in malloc () to implement inefficient systems. The idea behind memsys6 is to reduce the number of calls to the system malloc. Memsys6 is available only when sqlite_enable_memsys6 is used to compile SQLite, and needs to be called at startup:
Sqlite3_config (sqlite_config_chunkalloc );
Memsys6 is added in SQLite 3.6.1. It is very experimental and has been removed since 3.6.5. Other experimental memory splitters may be added in future SQLite versions, such as memsys7 and memsys8.
(5) memory distributor defined by the application
The application can provide its own memory distributor to SQLite at startup. To enable SQLite to use a new memory distributor, the application needs to call:
Sqlite3_config (sqlite_config_malloc, pmem );
Pmem points to a sqlite3_mem_methods object, which defines the memory distributor interface specific to the application. The sqlite3_mem_methods object only contains the structure of a series of function pointers, pointing to a variety of custom memory allocation functions. In multi-threaded applications, sqlite3_mem_methods is serialized only when sqlite_config_memstatus is activated. If sqlite_config_memstatus is disabled, methods in sqlite3_mem_methods need to pay attention to serialization.
(6) memory distributor Cover Layer
The application can insert a overwriting layer between the SQLite kernel and the underlying memory distributor. For example, the OOM test logic can simulate memory allocation failure through overwriting. The override layer is created using the following interfaces:
Sqlite3_config (sqlite_config_getmalloc, poldmem );
This interface gets the pointer of the existing memory distributor and saves it for actual memory allocation. Then, you can use sqlite3_config (sqlite_config_malloc,...) to insert the overlay layer to the existing memory distributor.
(7) Empty operation memory distributor
If SQLite uses the sqlite_zero_malloc option for compilation, the default memory distributor will be ignored and replaced by a pile memory distributor, which will not allocate any memory. Any call to the pile distributor will return a report with no available memory. This empty operation memory distributor serves as a placeholder so that SQLite can link some custom memory distributors that do not use malloc (), free (), or realloc. Before using SQLite, applications compiled with the sqlite_zero_malloc option must use sqlite3_config () and use sqlite_config_malloc or sqlite_config_heap to specify the new available memory distributor.
3.2 temporary memory
SQLite occasionally needs a large block of "temporary" memory to execute some temporary calculations. For example, when you rebalance a B-tree, you need to use temporary memory. These temporary memories are usually about 10 KB and used for a single, short-lived function call. In earlier versions of SQLite, temporary memory was obtained from the processor stack, which can work well on workstations with large stacks. However, in an embedded system with only a small processor stack (usually 4 K or 8 K), applying for a large buffer from the stack may cause problems. Therefore, SQLite is modified to get the temporary memory from the stack. The method for setting a temporary memory distributor is as follows:
Sqlite3_config (sqlite_config_scratch, pbuf, SZ, N );
Pbuf points to a continuous memory, and SQLite uses it for temporary memory allocation. The continuous memory size must contain at least SZ * n Bytes. The "SZ" parameter is the maximum number of bytes allocated for each temporary memory, and N is the maximum number of times the temporary memory is allocated at the same time. The "SZ" parameter value should be about 6 times of the maximum database page, and N should be about 2 times of the number of running threads in the system. No thread will request more than two temporary memory allocations at a time, so the value of N can ensure sufficient temporary memory allocations. If the temporary memory settings do not provide enough memory, SQLite will return to the normal memory distributor for temporary memory allocation. The default values are SZ = 0 and n = 0, indicating that the normal memory distributor is used as the default behavior.
3.3 page cache memory
In many applications, the SQLite database page cache subsystem uses dynamic memory allocation more frequently, or even more frequently than 10 times that of other subsystems. SQLite can be configured to allocate page cache memory from an independent memory pool with fixed slots. There are two advantages:
(1) because all the allocations are of the same size, the memory splitter may work faster and there is no need to merge adjacent idle slots or find suitable slots. All unallocated memory slots are stored in a linked list. When allocated, the first memory slot in the linked list is removed. When released, the memory slot is directly added to the head of the linked list.
(2) because there is only one allocation size, Robson proves that the n parameter is 1, and the entire memory space required by the Allocator (n) is exactly equal to the maximum memory used (m ). There is no additional memory fragment overhead, thus reducing the memory demand. This is especially important for page cache memory, because most of SQLite's memory requirements come from page cache.
The page cache memory distributor is disabled by default. The application can enable it at startup:
Sqlite3_config (sqlite_config_pagecache, pbuf, SZ, N );
Pbuf points to a continuous memory, and SQLite uses it to allocate page cache memory. The continuous memory size must contain at least SZ * n Bytes. The "SZ" parameter indicates the number of bytes allocated to the cache memory on each page. N indicates the maximum number of times that memory can be allocated. If SQLite requires page cache memory larger than SZ bytes, or requires allocation of more than N blocks, it is returned to the normal memory distributor.
3.4 backup memory distributor
SQLite database connections perform many small, short-term memory allocations. This is the most common case when SQL statements are compiled using sqlite3_prepare_v2. These small memory allocations are used to store table names and column names, Parse Tree nodes, separate query results, and B-tree cursor objects. This will cause frequent calls to malloc () and free () and use most of the CPU time slices allocated to SQLite.
SQLite 3.6.1 introduces a backup memory distributor to help reduce the load of memory allocation. In the backup distributor, each database connection is pre-allocated with a large block of memory (usually 50 to 200 KB), and then divided into multiple memory slots of a fixed size of 50 to bytes, this changes to the backup memory pool. The small memory allocated for database connection uses a slot in the backup pool, while the large memory allocation continues to use the general memory distributor. When the backup pool is used up, it will also switch to the general memory distributor. However, in many cases, the backup pool is sufficient for small memory allocation.
Because the reserve memory allocation is always the same size, the allocation and release algorithms are very fast, and there is no need to merge adjacent idle slots or find suitable slots. Each database connection maintains a single-chain table with idle slots. During allocation, the first memory slot in the linked list is directly removed, and the memory slot is directly added to the head of the linked list during release. In addition, each database connection has been run under a single thread (A mutex lock has been placed to force this), and no additional mutex lock is required to serialize the access to the list of backup slots. Therefore, the allocation and release of backup memory allocation are very fast. The speed test is performed on Linux and Max OS X workstation to display the configured backup memory load, the overall performance of SQLite can be increased by 10% to 15%.
The size of the backup memory pool has a global default value, but can be set to different values. You only need to use the following interfaces at startup:
Sqlite3_config (sqlite_config_lookaside, SZ, CNT );
SZ indicates the number of bytes in each backup slot. The default value is 100 bytes. CNT is the total number of memory slots for each database connection. The default value is 500. Obviously, the backup memory size for each database connection is SZ * CNT, and the default value is 50kb. Note that these default values are for SQLite 3.6.1 and may change in future versions.
The backup pool for a single database connection can be changed. Use the following call:
Sqlite3_db_config (dB, sqlite_dbconfig_lookaside, pbuf, SZ, CNT );
Pbuf points to the backup memory pool space. If pbuf is null, SQLite uses sqlite3_malloc () to obtain the required memory pool space. SZ and CNT are the size of each backup slot and the total number of slots. If pbuf is not null, it must point to at least SZ * CNT bytes.
The backup memory pool configuration on the database connection can be changed only when the memory has not been allocated. Therefore, the configuration settings should be performed immediately before any SQL statement is executed after the database connection is created using sqlite3_open () (or other creation functions.
3.5 memory status
By default, SQLite calculates its memory usage. These statistics determine how much memory an application actually needs, or use a high-reliability system to determine whether memory usage is about to be disabled or exceeds the limits demonstrated by Robson, resulting in a memory allocation subsystem crash. Many memory statistics are global, so mutex locks must be used for serialization. Statistics are enabled by default, but can be disabled. In this way, the mutex lock can be avoided during memory allocation or release, thus saving costs. Call the following interface:
Sqlite3_config (sqlite_config_memstatus, Onoff );
When the "Onoff" parameter is true, the memory statistical trace is activated. If the parameter is false, the memory statistical trace is disabled. If the statistics are active, you can use the following routine to access them:
Sqlite3_status (verbsqlite3_status (verb, & Current, & highwater, resetflag );
The "verb" parameter determines which statistical information to access. There are various verb definitions. For details, see http://sqlite.org/c3ref/c_status_malloc_count.html#sqlitestatusmemoryused. The currently selected value is written to the current integer parameter, and the highest historical value is written to the highwater parameter. If resetflag is true, the high-water flag is reset to the selected value when the returned value is called.
For the statistics of a single database connection, there are different interfaces:
Sqlite3_db_status (dB, verb, & Current, & highwater, resetflag );
This interface function is similar, except that there is one more database connection parameter, and the memory statistics of this connection are returned, rather than the entire SQLite database. The sqlite3_db_status () interface recognizes only one sqlite_dbstatus_lookaside_used verb. More Verbs may be recognized in the future.
The statistics for each connection do not use global variables, so no mutex lock is required to access and update the data. Even if sqlite_config_memstatus is disabled, the memory statistics for each connection continue.
3.6 set the memory usage limit
The sqlite3_soft_heap_limit64 () interface is used to set the upper limit of the total memory heap size that can be allocated by a general-purpose memory distributor. If the allocated memory exceeds this weak heap limit, SQLite will release the cache memory before continuing to allocate the request. The weak heap restriction mechanism only works when memory statistics are activated. If sqlite_enable_memory_management is used during compilation, it can achieve the best performance.
Weak heap restrictions are called "soft" because SQLite will continue to allocate additional memory if it cannot release enough secondary memory to meet this limit, and exceeds this limit. This is based on the theory that it is better to use additional memory than to completely fail. In SQLite 3.6.1, the weak heap restrictions can only be applied to general-purpose memory splitters. They cannot interact with temporary memory splitters, page cache memory splitters, or backup memory splitters. This problem will be solved in future versions.
4. mathematical guarantee to prevent memory allocation failure
J. M. Robson has made a systematic research on dynamic memory allocation and memory allocation failure. The results are published in the following paper:
J. M. Robson. "bounds for some functions concerning Dynamic Storage Allocation". Journal of the Association for Computing Machinery, Volume 21, number 8, July 1974, pages 491-499.

We use the following mark (similar to the mark of Robson, but not exactly the same ):
N: The amount of original memory required by the memory allocation system to ensure that the allocation will not fail.
M: The maximum amount of memory that an application can fetch at any point in time.
N: Ratio of maximum memory allocation to minimum memory allocation. We assume that each memory allocation is an integer multiple of the minimum allocation size.
Robson proves the following results:
N = m * (1 + (log2 N)/2)-N + 1
In general, Robson proves that any memory distributor must use a memory pool of N to prevent memory allocation failures, it exceeds the previously used maximum memory M multiplied by a multiple depending on N. That is to say, unless all memory allocations are of the same size, the system needs to access a larger memory than previously used. In addition, we can see that the remaining memory will increase rapidly with the increase of the ratio N, so we should keep all memory allocation as much as possible.
Robson proved to be constructive. He proposed an algorithm to calculate an allocation and release operation series because of memory fragments (available memory larger than 1 byte but smaller than N bytes) will cause allocation failure. Robson also proves that if the available memory is N or more bytes, a "2 power, the first hit" memory distributor will never fail to allocate memory.
Values m and n are attributes of the application. If the M and N values are known when an application is created, or at least the upper limit is known, and if the application uses the memsys5 memory distributor, sqlite_config_heap provides n Bytes of available memory space, robson proves that the memory allocation request will not fail in the application. That is to say, the application developer can select an N value to ensure that sqlite_nomem is not returned for calls to any SQLite interface, and the memory pool does not have fragments so that it cannot meet new memory allocation requests. This is critical in applications where software faults can cause damage or critical data loss.
4.1 calculation and control parameters m and n
Robson proves that it can be used in the following memory distributors of SQLite:
* General memory distributor (memsys5)
* Temporary memory distributor
* Page cache memory distributor
* Backup memory distributor
For other memory distributors except memsys5, all memory allocations are of the same size, so n = 1, n = m. That is to say, the memory pool does not need to be larger than the maximum amount of memory used at any time.
SQLite ensures that no threads use more than two temporary memory slots at the same time. Therefore, if the application allocates a temporary memory slot of twice the number of threads and each slot is large enough, the temporary Memory Allocator will not overflow. The maximum size of the temporary memory allocation is six times the maximum size of the page. Therefore, it is easy to ensure that the temporary memory allocation does not fail.
In SQLite 3.6.1, the use of page cache memory is difficult to control. The only way to control page cache memory is to use the cache_size Pragma command. In later SQLite versions, page cache memory control will become easier.
Applications with security controls usually modify the default backup memory pool configuration, so that when sqlite3_open () allocates the initial backup memory buffer, the memory allocation is not large because the N parameter is too large. To make the N value controllable, the maximum memory allocation should be less than 2 or 4 kb. Therefore, it is best to set a reasonable default value for the backup memory distributor as one of the following values:
Sqlite3_config (sqlite_config_lookaside, 32, 32);/* 1 K */
Sqlite3_config (sqlite_config_lookaside, 64, 32);/* 2 K */
Sqlite3_config (sqlite_config_lookaside, 32, 64);/* 2 K */
Sqlite3_config (sqlite_config_lookaside, 64, 64);/* 4 K */
Another method is to disable the backup memory distributor at the beginning:
Qlite3_config (sqlite_config_lookaside, 0, 0 );
Then, let the application maintain a separate large backup memory buffer pool and allocate them when creating database connections. Normally, applications only have a single database connection, so that the backup memory pool can be composed of a single large buffer.
Sqlite3_db_config (dB, sqlite_dbconfig_lookaside, astatic, 256,500 );
The backup memory distributor is mainly used for performance optimization, rather than preventing memory allocation failure. Therefore, it is reasonable to completely disable the backup memory distributor for security-free applications.
The general-purpose memory distributor is the most difficult to manage the memory pool because it supports allocation of different sizes. Because N is a multiple of M, we need to make N as small as possible, and keep the minimum allocation size of memsys5 as large as possible. In many applications, the backup memory distributor can process the allocation of small memory blocks. Therefore, it is reasonable to set the minimum size of memsys5 to 2, 4, or 8 times the maximum size of the backup memory. The minimum size of 512 bytes is reasonable.
In addition, to make n small, we may want the maximum memory allocation to be within the controllable range. Large allocation requests on a general-purpose memory distributor may come from the following sources:
(1) rows in the SQL table that contain long strings or blobs.
(2) Compile complex SQL queries into large pre-processing statement objects.
(3) SQL parsing object used inside sqlite3_prepare_v2.
(4) storage space connected to the database.
(5) Temporary memory allocation that flows out to the universal memory distributor.
(6) page cache memory allocation that flows out to the general memory distributor.
(7) Reserve memory allocation for new database connections.
You can control or eliminate the last three conditions by configuring the temporary memory distributor, page cache memory distributor, and backup memory distributor. The storage space required for the database connection object depends on the length range of the database file name, however, 32-bit systems seldom exceed 2 kb (in 64-bit systems, more space is required due to the increase in Pointer size ). Each resolution object uses approximately KB of memory. Therefore, in the above case (3) to (7), the maximum memory allocation size can be controlled below 2 kb. If an application is designed to manage data in small blocks, the database does not contain any long string or blobs. In this case, (1) is not a factor. If the database contains a long string or blobs, use incremental blob.
I/O to read them, and its update should also use the incremental blob I/O method, instead of other methods. Otherwise, sqlite3_step () will have to read the entire row into the continuous memory at a certain time point, which involves at least one large memory allocation.
The last source of memory allocation is the pre-processing statement object compiled by complex SQL queries. What SQLite developers are doing is to reduce the memory space requirements. However, large and complex SQL queries still require preprocessing statement objects of several kb. The current work und is to separate complex SQL operations into multiple small and simple operations to include them in each separate pre-processing statement object.
In all cases, applications can usually keep the maximum memory allocation at 2 K or 4 K. This makes the value of log2 (n) 2 or 3, and limits n to 2 to 2.5 times of M.
The maximum size of a common memory allocation required by an application depends on the number of opened database connections, pre-processing statement objects, and the complexity of pre-processing statement objects. For any given application, these factors are usually fixed and can be determined using sqlite_status_memory_use. A typical application may only use 40 kb of general memory, which makes the value of N approximately kb.
4.2 damage
If the memory allocation subsystem of SQLite is configured with no memory allocation failure, but the actual memory usage exceeds the limit proved by Robson, SQLite continues to operate normally. Temporary memory distributor, page cache memory distributor, and backup memory distributor are automatically switched to memsys5 universal memory distributor. Generally, the memsys5 memory distributor will continue to execute the allocation function, even if M or N exceeds the configured limit, there will be no memory fragments. Robson proves that memory allocation may fail in this case, but such a failure requires a special order of allocation and release, and there is no such order of allocation and release in SQLite. Therefore, in practice, the Robson limit can be exceeded without any negative impact.
However, application developers should remember to monitor the status of the memory allocation subsystem and issue an alarm when the memory usage exceeds the limit so that the application can provide various warning information before the failure. SQLite memory statistics interface provides all the mechanisms for applications to complete monitoring tasks.
5. Memory Interface Stability
In SQLite 3.6.1, all available memory splitters and mechanisms are experimental and not completely stable. Since SQLite 3.7.0, these interfaces are stable.