The dynamic memory allocation of SQLite profiling

SQLite uses dynamic memory allocations to obtain the memory needed for various objects, such as database connections and SQL preprocessing statements, to establish the memory cache for database files, and to save query results. We've done a lot of work to make SQLite's dynamic memory allocation subsystem reliable, predictable, robust, and efficient. This article outlines the dynamic memory allocation for SQLite, which allows software developers to get the best performance when using SQLite.

1. Characteristics
The SQLite kernel and its memory allocation subsystem provide the following features:
(1) Robust handling of memory allocation failures. If a memory allocation request fails (that is, malloc () or realloc () returns null), SQLite frees the non-associative cache page and then re-allocates the request. If it fails, SQLite returns SQLITE_NOMEM to the application.
(2) No memory leaks. The application is responsible for destroying any objects that have been allocated. For example, an application must use Sqlite3_finalize () to end each preprocessed SQL statement and use Sqlite3_close to close each database connection. As long as the application mates, SQLite will never leak memory even in the event of a memory allocation failure or system error.
(3) Memory usage limit. The Sqlite3_soft_heap_limit64 () mechanism allows the application to set the memory usage limit for SQLite. SQLite reuses memory from the cache instead of allocating new memory to meet the set limits.
(4) 0 allocation options. The application can provide SQLite with a few large buffers of memory at startup, and SQLite will use these buffers as a requirement for all of its memory allocations, no longer invoking the system's malloc () and free ().
(5) The application provides a memory allocator. At startup, the application can provide SQLite with an optional memory allocator in place of the system's malloc () and free ().
(6) provides a formal guarantee against memory allocation failure or fragmentation of heap memory. SQLite can be configured to ensure that memory allocation failures or memory fragmentation are not present. This feature is critical for long-running, high-reliability embedded systems where a memory allocation error can cause the entire system to fail.
(7) Memory usage statistics. The application can count how much memory SQLite uses, and can detect whether memory usage is approaching or exceeding design limits.
(8) Call the allocator as few as possible. The system's malloc () and free () implementations are inefficient on many systems. SQLite reduces the overall processing time by minimizing the use of malloc () and free ().
(9) Open access. Pluggable SQLite extensions or applications can access the underlying memory allocator used by SQLite through the Sqlite3_malloc (), Sqlite3_realloc (), and Sqlite3_free () interfaces.

2. Testing
The test infrastructure verifies that SQLite does not erroneously use dynamically allocated memory by using a special detection memory allocator. The Detect memory allocator is activated by compiling with the sqlite_memdebug option, which is slower than the default memory allocator and is therefore not recommended for use in the product. However, when the test is activated, the following checks can be made:
(1) Boundary check. Detects that the memory allocator places Sentinel values at the end of each memory allocation to verify that SQLite does not have any routines that write out the assigned boundaries.
(2) Memory usage after release. When each chunk of memory is freed, each byte is populated with some useless bit patterns. This ensures that any traces that have been used are never left behind when the inside is released.
(3) The release of memory obtained from non-malloc. Each memory allocation from the detection memory allocator contains a sentinel value to verify that each allocation is released from the previous malloc.
(4) Uninitialized memory. Detecting the memory allocator initializes the allocation of each block of memory with some useless bit patterns to ensure that the user cannot make any assumptions about the contents of the allocated memory.
SQLite tracks how much memory is currently removed, regardless of whether the detection memory allocator is used. There are hundreds of test scripts used to test SQLite. At the end of each script, all objects are destroyed and a test is made to ensure that all memory is freed. This is the way to detect memory leaks. Note that the detection of memory leaks is done at all times in large quantities, including test builds and product build processes. Whenever a developer runs any single test script, memory leak detection is activated. Therefore, the memory leaks introduced during the development process can be detected and repaired quickly.
SQLite tests for out-of-memory (OOM) errors using a special memory allocator overlay that simulates memory failure. The overlay is inserted between the memory allocator and the SQLite kernel, which passes the memory allocation request directly to the underlying allocator and transmits the results back to the requestor. The overlay can be set to fail the nth memory allocation. In order to run the Oom test, the overlay first sets the first allocation failure and runs some test scripts to verify that the allocations are captured and processed correctly. The overlay then sets the second allocation failure and runs the test repeatedly. The failure point continues to notify one allocation at a time until the entire test process is complete and no memory allocation errors occur. This complete test process runs two times, the first time the overlay only sets the nth allocation failure, and the second time the overlay sets nth and subsequent allocation failures. Note Even if you encounter an oom overlay, the memory leak detection logic continues to work to verify that SQLite does not leak memory even if it encounters a memory allocation error. The Oom overlay can work with any underlying memory allocator, including detecting the memory allocator (at which point the Oom error does not introduce other memory usage errors).
The detection memory allocator and the memory leak detection logic work on the entire SQLite test suite. The TCL test Suite provides 99% of the statement test coverage, and the TH3 test provides 100% of the branch test coverage to ensure no memory leaks. So SQLite's dynamic memory allocator can work correctly in a variety of situations.

3. Configuration
3.1 Optional bottom-level memory allocator
The SQLite source code contains several different memory allocator modules that can be selected at compile time or as a limited extension at startup.
(1) Default memory allocator
By default, SQLite allocates memory using malloc (), realloc (), and free () routines in the C standard library. The implementation also makes a thin wrapper over these routines to provide a memsize () function that returns the size of an existing allocation. Memsize () also accurately tracks the number of bytes of memory that are not returned, and it determines how many bytes are never returned in memory when an allocation is freed. The memsize () implementation in the default memory allocator is allocating an additional 8 bytes to the head on each malloc () request and saving the allocated size to the 8-byte header.
In most applications we recommend using the default memory allocator, which is the default if there is no mandatory requirement to use the optional memory allocator.
(2) Debug memory allocator
If SQLite compiles using the Sqlite_memdebug compile-time option, a different memory allocator is used for heavy-duty packaging of System malloc (), realloc (), and free (). The heavy-duty wrapper allocates more than 100 bytes of extra space to each allocation request to place the Sentinel value at the end of the allocation. When an allocation is released, check these sentinel values to make sure that the SQLite kernel does not exceed the two ends of the buffer. When the system library comes from glibc, the heavy wrapper also uses the GNU BackTrace () function to check the stack and record the ancestor function of the malloc () call. When you run a test suite, the heavy wrapper also records the name of the current test case. These two features are useful for tracking memory leaks.
Heavy-duty Packers are only used for sqlite testing, analysis and commissioning. It has significant performance and memory overhead and is generally not used in the final product.
(3) 0 allocating memory allocator
When SQLite compiles with the SQLITE_ENABLE_MEMSYS5 option, it contains an optional memory allocator that does not use malloc (). SQLite developers call it "MEMSYS5". Even if included in the version, MEMSYS5 is disabled by default. The application must invoke the following SQLite interface at startup:
Sqlite3_config (Sqlite_config_heap, PBuf, Szbuf, mnreq);
Where pbuf points to a large contiguous block of memory, SQLite uses it to satisfy all memory allocation needs. Pbuf can also point to a static array or a section of memory obtained from other application-specific mechanisms. Szbuf is the number of bytes of pbuf memory. Mnreq is the minimum number of bytes allocated at one time. Any call to Sqlite3_malloc (n) is rounded up to mnreq when N is less than mnreq. Nmreq must be a power of 2. We will see later that the Mnreq parameter is critical for reducing the N value and minimum memory requirements in the Robson proof.
The MEMSYS5 dispenser is designed for use in embedded systems and of course there are no restrictions that may not be used on workstations. Szbuf typically between hundreds of KB and dozens of MB, depending on system needs and memory budget. The algorithm used by MEMSYS5 can be summed up as "power of 2, first hit". The size of all memory allocation requests is rounded to a power of 2, which is allocated using the first large enough free block of memory in Pbuf. Use the partner system to merge contiguous blocks of free memory. If used properly, this algorithm can provide a mathematical guarantee to avoid memory fragmentation and internal crashes (see the following description).
(4) Experimental memory splitter
As you can see from the name "MEMSYS5", there may be other optional memory allocators, indeed. The default memory allocator is called "memsys1", and the debug memory allocator is called "Memsys2". If SQLite compiles using SQLITE_ENABLE_MEMSYS3, then another 0 allocation memory allocator MEMSYS3 included in the version. It is similar to MEMSYS5 and must be called Sqlite3_config (Sqlite_config_heap,...) To activate. MEMSYS3 uses a memory buffer to satisfy all memory allocation needs. The difference between it and MEMSYS5 is the use of different memory allocation algorithms, which seem to work well in practice, but do not provide a mathematical guarantee to avoid memory fragmentation and memory crashes. MEMSYS3 is MEMSYS5 's predecessor, and SQLite developers believe that MEMSYS5 is better than MEMSYS3, and all applications that require 0 allocation of memory allocators should use MEMSYS5 instead of MEMSYS3. MEMSYS3 may be removed in future versions of SQLite.
In SQLite 3.6.1, the code for MEMSYS4 is still in the source tree and has been removed from 3.6.5. Memsys4 tries to get memory with mmap (), using Madvise () to release unused pages to the operating system so that they can be used by other processes. Now MEMSYS4 has been abandoned. MEMSYS6 uses system malloc () and free () to get the required memory, which is an aggregator. MEMSYS6 only calls malloc () to get large amounts of memory, and then divides these large-capacity memory into chunks of small memory to meet the needs of the SQLite kernel. MEMSYS6 is used in malloc () for inefficient systems. The idea behind MEMSYS6 is to reduce the number of calls to the system malloc (). MEMSYS6 is only available when you compile SQLite with SQLITE_ENABLE_MEMSYS6, and it needs to be called at startup:
Sqlite3_config (Sqlite_config_chunkalloc);
MEMSYS6 is added in SQLite 3.6.1, it is very experimental, and has been removed from the beginning of 3.6.5. Other experimental memory allocators may be added in a future version of SQLite, the name may be memsys7, MEMSYS8, and so on.
(5) application-defined memory allocator
The application can provide its own memory allocator to SQLite at startup. In order for SQLite to use the new memory allocator, the application calls:
Sqlite3_config (Sqlite_config_malloc, Pmem);
Where Pmem points to a Sqlite3_mem_methods object that defines an application-specific memory allocator interface. The Sqlite3_mem_methods object is simply a structure that contains a series of function pointers, pointing to a variety of custom memory allocation functions. In multithreaded applications, Sqlite3_mem_methods is serialized only if and when Sqlite_config_memstatus is activated. If Sqlite_config_memstatus is disabled, the methods in Sqlite3_mem_methods need to focus on serialization.
(6) Memory allocator overlay
Applications can insert overlays between the SQLite kernel and the underlying memory allocator. For example, the Oom test logic can simulate the failure of memory allocation by using the overlay layer. The overlay uses the following interfaces to create:
Sqlite3_config (Sqlite_config_getmalloc, Poldmem);
The interface obtains a pointer to an existing memory allocator and saves it for actual memory allocation. Then by using a similar sqlite3_config (Sqlite_config_malloc,...) Place the overlay into the existing memory allocator.
(7) Empty operation Memory Splitter
If SQLite compiles using the SQLITE_ZERO_MALLOC option, the default memory allocator is ignored and replaced by a pile memory allocator, which does not allocate any memory. Any call to the pile allocator will return a report with no memory available. This empty operation memory allocator is only used as a placeholder so that SQLite can link some custom memory allocators that do not use malloc (), free (), or realloc (). Applications compiled with the SQLITE_ZERO_MALLOC option need to use Sqlite3_config (), combined with sqlite_config_malloc or sqlite_config_, before using SQLite Heap to specify a new optional memory allocator.

     3.2 Temporary memory
    sqlite occasionally requires a chunk of "temporary" memory to perform some temporary calculations. For example, when rebalancing a b-tree, you need to use temporary memory. These temporary memory is usually around 10KB for a single, short-lived function call. In earlier versions of SQLite, temporary memory was taken from the processor stack, which worked well on workstations with large capacity stacks. However, in an embedded system with only a small capacity processor stack (typically 4 K or 8K), requesting a large buffer from the stack can cause problems. Therefore, SQLite is modified to get temporary memory from the heap. The temporary memory allocator is set up as follows:
    sqlite3_config (Sqlite_config_scratch, PBuf, SZ, N);
     where pbuf points to a contiguous memory, SQLite uses it for temporary memory allocations. This contiguous memory must be at least sz*n bytes in size, and the "sz" parameter is the maximum number of bytes per temporary memory allocation, and n is the maximum number of concurrent temporary memory allocations. The "sz" parameter value should be about 6 times times the maximum database page, and N should be about twice times the number of threads running in the system. No thread requests temporary memory allocations more than two times at a time, so the value of n ensures sufficient temporary memory allocations. If the temporary memory setting does not provide enough memory, SQLite will revert back to using a normal memory allocator for temporary memory allocations. The default setting is Sz=0, n=0, which indicates that a normal memory allocator is used as the default behavior.

3.3-page cache memory
In many applications, SQLite's database page cache subsystem uses dynamic memory allocations more frequently, even more frequently than 10 times times that of other subsystems. SQLite can be configured to allocate page cache memory from a separate, slot-sized pool of memory. This has two advantages:
(1) Because all allocations are the same size, the memory allocator may work faster, and the allocator does not need to merge adjacent idle slots or find the right size slots. All unallocated memory slots are stored in a linked list, and when allocated, the first memory slot in the list is removed, and the memory slots are added directly to the head of the linked list when released.
(2) because there is only one allocation size, the N parameter in the Robson certificate is 1, and the entire memory space (N) required by the allocator is exactly equal to the maximum memory used (M). There is no additional memory fragmentation overhead, so memory requirements are reduced. This is especially important for page cache memory, because most of SQLite's memory requirements come from the page cache.
The page cache memory allocator is disabled by default, and the application can open it at startup:
Sqlite3_config (Sqlite_config_pagecache, PBuf, SZ, N);
Where pbuf points to a contiguous memory, SQLite uses it to allocate page cache memory. This contiguous memory must be at least sz*n bytes in size, and the "sz" parameter is the number of bytes allocated per page cache memory, and n is the maximum number of times that can be allocated. If SQLite needs more than the SZ bytes of page cache memory, or if more than n blocks of allocations are required, then fallback to use the usual memory allocator.

3.4 Backup Memory splitter
The SQLite database connection makes many small, short-term memory allocations. This is most often the case when compiling SQL statements with SQLITE3_PREPARE_V2 (). These small memory allocations are used to store such things as table name and column names, parse tree nodes, individual query results, and B-tree cursor objects. This results in frequent calls to malloc () and free () and the use of most of the CPU time slices allocated to SQLite.
SQLite 3.6.1 introduces a fallback memory allocator to help reduce memory allocation load. In the fallback allocator, each database connection is pre-allocated a chunk of memory (typically 50 to 100KB) and then splits the memory block into a fixed-size memory slot of 50 to 200 bytes, thus becoming a fallback memory pool. Small memory allocations for database connections use one slot in the fallback pool, while large memory allocations continue to use the common memory allocator. The common memory allocator is also used when the backup pool is exhausted, but in many cases the use of small memory allocations by the fallback pool is sufficient.
Because the fallback memory allocation is always the same size, the allocation and deallocation algorithms are very fast, eliminating the need to merge adjacent idle slots or find the right size slots. Each database connection maintains a single linked list of free slots. The first memory slot in the linked list is removed directly when it is allocated, and the memory slot is added to the head of the list directly when released. In addition, each database connection has been run under a single thread (a mutex has been placed to force this) and no additional mutex is required to serialize access to the list of fallback slots. As a result, the allocation and release of backup memory allocations is very fast, with speed testing on Linux and Max OS x workstations, showing that the overall performance of SQLite can be increased by 10% to 15% depending on the configured backup memory load.
The size of the backing memory pool has a global default value, but can be configured to a different value. As long as the following interfaces are used at startup:
Sqlite3_config (Sqlite_config_lookaside, SZ, CNT);
Where SZ is the number of bytes per fallback slot, the default is 100 bytes. CNT is the total number of backup memory slots per database connection, with a default value of 500 slots. It is clear that the backing memory size for each database connection is sz*cnt bytes, which defaults to 50KB. Note that these defaults are for SQLite 3.6.1 and may change in future releases.
A backup pool for a single database connection can be changed, using the following call:
Sqlite3_db_config (db, Sqlite_dbconfig_lookaside, PBuf, SZ, CNT);
Where Pbuf points to the backup memory pool space. If Pbuf is Null,sqlite, Sqlite3_malloc () will be used to obtain the required memory pool space. SZ and CNT are the total size and number of slots for each backup slot. If PBUF is not NULL, it must point to at least sz*cnt bytes of space.
The fallback memory pool configuration on a database connection can only be changed if the memory is not yet allocated. Therefore, the configured settings should be made immediately after the database connection is created with Sqlite3_open () (or other creation function) and before any SQL statements are executed.

3.5 Memory Status
By default, SQLite counts its memory usage. These statistics can determine how much memory an application really needs, or it can be used in high-reliability systems to determine if memory usage is about to close or exceed the Robson proof limit, causing the memory allocation subsystem to crash. Many memory statistics are global, so mutexes must be serialized. The statistics default is open, but you can disable it so that you can avoid using mutexes in memory allocation or deallocation, thus saving overhead. Call the following interface:
Sqlite3_config (Sqlite_config_memstatus, onoff);
When the "OnOff" parameter is true, the memory statistics trace is activated and the memory statistics trace is disabled when false. If the statistics are active, you can use the following routines to access them:
Sqlite3_status (verbsqlite3_status (verb, &current, &highwater, Resetflag);
The "verb" parameter determines which statistics are accessed, and there are a variety of verb verb definitions, which can be referred to http://sqlite.org/c3ref/c_status_malloc_count.html#sqlitestatusmemoryused. The currently selected value is written to the current integer parameter, and the highest historical value is written to the Highwater parameter. If Resetflag is true, the High-water flag is reset to the currently selected value when the call returns.
For the statistics of a single database connection, there are different interfaces:
Sqlite3_db_status (db, verb, &current, &highwater, Resetflag);
This interface function is similar, just one more database connection parameter, and returns the memory statistics of this connection, not the entire SQLite library. The Sqlite3_db_status () interface currently recognizes only one verb sqlite_dbstatus_lookaside_used and may recognize more verbs in the future.
The statistics for each connection do not use global variables, so no mutex is required to access and update. The memory statistics for each connection will continue even if the sqlite_config_memstatus is closed.

3.6 Setting Memory usage limits
The Sqlite3_soft_heap_limit64 () interface is used to set the total amount of memory heap that the universal memory allocator can allocate. If the allocated memory exceeds this weak heap limit, SQLite frees the cached memory before continuing to allocate the request. The weak heap throttling mechanism works only if the memory statistics are active, and if Sqlite_enable_memory_management is used at compile time, it can get the best performance.
The weak heap limit is called weak (soft) because if SQLite does not release enough secondary memory to meet this limit, it will continue to allocate additional memory and exceed this limit. This is based on the theory that using extra memory is better than a complete failure. In SQLite 3.6.1, the weak heap limit can only be applied to the universal memory allocator, and it cannot interact with the temporary memory allocator, the page cache memory allocator, or the fallback memory allocator. This problem will be resolved in a future release.

4. Mathematical guarantee to prevent memory allocation failure
For the dynamic memory allocation problem, memory allocation failure problem, J.m.robson conducted a systematic study, the results are published in the following papers:
J. M. Robson. "Bounds for Some Functions concerning Dynamic Storage Allocation". Journal of the Association for Computing Machinery, Volume, number 8, July 1974, pages 491-499.
We use the following notation (similar to the Robson notation, but not exactly the same):
N: The memory allocation system in order to ensure that there is no allocation failure and the amount of raw memory required.
M: The maximum amount of memory that the application has ever taken out at any point in time.
N: The ratio of the maximum memory allocation to the minimum allocation. We assume that each memory allocation size is an integer multiple of the minimum allocation size.
The Robson proves the following results:
n = m* (1 + (log2 N)/2)-n + 1
In layman's terms, Robson proves that in order to prevent memory allocations from failing, any memory allocator must use a memory pool of size n, which exceeds the maximum memory m ever used multiplied by a multiple of N. That is, unless all memory allocations are the same size, the system needs to access larger memory than was ever used. In addition, we see that the amount of remaining memory needed increases rapidly with the increase in the ratio n, so we should keep all memory allocations as large as possible.
The Robson proves to be structural in nature. He proposes an algorithm to compute an allocation and release operation series due to memory fragmentation (free memory greater than 1 bytes but small n bytes) will result in allocation failure. Robson also proves that if the available memory is n or more bytes, a "power of 2, first hit" Memory allocator will never have memory allocation failures.
The M and N values are properties of the application. If the M and N values are known at the time of the application creation, or at least the upper values are understood, and if the application uses the MEMSYS5 memory allocator, providing n bytes of free memory space through Sqlite_config_heap, Robson proves that memory allocation requests do not fail in the application. That is, the application developer can select an N value to ensure that calls to any interface on SQLite do not return SQLITE_NOMEM, and that the memory pool is not fragmented so that new memory allocation requests are not met. This is critical in applications where a software failure can lead to corruption or critical data loss.

4.1 calculation and control parameters m and N
Robson proof can be used in SQLite's following memory allocator:
* Universal Memory Splitter (MEMSYS5)
* Temporary memory Allocator
* Page Cache memory Allocator
* Backup Memory Splitter
For a memory allocator other than MEMSYS5, all memory allocations are of the same size and therefore n=1,n=m. In other words, the memory pool does not need to be larger than the maximum amount of memory used at any time.
SQLite guarantees that no thread uses more than two temporary memory slots at the same time, so if the application allocates a temporary memory slot twice as many threads, and each slot is large enough, the temporary memory allocator does not overflow. The upper limit for temporary memory allocations is 6 times times the maximum page size, so it is easy to ensure that the temporary memory allocator does not have an assignment failure.
In SQLite 3.6.1, the use of page cache memory is more difficult to control, and the only way to control page cache memory is to use cache_size pragma directives. In later versions of SQLite, page cache memory control is made easier.
A secure application typically modifies the default fallback memory pool configuration so that when Sqlite3_open () allocates the initial fallback memory buffer, the memory allocation is not large enough because the n parameter is too large. In order for the N value to be controlled, the maximum memory allocation is preferably below 2 or 4KB. Therefore, the reasonable default value setting for the fallback memory allocator is preferably one of the following values:
Sqlite3_config (Sqlite_config_lookaside, 32, 32); /* 1K */
Sqlite3_config (Sqlite_config_lookaside, 64, 32); /* 2K */
Sqlite3_config (Sqlite_config_lookaside, 32, 64); /* 2K */
Sqlite3_config (Sqlite_config_lookaside, 64, 64); /* 4K */
Another way to do this is to disable the fallback memory allocator at the beginning:
Qlite3_config (sqlite_config_lookaside, 0, 0);
Then let the application maintain a separate large backup memory buffer pool that is assigned to them when the database connection is created. Typically, an application has only a single database connection so that the backing memory pool can consist of a single large buffer.
Sqlite3_db_config (db, Sqlite_dbconfig_lookaside, Astatic, 256, 500);
The fallback memory allocator is primarily used for performance tuning, rather than preventing memory allocation failures. It is therefore reasonable to disable the fallback memory allocator completely for secure applications.
The universal memory allocator is the hardest to manage a memory pool because it supports allocations of different sizes. Since n is a multiple on M, we need to make N as small as possible and keep the minimum allocation size of MEMSYS5 as large as possible. In many applications, the fallback memory allocator is able to handle the allocation of small chunks of memory. Therefore, it is reasonable to set the minimum allocation size of MEMSYS5 to 2,4 or 8 times times the maximum allocation of the backing memory, and the minimum allocation size of 512 bytes is reasonable.
Also, in order for N to be small, we may want the maximum memory allocation size to be within the controllable range. There are several possible sources of large allocation requests on the universal memory allocator:
(1) The SQL table contains long strings or blobs rows.
(2) a complex SQL query compiled into a large preprocessing statement object.
(3) SQL parsing object used internally by SQLITE3_PREPARE_V2 ().
(4) Storage space for database connections.
(5) A temporary memory allocation that flows out into the universal memory allocator.
(6) Page cache memory allocation to the universal memory allocator.
(7) Fallback memory allocation for new database connections.
The last three cases can be controlled or excluded by properly configuring the temporary memory allocator, the page cache memory allocator, and the fallback memory allocator. The storage space required for a database connection object depends on the length range of the database file name, but rarely exceeds 2KB on 32-bit systems (more space is required on 64-bit systems due to increased pointer size). Each parsed object probably uses 1.6KB of memory. Therefore, the above scenario (3) to (7) can control the maximum memory allocation size below 2KB. If the application is designed to manage data in small chunks, then the database will not contain any long strings or blobs, so the case (1) is not a factor. If the database contains long strings or blobs, you should read them with incremental blob I/O, and their updates should also use the incremental blob I/O method instead of the other methods. Otherwise, Sqlite3_step () will have to read the entire row into contiguous memory at some point, which involves at least one large memory allocation.
The last source of memory allocation is a preprocessed statement object compiled by a complex SQL query. The work being done by SQLite developers is to reduce this part of the memory space requirement. However, large, complex SQL queries still require a few kilobytes of preprocessed statement objects. The current workaround is to separate complex SQL operations into small, simple operations that are included in individual preprocessing statement objects.
After considering all the scenarios, the application can usually keep the maximum memory allocation at 2 K or 4 K. This makes the value of LOG2 (n) 2 or 3, limiting N to 2 to 2.5 times times the M.
The maximum amount of common memory allocation required by an application depends on how many open database connections, preprocessed statement objects, and the complexity of the preprocessing statement objects. These factors are usually fixed for any given application and can be determined experimentally using Sqlite_status_memory_use. A typical application might use only 40KB of common memory, which makes the value of n approximately 100KB.

4.2 Ductile Destruction
SQLite will continue to operate normally if the memory allocation subsystem of SQLite is configured so that no memory allocation fails, but the actual memory usage exceeds the limit of the Robson proof setting. The temporary memory allocator, the page cache memory allocator, and the fallback memory allocator automatically switch to the MEMSYS5 universal memory allocator. Typically, the MEMSYS5 memory allocator will continue to perform the allocation function, even if m or n exceeds the set limit and there is no memory fragmentation. The Robson proves that in this case a memory allocation may fail, but such a failure requires a special allocation and release order, which does not have the order of allocation release in SQLite. So in practice, the Robson limit can be exceeded without a bad effect.
However, application developers should remember to monitor the state of the memory allocation subsystem and issue an alert when memory usage exceeds the limit so that the application can provide various warning messages before it fails. SQLite's memory statistics interface provides the application with all the mechanisms to complete the monitoring task.

5, the Memory interface stability
In SQLite 3.6.1, all the optional memory allocators and mechanisms are experimental and not fully stable. Starting with SQLite 3.7.0, these interfaces are stable.

The dynamic memory allocation of SQLite profiling