From Python source shallow to dissect the memory management of Python

Python's memory Management Architecture (OBJECTS/OBMALLOC.C):
Copy CodeThe code is as follows:

_____ ______ ______ ________
[INT] [Dict] [List] ... [String] Python Core |
+3 | <-----object-specific Memory-----> | <--non-object Memory--|
_______________________________ | |
[Python ' s object Allocator] | |
+2 | ####### Object Memory ####### | <------Internal buffers------> |
______________________________________________________________ |
[Python ' s raw memory allocator (pymem_ API)] |
+1 | <-----Python Memory (under PYMEM Manager ' s control)------> | |
__________________________________________________________________
[Underlying general-purpose allocator (EX:C library malloc)]
0 | <------Virtual Memory allocated for the Python process-------> |

0. Interfaces provided by C-language library functions

1. The Pymem_* family is a simple package of malloc, ReAlloc, and free in C that provides the underlying control interface.

2. Pyobject_* family, advanced memory control interface.
3. Object type-related management interfaces

Pymem_*

PYMEM_ family: Low-level memory allocation interface (low-level allocation interfaces)

Python provides a simple encapsulation of malloc, realloc, and free in C:

Why so many times in a swoop:

• Different C implementations have different clubs for malloc (0), while Pymem_malloc (0) turns to malloc (1).
• There are potential problems with the use of malloc and free mixed implementations of the C implementation. Python provides encapsulation to avoid this problem.
• Python provides macros and functions, but macros cannot avoid this problem, so writing extensions should avoid using macros

Source:

  Include/pymem.h#define Pymem_malloc (N) ((size_t) (n) > (size_t) Py_ssize_t_max? NULL \             : malloc ((n)? (n): 1) #define PYMEM_REALLOC (P, N) ((size_t) (n) > (size_t) Py_ssize_t_max? NULL \              : ReAlloc ((P), (n)? (n): 1) #define Pymem_free free  objects/object.c/* Python's malloc wrappers (see PYMEM.H) */void *pymem_malloc (size_ T nbytes) {  return pymem_malloc (nbytes);} ...

In addition to a simple package for C, Python provides 4 macros

Pymem_new and Pymem_new

Pymem_resize and Pymem_resize

They can perceive the size of the type

#define PYMEM_NEW (type, n) \ (((size_t) (n) > py_ssize_t_max/sizeof (type))? NULL:   \    ((type *) Pymem_malloc ((n) * sizeof (type)))) #define PYMEM_RESIZE (p, type, n) \ ((P) = ((size_t) (n) &G T Py_ssize_t_max/sizeof (type))? NULL:    \    (type *) Pymem_realloc ((p), (n) * sizeof (type))) #define Pymem_del        Pymem_free#define Pymem_del        Pymem_free

Some of the functions described below are still functions and macros exist at the same time, with underscores all uppercase characters are macros, which are no longer specifically explained.
Pyobject_*

Pyobject_* family, is an advanced memory control interface (high-level object memory interfaces).

Attention

• Don't mix with the Pymem_* family!!
• Unless there is a special internal management requirement, you should insist on using pyobject_*

Source

  Include/objimpl.h#define pyobject_new (type, typeobj) \        ((type *) _pyobject_new (typeobj)) #define PYOBJECT_NEWVAR ( Type, typeobj, n) \        ((type *) _pyobject_newvar ((typeobj), (n)))  Objects/object.cpyobject *_pyobject_new ( Pytypeobject *tp) {  pyobject *op;  OP = (Pyobject *) Pyobject_malloc (_pyobject_size (TP));  if (op = = NULL)    return pyerr_nomemory ();  Return Pyobject_init (OP, TP);} Pyvarobject *_pyobject_newvar (Pytypeobject *tp, py_ssize_t nitems) {  pyvarobject *op;  Const size_t size = _pyobject_var_size (TP, nitems);  OP = (Pyvarobject *) pyobject_malloc (size);  if (op = = NULL)    return (Pyvarobject *) pyerr_nomemory ();  Return Pyobject_init_var (OP, TP, Nitems);}

They do two things:

1. Allocating Memory: Pyobject_malloc
2. Partially initialized objects: Pyobject_init and Pyobject_init_var

Initialization is nothing to look at, but this malloc is a little more complicated ...
Pyobject_{malloc, free}

This is not the same as the pymem_* of 3, but the complexity of the great!

void * Pyobject_malloc (size_t nbytes) void * Pyobject_realloc (void *p, size_t nbytes) void Pyobject_free (void *p)

Python programs frequently need to create and destroy small objects, and in order to avoid a large number of malloc and free operations, Python uses the technology of the memory pool.

• A series of arena (each managed 256KB) forms a list of memory areas
• Each arena has a number of pool (each 4KB) composition
• Each memory request release will be performed within a pool

One-time request memory block

When requesting memory size between 1~256 bytes, use a memory pool (when requesting more than 0 or 257 bytes, you will retire with the Pymem_malloc we mentioned earlier).

Each time the application is applied, the actual allocated space will be aligned according to a certain number of bytes, and the following table is 8 bytes (for example, Pyobject_malloc (20) bytes will be allocated 24 bytes).
Copy the Code code as follows:

Request in bytes size of allocated block size class IDX
----------------------------------------------------------------
1-8 8 0
9-16 16 1
17-24 24 2
25-32 32 3
33-40 40 4
... ... ...
241-248 248 30
249-256 256 31

0, 257 and up:routed to the underlying allocator.

These parameters are controlled by some macros:

#define ALIGNMENT        8/        * must be 2^n *//* Return the number of bytes in size class I, as a uint. */#define Index2siz E (i) (((UINT) (i) + 1) << alignment_shift) #define SMALL_REQUEST_THRESHOLD 256

Pool

Each memory block requested is allocated in the pool, and the size of a pool is 4k. controlled by the following macros:

#define SYSTEM_PAGE_SIZE (4 * 1024)
#define POOL_SIZE system_page_size/* Must be 2^n */

The head of each pool is defined as follows:

struct Pool_header {  union {block *_padding;      UINT Count; } ref;     /* Number of allocated blocks  */  block *freeblock;          /* pool ' s free list head     */  struct pool_header *nextpool;    /* Next pool of this size class */  struct pool_header *prevpool;    /* Previous pool    ""    */  uint Arenaindex;          /* index into Arenas of base ADR */  UINT Szidx;             /* Block Size class index    */  uint Nextoffset;          /* bytes to Virgin block     */  uint Maxnextoffset;         /* Largest valid Nextoffset   */};

Notice that there is a member Szidx that corresponds to the Size class idx of the last column in the previous list. This also indicates a problem: each pool can only allocate a fixed-size block of memory (for example, only 16 bytes of blocks are allocated, or only 24-byte blocks are allocated ...).

To be able to allocate memory blocks of various sizes in the preceding list, you must have more than one pool. The pool of the same size is assigned, and a new pool is also required. Multiple pool form a linked list in turn
Arena

Multiple pool objects are managed using something called arena.

struct Arena_object {  uptr address;  block* pool_address;  UINT Nfreepools;  UINT Ntotalpools;  struct pool_header* freepools;  struct arena_object* Nextarena;  struct arena_object* Prevarena;};

The size of the memory controlled by the Arean is controlled by the following macros:

#define Arena_size       (<<)/   * 256KB */

A series of arena form a linked list.
Reference counting and garbage collection

The life cycle of most objects in Python is controlled by reference counting, which enables dynamic management of memory.

But the reference count has a fatal problem: a circular reference!

To break the circular reference, Python introduced garbage collection techniques.

