Introduction, Memory Exploration common functions
ID (), query object ID, usually returns the address of the object
sys.getsizeof (), returns the amount of space occupied by this object , and for arrays, the array object stores the array length, array type, and other information, in addition to the space occupied by each value in the array.
Numpy.ndarray.ctypes.data property that returns the memory location of the NumPy array
Array.array.buffer_info (), array object memory information, return element start address and number of elements
Help (Array.buffer_info)
‘‘‘
Buffer_info (self,/)
Return a tuple (address, length) giving the current memory address and the length in items of the buffer
Used to hold array ' s contents.
‘‘‘
numpy.ndarray.nbytes, gets the space size of the values stored in the array
Numpy.ndarray.items, gets the amount of space that each value in the array occupies
Some of the functions are tried as follows, interesting is the iteration of the elements in the NumPy array, whose ID is the address of two loops, the reason is unknown ,
A = Np.array ([1,2,3,4,5]) a.ctypes.data# 2199487215904id (a) # 2199565580288sys.getsizeof (a) # 116a.nbytes# 20a.itemsize # 4for I in a: print (ID (i), type (i), i) # 2199565034888 <class ' Numpy.int32 ' > # 2199565034912 <class ' numpy.i Nt32 ' > 2199565034888 <class ' numpy.int32 ' > 3# 2199565034912 <class ' numpy.int32 ' > 4# 2199565034888 < Class ' Numpy.int32 ' > 5
I. Implementing dynamic arrays using Array.array and Numpy.frombuffer
Using the Array.array array object's memory is to determine the continuous characteristics (This is why the list is not why, in fact, the Buffer_info method of the array can see the memory consumption, return), use NP access to this memory, and specify the parsing mode , create an array from memory,
Creating an array from memory
At this point two objects share memory, and modifying the data affects both.
Import numpy as Npfrom array import Arraya = Array (' d ', [1,2,3,4]) na = Np.frombuffer (A, dtype=np.float) print (A, NA) na[1] = 20print (A, NA)
Array (' d ', [1.0, 2.0, 3.0, 4.0]) [1. 2.3. 4.]
Array (' d ', [1.0, 20.0, 3.0, 4.0]) [1. 20.3. 4.]
Multidimensional array creation and dynamic addition of element demos
Multidimensional is a one-dimensional np.array after the creation of reshape on the line,
Import mathbuf = Array (' d ') for I in range (5): buf.append (Math.sin (i*0.1)) buf.append (Math.Cos (i*0.1)) data = Np.frombuffer (buf, dtype=np.float). Reshape ( -1,2) print (data)
[0. 1. ] [0.09983342 0.99500417] [0.19866933 0.98006658] [0.29552021 0.95533649] [0.38941834 0.92106099]]
Array memory allocation mechanism and considerations for dynamic array construction
Import Sysa = Array ("D") for I in range: a.append (i) if i = = 2: na = Np.frombuffer (A, dtype=float) Prin T (A.buffer_info (), sys.getsizeof (a)) if i = = 4: print ()
(140269607002576, 1) 96 (140269607002576, 2) 96 (140269607002576, 3) 96 (140269607002576, 4) 96 (140269607286768, 5) 128 ( 140269607286768, 6) 128 (140269607286768, 7) 128 (140269607286768, 8) 128 (140269607021360, 9) 192 (140269607021360, 10) 192
You can see that a space is initialized after initialization and will continue to allocate a new space when it is not enough.
NumPy array pointers are initialized to i=2, so not updated to the latest data, you can see that the data parsing is not what we want,
Print (Na.ctypes.data, ' \ n ', NA)
140269607002576
[6.93023941e-310 1.01768729e-316 0.00000000e+000]
Second, using struct structure to realize dynamic structure array
Using the struct structure to receive the row, using ByteArray to store binary data, using Frombuffer to read data, which uses the need to pay attention to the data parsing format settings to conform to the struct receive format settings
Import structbuf = ByteArray () for I in range (5): buf + = Struct.pack (' =hdd ', I,math.sin (i*0.1), Math.Cos (i*0.1)) # ' = ' means no data memory usage alignment (shaping and floating-point consumption memory is different) Dtype = Np.dtype ({' names ': [' id ', ' sin ', ' cos '], ' formats ': [' h ', ' d ', ' d ']},align=false # Align Indicates if the data has been aligned with the Np.frombuffer (buf,dtype=dtype) print (data[' id '), ' \ n ', data[' sin ', ' \ n ', data[' cos '])
[(0, 0. , 1. )
(1, 0.09983342, 0.99500417) (2, 0.19866933, 0.98006658)
(3, 0.29552021, 0.95533649) (4, 0.38941834, 0.92106099)]
[0 1 2 3 4] [0. 0.09983342 0.19866933 0.29552021 0.38941834] [1. 0.99500417 0.98006658 0.95533649 0.92106099]
At this point the saved array resembles a table, similar to the dataframe format of PD, the row index is 0~n, and the column index can be either 0~m or Names field , as shown in the next article.
"Numpy" Memory Analysis _ Create an array with shared memory