"Numpy" Memory Analysis _ Create an array with shared memory

Source: Internet
Author: User

Introduction, Memory Exploration common functions

ID (), query object ID, usually returns the address of the object

sys.getsizeof (), returns the amount of space occupied by this object , and for arrays, the array object stores the array length, array type, and other information, in addition to the space occupied by each value in the array.

Numpy.ndarray.ctypes.data property that returns the memory location of the NumPy array

Array.array.buffer_info (), array object memory information, return element start address and number of elements

Help (Array.buffer_info)
‘‘‘
Buffer_info (self,/)
Return a tuple (address, length) giving the current memory address and the length in items of the buffer
Used to hold array ' s contents.
‘‘‘

numpy.ndarray.nbytes, gets the space size of the values stored in the array

Numpy.ndarray.items, gets the amount of space that each value in the array occupies

Some of the functions are tried as follows, interesting is the iteration of the elements in the NumPy array, whose ID is the address of two loops, the reason is unknown ,

A = Np.array ([1,2,3,4,5]) a.ctypes.data# 2199487215904id (a) # 2199565580288sys.getsizeof (a) # 116a.nbytes# 20a.itemsize # 4for I in a:    print (ID (i), type (i), i) # 2199565034888 <class ' Numpy.int32 ' > # 2199565034912 <class ' numpy.i Nt32 ' > 2199565034888 <class ' numpy.int32 ' > 3# 2199565034912 <class ' numpy.int32 ' > 4# 2199565034888 < Class ' Numpy.int32 ' > 5
I. Implementing dynamic arrays using Array.array and Numpy.frombuffer

Using the Array.array array object's memory is to determine the continuous characteristics (This is why the list is not why, in fact, the Buffer_info method of the array can see the memory consumption, return), use NP access to this memory, and specify the parsing mode , create an array from memory,

Creating an array from memory

At this point two objects share memory, and modifying the data affects both.

Import numpy as Npfrom array import Arraya = Array (' d ', [1,2,3,4]) na = Np.frombuffer (A, dtype=np.float) print (A, NA) na[1] = 20print (A, NA)

Array (' d ', [1.0, 2.0, 3.0, 4.0]) [1. 2.3. 4.]

Array (' d ', [1.0, 20.0, 3.0, 4.0]) [1. 20.3. 4.]

Multidimensional array creation and dynamic addition of element demos

Multidimensional is a one-dimensional np.array after the creation of reshape on the line,

Import mathbuf = Array (' d ') for I in range (5):    buf.append (Math.sin (i*0.1))    buf.append (Math.Cos (i*0.1)) data = Np.frombuffer (buf, dtype=np.float). Reshape ( -1,2) print (data)
[0.          1.        ] [0.09983342  0.99500417] [0.19866933  0.98006658] [0.29552021  0.95533649] [0.38941834  0.92106099]]

Array memory allocation mechanism and considerations for dynamic array construction

Import Sysa = Array ("D") for I in range:    a.append (i)    if i = = 2:        na = Np.frombuffer (A, dtype=float)    Prin T (A.buffer_info (), sys.getsizeof (a))    if i = = 4:        print ()
(140269607002576, 1) 96 (140269607002576, 2) 96 (140269607002576, 3) 96 (140269607002576, 4) 96 (140269607286768, 5) 128 ( 140269607286768, 6) 128 (140269607286768, 7) 128 (140269607286768, 8) 128 (140269607021360, 9) 192 (140269607021360, 10) 192

You can see that a space is initialized after initialization and will continue to allocate a new space when it is not enough.

NumPy array pointers are initialized to i=2, so not updated to the latest data, you can see that the data parsing is not what we want,

Print (Na.ctypes.data, ' \ n ', NA)

140269607002576

[6.93023941e-310 1.01768729e-316 0.00000000e+000]

Second, using struct structure to realize dynamic structure array

Using the struct structure to receive the row, using ByteArray to store binary data, using Frombuffer to read data, which uses the need to pay attention to the data parsing format settings to conform to the struct receive format settings

Import structbuf = ByteArray () for I in range (5):    buf + = Struct.pack (' =hdd ', I,math.sin (i*0.1), Math.Cos (i*0.1))  # ' = ' means no data memory usage alignment (shaping and floating-point consumption memory is different) Dtype = Np.dtype ({' names ': [' id ', ' sin ', ' cos '], ' formats ': [' h ', ' d ', ' d ']},align=false  # Align Indicates if the data has been aligned with the Np.frombuffer (buf,dtype=dtype) print (data[' id '), ' \ n ', data[' sin ', ' \ n ', data[' cos '])
[(0,  0.        ,  1.        )
(1, 0.09983342, 0.99500417) (2, 0.19866933, 0.98006658)
(3, 0.29552021, 0.95533649) (4, 0.38941834, 0.92106099)]
[0 1 2 3 4] [0. 0.09983342 0.19866933 0.29552021 0.38941834] [1. 0.99500417 0.98006658 0.95533649 0.92106099]

At this point the saved array resembles a table, similar to the dataframe format of PD, the row index is 0~n, and the column index can be either 0~m or Names field , as shown in the next article.

"Numpy" Memory Analysis _ Create an array with shared memory

Contact Us

The content source of this page is from Internet, which doesn't represent Alibaba Cloud's opinion; products and services mentioned on that page don't have any relationship with Alibaba Cloud. If the content of the page makes you feel confusing, please write us an email, we will handle the problem within 5 days after receiving your email.

If you find any instances of plagiarism from the community, please send an email to: info-contact@alibabacloud.com and provide relevant evidence. A staff member will contact you within 5 working days.

A Free Trial That Lets You Build Big!

Start building with 50+ products and up to 12 months usage for Elastic Compute Service

  • Sales Support

    1 on 1 presale consultation

  • After-Sales Support

    24/7 Technical Support 6 Free Tickets per Quarter Faster Response

  • Alibaba Cloud offers highly flexible support services tailored to meet your exact needs.