Memory growth mode for "Python" Memory analysis _list and array

"Python" Memory analysis _list object Memory footprint Analysis

In Python, the list is a dynamic array of pointers, and the array object provided by the array module is a dynamic arrays of values that hold the same type. Because the array directly holds the value, it uses less memory than the list. Both the list and array are dynamic arrays, so when you add new elements to them, and there is no space to save the new elements, they automatically reallocate the memory blocks and copy the original memory values into the new memory block. To reduce the number of times the memory is redistributed, the size is usually the same as the original K times each time it is reassigned. The larger the K value, the less memory is redistributed, but the more wasted space. This section uses a series of experiments to observe the memory allocation patterns of lists and arrays.

First, the growth mode of the list is calculated by getsizeof () Step1

Sys.getsizeof () can get the memory size that the list occupies. Write a program that calculates a list of length 10000, each of which holds the size of the list when it grows to this subscript:

Import sys# "Your program" calculates the size list, each of its small labels is saved to grow to this subscript when the sizes list is sized to []for I in Range (10000):    size.append (Sys.getsizeof ( size)) Import Pylab as Plpl.plot (size, lw=2, c= ' B ') pl.show ()

The position of each step in the diagram represents a memory allocation, and the length of each step indicates the size of the memory allocation.

Step2

Write your program to calculate the Resize_pos array that represents the memory size of the list each time the memory is allocated:

Import NumPy as np# "Your program" calculates Resize_pos, each of its elements is the location of each allocated memory in size # you can use NumPy diff (), where (), nonzero () to quickly complete this calculation. size = []for i in Range (10000):    size.append (sys.getsizeof (size)) size = Np.array (size) new_size = Np.diff (size) Resize _pos = Size[np.where (new_size)]# Resize_pos = Size[np.nonzero (new_size)]pl.plot (Resize_pos, lw=2) pl.show () print (" List increase rate: ") tmp = Resize_pos[25:].astype (np.float)  ? Print (Np.average (tmp[1:]/tmp[:-1)) #?

The graph shows that the curve is exponential and the 45th time the memory is allocated, the list size is close to 10000.

In order to calculate the growth rate, it is only necessary to calculate the average quotient of two values before and after the Resize_pos array.

To improve accuracy, we only calculate the mean of the second half, and note that the Astype () method is required to convert an array of integers to a floating-point array. The output of the program is as follows:

List Increase Rate:

1.12754776209

"Note" The two usages of Np.where index positioning, Np.nonzero the use of non-0-value bool judgments, Np.diff the use of difference functions.

Step3

We can fit the Resize_pos array with Scipy.optimize.curve_fit (), and the fitted function is an exponential function:

Please write a program to fit the Resize_pos array with the above formula:

From scipy.optimize import curve_fit# "Your program" to fit the Resize_pos array with the exponential function def func (x, A, B, C, D):      return a * NP.EXP (b * x + C) + Dxdata = range (len (resize_pos)) Ydata = resize_pospopt, Pcov = Curve_fit (func, XData, ydata) y = [Func (i, *popt) for I I N Xdata]pl.plot (xdata, y, lw=1, c= ' R ') Pl.plot (XData, Ydata, lw=1, c= ' B ') pl.show () print ("list increase rate by Curve_fit:" ) print (10**popt[1])

List increase rate by curve_fit:1.31158606108

"Attention" In this procedure, the exponential fitting in scipy is demonstrated.

Estimation of array memory allocation by computing time

Unfortunately, no matter how long the array object is, the result of sys.getsizeof () is unchanged. Therefore, the growth factor of the array object cannot be computed using the method in the previous section.

Because memory allocations can take a long time, it is possible to find the length of memory allocations by measuring the time of each additional element. Please write a program to measure the time of adding elements:

From array import arrayimport time# "Your program" calculates the time to add elements to an array timestimes = []times_step = []arrays = [Array (' l ') for I in range (1000)] Start = Time.time () for I in range (£):    start_step = Time.time ()    [A.append (i) for a in arrays]    end = Time.tim E ()    times_step.append (end-start_step)    times.append (End-start) pl.figure () Pl.plot (Times) pl.figure () Pl.plot (Times_step) pl.show ()

Output two pictures, the previous program to indicate the number of elements corresponding to the total time-consuming, followed by the process of adding elements each time the time-consuming, note that this map only when the number of arrays is larger than the shape of the array when the number is not enough when the line chart difference is very large.

Further, we analyze a line chart that takes time that is significantly longer than the nearby point (maximum value) to correspond to the number of elements at this time.

ts = Np.array (times_step) le = Range (np.sum (ts>0.00025)) si = Np.squeeze (np.where (ts>0.00025)) Pl.plot (le,si,lw=2 ) Pl.show ()

Memory growth mode for "Python" Memory analysis _list and array