Python calculates non-built-in data types that consume memory

The limitations of getsizeof

Python objects with non-built-in data types cannot get the true size with sys.getsizeof (), for example:

Import Networkx as NX Import  = for in xrange (10000)]print"size of L: " , Sys.getsizeof (L) g.add_nodes_from (L) Print " size of graph: ", sys.getsizeof (G)

Results

Size of l:8763264

Analysis

Graph graph contains point sequence l, and size is not as large as l, so it is not allowed to use getsizeof to calculate the size of Python's non-built-in type object.

Obtaining memory using the Psutil module

Example 1:

ImportNetworkx as NXImportPsutilImportSYSImportOsG=NX. Graph () L= [I forIinchXrange (10000)]Print "size of L:", Sys.getsizeof (L) g.add_nodes_from (L)Print "size of graph:", sys.getsizeof (G) process=Psutil. Process (Os.getpid ()) Max_mem=process.memory_info (). RSSPrint 'Max_mem:', Max_mem

There's a problem with this, and we need to get rid of the memory of the first system.

ImportPsutilImportSYSImportosprocess=Psutil. Process (Os.getpid ()) Max_mem_1=process.memory_info (). RSSPrint 'Max_mem:', max_mem_1g=NX. Graph () L= [I forIinchXrange (10000)]g.add_nodes_from (l) max_mem_2=process.memory_info (). RSSPrint 'max_2:', Max_mem_2Print 'Max_mem:', max_mem_2-max_mem_1

Results

max_mem:23724032316375047913472

Example 2:

ImportPsutilImportOSImportSYS fromDatetimeImportdatetimeprocess=Psutil. Process (Os.getpid ()) Max_mem_1= Process.memory_info (). rss/1024.0/1024.0/1024.0Print 'Max_mem 1:', Max_mem_1all_road_nx='a'* 1024 * 1024 * 1024 * 10;Print 'Size:', sys.getsizeof (ALL_ROAD_NX)/1024.0/1024.0/1024.0Print 'len All_road_nx:', Len (all_road_nx) max_mem_2= Process.memory_info (). rss/1024.0/1024.0/1024.0Print 'Max_mem 2:', Max_mem_2Print 'Max_mem 3:', max_mem_2-max_mem_1

Results:

Max_mem 1:0.0086250305175810.0000000345107374182402:10.00865173343: 10.0000267029

Python Module Psutil Introduction

The Psutil provides an interface that can be used to obtain information, including:

• The currently running process
• System (Resource usage) information
• Cpu
• Memory
• Disk
• Internet
• User

Psutil implements a number of functions, including the following tools:

• Ps
• Top
• Df
• Kill
• Free
• Lsof
• Free
• Netstat
• Ifconfig
• Nice
• Ionice
• Iostat
• Iotop
• Uptime
• Pidof
• Tty
• W.H.O.
• Taskset
• Pmap

Psutil Obtaining System Status examples

#! Coding:utf-8ImportNetworkx as NXImportPsutilImportSYSImportOSP=Psutil. Process (Os.getpid ()) Psutil.pids ()#View all system processesp = psutil. Process (6241)#View all system processesPrint "Name:", P.name ()#Process NamePrint "Bin Path", P.exe ()#the bin path of the processPrint "Process Absolute Path", P.CWD ()#The working directory absolute path of the processPrint "process Status", P.status ()#process StatusPrint "Process creation Time", P.create_time ()#Process creation TimePrint "Process UUID Information", P.uids ()#Process UID InformationPrint "Process GID Information", P.gids ()#GID information for the processPrint "CPU time information for the process", P.cpu_times ()#CPU time information for the process, including User,system two CPU informationPrint "get process CPU affinity", P.cpu_affinity ()#get process CPU affinity, if you want to set the CPU affinity, the CPU number as a reference is goodPrint "Process Memory Utilization", P.memory_percent ()#Process Memory UtilizationPrint "Process Memory Rss,vms information", P.memory_info ()#Process Memory Rss,vms informationPrint "IO information for the process", P.io_counters ()#IO information for the process, including read/write IO numbers and parametersPrint "Process List", P.connections ()#return to List of processesPrint "number of threads opened by the process", P.num_threads ()#number of threads opened by the process

Results

Name:pythonbin Path/home/tops/bin/python2.7Process Absolute Path/home/wzh94434/Iu_iso_test process state sleeping process creation time1463322002.74process UUID Information puids (real=124674, effective=124674, saved=124674) Process GID information pgids (real=100, effective=100, saved=100) Process CPU time information pcputimes (user=14.38, system=2.38, children_user=0.0, children_system=0.0) Get process CPU affinity [0,1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16, 17, 18, 19, 20, 21, 22, 23, 24, 25, 26, 27, 28] Process Memory Utilization0.0432284208934Process Memory Rss,vms information Pmem (RSS=58548224, vms=534482944, shared=6922240, text=1536000, lib=0, data=268894208, dirty=0) Process IO information pio (Read_count=4166, write_count=1192, Read_bytes=0, write_bytes=0) Process list [Pconn (FD=3, family=2, type=1, Laddr= ('10.184.70.11', 57785), raddr= ('10.184.70.13', 8018), status='established')] The number of threads that the process is opening4

Python calculates non-built-in data types that consume memory