Python shared memory

Shared memory is the simplest way to communicate between processes. It allows multiple processes to access the same memory. After a process changes its data, data changes can be seen in other processes.

Shared memory is the fastest way to communicate between processes:
The Worker Process shares the same memory space.
The ingress accesses shared memory as quickly as the private memory.
Worker does not require system calls and kernel entry.
Memory does not cause unnecessary memory replication.

The kernel does not synchronize access to shared memory. Therefore, programmers must provide their own synchronization.

Use shared memory:
Allocate a process to allocate memory segments.
Worker processes that use this memory segment need to connect (attach) This memory segment.
After each process uses the shared memory segment, detach the memory segment.
A worker must have a process to destroy the memory segment somewhere.

Linux memory model:
The virtual memory of each process is divided into pages ).
Every process maintains the ing between its memory address and the virtual memory page.
The actual worker data exists on the memory address of the process.
Although each process has its own address space, the ing of multiple processes can still point to the same page.

The size of all shared memory segments is an integer multiple of the Linux memory page size.
The page size in Linux is 4 kb, but programmers should use the getpagesize function to obtain this value.

Allocation: shmget
The first parameter of the marker is an integer key used to specify the segment to be created. Unrelated processes can access the same shared memory by specifying the same key.
Secret uses the constant ipc_private as the first parameter to avoid key conflicts.
The second parameter of bytes is the size of the allocated segment (number of bytes ). The actual number of bytes allocated will discard the excess part to an integer multiple of the page size.
The third parameter of marker is a bit flag used to indicate the created option.
Required bytes ipc_create: indicates that a new shared memory is to be created.
`Ipc_excl: always used with the previous flag. If the shared memory segment of the specified key already exists, this flag will cause the call to fail. If this flag is not specified, the call will return the shared memory segment that has occupied the key.
Zookeeper mode flag: 9 BITs, which is the same as the System File Permission, but the execution flag is invalid. These flags are defined in <sys/STAT. h>.
The values returned by the delimiter are the newly created or acquired memory segment identifier (shmid ).

Connection: shmat
The first parameter of the token is the shmid sign obtained by shmget ).
The second parameter of worker is the pointer to the address space of the process you want to map. If null is specified, Linux selects an available address.

========================================================== ==============

Python shared memory

The Python multiprocessing module provides two types of shared memory: sharedctypes and manager, which are less efficient but support remote memory sharing. Sharedctypes is highly efficient, with two orders of magnitude for fast manager. It is equivalent to common memory access when multiple processes are accessed.

The results are compared as follows: Test arrayelapsed 0:00:00. 119707 test dictelapsed 0:00:00. 152856 test shared manager listelapsed 0:00:37. 87666 test sharedctypes list in main processelapsed 0:00:00. 154170 test sharedctypes list in subprocesselapsed 0:00:00. 303328 elapsed 0:00:00. 327261 elapsed 0:00:00. 349312 elapsed 0:00:00. 318946 elapsed 0:00:00. 364301 elapsed 0:00:00. 370176 elapsed 0:00:00. 343588 elapsed 0:00:00. 348737

The Code is as follows:

import array from datetime import datetime, timedelta size = 1000000 def tranverse(a):     t = datetime.now()     for i in range(size):         a[i]     print 'elapsed %s'% (datetime.now()- t) a = array.array('i', [i for i in range(size)]) print 'test array' tranverse(a) a = {} for i in range(size):     a[i] = i print 'test dict' tranverse(a) from multiprocessing import Manager manager = Manager() a = manager.list([i for i in range(size)]) print 'test shared manager list' tranverse(a) from multiprocessing.sharedctypes import RawArray a = RawArray( 'i', [i for i in range(size)] ) print 'test sharedctypes list in main process' tranverse(a) from multiprocessing import Process ps = [Process(target=tranverse, args=(a, )) for i in range(8)] print 'test sharedctypes list in subprocess' for p in ps:     p.start() for p in ps:     p.join()

Refer:
Http://hi.baidu.com/%B1%D8%BC%C7%CE%B4%B6%C1/blog/item/f47e99d5cf0edfc051da4b8b.html
Http://blogold.chinaunix.net/u3/108641/showart_2267237.html
Http://www.rainsts.net/article.asp? Id = 1018