Python uses sparse matrix to save memory instances

This article mainly introduces how to use sparse matrix to save memory. The number of non-zero elements in the matrix is much smaller than the total number of matrix elements, and the distribution of non-zero elements is irregular, this matrix is called a sparse matrix. If you need a friend, you can refer to the recommendation system that often needs to process data such as user_id, item_id, and rating. It is actually a sparse matrix in mathematics, scipy provides the sparse module to solve this problem, but scipy. sparse is not suitable for many problems:

1. data [I,...], data [..., j], and data [I, j] cannot be supported at the same time;
2. Because the data is stored in the memory, it cannot support massive data processing.

To support data [I,...], data [..., j] fast slicing requires centralized storage of I or j data. At the same time, to save massive data, you also need to put a part of the data on the hard disk and use the memory as the buffer. The solution here is relatively simple. It uses Dict-like things to store data. For an I (such as 9527), its data is stored in dict ['i9527, similarly, for a j (such as 3306), all its data is stored in dict ['j3306'], and data [9527,...] needs to be retrieved. you only need to retrieve dict ['i9527']. dict ['i9527'] is originally a dict object and stores the value corresponding to a j. To save memory space, we store this dict in the form of a binary string and directly use the code:

The Code is as follows:

'''
Sparse Matrix
'''
Import struct
Import numpy as np
Import bsddb
From cStringIO import StringIO

Class DictMatrix ():
Def _ init _ (self, container ={}, dft = 0.0 ):
Self. _ data = container
Self. _ dft = dft
Self. _ nums = 0

Def _ setitem _ (self, index, value ):
Try:
I, j = index
Except t:
Raise IndexError ('invalidindex ')

Ik = ('I % d' % I)
# To save memory, we pack j and value into a binary string.
Ib = struct. pack ('if', j, value)
Jk = ('J % d' % j)
Jb = struct. pack ('if', I, value)

Try:
Self. _ data [ik] + = ib
Except t:
Self. _ data [ik] = ib
Try:
Self. _ data [jk] + = jb
Except t:
Self. _ data [jk] = jb
Self. _ nums + = 1

Def _ getitem _ (self, index ):
Try:
I, j = index
Except t:
Raise IndexError ('invalidindex ')

If (isinstance (I, int )):
Ik = ('I % d' % I)
If not self. _ data. has_key (ik): return self. _ dft
Ret = dict (np. fromstring (self. _ data [ik], dtype = 'i4, f4 '))
If (isinstance (j, int): return ret. get (j, self. _ dft)

If (isinstance (j, int )):
Jk = ('J % d' % j)
If not self. _ data. has_key (jk): return self. _ dft
Ret = dict (np. fromstring (self. _ data [jk], dtype = 'i4, f4 '))

Return ret

Def _ len _ (self ):
Return self. _ nums

Def _ iter __(

Test code:

The Code is as follows:

Import timeit
Timeit. Timer ('foo = _ main _. data [9527,...] ', 'import _ main _'). timeit (number = 1000)

It takes 1.4788 seconds to read a piece of data, which is about 1.5 ms.
Another advantage of using Dict to store data is that you can use the memory Dict or any other form of DBM, or even the legendary Tokyo Cabinet ....

Okay. The code is complete.