Python saves memory instances using sparse matrices

Recommendation system often need to deal with similar user_id, item_id, rating such data, in fact, is the sparse matrix in mathematics, SCIPY provides a sparse module to solve this problem, but scipy.sparse a lot of problems are not very suitable:

1, not very good at the same time support data[i, ...], data[..., j], Data[i, j] fast slicing;
2, because the data is kept in memory, can not be very good to support the massive data processing.

To support Data[i, ...], data[..., j] fast slicing, need I or J of the data set storage, and in order to save a huge amount of data, also need to put part of the data on the hard disk, memory buffer. The solution here is simple, with a class dict to store the data, for a certain I (such as 9527), its data is stored in the dict[' i9527 '), the same, for a J (such as 3306), its entire data stored in the dict[' j3306 ') inside, Need to remove data[9527, ...] , just take out dict[' i9527 '), dict[' i9527 '] is originally a Dict object, store a value of J corresponding, in order to save memory space, we put this dict as a binary string storage, directly on the code:

The code is as follows:

'''
Sparse Matrix
'''
Import struct
Import NumPy as NP
Import BSDDB
From Cstringio import Stringio

Class Dictmatrix ():
def __init__ (self, container = {}, DFT = 0.0):
Self._data = Container
SELF._DFT = DFT
Self._nums = 0

def __setitem__ (self, Index, value):
Try
I, j = Index
Except
Raise Indexerror (' Invalid index ')

IK = (' i%d '% i)
# To save memory, we package J, value into a binary string of characters
IB = Struct.pack (' If ', j, value)
JK = (' j%d '% j)
JB = Struct.pack (' If ', I, value)

Try
Self._data[ik] + + IB
Except
Self._data[ik] = IB
Try
SELF._DATA[JK] + = JB
Except
SELF._DATA[JK] = JB
Self._nums + = 1

def __getitem__ (self, Index):
Try
I, j = Index
Except
Raise Indexerror (' Invalid index ')

if (isinstance (i, int)):
IK = (' i%d '% i)
If not Self._data.has_key (IK): return self._dft
ret = Dict (np.fromstring (Self._data[ik], Dtype = ' i4,f4 '))
if (Isinstance (j, int)): Return Ret.get (J, Self._dft)

if (Isinstance (j, int)):
JK = (' j%d '% j)
If not Self._data.has_key (JK): Return SELF._DFT
ret = Dict (np.fromstring (SELF._DATA[JK], Dtype = ' i4,f4 '))

return ret

def __len__ (self):
Return self._nums

Def __iter__ (

Test code:

The code is as follows:

Import Timeit
Timeit. Timer (' foo = __main__.data[9527, ...] ', ' Import __main__ '). Timeit (number = 1000)

Consumes 1.4788 seconds and probably reads a data 1.5ms.
Another benefit of using class Dict to store data is that you can use memory dict or any other form of dbm, even the legendary Tokyo Cabinet ....

OK, the yard is finished.

