Python saves memory instances with sparse matrices _python

The recommendation system often needs to deal with similar user_id, item_id, rating such data, is in fact the sparse matrix in mathematics, SCIPY provides a sparse module to solve this problem, but Scipy.sparse has a lot of problems are not suitable:

1, not very good at the same time support data[i, ...], data[..., j], Data[i, j] fast slicing;
2, because the data stored in memory, can not be very good support for massive data processing.

To support the fast slicing of data[i, ..., data[, j], you need to store the data in I or J, and in order to save a huge amount of data, you need to put a portion of the data on your hard disk and use memory to do the buffer. The solution here is relatively simple, to store data with a class dict, for an I (such as 9527), its data is kept in dict[' i9527 ', and for a J (for example, 3306), all its data is stored in dict[' j3306 '. Need to remove data[9527, ...] , as long as remove dict[' i9527 '] can, dict[' i9527 '] Originally is a Dict object, store a certain J corresponding value, in order to save memory space, we put this dict in binary string form storage, directly on the code:

Copy Code code as follows:

'''
Sparse Matrix
'''
Import struct
Import NumPy as NP
Import BSDDB
From Cstringio import Stringio

Class Dictmatrix ():
def __init__ (self, container = {}, DFT = 0.0):
Self._data = Container
SELF._DFT = DFT
Self._nums = 0

def __setitem__ (self, Index, value):
Try
I, j = Index
Except
Raise Indexerror (' Invalid index ')

IK = (' i%d '% i)
# To save memory, we package J, value into a character binary string
IB = Struct.pack (' If ', j, value)
JK = (' j%d '% j)
JB = Struct.pack (' If ', I, value)

Try
Self._data[ik] + = IB
Except
Self._data[ik] = IB
Try
SELF._DATA[JK] = JB
Except
SELF._DATA[JK] = JB
Self._nums + 1

def __getitem__ (self, Index):
Try
I, j = Index
Except
Raise Indexerror (' Invalid index ')

if (isinstance (i, int)):
IK = (' i%d '% i)
If not Self._data.has_key (IK): return self._dft
ret = Dict (np.fromstring (Self._data[ik], Dtype = ' i4,f4 '))
if (Isinstance (j, int)): Return Ret.get (J, Self._dft)

if (Isinstance (j, int)):
JK = (' j%d '% j)
If not Self._data.has_key (JK): Return SELF._DFT
ret = Dict (np.fromstring (SELF._DATA[JK], Dtype = ' i4,f4 '))

return ret

def __len__ (self):
Return self._nums

Def __iter__ (

Test code:

Copy Code code as follows:

Import Timeit
Timeit. Timer (' foo = __main__.data[9527, ...] ', ' Import __main__ '). Timeit (number = 1000)

Consumes 1.4788 seconds, probably reads a data 1.5ms.
Another benefit of using class Dict to store data is that you can easily use memory dict or any other form of dbm, or even the legendary Tokyo Cabinet ....

All right, finish the code.