A sparse matrix Python storage scheme for comparing memory-saving

Recommendation system often need to deal with similar user_id, item_id, rating such data, in fact, is the sparse matrix in mathematics, SCIPY provides a sparse module to solve this problem, but scipy.sparse a lot of problems are not very suitable: 1, Not very good at the same time support data[i, ...], data[..., j], Data[i, j] fast slicing; 2, because the data is stored in memory, not very good support for massive data processing.

To support Data[i, ...], data[..., j] fast slicing, need I or J of the data set storage, and in order to save a huge amount of data, also need to put part of the data on the hard disk, memory buffer. The solution here is simple, with a class dict to store the data, for a certain I (such as 9527), its data is stored in the dict[' i9527 '), the same, for a J (such as 3306), its entire data stored in the dict[' j3306 ') inside, Need to remove data[9527, ...] , just take out dict[' i9527 '), dict[' i9527 '] is originally a Dict object, store a value of J corresponding, in order to save memory space, we put this dict as a binary string storage, directly on the code:

' Sparse Matrix ' ' Import structimport numpy as Npimport bsddbfrom Cstringio import Stringio class Dictmatrix (): def _      _init__ (self, container = {}, DFT = 0.0): Self._data = Container SELF._DFT = DFT self._nums = 0 def __setitem__ (self, Index, value): try:i, j = index except:raise indexerror (' I        Nvalid index ') ik = (' i%d '% i) # to save memory, we pack J, value into a word binary string ib = Struct.pack (' If ', j, value)            JK = (' j%d '% j) jb = Struct.pack (' If ', I, value) Try:self._data[ik] + = IB except:        Self._data[ik] = IB TRY:SELF._DATA[JK] + = JB EXCEPT:SELF._DATA[JK] = JB Self._nums + = 1 def __getitem__ (self, Index): try:i, j = Index Except:rai Se indexerror (' invalid index ') if (isinstance (i, int)): IK = (' i%d '% i) if not self._data. Has_key (IK): return self. _DFT ret = dict (np.fromstring (Self._data[ik], Dtype = ' i4,f4 ')) if (Isinstance (j, int)): return ret  . Get (J, Self._dft) if (Isinstance (j, int)): JK = (' j%d '% j) if not Self._data.has_key (JK): return SELF._DFT ret = dict (np.fromstring (SELF._DATA[JK], Dtype = ' i4,f4 ')) return ret def __len __: Return Self._nums def __iter__ (self): Pass "generates matrix from file considering the performance of dbm reading and writing is not as good as memory, we did some Cache, batch write every 1000W times considering the string stitching performance is not very good, we use Stringio to do splicing "Def from_file (self, FP, Sep = ' t '): CNT = 0 CA                Che = {} for L in fp:if 10000000 = = Cnt:self._flush (cache) CNT = 0 cache = {} I, J, v = [Float (i) for i in L.split (Sep)] ik = (' i%d '% i) IB = S                Truct.pack (' If ', J, v) JK = (' j%d '% j) jb = Struct.pack (' If ', I, v) try:   Cache[ik].write (IB)         Except:cache[ik] = Stringio () cache[ik].write (IB) Try:              Cache[jk].write (JB) EXCEPT:CACHE[JK] = Stringio () cache[jk].write (JB)         CNT + = 1 Self._nums + 1 Self._flush (cache) return self._nums def _flush (self, cache): For k,v in Cache.items (): V.seek (0) s = v.read () try:self._data[k ] + = s except:self._data[k] = s if __name__ = = ' __main__ ': db = Bsddb.btopen (None, Cachesiz E = 268435456) data = Dictmatrix (db) data.from_file (open ('/path/to/log.txt ', ' R '), ', ')

Test 4500W rating data (shaping, integer, floating point format), 922MB text file import, using memory Dict storage, 12 minutes to build, Consumption of memory 1.2G, in the sample code of BDB storage, 20 minutes to build, take up memory 300~400mb about, more than CacheSize, data read test:

Import Timeittimeit. Timer (' foo = __main__.data[9527, ...] ', ' Import __main__ '). Timeit (number = 1000)

Consumes 1.4788 seconds and probably reads a data 1.5ms.

Another benefit of using class Dict to store data is that you can use memory dict or any other form of dbm, even the legendary Tokyo Cabinet ....

OK, the yard is finished.

