Solving a Python dictionary structure memory spike problem

Background: When reading a key value data, the Python dictionary structure causes memory usage to expand by about 10 times times and is not tolerated. This article solves this problem

Data: Word2vec The result of the training, word corresponds to a 400-D word vector. A total of 1.6G around the glossary

Solution: Use Python's class array for resolution.

Step: 1, convert raw data into Key,index and binary Value data block file, where binary Value data block size is 750M

2, the array after loading in accordance with the index query effect: Only about 800M of memory, greatly reducing the use of memory. Query using: vec[i] + = Word2vec_array[index*word2vec_dim + i]

Pei Zhi

Solving a Python dictionary structure memory spike problem