Redis batch import data

Environment:

An openstack Virtual Machine with 8-core CPU, 16 GB memory, and GB disk

Ubuntu 12.04x64 Server

Install redis2.6.7

Process:

During the experiment, we need to import 0.21 billion key-value data into redis for fast query.

After installing, configuring, and starting redis, You need to import data to redis.

If you try to use jedis to write the client to insert data into redis one by one, the speed is very slow. Just insert tens of thousands of data entries in one minute, thank God, it takes several days for 0.21 billion records (I guess the insert speed of 0.1 million records per second in redis is the batch insert speed, and the insert speed is relatively short ).

Redis batch plug, check, some use a tool named redis_import_tools https://github.com/unbracketed/redis-import-tools

I wrote it using python. I thought it was a little troublesome to read readme. It was faster than the pipe line function provided by redis. I still don't believe it, try how slow the pipe line of redis can be:

Reference http://redis.io/topics/mass-insert

You can easily generate a data file in redis protocol format. It takes about 12 minutes to import 0.21 billion entries (9 GB, with less than 4 GB of pure data, after the import, the redis-server process consumes about 90% of the memory (16 GB in total ). It seems that:

1. redis's own pipe is still very powerful;

2. redis is very memory-consuming. The physical memory should be more than 4 times the data (and the physical memory is not occupied by other processes). Otherwise, the system's memory allocation method should be set, this allows redis to use virtual memory, but the query speed may be reduced.