Redis Data Import Tool Optimization Process summary

Source: Internet
Author: User

Redis Data Import Tool optimization Process summary background

Developed a Redis data import tool using C + +
Import all table data into Redis from Oracle;
Not pure data import, the original records in each Oracle need to be processed by the business logic,
and add an index (Redis collection);
When the tool is finished, performance is a bottleneck;

Optimization effect

2 Sample Data tests were used:
Sample Data a table 8,763 records;
b table 940,279 Records;

Before optimization, a watch time is 11.417s;
After optimization, the A-meter time consuming 1.883s;

Tools to use

Gprof, Pstrace,time

Use the time tool to see how long each execution takes, including user and system time, respectively;
Use Pstrace Print to run in real time, query the main system call of the process, discover time-consuming point;
Use the time-consuming summary of the GPROF statistics program to focus on optimizing the most time-consuming areas;
Introduction to use:
1. All editing and connection options for g++ must be added-PG (the first day because there is no-PG option at the connection, resulting in no statistical report);
2. After executing the procedure, this catalogue will produce gmon.out files;
3.gprof Redistool gmou.out > Report, generate a readable file report, open the report centralized optimization of the most time-consuming function;

Optimization process

11.417s before optimization:

time ./redistool im a a.csvreal    0m11.417suser    0m6.035ssys     0m4.782s (发现系统调用时间过长)
File Memory Mapping

The system call time is too long, mainly is the file reads and writes, the preliminary consideration is reads the file, calls the API frequency too frequently;
Read the sample is a file fgets line of reading, using the file memory map Mmap, you can directly use the pointer operation of the entire file memory fast;

Log switch in advance

Improved the file read and write, found that the optimization effect is relatively limited (increased by about 2s); fgets is a C file read library function, compared to the system read (), is a buffer, should not be too slow (online test, file memory mapping compared to fgets () can be faster than the order of magnitude, Feel the scene should be more special);

Later through the Pstrace tool found Log.dat open too many times, the original debug log switch written to the back, resulting in the debug log will open the log file opened ("Log.dat");
To advance the log switch; after the improvement, 3.53s

time ./redistool im a a.csvreal    0m3.530suser    0m2.890ssys     0m0.212s
Vector space Pre-allocation

Later through GPROF analysis, a function of vector memory allocation of many times, and there are many replication times:
Improve the following line of code:

vector <string> vSegment;

Use static vector variables and pre-allocate memory:

static vector <string> vSegment;vSegment.clear();static int nCount = 0;if( 0 == nCount){    vSegment.reserve(64);}++nCount;

improved to 2.286s after optimization

real    0m2.286suser    0m1.601ssys     0m0.222s

Similarly, a member vector in another class uses the pre-allocated space (in the constructor):


improved to 2.166s after optimization;

real    0m2.166suser    0m1.396ssys     0m0.204s
function rewrite && inline

Continue executing the program, discovering that the Sqtoolstrsplitbych () function is consuming too much, overwriting the entire function logic, and then inline the rewritten function:
improved to 1.937s after optimization

real    0m1.937suser    0m1.301ssys     0m0.186s
Remove debugger and optimize monitoring symbols

Finally, after the debug and PG debugging symbols are removed, the final effect is 1.883s;

real    0m1.883suser    0m1.239ssys     0m0.191s
Meet Production requirements

The last steps above appear to be the millisecond level of ascension, the effect is obvious after enlarging to the whole table data;
After optimization, the production of a table is 152w, the import time-consuming about 326s (minutes);
b table Data 420w, import time is about 1103s (~18 minutes)

Posted by: Big CC | 28jun,2015
Blog: [Subscribe]
Github: Big cc

Redis Data Import Tool Optimization Process summary

Related Article

Contact Us

The content source of this page is from Internet, which doesn't represent Alibaba Cloud's opinion; products and services mentioned on that page don't have any relationship with Alibaba Cloud. If the content of the page makes you feel confusing, please write us an email, we will handle the problem within 5 days after receiving your email.

If you find any instances of plagiarism from the community, please send an email to: and provide relevant evidence. A staff member will contact you within 5 working days.

A Free Trial That Lets You Build Big!

Start building with 50+ products and up to 12 months usage for Elastic Compute Service

  • Sales Support

    1 on 1 presale consultation

  • After-Sales Support

    24/7 Technical Support 6 Free Tickets per Quarter Faster Response

  • Alibaba Cloud offers highly flexible support services tailored to meet your exact needs.