Record a project optimization process

This is a creation in Article, where the information may have evolved or changed.

Recently, using parse to do data collection, the background is MongoDB. One requirement is to analyze the number of user cities that have been collected in the database for more than 10 million of users in the specified time period, so as to determine whether the user is a travel user for accurate push. Logical analysis This piece has nothing to say, the point is to insert the database phase.

First version

The beginning of the version did not think too much, according to the conventional idea to do, after the data analysis is completed, according to the user ID send a request to parse query whether the user exists, if not exist then insert, there is a comparison of user tags are changed, no changes are skipped. The code is soon written, according to the log shows the total data at about 11 million, the last statistic out of the independent user in 6 million. Look at the output request is very slow, so after work let the script run in the background. It turned out that 3 million of the data was inserted after the next morning's visit. This speed is certainly not good, the leader means that this script runs once a week, we live in 10 million or so, after the later data volume, it is not to run a few days! So I started thinking about the space to optimize

A second version

Considering that the first phase of the data-counting phase is time consuming in minutes, the time is mainly spent on the parse request. The user ID is indexed in MongoDB, and the data is cached from the MongoDB everywhere before entering and exiting the data. Comparing the results in memory, there is a need to insert again, reducing the number of requests. The result is good, the speed is obviously a lot faster, a day or so should be able to run out. But I still can't accept the result, so I'm starting to wonder if there's room for optimization.

A third version

View Log Discovery Data processing is fast, and time is spent primarily on requests and waiting for the return of the request. So it's natural to think of multi-threading, using multiple threads to fetch data from the queue to send requests. At first, 2000 threads were opened, and the result was that it was not up to the end of the 900 thread to run. Because I did not write the code in Python before, with C + + and Golang more, especially golang inside the convenience of the association and channel cause I quickly forget the traditional multithreading should be how to write. Locked up all kinds of crash, but the result is good, clear all data re-import should be able to import within 1 hours to complete.

Fourth version

In fact, I was quite satisfied with the previous version, but when I looked at the log, I found that the data processing queue was often full, causing the thread that inserted it to hang up and unlock it. So I went over the parse interface document to see if there was a bulk import of the interface results really let me find out. So sure enough, every time you submit a maximum of 30 data imports. This time found that 900 threads have been waiting for the thread, so the volume of the number of data from the previous 10,000 to 30,000, which almost reached a balance. The time was also shortened to about 30 minutes.

Fifth version

The fifth version is completely unexpected, is that I in the data when the way to open top to see the performance, the results found that 32G of memory was eaten by the script 50% or so, and MongoDB resident itself compared to eat memory also accounted for 46% of the space, considering the future data may also rise, This method of importing MongoDB data into memory seems quite inappropriate. After pondering, the final decision is to iterate the data with a loop iteration, compare the extracted data with the statistical results, and then delete the record from the statistical results after the comparison is completed. Finally unified processing of the remainder of the statistical results, this basic only need to bulk insert on the line, do not consider the update things. This version is a relatively perfect version, the import time of about 40 minutes, memory consumption of about 10% compared to the previous significantly reduced, the time has not increased a lot.

Summarize

The

  producer consumer model is common but is a practical approach in engineering. In the face of this big data processing work in fact should think of using multi-threaded to send requests, but after all, not familiar with Python, multithreading is never used to lazy. Cache is a good thing but really too much memory, the same keyword should consider the value of the cache one copy, and the other in real-time extraction method to do.