Compute optimization for &io-intensive tasks

Cause of the problem:

Recently, due to the actual needs of the work, the calculation method of a computational unit needs to be reconstructed. The reason is that because the calculation unit is computationally time-consuming, a single calculation takes approximately between 1s-2s, and new requirements require that approximately 1500 compute units be calculated within 20s. If the calculation method of the original calculation unit is not optimized and efficiency improvement, then 8-core CPU (Hyper-thread 16 thread), in a single computing 1s ideal conditions, server 16 thread to complete the task of the theoretical limit also need to 90s+, not to mention multithreading is not a simple efficiency overlay, the actual test case, Time consuming is often above 150s. Therefore, the calculation optimization of the original calculation unit is necessary.

Problem Analysis:

Through the realization process of the original calculation unit, there are a large number of database reads and a large number of calculation tasks, including the number of data tables ranging from hundreds of to millions, the data tables have a similar feature is relatively fixed, not real-time business data. Through the analysis of the original calculation process, the main time-consuming is in various conditions and compared to the database IO operation.

Therefore, to improve efficiency, the first thing to think about is how to reduce the number of database Io, but the actual computing task is a very strict time-series logic, that is, each step of the processing input is the data processing results of the previous step. Therefore, it is difficult to transform parallel computing in a single computing task, and because the time overhead of single database Io is not very good, it is not appropriate to make asynchronous transformation, but it will increase the complexity of the code.

So ultimately, the idea is to focus on how to optimize the computational task itself.

How to resolve:

After analysis of the computational task, in this scenario context, the decision to hash the location as the solution (this is a possible solution, but not necessarily optimal).

With the previous analysis of the calculation task, the data involved are relatively fixed, so first consider loading all the data into memory (because the amount of data is not very large, the server memory can withstand, can be loaded into mem or Redis according to the actual demand). If you just load the data into memory, and then replace the original database IO with Linq2object, it is not much better, because the most time-consuming operation in the computational logic is to query the range of the data, that is, the data does not match exactly, but it needs to find the upper and lower bounds of the corresponding data of the target value and perform the linear interpolation.

If you can implement the work of the scope data query in a faster and more precise manner, the maximum time overhead in the computational logic is saved. So consider how to use hash positioning.

The concrete transformation process does not repeat, the work difficulty mainly lies in the structure of the hash key, and how to implement the ' > and < ' condition operation in database query through hash addressing. Specifically, by expanding the range of dimensions into integers, and the minimum step in advance to do linear interpolation, can be formed to meet the requirements of the hash key, at the same time, by the need to locate the value of the length of the division to take the whole, you can get the lower value of the target value, and then the lower limit value plus step, you can get The result of the ' > and < ' operation before the database is obtained through a hash address.

Solution Results:

Through the above transformation, in this computational task scenario, the time cost of calculating the 1000+ compute unit has been reduced to 1-4 seconds (because it is a WCF service call, so it needs to depend on the condition of network communication, etc.), can satisfy the demand completely.

Through the reconstruction of this computational task, it can be seen that the optimization methods such as computational-intensive/io-intensive tasks, asynchronous and parallel computing are very difficult to carry out, and the improvement is very limited (computationally intensive task), so it is an important idea to achieve the ultimate goal by optimizing the atomic task itself.

Compute optimization for &io-intensive tasks