Large-scale query MySQL scheme in log monitoring system

Recent development has encountered a problem: the need to query a large time period of data, divided into 1000 small segments, that is, 1000 points. The x-axis is the time, and the y-axis is the statistical data for that small time period. Note: The data return is a list where each object return value is statistically calculated for that small period of time and needs to be returned in the order of entry (so that the front-end display is convenient). For example, to query 12 to 1 points of data, the query frequency is 30 minutes, then you need to query 11:30-12:00,12:00-12:30,12:30-1:00 these three pieces of data (because the monitoring system is to query the past data, so the value of 12 points should be the first half an hour). Here's the problem,

Plan one: Direct hot-check database, 1000 times, the final 30s+, elimination!

Scenario two: Because of many times to check the library, 1000 times is certainly time-consuming, so the use of a library, the conditions with a SQL, with the union all to each small time period to query out the data and set, time consuming 20S, and there is a problem of SQL too long error (default 1M)

Scenario Three: Directly query a large time period of data, not statistics, directly back in the library data, in the code (that is, in memory) to go back to the List, for each entity, traversing the time period, the entity to match to this time period, map (key,list), key is the segment number, The list saves the entities for that time period. The data is ultimately counted for each time period. The third scenario, which takes 1-2 seconds, proves again the truth of time-consuming io> memory. Here is a detail: in order to ensure the return of data orderly (time period successively), here Map uses treemap, natural sort can.

Large-scale query MySQL scheme in log monitoring system