Redis source code analysis () --- latency analysis and processing

When we mention latency statistics, we must come up with the term "Performance Testing". That's right. In redis's redis_benchmark file, we did use the relevant information in the latency file. The official explanation of this file in redis is as follows:

/* The latency monitor allows to easily observe the sources of latency * in a redis instance using the latency command. different latency * sources are monitored, like disk I/O, execution of commands, fork * system call, and so forth. ** the latency listener can listen on many simple resources in redis, such as I/O disk operations, execute some commands, and * fork creates sub-thread operations. *----------------------------------------------------------------------------

In the delayed operation of redis, the entire process is very simple. It maintains a statistical list for each event. Each list contains a series of samples collected, each of which includes, the creation time of this sample and the delay time of this sample. Event = is a dictionary ing relationship for sampleserieslist. Next, let's take a look at the key collection point, the structure definition of the collection point named latencysample:

/* Representation of a latency sample: the sampling time and the latency * observed in milliseconds. * // * example of delayed sample */struct latencysample {// int32_t time when the delayed sample is created;/* we don't use time_t to force 4 bytes usage everywhere. * /// the specific delay time, in milliseconds uint32_t latency;/* latency in milliseconds. */};

What is maintained in the dictionary is not a sample node, but a node list structure:

/* The latency time series for a given event. * // * a series of delayed samples collected for an event */struct latencytimeseries {// The subscript int idx of the next delayed sample;/* index of the next sample to store. * /// maximum latency uint32_t Max;/* max latency observed for this event. * /// the latest latency record struct latencysample samples [latency_ts_len];/* latest history. */};

In the design of redis code, the latency is used for testing and result analysis. Therefore, the author also designs the data statistics struct used in subsequent analysis reports;

/* Latency statistics structure. * // * statistics result structure of the delayed sample */struct latencystats {// the absolute maximum delay time uint32_t all_time_high;/* absolute max observed since latest reset. * /// average sample Delay Time uint32_t AVG;/* Average of current samples. * // minimum latency uint32_t min of the sample;/* min of current samples. * /// maximum latency of the sample uint32_t Max;/* max of current samples. * /// mean relative error, compared with the average latency uint32_t mad;/* mean absolute deviation. * // the total number of samples uint32_t samples;/* Number of non-zero samples. * /// the creation time of the earliest delay Record Point time_t period;/* number of seconds since first event and now. */};

The meaning is very direct, so how does one perform event detection in a simple sample?

/* Start monitoring an event. we just set the current time. * // * set the listener for an event, that is, set the current time */# define latencystartmonitor (VAR) if (server. latency_monitor_threshold) {Var = mstime ();} else {Var = 0;}/* end monitoring an event, compute the difference with the current time * to check the amount of time elapsed. * // * end the listener and determine the time passed */# define latencyendmonitor (VAR) if (server. latency_monitor_threshold) {Var = mstime ()-var ;}

It is very simple: record start time, record end time, the difference in the middle is the delay time, if it exceeds the given time range, it will be added to the delay list:

/* Add the sample only if the elapsed time is> = To the configured threshold. * // * If the latency exceeds the server. latency_monitor_threshold, add the sample to the latency list */# define latencyaddsampleifneeded (event, VAR) if (server. latency_monitor_threshold & (VAR)> = server. latency_monitor_threshold) latencyaddsample (event), (VAR ));

We will pay attention to latencyaddsample, which is to add the sampling node to the record. The steps are as follows:

1. Based on the input event, find the Val where key is the event in server. latency_events, that is, a latencytimeseries

2. Add a new sample in struct latencysample samples [latency_ts_len] of latencytimeseries.

The implementation code is as follows:

/* Add the specified sample to the specified time series "Event ". * This function is usually called via latencyaddsampleifneeded (), that * is a macro that only adds the sample if the latency is higher than * server. latency_monitor_threshold. * // * Add the sample to the sample list of the specified event object */void latencyaddsample (char * event, mstime_t latency) {// locate the delayed sample record struct latencytimeseries * Ts = dictfetchvalue (server. latency_events, event); time_t now = Time (null); int Prev;/* Create the time series if it does not exist. */If (TS = NULL) {Ts = zmalloc (sizeof (* TS); TS-> idx = 0; TS-> max = 0; memset (ts-> samples, 0, sizeof (ts-> samples); // If ts is empty, add it again. An event corresponds to a latencytimeseries dictadd (server. latency_events, zstrdup (event), TS);}/* If the previous sample is in the same second, we update our old sample * If this latency is> of the old one, or just return. */Prev = (ts-> idx + latency_ts_len-1) % latency_ts_len; If (ts-> samples [Prev]. time = now) {If (latency> TS-> samples [Prev]. latency) ts-> samples [Prev]. latency = latency; return;} // assign ts-> samples [ts-> idx] to the sample. time = Time (null); TS-> samples [ts-> idx]. latency = latency; If (latency> TS-> MAX) ts-> max = latency; TS-> idx ++; If (ts-> idx = latency_ts_len) ts-> idx = 0 ;}

After all the nodes come out, of course, the structure analysis and statistics will be performed, and the latencystats struct will be used;

/* Analyze the samples avaialble for a given event and return a structure * populate with different metrics, average, mad, Min, Max, and so forth. * Check latency. h definition of struct latenctstat for more info. * If the specified event has no elements the structure is populate with * zero values. * // * analyze the delay result of an event at a time, and save the result information to the latencystats struct */void analyzelatencyforevent (char * event, struct latenc Ystats * ls) {struct latencytimeseries * Ts = dictfetchvalue (server. latency_events, event); Int J; uint64_t sum; // initialize the variable LS-> all_time_high = ts? Ts-> MAX: 0; LS-> AVG = 0; LS-> min = 0; LS-> max = 0; LS-> mad = 0; ls-> samples = 0; LS-> period = 0; If (! TS) return;/* first pass, populate everything but the mad. */SUM = 0; For (j = 0; j <latency_ts_len; j ++) {If (ts-> samples [J]. time = 0) continue; LS-> samples ++; If (LS-> samples = 1) {LS-> min = LS-> max = ts-> samples [J]. latency;} else {// find the longest and least latencies. If (LS-> min> TS-> samples [J]. latency) ls-> min = ts-> samples [J]. latency; If (LS-> max <ts-> samples [J]. latency) ls-> max = ts-> samples [J]. latency;} sum + = ts-> samples [J]. latency;/* track the oldest event time in LS-> period. */If (LS-> period = 0 | ts-> samples [J]. time <ls-> period) // The Creation Time of the earliest delay record point ls-> period = ts-> samples [J]. time;}/* so far AVG is actually the sum of the latencies, and period is * The oldest event time. we need to make the first an average and * The second a range of seconds. */If (LS-> samples) {LS-> AVG = sum/LS-> samples; LS-> period = Time (null)-ls-> period; if (LS-> period = 0) ls-> period = 1;}/* second pass, compute mad. * // calculate the average relative error, sum = 0 compared to the average latency; For (j = 0; j <latency_ts_len; j ++) {int64_t delta; if (ts-> samples [J]. time = 0) continue; Delta = (int64_t) ls-> AVG-ts-> samples [J]. latency; If (delta <0) Delta =-delta; sum + = delta;} If (LS-> samples) ls-> mad = sum/LS-> samples ;}

Of course, you can also use these collected points to draw a micro-line chart to better display the image:

# Define latency_graph_cols 80/* use the delayed sample point to draw the corresponding line chart */SDS latencycommandgensparkeline (char * event, struct latencytimeseries * TS) {Int J; struct sequence * seq = createsparklinesequence (); SDS graph = sdsempty (); uint32_t min = 0, max = 0; For (j = 0; j <latency_ts_len; j ++) {int I = (ts-> idx + J) % latency_ts_len; int elapsed; char * label; char Buf [64]; If (ts-> samples [I]. time = 0) continue;/* update min and Max. */If (SEQ-> length = 0) {min = max = ts-> samples [I]. latency;} else {If (ts-> samples [I]. latency> MAX) max = ts-> samples [I]. latency; If (ts-> samples [I]. latency <min) min = ts-> samples [I]. latency;}/* use as label the number of seconds/minutes/hours/days * ago the event happened. */elapsed = Time (null)-ts-> samples [I]. time; If (elapsed <60) snprintf (BUF, sizeof (BUF), "% ds", elapsed); else if (elapsed <3600) snprintf (BUF, sizeof (BUF), "% DM", elapsed/60); else if (elapsed <3600*24) snprintf (BUF, sizeof (BUF), "% DH ", elapsed/3600); else snprintf (BUF, sizeof (BUF), "% dd", elapsed/(3600*24); label = zstrdup (BUF); sparklinesequenceaddsample (SEQ, ts-> samples [I]. latency, label);} graph = sdscatprintf (graph, "% s-high % lu MS, low % lu MS (all time high % lu MS) \ n", event, (unsigned long) Max, (unsigned long) min, (unsigned long) ts-> MAX); For (j = 0; j <latency_graph_cols; j ++) graph = sdscatlen (graph, "-", 1); graph = sdscatlen (graph, "\ n", 1 ); // call the sparkline function to draw a line chart graph = sparklinerender (graph, seq, latency_graph_cols, 4, sparkline_fill); freesparklinesequence (SEQ); // return the line chart string return graph ;}

Some commands are encapsulated in redis for external calls, which will not be analyzed here. It is a compound call to the above method:

/* -------------------------- Latency API quota */void latencymonitorinit (void)/* initialize the delayed listener and create the event dictionary object */void latencyaddsample (char *, event mstime_t latency) /* Add the sample to the sample list of the specified event object */INT latencyresetevent (char * event_to_reset)/* reset the latency of the event, delete the event record in the dictionary */void analyzelatencyforevent (char * event, struct latencystats * ls)/* analyze the delay result of the event at a certain time, the result information is stored in the latencystats structure */SDS createlatencyreport (void)/*. Based on the results of the delayed sample, the analysis report with good readability is created */void latencycommandreplywithsamples (redisclient * C, struct latencytimeseries * TS) void Merge (redisclient * C) SDS latencycommandgensparkeline (char * event, struct latencytimeseries * TS) void latencycommand (redisclient * C)

The Analysis of redis latency files is also over. After analyzing the redis code for such a long time, I feel that every piece of code has its own highlights. I have analyzed more than 30 periods, I still learned a lot of things I couldn't learn on the Internet. I learned more about the mainstream ideas of redis on the Internet, such as some small points. I can really understand them only by my taste.

Redis source code analysis () --- latency analysis and processing