Performance tests based on user experience: Chapter 2 simulates the latency of individual users

Copyright:Based on the spirit of sharing, this article can be reproduced to benefit more test peers. Please retain this copyright statement upon reprinting and ensure the integrity of the article, but shall not be used for any commercial or other purposes for profit purposes.

Original article name: user experience, not metrics
Original Author: Scott barber
Source: http://www.perftestplus.com/resources/UENM2.pdf

Translation name: Performance Test Based on User Experience
Translation: pent
Http://www.cnblogs.com/pent/archive/2007/07/01/802117.html.

 

Chapter 2 Simulate latency of individual users

Website Visitors think, read, and type at different speeds. As part of the testing process, you are responsible for solving the different speeds of simulation and scripting. TheseRational teststudioThere are many ways to achieve this 《User Experience, not metricsIn chapter 2 of the series, I learned some of them. This seriesArticleThe focus is to associate customer satisfaction with the performance experience of external users, which has been explained in the first chapter of this article. This article and subsequent articles will provide necessary theories, including how to simulate actual users based on testing purposes, and demonstrate these principles using recording or modifying scripts as examples based on years of performance testing experience.

Some industry experts say that the changes in user latency are irrelevant to the actual test results. When there are many users20Second Delay will also get the same result. I will argue that if the single server architecture and staticHtmlI may agree with the script, but this is impossible for today's multi-level and multi-functional websites. Scientific website load testing (The science of web-site Load Testing)Alberta SavoiaWhy?20The second theory is not a fact that is explained in quite detail and provides examples and mathematical proofs of the real world.

This article describesTeststudioUsers are helpful, but it is especially helpful for intermediate and middle-level users. Here I will introduce different methods of scripted mathematical distribution, but you have to decide which one can match the specific scenario with the highest accuracy. The accuracy of statistics is significant and unique for every website to simulate the end user group.

1. Determine user latency

How long does it take for a user to log on to, browse the home page, and enter a form? You can obtain the user Delay Time in combination with the user activity on the website in several ways. The best way is to use the real data collected at the production site, but this is almost impossible, because testing usually occurs before the website is released for use. Because of this, sometimes we need to make reasonable speculation based on the website's behavior. The four most acceptable methods are as follows:

• L When the tested website is in production, you can know the browsing (or typing) time from the log files on each page, extract the mean difference and standard deviation, and determine the actual value and distribution status. With this information, the user delay time on each page can be easily determined. Your production site may also have website traffic monitoring software that can directly provide such information, for example, Webtrends or livestat .
• LWhen you cannot perform detailed log analysis in the absence of log files, time, and resource records, you can use the metrics and statistics collected by some companies. These statistics come from objective sampling of users or sites and provide the average page browsing time and user session duration. Although these numbers are not from your specific site, they are good for approximation. Figure1Yes fromNetratings.
  

 Figure1 netratingsProvided sample data

• As you can see, this is a very high-level information that may be useful or useless for simulating actual users. The most useful information in the figure is "viewing Page Duration "(Duration of a pageViewed), Reports the average time the user views all types of web pages. The remaining statistical information is consolidated by a combination of a large user group that browses the website.
• LIf you do not have a log file or think that metrics from other companies are not representative, you can use simple internal experiments by employees, customers, friends, or family members. For example, the page viewing time for new and old users is different. This method is also considered "Clipboard and Stopwatch ". I think this is a very effective data collection method for websites that have not yet been put into operation or data collected through other methods for verification.
• LThe final means is to evaluate the scope of these latencies through your intuition or speculation. For realistic load tests, even this final method is much better than ignoring changes to user latency. Each user consumes the same time on each page. Such a load test is simple but not realistic, and may produce misleading results.

2. Understand the latency range and distribution

Now you know the time and scope of the user's time on the page, but you have not finally completed the simulation of user latency. You must diversify the user's latency, or the server will display2Load.

 

 Figure2Striped response Diagram

In the above response diagram, each red dot represents a user activity (here is a page request), and the horizontal axis is time, in seconds.,Each virtual user is listed on the vertical axis as calculated from the test start. This response graph is a typical "striped "(Striping) Requests, which must be avoided in the load test. It is a good example of stress testing, but not a load test. If you place a ruler vertically on the graph and slowly move from left to right, in the vertical direction, you will see that the red spots representing user activities are almost the same online, this also indicates the user requests received by the server at the same time. This is a very poor description of the actual user group. Now look at the figure3.

 

 Figure3Non-striped response Diagram

If you use a ruler to measure the amount, you will find that the red dot distribution is more even this time. Although it is still not complete, it is obviously closer to actual users. The only difference between the two tests is that the latter has a certain random delay, rather than an absolute delay.

Before the preceding two examples are adjusted in a planned manner3Point information is required. In the first example, I only used the average latency during script recording. Second, I added the minimum and maximum latency values and the distribution. There are many mathematical models for this distribution. I will focus on the two most commonly used: even distribution (Uniform Distributions) And normal distribution (Normal distributions). For the sake of completeness, the negative exponential distribution is also introduced. This distribution is not often used, but it is not useless. AdvancedProgramThis distribution can be combined with other mathematical functions to simulate certain user models.

The average distribution between the minimum and maximum values is the easiest to simulate. This distribution model simply selects random numbers from the top and bottom to form a uniform distribution, which means that the generated data is not only in the middle or both sides of the data range. Figure4Shown1000Items0To25The average distribution between them.

 

Figure4Uniform distribution between the maximum and minimum values

Normal Distribution, also known as bell curve, is more difficult to simulate but more accurate. This distribution model selects numbers randomly and weighting the frequency of the center value or average value. Figure5Shown in0To25Generated1000Normal Distribution of numbers (average value:12.5, Standard deviation is3.2). Normal Distribution is generally considered the most accurate mathematical model when no actual data is available.

 

Figure5Ideal Normal Distribution

Distribution of the negative Index Model6As shown in, the delay frequency of this model is obviously skewed to the end. I think this distribution is rarely useful, but it is included for integrity considerations. Figure6Shown in0To25Between1000Negative Exponential Distribution of numbers.

 

Figure6Negative Exponential Distribution

3. Time Delay generated in teststudio

UseRational teststudioOfVucLanguage, there are many ways to generate time delay. How to UseThink_avgAnd related commands and parameters,RationalThe documents and training materials have been described in detail. I will not discuss them in depth here, but I will only give a brief introduction. This method is only available inThink_avgThe command is not in the timer (Timer. I foundThink_avgCommands are easier to represent the processing process of the client, but they are not suitable for the user model. The timer and client processing time models will be discussed in detail in subsequent articles.

When the user delay in the script is not included in the counterDelayCommand. UseDelayCommand to create static values and even, negative index, normal distribution for discussion of user latency.

If it is clear that some activities consume all users8Seconds (very small, but possible), the following command can be used:

Delay (8000 );

The values in the brackets are static latency values, in milliseconds. The syntax of this command isDelay (value );Static latency is generally only used in the exact model.

If you are sure that the time when the user executes the activity on the website is6To12In seconds, the even distribution can accurately represent the actual user. The command is as follows:

Delay (uniform (6000,12000 ));

In this example,6000Is the minimum value while12000Is the maximum value, in milliseconds. The syntax of this command isDelay (uniform (min_value, max_value )).

If you determine the time when the user is active on the website9To15Seconds, but most users are close9Seconds instead15Seconds, and no fewer users9In seconds, the negative exponential distribution is the most accurate. For example, only when9After a short video is played in seconds, the next button is displayed. The command is as follows:

Delay (negexp (9000,15000 ));

In this example,9000Is the minimum value while15000Is the maximum value, in milliseconds. The syntax of this command isDelay (negexp (min_value, max_value )).

InVucThe function does not have a latency command to create a normal distribution. Therefore, we use the following functions andDelayCommand combinations constitute a normal distribution. Note,NormdistThe function must be placed directly in each script# IncludeCommand.

Int func normdist (Min, Max, STDev)/* specifies input values
For normdist function */

/* Min: Minimum value; max: maximum value;
STDev: Degree of deviation */
Int min, Max, STDev;
{

/* Declare range, iterate and result as integers-vuc
Does not support floating point math */
Int range, iterate, result;

/* Range of possible values is the difference between
Max and Min values */
Range = max-min;

/* This number of iterations ensures the proper shape
The resulting curve */
Iterate = range/STDev;

/* Integers are not automatically initialized to 0
Upon declaration */
Result = 0;

/* Compensation for integer vs. floating point math */
STDev + = 1;
For (C = iterate; C! = 0; c --)/* loop through iterations */
Result + = (uniform (1,100) * STDev)/100;

/* Calculate and tally result */
Return result + min;/* send final result back */
}

Use this function0Millisecond is the minimum value,25000Millisecond is the maximum value,3200The standard deviation in milliseconds is executed.1000Generation Normal Distribution7. Note that the same parameters as the ideal normal curve are used here, but the unit is changed from second to millisecond. We can see that this graph and Graph5The ideal normal distribution is almost the same.

 

Figure7UseNormdistNormal Distribution of function generation

DelayFunctions are most frequently used in normal distribution. Run the following command to simulate10Seconds35Seconds, standard deviation is3.2Second, the user latency is normally distributed (for the above example, the time is only shifted to the right10Seconds ):

Delay (normdist (Listen, 25000, 3200 ));

Here10000Is the minimum value while25000Is the maximum value,3200Is the standard deviation, in milliseconds. The syntax of this command is:Delay (normdist (min_value, max_value, std_deviation )). Remember, if there is noNormdistFunctionCodeAn error will be reported.

4. Now you try it

To prove the simplicity and effectiveness of these concepts, I suggest you follow the exercises below, provided that you know how to record and play backVUScript and how to insert it during recordingTimer.

4.1. Exercise for determining user latency and distribution

Select a completely static website, because every time a website changes, it will only hinder the learning process. Then determine a navigation path, suchOnblestat.comOn the homepage of the website, clickAbout usAnd then clickEssentialsAnd clickHeritage. First, write down the possible user distribution and latency of each page in the notepad, and then find some colleagues to perform operations based on the instructions on the paper, and record their stay time on these pages. Check whether the actual time and distribution are close to the records in notepad.

4.2. Simulate user latency and distribution in vuc exercises

I used up the test time for this website.4Methods To determine the user latency and distribution. In all cases, different time values are obtained based on the normal distribution curve. For the sake of this example, we suppose this is not always the case so that you can see how to use the built-inCFunctions and the newNormdistFunction to generate a time delay.

I reiterate that the latency and distribution in this example do not represent the actual website interaction. I will create exercises in this article to demonstrate all topics.

First, recording a simpleVUScript (I useNoblestat.com). Used for recordingTimerInclude each page. (You can also useTimer BlockBut the recorded original script is a little different.5ChapterTimer Block). Recording3Pages,Home Page,Page1,Page2.

Use1Make sure that the script is successfully played back before modification. And then useRobotOpen the script and find the firstStop_timeCommand. The script for this part is as follows:

Http_header_recv ["rdn_on _~ 233 "] 304;/* not modified */
Http_nrecv ["rdn_on _~ 234 "] 100% %;/* 238 bytes-from cache */
Stop_time ["home page"];
Start_time ["page1"];
Sets think_avg = 12342;
/* Keep-alive request over connection www_noblestar_com */
Http_request ["rdn_on _~ 235 "]

This part of the code stops the timer statistics on the loading time of the home page, and starts loadingPage1Timer statistics, and then wait for almost12Seconds laterPage1. In fact, you don't want12The second delay is also included in the timer, and there is still no hope12Second is a static value, but6Seconds18The normal distribution of the second delay range. You can delete or comment outStart_timeCommandSet think_avgThis line, at the same time inStop_timeAndStart_timeAdd between commandsDelay (uniform (6000,18000 )). The code is shown as follows:

Http_header_recv ["rdn_on _~ 233 "] 304;/* not modified */
Http_nrecv ["rdn_on _~ 234 "] 100% %;/* 238 bytes-from cache */
Stop_time ["home page"];
Delay (uniform (6000,18000);/* added to replace think_avg below */
Start_time ["page1"];
/* Set think_avg = 12342;-replaced by delay above */
/* Keep-alive request over connection www_noblestar_com */
Http_request ["rdn_on _~ 235 "]

Now this code will count the homepage andPage1The actual loading time of the page, which will not be accessed until the user has read the Home PagePage1Page, and the wait time is6To18The random time in the second normal distribution.

FindPage1OfStop_timeCommand, the original code is as follows:

Http_header_recv ["rdn_on _~ 242 "] 200;/* OK */
Http_nrecv ["rdn_on _~ 243 "] 100% %;/* 9997 bytes */
Stop_time ["page1"];
Start_time ["page2"];
Sets think_avg = 8536;
Set server_connection = www_noblestar_com_1;
/* Keep-alive request over connection www_noblestar_com_1 */
Http_request ["rdn_on _~ 244 "]

Like above, this script stopsPage1Timer, startedPage2And delay8.5About seconds. Here, you want8The static latency of seconds is changed to the minimum value.6Seconds, maximum14Second, standard deviation2The normal distribution delay of seconds. If you have not calculated the standard deviation of the latency range, multiply the maximum latency time minus the minimum latency time25% Is a more accurate and acceptable method.

SetNormdistCopy the function to the script.# IncludeThe code below is as follows:

# Include
Int func normdist (Min, Max, STDev)
Int min, Max, STDev; // min: Minimum value; max: maximum value;
STDev: Degree of deviation allowed
{
Int range, iterate, result;
Range = max? Min;
Iterate = range/STDev;
Result = 0;
STDev + = 1;
For (C = iterate; C! = 0; c --)
Result + = (uniform (1,100) * STDev)/100;
Return result + min;
}
{
Push http_control = http_partial_ OK | http_cache_ OK | http_redirect_ OK;

Delete or comment outStart_timeCommandSet think_avgThis line, and inStop_timeAndStart_timeAdd between commandsDelay (normdist (2000 )). The Code is as follows:

Http_header_recv ["rdn_on _~ 242 "] 200;/* OK */
Http_nrecv ["rdn_on _~ 243 "] 100% %;/* 9997 bytes */
Stop_time ["page1"];
Delay (normdist (6000,140 00, 2000);/* added to replace think_avg below */
Start_time ["page2"];
/* Set think_avg = 8536;-replaced by delay above */
Set server_connection = www_noblestar_com_1;
/* Keep-alive request over connection www_noblestar_com_1 */
Http_request ["rdn_on _~ 244 "]

Again, noNormdistIf the function is used, an error occurs during script execution. Now this code can be correctly calculated.Page1,Page2Loading time, and readPage1There is a random normal distribution time. If you want to, check all the modified scripts.

5. Advanced Users: Dual-peak Normal Distribution

You should haveRational teststudioPerformance testing experience andCA basic understanding of the language. Many websites I have tested have such pages, and their user visits are normally distributed in dual peaks. Imagine a page with a lot of text. When you access the page for the first time, you may read the text in detail. Next time, you will skip the page directly to the next page. User latency of such pages8.

 

Figure8Dual-peak Normal Distribution of text Web pages

In this example, assume that60% Of users spent8Second to browse this page and find the next link.40% Of users spent45Second to read the entire page in detail. You will find that there are two normal distributions with different minimum, maximum, and standard deviation. Generate0To10And create a control logic when the random number falls1~6The left-side latency is achieved in7~10To the right.

Y = uniform (1, 10);/* randomly select a number
Between 1 and 10 */
If (Y <= 6)/* If 1 through 6 */
{
Delay (normdist (0,16000, 2000);/* Delay Based on left side hump */
}
Else/* If 7 through 10 */
{
Delay (normdist (34000,64000, 6000);/* Delay Based on right side hump */
}
Start_time ["page2"];

This Code Execution1000The next graph will be displayed9.

 

Figure9The dual-peak Normal Distribution generated by our script

As you can see, using the same method, you can use this code to easily change the distribution type and value, and control the different user proportions in each peak value. By putting these principles together to form a variety of combinations, it is possible to simulate the latency distribution of almost all users and achieve a considerable level of accuracy. For example, you can add another10Seconds2Evenly distributed by minutes to simulate other15% Of users, and this possibility is endless.

6. Summary

The key point of this article is that the more accurate the simulated users are, the more reliable the performance test results are. The first step of accurate user simulation is to simulate user latency. This article discusses how to determine the user latency and how to useRational teststudioApply it to your virtual user script. Next2This article describes how to determine and simulate the website usage patterns of individual users and how to integrate these into a single user group model.

7. Appendix (word)

It goes without saying: Needless to say
Make educated guesses: Make reasonable assumptions
In-house experiments: Internal Experiment
In case: in case
Realistic: lifelike
Significantly: Significant and noteworthy
Bell curve: Bell Curve
Negative Exponential: Negative Index
Skew: Skewed
Programmatic: programmatic and planned
In-depth discussion: In-depth discussion
Represent: indicates
Parenthesis/parentheses: parentheses
Jot down: jot down, summary,
Co-worker: collaborators and colleagues
Come up with: Proposed, come up
Built-in: built-in
Reiterate: redo
Bracket: enclosed together, arc, Bracket
Proportion: proportional, balanced, area, partial

Returned topic: [translation] performance tests based on user experience