Research on "Turn" performance test design and LR principle

4 iterations of performance testing , without the need to step hard, the code and framework independently developed from 0 to 10,000 lines of Code test tool (script), as a performance test tool leader LR, I sometimes take his reference, cottage it a lot of things, There are also a lot of questions about why it was tested for performance over a 3-month performance test trip. Here are the detailed questions I came up with comparing test scripts and LR:

1, how to calculate the number of processing packets per second

I've been studying this for a long time. In the case of multi-threaded, when the server is pressed to create a thread to collect this information, or in each thread to achieve this time. Later I took the latter. Because after reaching a bottleneck, this period of time changes is very small but can not be ignored.

For example, the following pseudo code 1:

Eachthread: BeginTime = Time.time () Count = 0 While point: If revpackage () = = true: Count = Count + 1 EndTime = Time.time () Runtime = Begintime–endtime Eachsecondrpackage = float (Count)/float (Runtime) Eachsecondrpackage = SumAll (eachsecondrpackage)

Pseudo Code 2:

count = 0 Eachthread:   &NB sp;     Global Count While point:         If Revpackage () = = true:                    Locking                    count = count + 1                    unlock Tracethread:         time.sleep ( Runtime)         eachsecondrpackage = count/runtime

The first is that each thread takes its own time, and then calculates the received packet per second at a point that is true, and then adds up the package for all threads. The second is that the thread does not do any algorithmic operations. Let the individual thread do it. The first benefit is that the values are accurate and there are no locks that affect performance at the critical point. The second one is accurate on the total execution time, but the lock is used inside. For 2 kinds of general first use more, but if the delay is relatively large package or focus on the whole event flow, the second algorithm is more accurate (note that sometimes the smaller the delay does not mean the greater the pressure). The question I have here is, how does LR set the number for this TPS? Are there 2 combinations or just one?

When it comes to locks, we deal with databases in many performance tests. We certainly want to build n multi-threaded to impact the database (whether the database is not the system being tested), but the database itself can accept the thread is limited, and its limit is very low, although our database operation with the thread lock is possible, but the disadvantage is that if the event stream a lot, create virtual data, and issued a timely command and more concurrent operations, this lock will allow many threads (especially the delay of small threads) will be stuck at the point of an event stream, resulting in a broken socket. It also affects the results of the data (because it is not known whether there are other events that result in errors). The solution is to try not to affect the test case to do the database data first done, real-time database recommendations at some point to do the set point, statistics enough to do the pressure shock. Here we use the concept of the set point of LR, and note that the start and end events of the averaging are to be grasped.

Speaking of databases, if your DB knowledge is not very good, test data LR is a great choice, but in some complex cases we are not applying LR testing to each of these use cases. At this point you need to be very clear about your testing needs. The following pseudo-code:

Python: For I in range (1000): Cursor.execute (SQL);

C++: for (I = 0;i<1000; i++) { Cursor.execute (SQL); }

Sql:
For I in 1.. LOOP
(SQL)
Commit
END LOOP;

The use of PY and C + +, as well as the database itself cycle, 3 kinds of cycle time is not the same. SQL is the fastest, C + + next, and then the PY, the requirements of different systems tested in different ways to test performance, if you directly test the database a stored procedure, you can use SQL to SQL, or other language calls when the loop is SQL, compared to the test product calls SQL, The difference between invoking the product being tested and invoking the database directly is one of the languages. Questions about LR If it tests a lot of SQL performance metrics, how does it solve the problem I mentioned above?

Speaking of loops, every time we finish our test report, the developers always ask the bottleneck is the bottleneck of the product, or the bottleneck of your test script? Therefore, as a Test scripting language is of course preferred, but the scripting language is not high efficiency is weak, so every time in the scripting language to do multi-threaded stress testing, each key loop as far as possible to call C + + and other efficient statements to do, while paying attention to the call time. LR this piece of its practical time biased event flow way to do, so like this abnormal pressure is actually less.

Speaking of multi-threading, this is one of the more places where I study LR. When I write my own scripts, I often delve into the impact of different hardware on the thread utilization of different operating systems , as well as the wired lock, and should not be matched to the queue, process to do the test. Of course the ideal is the more testing machines do distributed, even with the gimbal to do better. But the reality of the situation you do not just consider how many threads to open the number of test machines, but that 100 threads with 1 machines to run, and 10 machines to run the difference, testing product bottlenecks first to test the network, system bottlenecks. If a machine reaches 50 threads and 100 threads have the same throughput, then the best boot thread for this machine is 50. I heard that LR has a queue thread that has processes working together to make concurrent tests. I also design the script according to his load test method, but even the gimbal has the weakness of analyzing the operating system and hardware, assuming that LR is doing 1000 (assuming data) concurrency (regardless of the multi-condition) in a single server, how is it implemented concurrently?

cloud computing One of the concepts is virtualization computing, But it doesn't mean that every test machine uses resources to make a virtual machine, because the most critical thread, the software and operating system of the virtual machine itself, consumes some of the thread's site, so it's prudent to use virtualization computing for testing. Another thing is the performance command Top,ps,sar and so on, you have to pay attention to those useful, which is relatively accurate, although Linux provides a lot of performance commands, but does not mean that they are identical. Of course, LR is also by artificial coordination Analysis network, testing machine performance.

Finally, you analyze and report the test. LR report is very gorgeous, a lot of special performance test nouns are playing a pile, the cutest boss likes to watch this money-earning thing. But the truth-seeking Daniel is not so easy to cheat, the company network, all resources are used to do performance testing must see meaningful things. My script put a lot of Excel charts (interactive call lazy) to help me make a lot of diagrams. Performance testing is the most important analysis, I said many of the techniques are designed to accurately obtain data analysis. So now performance testing is evolving from single-machine to distributed to cloud-to-exact point. Many times with the report to face the development of the various parts of the boss, as a small QC how to put hundreds of megabytes of logs and data out with these experts report need to pay attention to a lot of details. Why is this phase curve not irregular? What is a bottleneck? Is there a problem? Does this data serve as reference data or is it a representative problem? Should the system be optimized? What is the task and program design for the next iteration? All these must be made clear to themselves. Compared to LR, the only advantage is that I know how to capture the data, but it's better to keep trying to narrow the gap than the authority's tools.

The next stage is no longer how to query the bottleneck, how to find the main bug, because agile to the end of the time, I need to become the role of selection engineer, optimize the framework and processing, analysis of the operating system is the main task to save our products cost, is one of the factors in performance testing.

Research on "Turn" performance test design and LR principle