Chapter 7: Live Video cloud SDK Performance Test Model and sdk performance test

Chapter 7: Live Video cloud SDK Performance Test Model and sdk performance test

There are a lot of technical articles on live broadcasting, and there are not many systems. We will use seven articles to give a more systematic introduction to the key technologies of live video in all aspects of the current hot season, and help live video entrepreneurs to gain a more comprehensive and in-depth understanding of live video technology, better technology selection.

This series of articles outlines as follows:

(1) Collection

(2) handling

(3) coding and Encapsulation

(4) streaming and transmission

(5) latency Optimization

(6) Principles of Modern players

(7) SDK Performance Test Model

This is the last article in the video broadcast technology detail series. Live Video cloud SDK Performance Test ModelThe SDK performance has a significant impact on the final App. SDK version iterations are fast. A system test must be performed before each release. The test must have consistent behaviors. A performance model should be used as the theoretical basis to quantitatively evaluate the SDK performance. This article discusses the indicators that affect the SDK performance and establishes the corresponding performance model.

 

Important parameters that affect video quality and size

Before testing, we need to specify several parameters that have the greatest impact on the video quality and size: frame rate, bit rate, and resolution.

1) how to set Frame Rate

A frame is a static image, and continuous frames form an animation, such as a TV image. We usually say the number of frames. Simply put, the number of images transmitted in one second can also be understood as the number of refresh times per second by the graphics processor, it is usually expressed by fps (Frames Per Second. Each frame is a static image, and the rapid continuous display of frames forms the illusion of motion. A higher frame rate allows you to get smoother and more realistic animations. The more frames (fps) each second, the smoother the displayed action.

2) how to set the bit rate

First, let's look at the video encoding purpose. It is to transmit videos as clearly as possible with limited bandwidth. We use 25 frames per second as an example, and the GOP group is defined in 25 frames, currently, there are three frame formats: I, B, and P. I frame is a key frame. You can imagine it is a JPEG compressed image, while B, P frames rely on I frames. If I frames, B and P frames are lost, no image is visible. B and P frames describe not actual image pixel content, but the amount of change in each relevant pixel, which is smaller than the amount of information in the I frame. The GOP group refers to the length of the group where a key frame I frame is located. Each GOP group has only one I frame.

Let's take a look at the relationship between the bitstream size of a group of pictures? When video encoding is compressed in the same way and definition requirements are the same, the length format of the GOP group determines the size of the code stream. For example, if the video is 25 frames per second, the length of the GOP group is 5, if the frame format is IBPBP, there will be 5 I frames, 10 B frames, and 10 P frames in one second. If the length of the GOP group is 15, the frame format is IBBPBBPBBPBBPBB, then there will be 2 I frames, 16 B frames, and 7 P frames in one second. Then, five I frames are more informative than two I frames, therefore, the length format of the GOP group also determines the code stream size.

3) How to specify resolution

Resolution Definition Video resolution refers to the size or size of the image produced by the video imaging product. Common video resolutions include 640x1088, X, and X. In the two sets of numbers, the former is the image length, the latter is the image width, and the two are multiplied to obtain the image pixels.

Indicators that affect SDK Performance

With the above knowledge, we can start to prepare and test the SDK performance. First, let's analyze which indicators can reflect the SDK performance, which is divided into two platforms: Android and iOS:

Android

• GC: GC logs can be used to record the frequency and time of Mirror GC and Full GC. Full GC may cause obvious lag and need to be evaluated.
• The UI Loop is the VSync Loop, which reflects the effect of the SDK on App smoothness. In theory, 60 fps is the smoothest value.
• Memory: indicates the Memory usage of the SDK.
• CPU Usage: indicates the size of computing resources occupied by the SDK.

IOS

• UI Loop: reflects the effect of SDK on App smoothness. Theoretically, 60 fps is the smoothest value.
• Memory: indicates the Memory usage of the SDK.
• CPU Usage: indicates the size of computing resources occupied by the SDK.

In addition to some of the above system-level indicators, the following are some of the unique indicators of the live video SDK. These indicators can reflect the core competitiveness of the SDK and some major differences, the definition and smoothness of videos can also be quantified.

1) video definition indicators

• Frame Rate
• Bit Rate
• Resolution
• Quantization parameter (compression ratio)

2) indicators that affect video Smoothness

• Bit Rate
• Frame Rate

3) Other important indicators

Live streaming is a large consumer of traffic and performance. It has some indicators that directly affect users' feelings. We also need to pay attention to them:

• Power Consumption
• Fever (it is difficult to quantify. In most cases, fever is directly proportional to power consumption, which can be temporarily replaced by power consumption)

Test Plan

Some test conditions need to be fixed during the test process, and then the test results are obtained based on different test conditions. Here, two of the most common conditions are selected, it is the statistical data we get from a large number of customers, which can reflect the scenarios where most live video applications are located. It mainly limits the resolution, video processing, bit rate, and network environment.
There are two types of test indicators: objective and subjective. The former reflects the consumption of the SDK to the system, however, although it is an objective indicator, it does not mean that it has no impact on the user, but that the resulting user experience is not obvious. Subjective indicators will directly affect the end user experience, but they are easy to be ignored in traditional tests. Because they are difficult to quantify, some quantitative methods are proposed here, hope to arouse the reader's thinking.

Test Condition

• Resolution: 480 p
• No watermarks, no beautify
• Bit Rate 1 M
• The Network guarantee is between 0.5 Mb and MB ~ 2 M

This condition reflects the usage scenarios of most low-speed networks and the basic performance of the SDK. It can be used as a benchmark test for SDK streaming and pulling, do not introduce too many test dependencies.

Test Condition B

• Resolution: 720 p
• No watermark, with beautify
• Bit Rate 1 M
• The Network guarantee is between 0.5 Mb and MB ~ 2 M

This condition reflects the application scenarios of most customers. It has a high resolution and beautify video processing, and can be used as an important basis for analysis of SDK competing products. The test results are very close to the real scenarios.

1) Objective indicator test plan
Indicators that objectively affect App stability and performance:

• Memory
• Test for 10 minutes, memory curve
• 1 hour test, TP99, TP95, TP90, archiving required
• Test for 1 hour, memory increment, check whether there is memory leakage, need to archive
• Reference Value: previous result
• CPU
• Test the CPU Usage curve for 10 minutes
• Testing for 10 minutes, TP99, TP95, and TP90, archiving required
• Reference Value: previous result
• Bit Rate
• Test for 10 minutes, TP99, TP95, and TP90. It is important to note that the bit rate control here needs to be separated. If network jitter causes lower bit rate, such points are not included, only test SDK Bit Rate Control and archive
• Reference Value: 1 M (all sizes are deviations)
• Power Consumption
• Test for one hour, record the total power consumption of processes, display power consumption on the screen, CPU power consumption, and archive
• Reference Value: previous result

2) subjective indicator test plan
Subjective indicators that affect App users:

• The maximum frame rate that the UI Loop App can achieve. Unlike the video frame rate, the reason for statistics is that our SDK may affect the smoothness of the entire App, which needs to be tracked.
• Test for 10 minutes, UI Loop Curve
• Test for 10 minutes. UI Loop TP99, TP95, and TP90 must be archived and compared repeatedly.
• Reference Value: 60 fps
• Android GC
• 1-hour test, record the frequency of Mirror GC and Full GC, record the TP99, TP95, and TP90 of GC duration, and archive and compare them
• Reference Value: previous result
• Frame Rate (fps)
• Test for 10 minutes, TP99, TP95, and TP90, which need to be archived and compared repeatedly
• Reference Value: 30 fps
• Comparing the video definition indicators with the SNR
• The test lasts for 10 minutes. You need to archive and compare the results. You can use a fixed video as the input.
• Reference Value: previous result

3) Result Display

• Table display metrics
• Curve displays raw data and timeline data
• Deviation between heat map display and reference value
• The heat map shows whether the last archive value is improved or deteriorated.

Through this automated and systematic testing of repeated iterations, we have worked out the SDK performance with a highly paranoid mind, only to bring the best live broadcast experience to end users, to help our customers maximize their business value through the media of the secondary age, we hope that we will always be with you on your way.

Author: qiniu cloud evangelist @ Buhe,Original article can goQiniu cloud official blogView.