Several common solutions for simulating harsh network environments

first, using Fiddler to simulate the poor network environment

 

In solving the daily support needs, often encounter some user feedback some can not easily reproduce the bug, a large part of the bug is due to the user's own network environment fluctuations, or its own network environment is relatively poor, and services in the face of this poor network environment robustness is not enough, Cause some unexpected bugs to occur. In the normal development self-test process, it is difficult to create such a bad network environment, making these bugs more difficult to detect and repair in advance. Other services in the poor network environment, although the situation will not be unavailable, but the user physical examination is very poor, in order to optimize the user experience in this situation, but also to local simulation of this environment for tuning.
So to reproduce these bugs, or even to find these bugs in advance, you need to be able to simulate a bad network environment in the development environment, so as to see the performance of services in this poor network environment. The current simulated bad network environment can be achieved by the following means: through the application layer or Transport layer Proxy server, through the proxy server set up some parameters to simulate the bad network environment, so that the traffic through these proxy server is converted into a bad network environment traffic. such as the use of Fiddler,charles and other proxy server functions of network traffic analysis software to achieve. Through the use of a number of lower-level drive services, through the control network card packet delivery behavior, to simulate the poor network environment. such as Dummynet's IPFW drive. By setting up a controllable gateway to deploy a program that simulates a harsh environment on the gateway, all traffic that needs to be forwarded through the gateways is simulated as bad network conditions. This kind of support is provided by Netem under Linux.

Here is the first approach, that is, the use of fiddler to simulate the poor network environment, the service testing, this means to achieve simple, more intuitive, but the disadvantage is only to support those who use HTTP for communication and interaction services. The latter two methods will be further described in the following article. "What's fiddler?"

Fiddler's official Website describes its own: the free web debugging proxy for any browser, system or platform, which is a cross-browser, cross-system, cross-platform, Web debug proxy server. When your HTTP browsing is fiddler, fiddler can monitor traffic, view various information about HTTP traffic, set breakpoints to view and modify HTTP data, and even construct various test HTTP packages and replay recorded packets. Its official website is http://www.fiddler2.com/fiddler2/, the above detailed description of fiddler exactly what. "Simply use fiddler speed limit to simulate harsh network environments"

The fiddler itself has been provisioned to provide an option to simulate modem speed in the following locations:
Rules–performances–simulate Modem Speeds

Tick this option, all traffic through the fiddler agent will become and years ago 56k Kitten when the internet generally slow.
Since fiddler is just an HTTP proxy, to visually see the speed limit effect, preferably run in the browser of the speed tool, here choose the speedtest.net provided by the speed measurement tool to test.
The first is the speed before the option is turned on:

After opening the simulate Modem speeds:

Speed has gone back to the unbearable low speed, notice here the ping value also has a significant increase, and in fact, the ping value is the ICMP layer control message, and will not be affected by fiddler, theoretically, the ping value will not be improved, Further analysis of the messages in the Fiddler can be seen in the clues:

In fact, the Web plugin does not implement the ability to send ICMP packets and get ping values, but instead uses the response time of the smaller HTTP GET requests to calculate the ping value, which is actually calculated as an average HTTP RTT value, So it is normal to be affected by the fiddler simulated harsh environment. "Tuning parameters for simulating harsh network environments"

Direct simulation modem speed is really slow explosion, in fact, in the case of poor signal, mobile phone network speed has exceeded the 56k modem speed, so using the default configuration simulation out of the environment is too bad, and does not necessarily meet the requirements, at this time, the need to adjust the speed limit parameters.
Fiddler itself provides a configuration file for adjusting these parameters, click:
Rules–customize Rules ...
Will be opened with a text editor Customrules.js file, the default is located in the user directory of the document directory \fiddler2\scripts location, the suffix is JS, the content is jscript.net--Microsoft to the ECMAScript specification implementation, JavaScript is under the same specification as the daily use, but there are some differences in the implementation of the details of the extension.
After you open the file, you can find a M_simulatemodem flag bit:

if (m_simulatemodem) {
    //Delay sends by 300ms per KB uploaded.
    osession["Request-trickle-delay"] = "a";
    Delay receives by 150ms per KB downloaded.
    osession["Response-trickle-delay"] = "Max"
}

This flag bit controls the setting of the osession two parameter values, when the simulate Modem speeds is checked, request-trickle-delay and Response-trickle-delay are set, Where the value in Request-trickle-delay indicates how many milliseconds will be delayed when the data per kb is uploaded, and how many milliseconds the data per KB will be delayed when the response-trickle-delay is downloaded, and if its speed is already quite fast, The value set here can approximate the upload and download bandwidth after opening the simulation, such as the default setting of the download delay of 150ms, upload delay of 300ms, corresponding to calculate the approximate analog bandwidth:

Upload bandwidth = (1*8/1000)/0.300≈0.053mbps
Download bandwidth = (1*8/1000)/0.150≈0.027mbps

However, the actual situation is twice the bandwidth of this value, presumably may be fiddler internal implementation of some and description of the difference, why this phenomenon is not very clear, so the above formula finally need to revise a 2.0 coefficient, namely:

Upload bandwidth = ((1*8/1000)/0.300) *2.0≈0.106mbps
Download bandwidth = ((1*8/1000)/0.150) *2.0≈0.053mbps

Assuming that we set two parameters to 50, we will get the load bandwidth is 0.32Mbps, the speed results are as follows:
"Writing Custom Scripts"

Further, we can extend the logic in the customrules.js, referring to JScript's documentation to incorporate more custom logic into the simulated harsh environment, where a random delay setting is set so that the network bandwidth is not constant to a low-speed value, but is randomly dithered within a certain range:

static function Randint (min, max) {
    return Math.Round (Math.random () * (max-min) +min);
}
if (m_simulatemodem) {
    //Delay sends by 300ms per KB uploaded.
    osession["Request-trickle-delay"] = "" +randint (1,50);
    Delay receives by 150ms per KB downloaded.
    osession["Response-trickle-delay"] = "" +randint (1,50);
}

1 2 3 4 5 6 7 8 9

The resulting test results are as follows:

The trend graph of the instantaneous velocity during the speed measurement is as follows:

You can see that the overall network speed limit has a certain degree of jitter.
By further expanding the custionrules.js can achieve a lot of harsh environment simulation scenarios, if the scene is more complex, you can also write the Fiddler plug-in Way, write C # plug-in code to further control the behavior of fiddler, here do not do more to repeat. Details can be referred to: Http://docs.telerik.com/fiddler/extend-fiddler/extendwithdotnet "Fiddler Simulation of the limitations of the harsh network environment"

Fiddler speed limit is relatively simple and flexible, configuration is more convenient, but because it is an application layer of HTTP proxy, can only simulate the behavior on the layer, for some complex network layer of packet loss, retransmission and other bad situations can not be very good simulation, and for the application of other protocols are not supported, Some other methods and software for simulating harsh environments will be introduced to compensate for these deficiencies.

second, using dummynet to simulate the poor network environment simulating harsh network environments with Dummynet

The previous article mentioned three ways to simulate the debug code in a bad network environment: the application layer or the Transport layer's proxy server transport layer or the Network Layer control Packet's gateway to the drive Network layer control packet

At the same time, in the previous article introduced the first method, that is, using the application layer of HTTP proxy fiddler to simulate the poor network environment, this method is simple and flexible, but it is in the application layer, limited, and there is no way from the bandwidth and delay two aspects to fine-grained to the harsh network environment to simulate, Here is the second method of--dummynet. dummynet Introduction

Dummynet's official website address is: http://info.iet.unipi.it/~luigi/dummynet/, official online description of Dummynet is this: Dummynet is a live network Emulation Tool, originally designed for testing networking protocols, and since then used for a variety of applicatio NS including bandwidth management. That is, Dummynet is a real-time network simulation tool, in fact dummynet is part of IPFW firewall, IPFW is a network layer of firewall, and built in FreeBSD.
Using the options dummynet option of IPFW, you can set up a series of pipes to control the network traffic. dummynet Fundamentals

As shown in the image provided on the official web site, dummynet controls network traffic by establishing a pipe in two network layers (typically between the transport and application layers, or between the network layer and the Transport Layer), and the network traffic in accordance with the set rules is introduced into these pipelines. This allows the dummynet to intervene in these flows, control bandwidth, delay, and even further control the loss of the packet rate and many other parameters. dummynet installation (Windows)

The

dummynet itself as part of the IPFW firewall, built on FreeBSD itself, which natively supports the installation and use of freebsd,osx,linux,windows multiple operating systems, Under OSX and Linux download source compiled installation generally can be used, here is the main introduction of Windows Dummynet installation.  
Dummynet exists as a service driver on a network card under Windows, and the driver's SYS and INF files are already included in the official binary package, but the driver does not contain a digital signature. When you do not introduce a Windows operating system (such as Windows XP) that drives a forced signature mechanism, you can complete the Ipfw+dummynet service-driven installation by downloading to the latest binary package directly to the official website and then following the steps to install it. However, it is not possible to install the Ipfw+dummynet service driver on an operating system that has introduced a drive-forcing signature mechanism (typically Windows 7 later, or a more stringent Group Policy setting), and there are two scenarios for solving this problem: Open the test mode of windows, turn off the drive force signing mechanism, and then install the unsigned driver  
This way of implementation also has two ways, one is to start Windows when the startup option to turn off the driver forced signature Check mechanism, you can refer to this article  http ://jingyan.baidu.com/article/7c6fb42879543380642c9036.html , the other way is to enter the test mode of Windows, you can refer to  http:// The description in the jingyan.baidu.com/article/acf728fd21c3e7f8e510a3ef.html  article. Self-signed or available signature   for unsigned ipfw+dummynet service drivers;
How to self-sign or make available signatures refer to the instructions on MSDN  https://msdn.microsoft.com/en-us /library/windows/hardware/ff544865 (v=vs.85). aspx , how the signature is not within the scope of this article.  
This article mainly uses the second way, for the Ipfw+dummynet service driver to sign the Baidu signature, can install on Windows 7 normally.

The installation is as follows: Find the network card you need to limit bandwidth and traffic, enter the properties:
Click Install below, select Services, then select Install from disk and locate the Ipfw+dummynet service-driven INF installation file in the browse:
After "This connection uses the following items" see Ipfw+dummynet, that is, on behalf of the Ipfw+dummynet Service driver installation is successful, if the installation is successful after encountering the network card can not be used, it is generally the previous uninstall did not uninstall completely, restart the computer can be resolved.
After the installation is complete, you can run Testme.bat under the binary directory with administrator privileges, and the driver installation is successful if there are no alarm messages or error messages in the running results:
simulating harsh network environments with Dummynet

With the dummynet pipe, you can set some specific rules that can be used to simulate a bad network environment, and these settings are transparent and automatic for applications running in the operating system, and do not need to set up an HTTP proxy like when using Fiddler for impersonation. Low bandwidth simulation
The most basic simulation method of the bad network environment is to simulate the situation under the small bandwidth, using the pipe setting in the dummynet to limit the bandwidth of all TCP and UDP streams to a certain level, and run an administrator's command line, The pipe in the dummynet can be set using the Ipfw.exe file in binary directory and execute the following command on the command line:
IPFW Add pipe 2 in Proto TCP
IPFW Pipe 2 config bw 2mbit/s
TCP downstream traffic can be limited to 2mbit/s bandwidth, where the same speed as used in the previous article speedtest.net to verify the effect of limiting bandwidth, before executing the following instructions before the results are as follows:

After the execution of the above instructions, the results of the velocimetry are as follows:
High-latency simulations
Execute the following command on the command line for administrator privileges:
IPFW Add pipe IP from any to any
IPFW Pipe 200 Config delay
This causes all packages to be delayed by 200ms, where the ping command is used to ping Baidu directly to test the delay, and the command executes the previous results as follows:

After the command executes, the results are as follows:
High packet loss rate simulation
Execute the following command on the command line for administrator privileges:
IPFW Add pipe IP from any to any
IPFW Pipe 0.1 Config PLR
This is set to 10% of the packet loss rate, the use of PING-T instructions to continue to ping Baidu, to observe the change in packet loss rate:
Prior to execution:

After execution:
More granular settings
It can be seen that compared to fiddler, the use of dummynet can be more refined from all angles to set a bad network environment parameters, the above mentioned in the bandwidth, delay and loss of the rate of the simulation of course can be combined to use, At the same time, the pipe rules can be set up to further elaborate rules, such as the specified IP segment, the specified protocol, such as this section of the directive:
IPFW Add pipe 4 src-ip 10.1.2.0/24 in
IPFW Pipe 4 config BW 1mbit/s delay 123 PLR 0.1
A pipe that only involves a packet from the 10.1.2.0/24 segment of the source IP, and this pipe limits the bandwidth of the 1mbit/s, the delay is 123ms, and has a 10% packet loss rate, which, by combining these parameters, can simulate a very complex network environment, So as to meet the needs of development and testing. The description and setting method of the relevant setting parameters can refer to the relevant documents and other documentation of the official website.

Finally, when you are done simulating a bad network environment, execute the following command:
IPFW-Q Flush
IPFW-Q Pipe Flush
To clear all the set pipe and rules, further if you need to restore the original environment, you can remove the driver and restart the computer.

three, clumsy 0.2 analog speed

Clumsy can artificially create unstable network conditions under the Windows platform, allowing you to debug your application's performance in extreme network conditions. Introduction

Using the Windivert Library of the encapsulated Winodws Filtering Platform, the clumsy can intercept the network packets received and sent by the system in real time, causing delay, substitution and tampering. Whether you are trying to reproduce a program error caused by a network exception, or evaluating your application's performance in a bad network situation, clumsy allows you to achieve the desired effect at the system level without the need to add additional code:

Features: Download ready to use, no need to install anything. No additional setup required, no need to modify the code of your program. System-level network control, which can be applied to any Windows application such as command line, graphical interface, etc. Not only HTTP is supported, any TCP, UDP network connection can be processed. Support for local debugging (both server and client are in localhost) "Hot swap", your program can run all the time, and clumsy can be turned on and off at any moment. Real-time adjustment of various parameters, detailed control network situation. Example

The following animation shows the case of clumsy acting on a local netcat UDP server/client. Careful observation you can see that the data according to the influence of the clumsy has produced corresponding changes. If you have a basic idea of what clumsy is for, you may want to download it by selecting the appropriate version for your system on the downloads page.

More Information

Clumsy first intercepts the specified network data based on the filter selected by the user. In filter, you can set the protocol (TCP/UDP) you are interested in, the port number, whether it is received or issued. You can also narrow the scope further by using simple logic statements. When clumsy is activated, only network data that meets these criteria will be processed, and data that you are not interested in will still be transmitted by the system normally.

When the filter network packet is intercepted, you can choose the function provided by clumsy to purposefully adjust the network situation: delay (lag), the packet is cached for a period of time before the issue, so as to simulate the network latency situation. Drop, randomly discarding some data. Throttling (throttle), which intercepts the data for a short period of time and sends it together at the same time. Resend (Duplicate), randomly copying some data and sending it with itself. Disorderly order (out of order), disrupting the sequence in which packets are sent. Tampering (Tamper), randomly modifying a small portion of the package content.

Although the current broadband network connection is very popular, but the network transmission itself is not always stable in nature. If your application does not handle a variety of situations, it is possible that a missing UDP package will cause your program to crash. Proper debugging of such behavior clearly requires careful design and processing of the code structure, and it can be a lot of effort. And in some tightly encapsulated development environments (Unity3d's own network library may be an example), it can be more cumbersome. Clumsy the goal of minimizing the burden on programmers, hoping to provide a simple (but not perfect) solution.