Linux Traffic Replication Tool

For some businesses with concurrent requirements, especially when docking external traffic, the product must be done before the launch is a stress test, but the normal pressure test can not cover all situations. Take Gemeter, AB,, Webbench, http_load for example, these through the simulation request of the pressure test tool, can only send specific parameters, for some parameter exception caused by the program processing exception is not considered, so need a can copy real traffic, and do not affect the online business tools.
There are many traffic replication tools, such as Gor, Tcpreplay, tcpcopy, and so on, these tools fit the real scene, can simulate the real traffic, and support the amplification or reduction of traffic, more easily test out the bottleneck and potential problems of the program.

Several traffic replication tools:

1. Gor:https://github.com/buger/goreplay
2. Tcpreplay:https://github.com/appneta/tcpreplay
3. Tcpcopy:https://github.com/session-replay-tools/tcpcopy
4. Nginx module Ngx_http_mirror_module, introduced in Nginx 1.13.4, please check the Nginx version before use

Nginx Module Ngx_http_mirror_module

The configuration is as follows:

server {    listen 8080;    access_log /home/work/log/nginx/org.log;    root html/org;}server {    listen 8081;    access_log /home/work/log/nginx/mir.log ;    root html/mir;}upstream backend {    server 127.0.0.1:8080;}upstream test_backend {    server 127.0.0.1:8081;}server {    listen       80;    server_name  localhost;    #charset koi8-r;    #access_log  logs/host.access.log  main;    location / {        mirror /mirror;        proxy_pass http://backend;    }    location /mirror {        internal;        proxy_pass http://test_backend$request_uri;    }}

Flow amplification, configuration of two mirror can be

location / {        mirror /mirror;        mirror /mirror;        proxy_pass http://backend;    }

Use is very convenient, but the online nginx generally carry more than one business, modified Nginx configuration need nginx -s reload to make it effective, this operation on-line or as far as possible to avoid.

Gor Https://github.com/buger/goreplay

Gor Overview

Gor is an HTTP real-time traffic replication tool written in Golang. More powerful, support traffic amplification, reduction, frequency limit, also support to log requests to files, easy playback and analysis, but also support and ElasticSearch integration, the traffic into ES for real-time analysis.

Download the installation, you can download the compiled binary files directly using

> wget https://github.com/buger/goreplay/releases/download/v0.16.1/gor_0.16.1_x64.tar.gz> tar xzvf gor_0.16.1_x64.tar.gz

Traffic copied to file

You can copy the traffic to a file, and then replay them. During playback, the traffic is maintained at the original time interval. If you use percentages for rate limiting, the rate of playback will increase or decrease accordingly. With this rate limit, Gor can be used for stress testing.

#write to filegor --input-raw :80 --output-file requests_origin.gor#read from filegor --input-file requests_origin.gor --output-http "http://localhost:8081"

You can use a timestamp to name the recording file, by default, the file is stored by "block", that is, after the file size reaches the upper limit, add a suffix, and create another file, as follows

gor --input-raw :80 --output-file %Y%m%d.gor#append false20140608_0.gor20140608_1.gor20140609_0.gor20140609_1.gor

The default is to press "block" to store the file, but you can configure the parameters,--output-file-append, after use as follows

gor --input-raw :80 --output-file %Y%m%d.gor --output-file-append#append true20140608.gor20140609.gor

Configuration description of the time format file name:

%Y: year including the century (at least 4 digits)%m: month of the year (01..12)%d: Day of the month (01..31)%H: Hour of the day, 24-hour clock (00..23)%M: Minute of the hour (00..59)%S: Second of the minute (00..60)默认格式是%Y%m%d%H

Traffic replay

Currently, this method only supports "Input-file", and the playback rate can only be controlled in percent. Please note that this playback rate ratio is relative to input. That is, the time stamp of the recorded traffic is played back.

Playback at twice times the rate
gor --input-file "requests_origin.gor|200%" --output-http "http://localhost:8081"

If "Input-flie" is multiple files, you can use regular to match,
gor --input-file "requests_origin*.gor|200%" --output-http "http://localhost:8081"

With the following configuration parameters, you can perform a better stress test
--input-file-looprepeating loop execution Input-file
--exit-after 30sAfter 30s stop, you can control the time of the stress test. The unit of the minute is M

Gor Common commands

Simple HTTP traffic Replication
> gor --input-raw :80 --output-http "http://localhost:8081"

HTTP traffic Replication Frequency control (get more than 10 requests per second)
> gor --input-tcp :28020 --output-http "http://localhost:8081|10"

HTTP traffic replication shrinks
> gor --input-raw :80 --output-tcp "http://localhost:8081|10%"

HTTP traffic logged to local file
> gor --input-raw :80 --output-file requests_origin.gor

HTTP traffic playback and pressure measurement
> gor --input-file "requests_origin.gor|200%" --output-http "http://localhost:8081"

HTTP Traffic filtering replication

> gor --input-raw :8080 --output-http http://localhost:8081 --output-http-url-regexp ^www.

Customizing parameters for some traffic replication

> gor --input-raw :80 --output-http http://localhost:8081 --http-allow-method POST --http-set-header 'User-Agent: Gor' -output-http-workers=1 -http-allow-url test.php

Replicate two copies of traffic to different test services

> gor --input-tcp :8080 --output-http "http://localhost:8081" --output-http "http://localhost:8082"

Distribute traffic to different servers like load balancer

> gor --input-tcp :8080 --output-http "http://localhost:8081" --output-http "http://localhost:8082" --split-output true

Gor configuration parameters

> Gor--help-http-allow-header valuegor--input-raw:8080--output-http localhost:8081--http-allow-header api-version:v1.1 uses a regular expression to match the HTTP header, and if the requested header is not matched, it is rejected-http-allow-method Valuegor--input-raw:8080--output-http localhost:8081--http-allow-method get is similar to a white-list mechanism to allow HTTP request methods to be passed, except that the methods are rejected.-http-allow-url Valuegor--input-raw : 8080--output-http localhost:8081--http-allow-url ^www A regular expression used to match the URL to filter the exact URL of the match, Outside of this is filtered out-http-disallow-header Valuegor--input-raw:8080--output-http localhost:8081--http-disallow-header " User-agent:replayed by Gor "with a regular expression to match the HTTP header, the matching request will be rejected-http-disallow-url Valuegor--input-raw:8080--output-http localhost:8081--http-disallow-url ^www uses a regular expression to match the URL, and if the request matches, it is rejected-http-set-header Valuegor--input-raw:8080-- Output-http localhost:8081--http-set-header ' User-agent:gor ' Set header information, if already exists will overwrite-http-set-param Valuegor--input-raw : 8080--output-http localhost:8081--http-set-param api_key=v1.1 Set request parameters, if already exists will overwrite more parameters please consult the official documentation https://github.com/ Buger/goreplaY/wiki