Web traffic replication and stress testing tools Gor

This is a creation in Article, where the information may have evolved or changed.

Gor Overview
Gor is an HTTP real-time traffic replication tool written in Golang. Simply executing a process on the LB or Varnish Portal Server allows you to copy the traffic from the production environment to any location, such as the Staging environment and the DEV environment. Perfectly solves the problem of HTTP layer real-time traffic replication and stress testing.

Image.png

Features of the Gor
Gor supports the amplification and reduction of traffic and the frequency limit so that it does not require a consistent server cluster for the build and production environment to be tested correctly. Gor also supports filtering traffic based on regular expressions, which means that an API service can be tested separately. You can also modify HTTP request headers, such as replacing user-agent, or adding some HTTP headers.
Gor can also log requests to files for playback and analysis. Gor supports and ElasticSearch integration, storing traffic in ES for real-time analysis.
Installation of Gor
Binary version
The official precompiled version is simple to use and is available out of the box.
Download the latest version from the following address
Https://github.com/buger/gor/releases
Linux

$ wget https://github.com/buger/gor/releases/download/v0.12.1/gor_0.12.1_x64.tar.gz$ tar xzvf gor_0.12.1_x64.tar.gz$ cp gor /usr/local/bin

Mac

$ wget https://github.com/buger/gor/releases/download/v0.12.1/gor_0.12.1_mac.tar.gz$ tar xzvf gor_0.12.1_mac.tar.gz

Compiling the installation
Build a standard go locale with reference to http://golang.org/doc/code.html
and set the $GOPATH environment variable
Get Source code

$ go get github.com/buger/gor

Compile

$ cd $GOPATH/src/github.com/buger/gor$ go build

Compiling completes results in a Gor binary file.
Gor Practice
Performance testing
You can copy the traffic to a file, and then replay them. During playback, the traffic is maintained at the original time interval. If you use percentages for rate limiting, the rate of playback will increase or decrease accordingly. With this rate limit, Gor can be used for stress testing.

#write to filegor --input-raw :80 --output-file requests.log#read from filegor --input-file requests.gor --output-http"http://staging.com"

• traffic is copied to a file
  You can use the timestamp to name the recording file, by default, the file is stored by "block", that is, after the file size reaches the upper limit, add a suffix, and create another file, as follows

    gor ...--output-file%y%m% D.log#append false20140608_0.log20140608_1.log20140609_0.log20140609_1.log  

  defaults to how the file is stored by "block", but can be configured with parameters, --output-file-append, use the following

    Gor ...--output-file%y%m%d.log--output-file-append#append True20140608.log20140609.log  

  The configuration description for the time format file name:

   %y:year including the century (at least 4 digits)%m  : Month of the Year (01..12)%d:day of the Month (01..31)%h:hour of the day, 24-hour clock (00..23)%m:minute of the Hour (00..59)%s:second of the Minute (00..60)  

  The default format is%y%m%d%h
• Traffic replay
  Currently, this method only supports "Input-file", and the playback rate can only be controlled in percent. Please note that this playback rate ratio is relative to input. That is, the time stamp of the recorded traffic is played back.

  # Replay from file on 2x speedgor --input-file "requests.gor|200%" --output-http "staging.com"

  The above command example is played back at a rate of twice times. If "Input-flie" is multiple files, you can use regular to match, such as "request*.gor|200%"
  With the following configuration parameters, can better reflect the effect of the pressure test.
  --input-file-loop repeating loop execution input-file
  The--exit-after 30s stops after 30s and can control the time of the stress test. The unit of the minute is M
  Performance testing is a common use for Gor, and other common commands refer to "Gor Common commands"

Gor common Commands
Simple HTTP traffic Replication
$ gor --input-raw :80 --output-http "http://staging.com"
HTTP traffic Replication Frequency control (get more than 10 requests per second)
$ gor --input-tcp :28020 --output-http "http://staging.com|10"
HTTP traffic replication shrinks
$ gor --input-raw :80 --output-tcp "replay.local:28020|10%"
HTTP traffic logged to local file
$ gor --input-raw :80 --output-file requests.gor
HTTP traffic playback and pressure measurement
$ gor --input-file "requests.gor|200%" --output-http "staging.com"
HTTP Traffic filtering replication
$ gor --input-raw :8080 --output-http staging.com --output-http-url-regexp ^www.
Customizing parameters for some traffic replication
$ gor --input-raw :80 --output-http 192.168.2.6:8000 --http-allow-method POST --http-set-header 'User-Agent: Gor' -output-http-workers=1 -http-allow-url test.php
Replicate two copies of traffic to different test services
$ gor --input-tcp :28020 --output-http "http://staging.com" --output-http "http://dev.com"
Distribute traffic to different servers like load balancer
$ gor --input-tcp :28020 --output-http "http://staging.com" --output-http "http://dev.com" --split-output true
Gor configuration Parameters

$ gor--help-cpuprofile stringwrite CPU profile to File-debug verbose turn on debug mode, showing traffic to all interfaces-http-allow-header Value matches the HTTP header with a regular expression, and if the requested header is not matched, it is rejected Gor--input-raw:8080--output-http staging.com--http-allow-header API-VERSION:^V1 (default [])-http-allow-method value is similar to a white-list mechanism to allow the HTTP request method to be passed, except that the method is rejected. Gor--input-raw:8080-- Output-http staging.com--http-allow-method GET--http-allow-method OPTIONS (default [])-http-allow-url Value A regular expression is used to match the URL to filter the exact URL, except that it is filtered out of Gor--input-raw:8080--output-http staging.com--http-allow-url ^www. (default [])-http-disallow-header value matches the HTTP header with a regular expression, and the matching request is rejected Gor--input-raw:8080--output-http staging.com- -http-disallow-header "user-agent:replayed by Gor" (default [])-http-disallow-url value matches the URL with a regular expression, if the request matches, Will be rejected Gor--input-raw:8080--output-http staging.com--http-disallow-url ^www. (default [])-http-header-limiter value read request, rejecting a certain percentage of special requests based on FNV32-1A Hash gor--input-raw:8080--output-http staging.com-- Http-header-imiter User-id:25% (default [])-http-original-host in the output of--output-http, usually Gor uses the HTTP header that supersedes the request, so you should disable this option to preserve the original host header-http-param-limiter  Valuetakes a fraction of requests, consistently taking or rejecting a request based on the FNV32-1A hash of a specific GET Param:gor--input-raw:8080--output-http staging.com--http-param-limiter user_id:25% (default [])-http-rewrite-url Valuerewrite the request URL based on a mapping:gor--input-raw:8080--output-http staging.com--http-rewrite-url/v1/use r/([^\/]+)/ping:/v2/user/$1/ping (default [])-http-set-header valueinject additional headers to HTTP Reqest:gor-- input-raw:8080--output-http staging.com--http-set-header ' User-agent:gor ' (default [])-http-set-param ValueSet Request URL param, if param already exists it'll be Overwritten:gor--input-raw:8080--output-http staging.com--http-s Et-param Api_key=1 (default [])-input-dummy valueused for testing outputs. Emits ' Get/' request every 1s (default [])-input-file value reads request Gor--input-file from a file./requests.gor --output-http staging.com (default [])-input-http value read request from an HTTP interface # Listen for HTTP on 9000gor--input-http:9000--out Put-http staging.com (default [])-input-raw valuecapture traffic from given ports (use raw sockets and require *sudo* acces s): # Capture traffic from 8080 Portgor--input-raw:8080--output-http staging.com (default [])-input-tcp value used to stream between multiple Gor Turn traffic # Receive requests from other Gor instances on 28020 port, and redirect output to Staginggor--input-tcp:28020--OUTPU T-http staging.com (default [])-memprofile stringwrite memory profile to the File-middleware stringused for modifying tra Ffic using external command-output-dummy value is used to test the input and print out the received data. (default [])-output-file value writes the incoming request to a file Gor--input-raw:80--output-file./requests.gor (default [])-output-http Value forwards incoming request to an HTTP address on # Redirect all incoming requests to staging.com Addressgor--input-raw:80--output-http Http://sta Ging.com (default [])-output-http-elasticsearch string sends the request and response status to Elasticsearch:gor--input-raw:8080--output-http staging.com--output-http-elasticsearch ' es_host:api_port/index_ The name '-output-http-redirects int sets how many redirects are allowed-output-http-stats to output the status of the output queue every 5 seconds-output-http-timeout Duration Specifies the Request/response timeout for HTTP, which defaults to 5 seconds-output-http-workers Intgor By default is the number of dynamic extension workers, you can also specify a fixed number of workers-output-tcp Value is used to flow traffic between multiple Gor # Listen for requests on ports and forward them to other Gor instance on 28020 Portgor--input-raw:8 0--output-tcp replay.local:28020 (default [])-output-tcp-stats reports the status of the TCP output queue once every 5 seconds-split-output trueby default each Output gets same traffic. If set to true it splits traffic equally among all outputs.-stats open the status of the output queue-verboseturn on more verbose output