Goreplay (Gor) Golang Flow Copy tool trial

This is a creation in Article, where the information may have evolved or changed. 1. Project Address

https://github.com/buger/goreplay

2. Installation

wget https://github.com/buger/goreplay/releases/download/v0.16.1/gor_0.16.1_x64.tar.gz
tar xvf  gor_0.16.1_x64.tar.gz
cp goreplay /usr/bin

3. Simple to use

// The test environment uses nginx. There is one machine and one reverse proxy. The specific installation configuration is relatively simple.

a. Copy the traffic and output to standard output
Goreplay --input-raw :80 --output-stdout
Response message
GET / HTTP/1.1
Host: XXXXXXXXX
User-Agent: Mozilla/5.0 (Windows NT 6.1; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/62.0.3202.75 Safari/537.36
Accept: text/html, application/xhtml+xml, application/xml; q=0.9, image/webp, image/apng, */*; q=0.8
Accept-Encoding: gzip, deflate
Accept-Language: zh-CN, zh; q=0.9, en; q=0.8
Cache-Control: max-age=0
If-Modified-Since: Sat, 04 Nov 2017 13:52:14 GMT
If-None-Match: W/"59fdc60e-232"
Proxy-Connection: keep-alive
Upgrade-Insecure-Requests: 1
X-Forwarded-For: XXXXXXXXX
X-Lantern-Version: 4.4.0

b. Traffic copy other environment

     Goreplay --input-raw :80 --output-http="http://xxxxx:port"
     
c. The traffic copy is stored as a local file (you can also use s3 storage, open source has minio)
    
     Copy:
     Goreplay --input-raw :80 --output-file=requests.gor
     Replay:
     Goreplay --input-file requests.gor --output-http="http://XXXXXXXX:port"

4. Available Plugins

a. input 

--input-raw - used to capture HTTP traffic, you should specify IP address or 
interface and application port. More about Capturing and replaying traffic.
--input-file - accepts file which previously was recorded using --output-file. 
More about Saving and Replaying from file
--input-tcp - used by Gor aggregation instance if you decided forward traffic 
from multiple forwarder Gor instances to it. 

b. output

--output-http - replay HTTP traffic to given endpoint, accepts base url
--output-file - records incoming traffic to the file. More about Saving and Replaying from file
--output-tcp - forward incoming data to another Gor instance, used in conjunction with --input-tcp. Read more about Aggregator-forwarder setup.
--output-stdout - used for debugging, outputs all data to stdout.

5. Expansion

Rate limiting

a. Limiting replay using absolute number

# staging.server will not get more than ten requests per second
gor --input-tcp :28020 --output-http "http://staging.com|10"

b. Limiting listener using percentage based limiter

# replay server will not get more than 10% of requests 
# useful for high-load environments
gor --input-raw :80 --output-tcp "replay.local:28020|10%"

c. Consistent limiting based on Header or URL param value

# Limit based on header value
gor --input-raw :80 --output-tcp "replay.local:28020|10%" --http-header-limiter "X-API-KEY: 10%"

# Limit based on header value
gor --input-raw :80 --output-tcp "replay.local:28020|10%" --http-param-limiter "api_key: 10%"


Request filtering

a. Allow url regexp

# only forward requests being sent to the /api endpoint
gor --input-raw :8080 --output-http staging.com --http-allow-url /api

b. Disallow url regexp

# only forward requests NOT being sent to the /api... endpoint
gor --input-raw :8080 --output-http staging.com --http-disallow-url /api

c. Filter based on regexp of header

# only forward requests with an api version of 1.0x
gor --input-raw :8080 --output-http staging.com --http-allow-header api-version:^1\.0\d

# only forward requests NOT containing User-Agent header value "Replayed by Gor"
gor --input-raw :8080 --output-http staging.com --http-disallow-header "User-Agent: Replayed by Gor"

d. Filter based on HTTP method

gor --input-raw :80 --output-http "http://staging.server" \
    --http-allow-method GET \
    --http-allow-method OPTIONS

6. References

// wiki 
https://github.com/buger/goreplay/wiki
https://github.com/buger/goreplay
// kafka
https://github.com/buger/goreplay/wiki/Streaming-from-and-to-Apache-Kafka
// Distributed-configuration
https://github.com/buger/goreplay/wiki/Distributed-configuration