Processing millions requests per minute with Golang

This is a creation in Article, where the information may have evolved or changed.

Translate the original link reprint/reprint please indicate the source

Original link @medium.com published on 2017/08/30

I have worked in anti-spam, anti-virus and anti-malware fields for 15 years and have worked in several companies before and after. I know that these systems will eventually become very complex to deal with massive amounts of data.

I am now the CEO of Smsjunk.com and the chief architect of KnowBe4. Both companies are very active in the area of cyber security.

Interestingly, in the past 10 years as a yard farm, all of the websites I have experienced have been used for almost all of the background development Ruby on Rails . Don't get me wrong, I like Ruby on Rails it and think it's a great development environment. Often after a while, you start to design the system in ruby fashion. At this point you forget to take advantage of multi-threading, parallel, fast execution (fast executions) and small memory overhead (small overhead), the software architecture becomes simple and efficient. For many years, I have been C/C++ , Delphi and C# the developer. I'm beginning to realize that working with the right tools can be a lot easier.

I'm not very enthusiastic about language and frameworks. I believe efficiency, output, and maintainability of code depend on how you architect a simple solution.

Problem

In developing our anonymous telemetry and analytics system, our goal is to be able to handle a large number of post requests from the millions of endpoints. The HTTP request handler receives a JSON document containing many loads (payloads). These loads (payloads) need to be written to Amazon S3 and then processed by the map-reduce system.

Typically we create a worker pool schema (Worker-tier architecture). Use some of the following tools:

• Sidekiq
• Resque
• Delayedjob
• Elasticbeanstalk Worker Tier
• RabbitMQ

Then set up two clusters, one for processing HTTP requests and the other for workers. This allows us to scale based on the amount of background work processed.

From the outset, our team felt that it should be done with go, because at the discussion stage we estimate that this could be a system that handles very large traffic. I have used the go language for two years and have developed some systems in my work, but none of them have handled such a large load.

We first created several structures to define the payload of the HTTP request. We receive these loads via a POST request and then upload a function to the S3 bucket.

type PayloadCollection struct {    WindowsVersion  string    `json:"version"`    Token           string    `json:"token"`    Payloads        []Payload `json:"data"`}type Payload struct {    // [redacted]}func (p *Payload) UploadToS3() error {    // the storageFolder method ensures that there are no name collision in    // case we get same timestamp in the key name    storage_path := fmt.Sprintf("%v/%v", p.storageFolder, time.Now().UnixNano())    bucket := S3Bucket    b := new(bytes.Buffer)    encodeErr := json.NewEncoder(b).Encode(payload)    if encodeErr != nil {        return encodeErr    }    // Everything we post to the S3 bucket should be marked 'private'    var acl = s3.Private    var contentType = "application/octet-stream"    return bucket.PutReader(storage_path, b, int64(b.Len()), contentType, acl, s3.Options{})}

Simple to use Go routines

In the beginning, we used the simplest method to implement the processing function of the POST request. We tried to process the request in parallel through Goroutine.

func payloadHandler(w http.ResponseWriter, r *http.Request) {    if r.Method != "POST" {        w.WriteHeader(http.StatusMethodNotAllowed)        return    }    // Read the body into a string for json decoding    var content = &PayloadCollection{}    err := json.NewDecoder(io.LimitReader(r.Body, MaxLength)).Decode(&content)    if err != nil {        w.Header().Set("Content-Type", "application/json; charset=UTF-8")        w.WriteHeader(http.StatusBadRequest)        return    }    // Go through each payload and queue items individually to be posted to S3    for _, payload := range content.Payloads {        go payload.UploadToS3()   // <----- DON'T DO THIS    }    w.WriteHeader(http.StatusOK)}

For the right amount of load, this solution should be no problem. But this method does not work well after the load is increased. When we deployed this version into our production environment, we encountered an order of magnitude larger than expected. We completely underestimated the flow.

This method is somewhat unsatisfactory. It has no control over the number of goroutine created. Because we received 1 million post requests per minute, the code above quickly ran out.

Try again

We need a different solution. At the outset, we discussed the need to write the HTTP request handler function succinctly, and then transfer the processing to the background. Of course, this is something you have to do in the Ruby on Rails world, or you will block all worker work (such as Puma,unicorn,passenger and so on, we will not continue to discuss JRuby here). We need to use a common solution such as RESQUE,SIDEKIQ,SQS. This list can be very long, because there are many ways to accomplish this task.

The second version is the creation of a buffered channel. So we can put the task in the queue and then upload it to S3. Because we can control the length of the queue and have sufficient memory, we think it should be no problem to cache the work task in the channel queue.

var Queue chan Payloadfunc init() {    Queue = make(chan Payload, MAX_QUEUE)}func payloadHandler(w http.ResponseWriter, r *http.Request) {    ...    // Go through each payload and queue items individually to be posted to S3    for _, payload := range content.Payloads {        Queue <- payload    }    ...}

Then we need to extract the work task from the queue and process it. The code shows:

func StartProcessor() {    for {        select {        case job := <-Queue:            job.payload.UploadToS3()  // <-- STILL NOT GOOD        }    }}

Frankly speaking, I don't know what we were thinking. It must have been the result of staying up late and drinking Red Bull. This method does not bring any help to us. The queue is just delaying the problem. Our processing function (processor) uploads only one load (payload) at a time, and the rate at which requests are received is much larger than the ability of a handler to upload S3, and the buffered channel quickly reaches its limit. This blocks the HTTP request handler to add more work tasks to the queue.

We are merely delaying the triggering of the problem. The system is in the countdown, and finally it crashes. After the problematic version is deployed, the system's latency is constantly increasing at a constant rate.

0_latency.png

Better way to deal with it

We decided to use the common programming model of Go channel. Use a two-level channel system, one to hold the task queue, and the other to control the amount of concurrency to handle the task queue.

The idea here is to parallelize S3 uploads based on a sustainable rate. This rate does not slow down the machine or cause S3 connection errors. We have chosen a job/worker mode. If you are familiar with the language, you Java C# can think of it as a pool of work threads implemented by the Go language channel.

var (maxworker = os. Getenv ("max_workers") maxqueue = OS. Getenv ("Max_queue")//job represents the job to being runtype job struct {Payload payload}//A buffered channel that we    Can send work requests On.var Jobqueue Chan job//worker represents the worker that executes the JobType worker struct { Workerpool Chan Chan job Jobchannel Chan Job quit Chan Bool}func newworker (Workerpool Chan Chan Job) work Er {return worker{workerpool:workerpool, Jobchannel:make (Chan Job), Quit:make (chan bool )}}//Start method starts the run loop for the worker, listening for a quit channel in//case we need to stop Itfunc (w Wo            Rker) Start () {go func () {for}//register the current worker into the the worker queue. W.workerpool <-W.jobchannel Select {Case job: = <-w.jobchannel://We have re                ceived a work request. If err: = job. PAYLOAD.UPLOADTOS3 (); Err! = Nil {log. Errorf ("Error uploading to S3:%s", err.)                Error ())} case <-w.quit://We had received a signal to stop Return}}} ()}//stop signals the worker to stop listening for work requests.func (W worker) Stop () {go func () {W.quit <-true} ()}

We have modified the HTTP request handler function to create a structure containing the payload (payload) and Job then send it to a JobQueue channel called. The worker will process them.

func payloadHandler(w http.ResponseWriter, r *http.Request) {    if r.Method != "POST" {        w.WriteHeader(http.StatusMethodNotAllowed)        return    }    // Read the body into a string for json decoding    var content = &PayloadCollection{}    err := json.NewDecoder(io.LimitReader(r.Body, MaxLength)).Decode(&content)    if err != nil {        w.Header().Set("Content-Type", "application/json; charset=UTF-8")        w.WriteHeader(http.StatusBadRequest)        return    }    // Go through each payload and queue items individually to be posted to S3    for _, payload := range content.Payloads {        // let's create a job with the payload        work := Job{Payload: payload}        // Push the work onto the queue.        JobQueue <- work    }    w.WriteHeader(http.StatusOK)}

At the time the service was initialized, we created one Dispatcher and called the Run() function to create the worker pool. These workers listen JobQueue for new tasks and work on them.

dispatcher := NewDispatcher(MaxWorker)dispatcher.Run()

Here is our dispatcher implementation code:

  type Dispatcher struct {//A pool of workers channels that is registered with the Dispatcher Workerpool Chan Chan job}func newdispatcher (maxworkers int) *dispatcher {pool: = Make (Chan Chan Job, maxworkers) return & Dispatcher{workerpool:pool}}func (d *dispatcher) Run () {//starting n number of workers for I: = 0; i < D.maxwo Rkers; i++ {worker: = Newworker (D.pool) worker. Start ()} go D.dispatch ()}func (d *dispatcher) dispatch () {for {select {case job: = <-jobqueu E://A job request has been received go func (Job Job) {//try to obtain a worker Jo                b Channel is available. This would block until a worker is idle jobchannel: = <-d.workerpool//Dispatch the JO B to the worker job channel Jobchannel <-Job}}}  

Here we provide the maximum number of workers created as parameters and add these workers to the worker pool. Because we've already used Amazon's Elasticbeanstalk in the Docker go environment and configured our production environment in strict accordance with the 12-factor method, these parameter values can be obtained from the environment variables. We can easily control the number of worker and the length of the task queue. We can quickly adjust these values without having to redeploy the entire cluster.

var (  MaxWorker = os.Getenv("MAX_WORKERS")  MaxQueue  = os.Getenv("MAX_QUEUE"))

After deploying the new version, we saw that the system delay dropped to a negligible magnitude. The ability to handle requests has also risen sharply.

1_latency.png

A few minutes after Elastic Load balancers warmed up, we saw the Elasticbeanstalk app start processing nearly 1 million requests per minute. Our traffic usually climbs to more than 1 million requests per minute in the morning. At the same time, we have reduced the number of servers from 100 to 20 units.

2_host.png

By properly configuring clusters and auto-scaling, we are able to configure only 4 EC2 C4. Large instance. Then use elastic auto-scaling to create a new instance when the CPU usage lasts for 5 minutes above 90%.

3_util.png

Conclusion

For me brevity (simplicity) is the first. We can design a complex system with countless queues, many back-end workers, and complex deployments, and eventually we use the power of Elasticbeanstalk auto-scaling and the simple way that the go language offers to deal with concurrency. Write to Amazon S3 with just 4 machines (probably not as powerful as my MacBook Pro) to handle 1 million post requests per minute.

Each task has a corresponding correct tool. When your Ruby on Rails system needs a very powerful HTTP request processor, try to look beyond the ruby ecosystem for more powerful options.