Golang Kafka clinet Memory leak problem handling

Go memory leaks

New version of the service ran on a day of memory consumption of 20g, apparently a memory leak

Memory leak problem is difficult to locate

Technical positioning relies mainly on pprof to generate statistical files

The previous write Web project is based on Net/http/pprof to see the run state information and assist in troubleshooting

Https://github.com/cclient/gowebframework/blob/f691321b898484327ceb0dcc4897315d5b8b0637/src/server/api/profiler.go#L4

Now service is a back-end service, pre-and no pprof module

Troubleshoot data flow and code changes first

Not limited to the go language, the reason for memory leakage is mainly three cases, 1 resource class (file, Io stream) is not freed, 2 "global" object, with a large number of additional references, the runtime only add no less, 3 combination of both, the resource class object has attached a lot of additional references.

The amount of 20g, basically the daily amount of data processing, the release of the major changes caused by the introduction of the Kafka processing module, guessing the location to the Kafka, data flow processing in the write Kafka part of the leak

Kafka handling Partial references Github.com/shopify/sarama

View Code initial positioning in

Datacollector,err: = newdatacollector ([]string{kfkhost})if err = = Nil    {=  Datacollector.sendmessages (Messages)}

View DataCollector found a Close method

type Syncproducer Interface {SendMessage produces a given message, andreturns only if it either hasSucceededorFailed to produce. It wouldreturnThe partition andThe offsetOf the produced message,orAn errorifThe message failed to produce. SendMessage (msg*producermessage) (partition Int32, offset int64, err error)Sendmessages produces a given set of messages, andreturns only if allMessagesinchThe set has either succeededorfailed. Note that messagesCan succeed andFail individually;ifSome succeed andsome fail,Sendmessages wouldreturnAn error. Sendmessages (msgs []*producermessage) ErrorClose shuts down the producer andWaits forAny buffered messages to be//flushed. You must call this function before a Producer object passes out of//scope, as it may otherwise leak memory. You must call this before calling//Close on the underlying client. Close () Error}

Add to

Datacollector,err: = newdatacollector ([]string{kfkhost})if err = = Nil    {=  Datacollector.sendmessages (Messages)}defer datacollector.close ()

On-line, problem solving

If code grooming doesn't pinpoint a problem, then rely on technical means

Previously applied only through NET/HTTP/PPROF.

Read some information.
Https://github.com/hyper0x/go_command_tutorial/blob/master/0.12.md
Introduction of Net/http/pprof is a more brief choice

Golang Kafka clinet Memory leak problem handling