Remember once Golang memory leak problem fix

This is a created article in which the information may have evolved or changed.

Objective

The following features are required for a company project:

1. Read the log generated by the main program.
2. Log processing to MongoDB

The implementation of the idea is as follows:

1. Use the Filewatch Monitor file to read the newly added data once the file has changed.
2. The read data is stored in MongoDB after processing.

Problem

After running for a period of time, the program discovers that the system becomes a huge card, and the process management discovers that this process consumes most of the memory and again estimates that there is a memory leak and begins to look for the problem.
Adding memory consumption analysis to the program

fm, err := os.OpenFile("./mem.out", os.O_RDWR|os.O_CREATE, 0644)if err != nil {    log.Fatal(err)}pprof.WriteHeapProfile(fm)fm.Close()

The analysis found that most of the memory allocations were in watcher. Waitforfilechange (), and then continue using net/http/pprof

_ net/http/pprofgo func() {    http.ListenAndServe("localhost:6060", nil)}()

Using Web Page view, it is found that there are a lot of goroutine blocked in watcher. Waitforfilechange (). This was discovered because file monitoring did not return after a file change, causing a large number of goroutine to be blocked here.

Solve

After the test found that the file monitoring this package must have this problem, after the use of file lock substitution, run for a period of time without a leak, to solve the problem.