Together with the Golang go program routines

This is a creation in Article, where the information may have evolved or changed.

There is too much information about the Golang basis systematically, and there is no longer a one by one to repeat. The idea of this paper is to explore the routines of Go program from another angle. After all, the paper on the end of shallow, so, can do not mouth. Sometimes a few days do not write code, suddenly one day put in to do something, just came to realize, also only knock code, can find lost their own, that can forget all the unhappy.

Hello World

package mainimport (    "fmt")func main() {    fmt.Println("hello world")}

Go program structure from the whole is like this, the first line looks like this must be the Baotou declaration, the program in the package as a unit, a folder is a package, a package may have multiple files, but the package name is the same. In the case of C + + program include, here is the import, followed by the other package name, a package definition of variables or types, this package is visible, if the initial letter size, it can be exported. If the introduction of the program does not use the package, the compilation will be error, error, wrong. The declaration does not use the same variable, yes, it will error. There is no semicolon at the end of the line, the left curly braces must be put, the indentation does not need you to worry about, etc., many of the coding style problems here are no longer a problem, yes, go FMT help you all done, so you see most go program style is very close. After writing a time code, you will find that this style is really simple and neat.

This article focuses on

Through some concepts of learning and introduction, design and implementation of a thread pool, I believe many places may use this model or a variety of variants.

Variable

Variables declaration, definition, assignment, pointers and so on don't want to wordy, go to other places to learn.

Structural body

Let's start by defining a struct.

package package1type User struct {    Name string    addr int    age  int}

You must have noticed that the name first letter is in uppercase, in the Package2 package, the import package1 can be passed through the user. Name accesses the name member, and name is exported. But addr and age in the Package2 can not directly access, these two are not exported, only in the Package1 package is directly accessible, that is, private. How do you get a member that is not exported in Package2? Let's take a look at the method.

Method

func (u User) GetAge() string {    return u.age}func(u *User) SetAge(age int){    u.age = age}

The use of methods is very similar to C + + or java. The type of User in the following code snippet is *user, and you will find that the method calls are used only when the receiver of the method is an object or a pointer.

user := &User{        Name: name,        addr: addr,        age:  age,}user.SetAge(100)fmt.Println(user.GetAge())

And the usual way to construct objects is this.

func NewUser(name string, addr string, age int) *User {    return &User{        Name: name,        addr: addr,        age:  age,    }}

    user := new(User)    user := &User{}//与前者等价    user := User{}

Combining and nesting

There is no inheritance in go, no polymorphism, and no templates. The long-debated succession and combinatorial problems are not a problem here, because there is no choice. For example, I want to implement a thread-safe integer (assuming only with + + and--), probably doing

type safepending struct {    pending int    mutex   sync.RWMutex}func (s *safepending) Inc() {    s.mutex.Lock()    s.pending++    s.mutex.Unlock()}func (s *safepending) Dec() {    s.mutex.Lock()    s.pending--    s.mutex.Unlock()}func (s *safepending) Get() int {    s.mutex.RLock()    n := s.pending    s.mutex.RUnlock()    return n}

You can also use nested notation

type safepending struct {    pending int    sync.RWMutex}func (s *safepending) Inc() {    s.Lock()    s.pending++    s.Unlock()}func (s *safepending) Dec() {    s.Lock()    s.pending--    s.Unlock()}func (s *safepending) Get() int {    s.RLock()    n := s.pending    s.RUnlock()    return n}

This way the safepending type will have sync directly. Rwmutex all the attributes in the type, which is a good convenient notation.

Interface

A interface type is a set of methods, and if other types implement all the interfaces in the interface type, then we can say that this type implements the interface type. For example: An empty Interface interface{} contains a set of methods that are empty, and you can say that any type implements it, that is, interface{} can represent any type, and the direct conversion of a type looks at the example below.

Implement a small top heap

First, a worker structure is defined, and the worker object holds a lot of pending request,pinding for the number of request to be processed, with the worker as the element, a small top heap, and each pop operation returns a worker with the lowest load.
The Golang Standard library provides a container for the heap structure, we only need to implement a few methods, we can implement a heap type of data structure, use only need to invoke the standard library provided init initialization interface, POP interface, push interface, you can get the results we want. The method we want to implement is Len, less, Swap, Push, Pop, see the code below. Also worth mentioning is that Hawthorn June is also through the standard library provided examples of this knowledge point.

type Request struct {    fn    func() int    data  []byte    op    int    c     chan int}type Worker struct {    req     chan Request    pending int    index   int    done    chan struct{}}type Pool []*Workerfunc (p Pool) Len() int {    return len(p)}func (p Pool) Less(i, j int) bool {    return p[i].pending < p[j].pending}func (p Pool) Swap(i, j int) {    p[i], p[j] = p[j], p[i]    p[i].index = i    p[j].index = j}func (p *Pool) Push(x interface{}) {    n := len(*p)    item := x.(*Worker)    item.index = n    *p = append(*p, item)}func (p *Pool) Pop() interface{} {    old := *p    n := len(*p)    item := old[n-1]    //item.index = -1    *p = old[:n-1]    return item}

Use of Pool

package mainimport (    "container/heap"    "log"    "math/rand")var (    MaxWorks = 10000    MaxQueue = 1000)func main() {    pool := new(Pool)    for i := 0; i < 4; i++ {        work := &Worker{            req:     make(chan Request, MaxQueue),            pending: rand.Intn(100),            index:   i,        }        log.Println("pengding", work.pending, "i", i)        heap.Push(pool, work)    }    heap.Init(pool)    log.Println("init heap success")    work := &Worker{        req:     make(chan Request, MaxQueue),        pending: 50,        index:   4,    }    heap.Push(pool, work)    log.Println("Push worker: pending", work.pending)    for pool.Len() > 0 {        worker := heap.Pop(pool).(*Worker)        log.Println("Pop worker:index", worker.index, "pending", worker.pending)    }}

The running result of the program is as follows, you can see that each pop result returns a work element with the smallest pending value.

2017/03/11 12:46:59 pengding 81 i 02017/03/11 12:46:59 pengding 87 i 12017/03/11 12:46:59 pengding 47 i 22017/03/11 12:46:59 pengding 59 i 32017/03/11 12:46:59 init heap success2017/03/11 12:46:59 Push worker: pending 502017/03/11 12:46:59 Pop worker:index 4 pending 472017/03/11 12:46:59 Pop worker:index 3 pending 502017/03/11 12:46:59 Pop worker:index 2 pending 592017/03/11 12:46:59 Pop worker:index 1 pending 812017/03/11 12:46:59 Pop worker:index 0 pending 87

Careful you will be able to find, not work, why not goroutine to run the task? Yes, Hawthorn. Here is only a demonstration of the construction and use of the small top heap, as to how to use Goroutine to run the task, I think about it first.
In fact, with the code similar to the below can be

func (w *Worker) Stop() {    w.done <- struct{}{}}func (w *Worker) Run() {    go func() {        for {            select {            case req := <-w.req:                req.c <- req.fn()            case <-w.done:                break            }        }    }()}

Concurrency of Golang

Golang in the concurrency mechanism is very simple, Master good Goroutine, channel and some programming routines, you can use very well. Of course, all the problems that exist in concurrent programming are language-independent, but the level of support for the infrastructure in each language is different, so be careful in the Go program.

Goroutine

Official description of the Goroutine:

They ' re called Goroutines because the existing terms-threads, coroutines, processes, and so on-convey inaccurate Connotati Ons. A goroutine have a simple model:it are a function executing concurrently with other goroutines in the same address space. It is lightweight, costing little more than the allocation of stack space. And the stacks start small, so they is cheap, and grow by allocating (and freeing) heap storage as required.
Goroutines is multiplexed onto multiple OS threads so if one should block, such as and waiting for I/O, others continue To run. Their design hides many of the complexities of thread creation and management.
Prefix a function or method call with the GO keyword to run the call in a new goroutine. When the call completes, the goroutine exits, silently. (The effect is similar to the Unix Shell's & notation for running a command in the background.)

To start a goroutine, the usage is simple:

go DoSomething()

Channel

Look at the channel description:

A channel provides a mechanism for concurrently executing functions to communicate by sending and receiving values of a SP ecified element type. The value of an uninitialized channel is nil.

In short, it provides a mechanism for synchronization and communication between Goroutine.

Shared memory? OR communication?

Don ' t communicate by sharing memory; Share memory by communicating

This is one of the most important routines in the GO program. Take a specific small application scenario: A map type of data structure, its additions and deletions can be done in multiple threads, we will use what kind of scheme to implement it?

1. Add lock when adding and removing and checking operation
2. Implementing a thread-safe map type
3. Add and remove changes to limit the operation of the thread T, other threads if you want to make a deletion operation, unified message to thread T, by thread T for adding and removing operations (assuming that other threads do not have a map query operation)

The

For Scenario 3 is actually a small application of the Go program, which is of course irrelevant to language, but the idea of sharing memory through "communication" in the go language is very easy to implement, with native-supported Goroutine, channel, SELECT, GC and other infrastructure, You might have a "big message" delivery scenario performance concern, but the channel is supported by the reference type of the pass, and will automatically help you to garbage collection, a large structure of the reference type may actually occupy more than 10 bytes of space. This is really to save a lot of Hawthorn June Kung Fu. See the specific procedure for Go program:

Type job struct {//Something}type jobpair struct {key string value *job}type worker struct {jobqueue map [String]*job//Key:username Jobadd Chan *jobpair}//is not a true map insert operation, only sends a message to another thread, func (w *worker) pushjob (user Stri Ng, Job *job) {pair: = &jobpair{Key:user, Value:job,} w.jobadd <-pair}//not really map D Elete operation, message only to another thread func (w *worker) removejob (user string) {W.jobdel <-user}func (w *worker) Run () {go func () {for {select {case Jobpair: = <-w.jobadd:w.insertjob (Jobpair.key, JOBPA            Ir.value) Case Delkey: = <-w.jobdel:w.deletejob (Delkey)//case other channel For _, Job: = Range W.jobqueue {//does something use job//log.    PRINTLN (Job)//}}} ()}func (w *worker) insertjob (key string, value *job) error { W.jobqueue[key] = value W.PEnding. INC () return Nil}func (w *worker) Deletejob (key string) {Delete (W.jobqueue, key) W.pending.dec ()}

Thread pool

Model See flowchart below

Thread pool model. png

By the specific business of the producer line generates into a different job, through a common balance equalizer, the job is assigned to different workers to deal with, each worker occupies a goroutine. In a scenario where the number of jobs is huge, this model is much better than a job that consumes a goroutine model. You can configure different worker numbers and the number of jobs each worker can handle, depending on the business characteristics and hardware configuration.

We can define the job structure first, depending on the business, the inside will contain different attributes.

type job struct {    conn     net.Conn    opcode   int    data     []byte    result     chan ResultType //可能需要返回处理结果给其他channel}type jobPair struct {    key   string    value *job}

Then look at the worker definition

Type worker struct {jobqueue map[string]*job//Key:username broadcast Chan DataType Jobadd Chan *jobpair Jobdel Chan string pending safepending index int done chan struct{}}func newworker (idx int, queue_ limit int, source_limit int, jobreq_limit int) *worker {return &worker{jobqueue:make (Map[string]*job, Qu Eue_limit), Broadcast:make (Chan DataType, source_limit),//4 exchanges Jobadd:make (Chan Jobpair, Jobreq_limit ), Jobdel:make (Chan string, jobreq_limit), pending:safepending{0, sync. rwmutex{}}, Index:idx, Done:make (Chan struct{}),}}func (w *worker) pushjob (User string, Job * Job) {pair: = jobpair{Key:user, Value:job,} w.jobadd <-Pair}func (w *worker) removejob (U Ser string) {W.jobdel <-user}func (w *worker) Run (WG *sync. Waitgroup) {WG. ADD (1) go func () {log. Println ("New Goroutine, worker index:", W.index) dEfer WG. Done () Ticker: = time. Newticker (time.  Second *) for {select {case data: = <-w.broadcast:for _, Job: = Range W.jobqueue {log. PRINTLN (Job, data)} case Jobpair: = <-w.jobadd:w.insertjob (Jobpair.key, JOBP Air.value) Case Delkey: = <-w.jobdel:w.deletejob (delkey) case <-ticker. C:w.loadinfo () Case <-w.done:log. Println ("Worker", W.index, "Exit") Break}}} ()}func (w *worker) Stop () {Go func ( {W.done <-struct{}{}} ()}func (w *worker) insertjob (key string, value *job) error {W.jobqueue[key] = Value W.pending.inc () return Nil}func (w *worker) Deletejob (key string) {Delete (W.jobqueue, key) w.pending.de C ()}

With the implementation of the small top heap mentioned above, we can implement a thread pool with load balancing.
A pattern can not be applied to all business scenarios, Hawthorn June think it is important for different business scenarios to design or optimize the programming model of the ability, the above is inappropriate, welcome to spit groove or correction, like can also play rewards.

Reference documents

1. Https://blog.golang.org/share-memory-by-communicating
2. Https://golang.org/doc/effective_go.html
3. http://marcio.io/2015/07/handling-1-million-requests-per-minute-with-golang/