Goroutine and Scheduler

This is a creation in Article, where the information may have evolved or changed.

http://skoo.me/go/2013/11/29/golang-schedule?hmsr=studygolang.com&utm_medium=studygolang.com&utm_source=studygolang.com

We all know that the go language is native-supported language-level concurrency, and the smallest logical unit of concurrency is goroutine. Goroutine is a user-state thread provided by the go language, which is, of course, a thread of user-state running on a kernel-level thread. When we create a lot of goroutine, and they are all running on the same kernel thread, we need a scheduler to maintain these goroutine, ensure that all goroutine use CPU, and use CPU resources as fairly as possible.
The principle and implementation of this scheduler is worth studying in depth. The main support of the entire scheduler has 4 important structures, namely M, G, P, Sched, the first three are defined in Runtime.h, Sched defined in PROC.C.
The SCHED structure is the scheduler, which maintains queues with storage m and G, as well as some state information of the scheduler.
M is a kernel-level thread, an M is a thread, Goroutine is running above M, and M is a large structure that maintains a lot of information such as the small Object memory cache (Mcache), the currently executing goroutine, the random number generator, and so on.
P Full name is processor, the processor, its main purpose is to perform goroutine, so it also maintains a goroutine queue, which stores all the goroutine that need it to execute, the role of P can be a bit confusing, At first it was easy to clash with M, and the following was the key to talking about their relationship.
G is the goroutine implementation of the core structure, g maintenance of the goroutine required stack, program counter and its location m and other information.

Understanding the relationship between M, P, G Three is very important to understand the whole scheduler, I have found a diagram on the network to illustrate its three relationships:

The Gopher (Gopher) carried a pile of bricks to be processed in a car. M can be seen as the ground mouse in the picture, p is the car, G is the car in the bricks. A picture wins thousands of words, figuring out the relationship between the three of them, and here we begin to focus on how the rat is moving the bricks.

startup process

When we are concerned about the internal principles of most programs, we try to figure out how to start the initialization process and understand that this process is critical to further analysis. The assembly code _RT0_AMD64 in the Asm_amd64.s file is the entire startup process, and the core process is as follows:
Call runtime args (SB) call Runtime Osinit (SB) Call Runtime Hashinit (SB) Call Runtime Schedinit (SB)//Create a new Goroutine to start Programpushq $runtime main F (SB)//Entrypushq $//arg sizecall runtime Newproc (SB) POPQ AXPOPQ ax//s Tart this MCALL runtime Mstart (SB)

After the boot process has done the scheduler initialization runtime Schedinit, call runtime Newproc to create the first goroutine, the Goroutine will execute the function is runtime Main, This first goroutine is the so-called Lord Goroutine. We wrote the simplest go program "Hello,world" is to completely run in this goroutine, of course, any go program entrance is from this goroutine start. The last call to runtime Mstart is the real execution of the main goroutine created in the previous step.
The Scheduler initialization runtime Schedinit function during startup mainly creates a batch of carts (p) based on the Gomaxprocs value set by the user, no matter how large the Gomaxprocs is set, and can only create up to 256 cars (p). These carts (p) are idle after initial creation, i.e. they are not yet in use, so they are placed in the scheduler structure (Sched) pidle
The list of field maintenance is stored for future needs.
Looking at the runtime main function, you can see that the first thing to do when the main goroutine is started is to create a new kernel thread (gopher m), but this thread is a special thread that is specifically responsible for doing specific things throughout the run-system monitoring (Sysmon). The next step is to go to the Go program's main function to start the GO program execution.
The GO program is now up and running. A really work go program, must create a lot of goroutine, so after the go program started running, will add goroutine to the scheduler, the scheduler will be responsible for maintaining the normal execution of these goroutine.

Create Goroutine (G)

In the Go program, there are often similar codes:
Go do_something ()

The Go keyword is used to create a goroutine, and the following function is the code logic that the Goroutine needs to execute. The Go keyword corresponds to the interface of the scheduler is runtime Newproc
。 The runtime Newproc is very simple, it is responsible for making a brick (g), and then put the Brick (g) into the current hamster (M) in the car (P).
Each new goroutine needs to have a stack of its own, the sched of the G-structure
The field maintains the stack address and program counters and other information, this is the most basic scheduling information, that is, the goroutine to abandon the CPU need to save this information, the next time you regain the CPU, you need to load this information into the corresponding CPU registers.
Assuming that a large number of Goroutne have been created at this time, it is up to the scheduler to maintain these goroutine.

Create kernel thread (M)

There are no language-level keywords in the GO program that let you create a kernel thread, you can only create goroutine, and kernel threads can only be created by runtime according to the actual situation. When does runtime create a thread? To the hamster transport brick diagram, brick (G) too much, the hamster (M) and too little, really busy, just have free car (p) not used, then borrow some land from elsewhere (m) come over until the car (p) to run out. Here is a gopher (m) not enough to borrow from elsewhere (m) process, this process is to create a kernel thread (m). The interface functions for creating m are:
void Newm (void (*FN) (void), P *p)

The core behavior of the NEWM function is to invoke the clone system call to create a kernel thread, where each kernel thread starts executing at the runtime Mstart function. The parameter p is a free car (p).
Each of the created kernel threads is executed from the runtime Mstart function, and they will be assigned to their own car to move bricks.

Dispatch Core

The NEWM interface simply assigns a free p to the newly created M, which is equivalent to telling the borrowed Gopher (M)--"The next day, you will use the number 1th car to move bricks, remember the number 1th car, you will go to the car park to get the car." ", Gopher (M) go get the car (P) the process is Acquirep
。 Runtime Mstart is entering schedule.
The code in the P,runtime Mstart function before the current m assembly is given:
} else if (m! = &runtime M0) {acquirep (M->NEXTP); m->nextp = nil;} Schedule ();

The content of the If branch is P,NEXTP on the current m assembly
Is the NEWM allocated to the free car (P), but only at this time to really get hands. No p,m is unable to carry out goroutine, just like the ground mouse does not have the car can not be transported bricks the same reason. Corresponding to the action of the Acquirep is Releasep, the M assembly of P to load off, the work is done, the hamster needs to rest, the car also to the parking lot, and then go to sleep.
When the Gopher (M) got his own car (P), he entered the factory and began to work, that is, the schedule
Call. The code for simplifying schedule is as follows:
Static Voidschedule (void) {G *gp; gp = Runqget (m->p), if (gp = = nil) GP = findrunnable (); if (m->p->runqhead! = m >p->runqtail && Runtime Atomicload (&runtime sched.nmspinning) = = 0 && Runtime atomicload ( &runtime Sched.npidle) > 0)//todo:fast atomic wakep (); Execute (GP);}

Schedule function was simplified by me too much, mainly I do not like to stick large sections of the code, so only the skeleton code is retained. Here's a 4-step logic:

Runqget
, the Gopher (M) tried to remove a brick (G) from his own car (P), and of course the result might have failed, that is, the hamster's car was empty, no bricks.
Findrunnable
, if the hamster's own car without bricks, that also can not idle work is it, so the hamster will try to run to the factory warehouse to take a brick to deal with, the factory warehouse may not have bricks ah, this situation, the land rat did not lazy stop work, but quietly run out, random stare on a small partner (gopher), Then he tried to steal half of the bricks from its car into his car. If many attempts to steal bricks have failed, it means that there is no brick to move, this time the hamster will return the car to the parking lot, and then to sleep
Take a break. If the hamster sleeps, the following process is of course stopped, and the hamster sleeps in the thread sleep.
Wakep
, to this process, the poor hamster found himself in the car there are many bricks ah, they can not handle it, and then look back at the parking lot there is idle car, immediately ran to the dorm, your sister, incredibly still have a small partner in sleep, directly to the bottom of a foot, "Your sister, incredibly still sleeping, Lao Tzu is almost exhausted, Get up and work and share the job. The little buddy woke up and took his car and went to work. Sometimes, poor hamster ran to the dorm but found no sleep in the small partner, so will be very disappointed, finally had to tell the factory owner-"Parking lot and idle car ah, I can not move, hurriedly from other factories to borrow a gopher to help it." Finally, the factory owner got a new hamster to work on.
Execute
, the hamster took the bricks and put them into the fire and practiced happily.

Note: "Gopher steal brick" called Work stealing, a scheduling algorithm.
Here, it seems that the entire factory is functioning normally, impeccable appearance. No, there is a doubt unresolved ah, suppose the rat's car has a lot of bricks, it put a brick into the stove, when to take it out, put in the second brick? Do you want to keep the first piece of brick on fire before you take it out? It is estimated that the brick behind is really waiting for the flowers to be thanked. Here is to really solve the Goroutine scheduling, context switching problem.

Dispatch point when we look through the channel's implementation code, we can see that the runtime Park function is triggered when the channel reads and writes. After Goroutine calls Park, the Goroutine is set to the bit waiting state, discarding the CPU. The Goroutine of Park is in waiting state, and this goroutine is not in the car (P), and if it is not called runtime ready, it will never be executed again. In addition to channel operation, the timer, network poll, etc. may be park goroutine.

In addition to park can discard the CPU, calling the runtime gosched function can also let the current goroutine abandon the CPU, but unlike park, Gosched is to set Goroutine to Runnable State, Then put into the scheduler global waiting queue (that is, the factory warehouse mentioned above, this will understand why the factory warehouse will have bricks (G) it).
In addition, it is the turn of the system call, some system calls will also trigger the rescheduling. The go language is completely its own package system call, so in the encapsulation system call, can do a lot of hands and feet, that is, enter the system call when the execution of Entersyscall, exit and execute the Exitsyscall function. Only system calls that encapsulate the entersyscall can trigger a reschedule, which will change the state of the Trolley (P) to Syscall. Do you remember the Sysmon thread that was mentioned at the beginning? This system monitoring thread will scan all the cars (p), found a car (p) in the state of the Syscall, it is known that the car (p) encountered Goroutine is making a system call, so the system monitoring thread will create a new Gopher (M) To get this car in the Syscall to Rob, began to work, so that all the bricks in the car (G) can bypass the previous system call waiting. The land mouse of the stolen car, such as the system call back, found that his car did not, can not continue to work, so can only execute system call Goroutine put back to the factory warehouse, sleep
went on.
From the Goroutine dispatch point can be seen, the scheduler is still quite rough, scheduling granularity is a bit too big, fairness also did not think so good. In short, this scheduler is still relatively simple.

On-site processing goroutine on the CPU swap-out, constant context switching, must be guaranteed to save the scene and restore the scene

Save the scene is when the goroutine abandon the CPU, the value of the relevant register to the memory, the recovery site is the Goroutine to regain the CPU, you need to put the previous register information from memory to the corresponding register.
Goroutine in the active abandonment of the CPU (park/gosched), will involve invoking the runtime McAll function, which is also a compilation implementation, mainly the Goroutine stack address and program counter saved to the G structure of the Sched
field, McAll completes the field save. The function to restore the scene is Runtime Gogocall, which is called primarily in execute
, which requires reloading the appropriate registers before executing goroutine.