Golang Dispatcher Source Code Analysis

This is a creation in Article, where the information may have evolved or changed.

Original: Golang Dispatcher source code analysis, author: unintentional disaster

Why does Golang need a scheduler?

Goroutine is introduced to facilitate the writing of high concurrent programs. When a goroutine is performing a blocking operation (such as a system call), the other goroutine in the current thread are handed over to the other thread to continue execution, thus preventing the entire program from blocking.

Because Golang introduces garbage collection (GC), Goroutine is required to be stopped when the GC is executed. The function can be implemented conveniently by implementing the scheduler by itself. Through multiple goroutine to achieve concurrent programs, both the advantages of asynchronous IO, but also multi-threaded, multi-process programming convenience.

The introduction of Goroutine also implies the introduction of great complexity. A goroutine contains both the code to execute and the stack and PC, SP pointers that are used to execute the code.

What problem does the scheduler solve?

Stack Management

Since each goroutine has its own stack, the corresponding stack is created at the same time when the Goroutine is created. The stack space will continue to grow when the goroutine is executed. Stacks typically grow continuously, because each thread in each process shares the virtual memory space, and when there are multiple threads, it is necessary to assign a stack of different start addresses for each thread. This requires estimating the size of each line stacks before allocating the stack. If the number of threads is very large, it is easy to stack overflow.

To solve this problem, we have split stacks technology: When you create a stack, you allocate only a small piece of memory, and if you make a function call that causes the stack space to be low, a new stack space is allocated elsewhere. The new space does not need to be contiguous with the old stack space. The parameters of the function call are copied to the new stack space, and the next function execution is performed in the new stack space.

Golang's stack management approach is similar, but for higher efficiency, the use of continuous stack (Golang continuous stack) implementation is the first allocation of a fixed-size stack, when the stack space is insufficient, allocate a larger stack, and the old stack all copied to the new stack. This avoids the frequent memory allocations and releases that the split stacks method can cause.

Preemptive scheduling

The execution of the goroutine can be preempted. If a goroutine has been occupying the CPU for a long time without being transferred, it will be preempted by the runtime, giving the CPU time to the other goroutine.

The design of the scheduler

The Golang Scheduler introduces three constructs to model the scheduling process:

• G represents a goroutine;
• M represents a thread of an operating system;
• P represents a CPU processor, usually the number of P equals the CPU core (Gomaxprocs).

All three are defined in Runtime2.go, and the relationship between them is as follows:

• G needs to be bound on M to run;
• M needs to bind p to run;
• More than one m in a program will not be in the same state at the same time, with up to gomaxprocs m executing.

Earlier versions of Golang were not p, and the dispatch was done by G with M. The problem is that whenever a goroutine is created, terminated, or dispatched, a global lock is required to protect the associated object (sched) of the schedule. The global lock severely affects the concurrency performance of Goroutine. (Scalable Go Scheduler)

By introducing p, a scheduling algorithm called Work-stealing is implemented:

• Each p maintains a G-queue;
• When a G is created, or becomes executable, it is placed in the executable queue of p;
• At the end of a G execution, p will take the G out of the queue, and if the queue of P is empty at this point, no other g can execute, randomly select another p, stealing half from its executable G-queue.

This algorithm avoids the use of global locks when scheduling goroutine.

Implementation of the Scheduler

Schedule () and findrunnable () functions

The Goroutine schedule is performed in P, and the schedule () function is called whenever the runtime needs to dispatch, and the function is defined in the Proc1.go file.

The schedule () function first calls Runqget () to fetch an executable g from the queue of the current p. If the queue is empty, continue calling the Findrunnable () function. The findrunnable () function obtains g in the following order:

1. Call Runqget () take g from the queue of the current P (same as the call in Schedule ());
2. Call Globrunqget () to fetch the executable g from the global queue;
3. Call Netpoll () takes the end of the asynchronous call G, the call is a non-blocking call, directly return;
4. Call Runqsteal () to "steal" from the queue of other p.

If the above four steps fail to succeed, continue with some low-priority tasks:

1. If the garbage collection mark stage, the garbage collection of the mark work;
2. Call Globrunqget () again to fetch the executable g from the global queue;
3. Call Netpoll () again at the end of the asynchronous call to G, which is called a blocking call.

If you have not obtained g, stop the execution of the current m and return to the beginning of the findrunnable () function for re-execution. If findrunnable () normally returns a G,shedule () function, the Execute () function is called to execute the G. The Execute () function calls the Gogo () function (defined in the assembly source file Asm_xxx.s, XXX represents the system architecture), and the Gogo () function recovers from the g.sched structure the Register field (SP, PC, etc.) that was last paused by the scheduler, and then proceeds to execution.

How to preempt?

Runtime when the program starts, it automatically creates a system thread that runs the Sysmon () function (as defined in Proc1.go). The Sysmon () function is executed throughout the entire program life cycle, monitoring the status of each goroutine, determining whether garbage collection is needed, and so on.

Sysmon () calls the retake () function, and the retake () function iterates through all p, which is preempted if a p is in the execution state and has been executed for a longer period of time. Retake () calls Preemptone () to set P's stackguard0 to Stackpreempt (for details on Stackguard, you can refer to Split Stacks), which will cause the G that is executing in this p to make the next function call. Causes the stack space check to fail. It then triggers Morestack () (Assembly code, located in Asm_xxx.s) and then makes a series of function calls, the main calling procedure is as follows:

1	Morestack () (Assembly Code) , Newstack( ), Gopreempt_m( ) Goschedimpl(), Schedule ()

In the Goschedimpl () function, G is unbound from m by calling DROPG (), and then Globrunqput () is added to the global runnable queue by calling G. Finally, call Schedule () to set the new executable g for the current p.

For further study of Golang preemptive scheduling, refer to Go preemptive Scheduler Design Doc.