How to Implement Fiber

Document directory

• Source code

Introduction

We know that a thread is the most basic unit for program execution. Any program execution depends on the thread execution. A thread is usually a basic component of the operating system. Generally, a thread is created. For example, on Win32, a thread is created using createthread. The operating system actually creates two objects, one is the user-state thread, the other is the kernel-state thread, and our code runs in the user-state thread. When a thread switches, such as calling waitforsingleobject or calling writefile to execute blocking Io, thread switching is usually involved, and switching needs to enter the kernel state, the resulting overhead is considerable. The fiber, that is, the fiber program, is completely running in the user State, and the switching of each thread is only completed in the user State, so the switching overhead is small. Thread Scheduling is usually completed by the thread scheduler of the operating system. In modern OS, preemptible scheduling policies are usually used. However, the call of a fiber process depends entirely on the programmer, that is, to implement a cooperative scheduling. The switch is only performed when the switch is proposed.

It is implemented in other languages, such as Python and Lua. Python provides generate, which can be used to simulate the fiber process. Another powerful greenlet library is another Implementation of the Fiber Process in Python. In Lua, we can use the built-in coroutine library.

Basic usage Win32

In Win32, we can use convertthreadtofiber/convertfibertothread, createfiber/deletefiber to manage the creation and destruction of fiber sets. For a thread, if you want to call a fiber program, you must call convert to convert a thread into a fiber program. When you need to switch to another fiber, you only need to call switchtofiber, so that some statuses of the existing thread will be saved and will be resumed later.

Linux

In Linux, context APIs can be used to perform similar functions. We call getcontext/makecontext to initiate the chemical fiber process and use swapcontext/setcontext to switch the fiber process.

Setjmp/longjmp

Speaking of this, we have to mention another old c API, setjmp/longjmp, which seems to be used to accomplish similar work. Setjmp is used to save the current execution environment, and longjmp is used to restore the last execution, so as to implement the non-local goto function. But there is a problem here, that is, when we call longjmp back to the status saved by setjmp for execution, if the longjmp caller and setjmp caller are not the same function, the status of the stack where longjmp is located will be undefined [1]. That is to say, we cannot save the status before longjmp and then return longjmp. This is an undefined action. Of course, on Win32, you can do this and work well (I used this technique in implementation ). However, in Linux, you will encounter a runtime error, prompting the stack
Smashing (GCC stack protection mechanism __fortify_fail), that is, the stack is destroyed (this has caused me to debug for a long time and finally give up. Many implementations on the Internet use this method, but it does not work ). Setjmp/longjmp has other traps. For example, on Win32, it does not save/restore the seh exception chain, and so on.

How fiber works

In fact, no matter which method, we only need to understand that if fiber is working, then we can implement our own fiber (of course, we need to consider other CPU-related situations ).

Similar to a thread, fiber has a stack to save the status required for the current call. Therefore, we need to create a stack for fiber first. Secondly, each fiber must need an entry function (just like a thread). During the switchover, you must enter this entry and then execute the function. In fact, when the code is executed on the x86 CPU, it is to modify the EIP pointer and direct it to this entry function. When switching the fiber process, that is, saving the stack status. on x86, ESP and EBP are two important registers, saving the current stack status. We also need to save other General registers, such as EBX, EDI, and ESI, because the registers are obviously modified for different fiber paths. The reason for not saving the other three registers: eax, ECx, and EDX is that these registers are all caller-save [4], that is, if the caller uses these registers, before calling other functions, you must save these registers.

Two problems are involved: modifying the EIP pointer and saving/restoring the register value.

Modify EIP

Because the EIP pointer can be modified only in privileged mode (the operating system works in privileged mode), we cannot directly modify user State programs. However, we know that the EIP can be indirectly modified in the JMP command. Another method is to use the RET command. the RET command will take the value from the top of the stack as the EIP value, so that the jump is realized. Here I chose to use the push + RET method to modify the EIP, because JMP requires the offset relative to the next instruction as the operand (relative jumpping ).

Save/restore GPR (general-purpose registers)

It is relatively simple to save and reply to GPR. You can use several Assembly commands.

Save/restore Stack pointer

When saving/restore EBP/ESP, you need to be extremely careful, because once we modify ESP, all subsequent stack operations will be performed on the new stack.

Implementation

Let's take a look at the members of fiber_context. This struct is used to save registers and stack pointers/sizes.

struct fiber_context{#ifdef FIBER_X86    // registers    uintptr_t ebp;    uintptr_t esp;    uintptr_t eip;    // callee-save general-purpose registers    // see http://www.agner.org/optimize/calling_conventions.pdf    uintptr_t ebx;    uintptr_t esi;    uintptr_t edi;#else#  error Unsupported platform#endif    char* stack;    int   stack_size;    void* userarg;};

All GPR and EBP/ESP/EIP of callee-save are saved. This is relatively simple.

Fiber_make_context

This function is used to create a context and initialize the execution environment.

void fiber_make_context(fiber_context* context, fiber_entry entry, void* arg){    assert(context && entry);    // default alignment of the stack    const int alignment = 16;    context->esp = (reinterpret_cast<uintptr_t>(context->stack) + context->stack_size) & ~(alignment - 1);    context->ebp = context->esp;    context->eip = reinterpret_cast<uintptr_t>(entry);    context->userarg = arg;    // push the argument onto the stack    char* top = reinterpret_cast<char*>(context->esp);    memcpy(top, &context->userarg, sizeof(void*));    // make space for the pushed argument    context->esp -= sizeof(void*);    // clear all callee-save general purpose registers    context->ebx = context->esi = context->edi = 0;}

Here, we initialize ESP and EBP and point them to the top of the stack. Note that the x86 CPU uses the full-descending stack, that is, the stack for reverse growth. Because the user can provide an optional parameter, we must push this parameter to the stack in advance, so that after the EIP is redirected, the entry function can normally access this parameter. Naturally, after the stack is pressed, we must reduce the ESP value and leave room for the parameter. The EIP value points to the address of the entry function, which is easy to understand. The rest is simply the default value of the GPR initialization.

Fiber_get_context

This function is used to save the current execution status. Because we can use fiber_set_context to resume execution from a context initialized by fiber_get_context, this special situation must be considered here. We cannot simply save the value of ESP/EBP, because the stack frame of this function will be destroyed once we return the value from fiber_get_context, in this case, if the saved ESP/EBP points to the frame of the fiber_get_context, a runtime error will obviously occur. The only thing we can do is to save the caller's stack.

For each C/C ++ function, the compiler inserts commands at the entrance to save the caller's EBP and modifies EBP/ESP to create a new stack frame. Therefore, we cannot write a common function to complete this job. We need a way to prevent the compiler from generating Prolog/epilog, so that we have more control. In VC, we can use the naked function. In GCC, we can only write the Assembly source code.

__declspec(naked) void fiber_get_context(fiber_context* context){    // TODO: how much space need to reserve for assert ?    //assert(context);    // save the current context in `context' and return    __asm    {        // save current stack pointer to context        mov ecx, dword ptr [esp + 0x4] ;        fixup, point to the argument, ignore return address        mov dword ptr [ecx], ebp ; context->ebp        mov eax, esp        // fixup esp, ignore return address, as the eip is set to the caller's address        add eax, 0x4        mov dword ptr [ecx + 0x4], eax ; context->esp        mov eax, dword ptr [esp]        mov dword ptr [ecx + 0x8], eax ; context->eip        // save callee-save general-purpose registers        mov dword ptr [ecx + 0xc],  ebx; context->ebx        mov dword ptr [ecx + 0x10], esi; context->esi        mov dword ptr [ecx + 0x14], edi; context->edi        ret    }}
Here, when saving the caller's ESP, I made some corrections to get the value between the call of the fiber_get_context (this value includes the parameter of the pressed stack ). On x86, when a function is called, the layout of the caller stack and the called Stack is as follows:
Therefore, to get the return value, you only need to add EBP to 4. However, because we are a naked function, here we can only use ESP to set the value, consider the eax and EBX of the stack, and add 8 to ESP to get the return address.
Fiber_set_context
This function is used to switch to another context. After this function is called, it will directly switch to the new fiber, and the control flow will not return. Therefore, we do not need to consider the usage of the stack, but just need a simple response.
void fiber_set_context(fiber_context* context){    __asm    {        ; restore the enviroment for context        mov eax, context        mov ebp, dword ptr [eax]        ; context->ebp        mov esp, dword ptr [eax + 0x4]  ; context->esp        ; restore callee-save general-purpose registers        mov ebx, dword ptr [eax + 0xc]  ; context->ebx        mov edx, dword ptr [eax + 0x10] ; context->edx        mov esi, dword ptr [eax + 0x14] ; context->esi        mov edi, dword ptr [eax + 0x18] ; context->edi        push dword ptr [eax + 0x8]      ; context->eip        ret        ; should never return here    }}
This function looks simple, but simply restores the register value and sets the EIP pointer.
Fiber_swap_context
This function saves the current context and switches to the new context. This function can be implemented using fiber_get_context/fiber_set_context.
void fiber_swap_context(fiber_context* oldcontext, fiber_context* newcontext){    assert(oldcontext && newcontext);    // save the current context in the oldcontext and set the current context from newcontext    __asm    {        push oldcontext        call fiber_get_context        // fixup oldcontext->esp, ignore pushed arguments, since we'll resume from restore        mov eax, oldcontext        add dword ptr[eax + 0x4], 0x4        // fixup return address        mov dword ptr[eax + 0x8], offset restore        // switch to newcontext        push newcontext        call fiber_set_context        restore:    }}
Note one point when saving the current execution environment, because the value of the Save EIP pointer when we call fiber_get_context should be the address of the next instruction of call, where it is push eax, however, we do not want this, because we will call fiber_set_context later. Therefore, we must modify the EIP to skip the call to fiber_set_context. Note that the ESP value is also worth noting. Because we call fiber_get_context to manually pressure the parameter stack, the ESP value also contains the oldcontext parameter during the recovery from restore. However, after restore, we directly return the result, so we must correct the ESP value and subtract the oldcontext of the Pressure stack (in fact, in theory, ESP is not corrected because in the epilog of the function, the EBP is automatically assigned to ESP, so the correction is meaningless. However, Because VC inserts the stack integrity check code at the end of the function, it must be corrected to prevent this error ).
Todo
Here we only save GPRS, and do not store other registers, such as floating point control registers.
Summary
In environments where a large number of threads cannot be created, the fiber process provides some solutions, because the fiber process is more lightweight and can achieve higher concurrency.
Source code
The code is stored on GitHub for easy download. The git url is GIT: // github.com/alexshen/fiber.git, and the webpage address is https://github.com/alexshen/fiber.
Reference
[1] setjmp. h wikipeida
[2] qemu coroutine
[3]
GCC inline assembly
[4] calling conventions for different C ++ compilers and Operating Systems