x86 function calling convention

The following excerpt from IDA Pro, it seems that some of the details are not explained clearly, need further thinking, practice.

After understanding the basic concepts of stack frames, the next step is to describe their structure in detail. The following example deals with the x86 architecture and behavior related to common x86 compilers, such as Microsoft Visual C + + or GNU gcc/g++. The most important step in creating a stack frame is to put the function into the stack by calling the function. The calling function must store the parameters required for the function being called, or it may cause serious problems. Each function chooses and follows a specific calling convention to indicate how they want to receive the parameter.

The calling convention specifies the exact location of the parameters required by the caller to place the function. Call a predetermined possible requirement to place parameters in a specific register, program stack, or register and stack. It is also important that, when passing parameters, the stack decides: who is responsible for removing these parameters from the stack after the called function has completed its operation. Some calling conventions stipulate that the caller is responsible for removing the parameters it places on the stack, while others require that the called function be responsible for deleting the parameters in the stack. Following the specified calling convention is especially important for maintaining the integrity of the program stack pointers.

1.C calling convention

Many C compilers of the x86 architecture use the default calling convention called the C calling convention. If the default calling convention is overridden, the commonly used _cdecl modifier in A/C + + program forces the compiler to take advantage of the C calling convention. From now on, we call this calling convention the cdecl calling convention. The cdecl calling convention stipulates that the caller will put the function arguments in a right-to-left order and that the caller (not the callee) is responsible for purging the parameters from the stack when the called function completes its operation.

One result of putting a parameter in the stack from right to left is that if the function is called, the leftmost (first) parameter will always be at the top of the stack. So no matter how many parameters the function requires, we can easily find the first parameter. Therefore, the cdecl calling convention is well suited for functions that have a variable number of arguments (such as printf).

Requiring the calling function to remove parameters from the stack means that you will often see that the instruction is immediately adjusted to the stack pointer after it is returned by the called function. If a function can accept a variable number of arguments, the caller is well suited for this kind of adjustment because it knows clearly how many parameters it passes as a function, making it easy to make the right adjustments. The called function has no way of knowing how many parameters it will receive in advance, making it difficult to make the necessary adjustments to the stack.

In the following example, we call a function that has the following prototype:

void demo_cdecl (intint. int int z)

By default, this function will use CDECL to call the reservation and want you to press 4 arguments in right-to-left order, while requiring the caller to clear the parameters in the stack, the compiler may generate the following code for the function's call:

;demo_cdecl (1, 2, 3, 4);//programer calls Demo_cdecl1.Push 4                ;push parameter Z   Push 3                ;Push parameter y   Push 2                ;Push parameter x   Push 1                ;Push parameter W   PagerDemo_cdecl;Call the function2.AddEsp -;Adjust ESP to its former value

A 4 push operation starting at 1 causes the program stack pointer (esp to occur) to change from 16 bytes (4*sizeof (int) on 32-bit architectures) and is revoked at 2 after returning from DEMO_CDECL. If DEMO_CDECL is called 50 times, an adjustment similar to 2 will occur after each call. The following example also complies with the cdecl calling convention, but after each call to DEMO_CDECL, the caller does not need to delete the parameters in the stack.

;demo_cdecl (1, 2, 3, 4)//PROGRAMEMR calls Demo_cdecl  mov[esp+ A],4   ;move parameter z to fourth position on stack  mov[esp+8],3   ;move parameter y to third position on stack  mov[esp+4],2   ;move parameter x to second position on stack  mov[ESP],1   ;move parameter w to top of stack  PagerDemo_cdecl;Call the function

In this example, in the "Prologue" phase of the function, the compiler has pre-allocated storage space at the top of the stack for DEMO_CDECL parameters. When the DEMO_CDECL parameter is placed on the stack, there is no need to modify the stack pointer, so you do not need to adjust the stack pointer after the call to Demo_cdecl ends. The GNU compiler (GCC and g++) is using this technique to put function parameters on the stack. Note that no matter which method is used, the stack pointer points to the leftmost parameter when the function is called.

2. Standard calling conventions

The standard here seems to be somewhat inappropriate, because it is the name Microsoft has for its own calling convention. This convention uses the modifier _stdcall in the function declaration, as follows:

void _stdcall demo_stdcall (intint int y);

To avoid confusion with the term "standard", in the remainder of this book, we refer to this calling convention as the stdcall calling convention.

As with the cdecl calling convention, the stdcall calling convention places function arguments on the program stack in a right-to-left order. The difference between using the stdcall calling convention is that when the function finishes executing, the function arguments in the stack should be deleted by the called function. For the function to be called, to accomplish this task, it must be clear how many parameters are in the stack, which is only possible if the number of parameters accepted by the function is fixed. Therefore, printf, a function that accepts a variable number of arguments, cannot use the stdcall calling convention. For example, the Demo_stdcall function requires 3 integer parameters and occupies 12 bytes on the stack (32-bit architecture is 3*sizeof (int)). The x86 compiler can use a special form of the RET instruction, extracting the return address from the top of the stack, and adding 12 to the stack pointer to clear the function parameters. Demo_stdcall may return to the caller using the following directive:

ret  a    ; return and Clear bytes from the stack

The main advantage of using stdcall is that after each function call, you do not need to clear the parameters from the stack through code, so you can generate a slightly smaller, faster program. By convention, Microsoft uses the StdCall convention for all functions that are fixed by the number of parameters that are output by shared library (DLL) files. If you are trying to generate a function prototype or binary compatible replacement for a shared library component, be sure to keep this in mind.

3.x86 Fastcall Convention

The Fastcall convention is a variant of the StdCall convention, which passes a maximum of two parameters to the CPU register (not the program stack). Microsoft Visual C + + and GNU gcc/g++ (3.4 and lower (should be higher?). --shijianyujingshen) compiler is able to recognize the fastcall modifier in a function declaration. If you specify to use the Fastcall convention, the first two parameters passed to the function are located in the ECX and edx registers respectively. The remaining parameters are placed on the stack from right to left in a manner similar to stdcall conventions. Similarly, similar to the StdCall convention, the Fastcall function is responsible for removing parameters from the stack when returning its callers. The Fastcall modifier is used in the following declaration:

void fastcall demo_fastcall (intint. int int z)

To invoke Demo_fastcall, the compiler may generate the following code:

;Demo_fastcall (1, 2, 3, 4); Programer calls demo FastcallPush  4                ;move parameter z to second position on stackPush  3                ;move parameter y to tio position on stackmovEdX2           ;move parameter x to edxmovEcx1           ;move parameter W to ECXPagerDemo_fastcall;Call the function

Note that after calling Demo_fastcall returns, it is not necessary to adjust the stack, because Demo_fastcall is responsible for clearing the parameters Y and Z from the stack when returning to the caller. Since two parameters are passed to the register, it is important to understand that the called function simply clears 8 bytes from the stack, even if the function has 4 parameters.

4.c++ calling convention

Non-static member functions in C + + classes are different from standard functions, and they need to use the this pointer, which points to the object used to invoke the function. The address of the object used to invoke the function must be provided by the caller, so it is supplied as a parameter when the non-static member function is called. It is not surprising that the C + + language standard does not specify how the this pointer should be passed to non-static member functions, so different compilers use different techniques to pass the this pointer.

Microsoft Visual C + + provides the thiscall calling convention, which passes the this pointer to the ECX register, and, as in stdcall, requires non-static member functions to purge parameters from the stack. The GNU g++ compiler regards this as the first implied parameter of any non-static member function, and in all other respects the same as the use of the CDECL Convention. Therefore, for code compiled with g++, this is placed at the top of the stack before a non-static member function is called, and the caller is responsible for removing the stack parameter (at least one parameter) when the function returns. Other features of compiled C + + code are discussed in chapter 8th.

5. Other calling conventions

To fully describe each of the existing calling conventions, you may need to write a book. The calling convention is typically language-specific, compiler, and CPU. If you encounter a more uncommon compiler-generated code, you may need to do some research yourself. However, the following situations require special attention: optimizing code, customizing assembly language code, and system invocation.

If an output function, such as a library function, is intended for use by other programmers, it must conform to the prevailing calling convention so that programmers can easily invoke these functions. In addition, if the function is used only by internal programs, the function requires a calling convention that is known only by the program of the function. In such cases, the optimization compiler chooses to use an alternate calling convention to generate faster-running code. Examples include using the/GL option in Microsoft Visual C + + and using the Regparm keyword in the GNU gcc/g++ .

If programmers are not afraid of trouble and use assembly language, they will have complete control over how to pass parameters to the functions they create. Unless they want to create functions for other programmers to use, assembly-language programmers can pass parameters in any way they deem appropriate. Therefore, be cautious when parsing custom assembly code. Custom assembly code can often be seen in fuzzy routines (obfuscation routine) and shellcode.

A system call is a special function call that is used to request an operating system service. Typically, system calls cause state transitions, which are entered into kernel mode by user mode so that the operating system kernel executes the user's request. The way system calls are initiated varies by operating system and CPU. For example, a Linux x86 system call starts with an int 0x80 instruction or sysenter instruction, while the other x86 operating systems may use only sysenter instructions. On many x86 systems (Linux is an exception), the parameters of the system call are on the runtime stack, and a system call number is placed in the EAX register before the system call is initiated. Linux system calls accept parameters that are located in a particular register, and sometimes, if the available registers cannot store all the parameters, it also accepts parameters that are in memory.

x86 function calling convention