Compiler stack protection technology in GCC

Source: Internet
Author: User

Compiler stack protection technology in GCC

Buffer overflow, represented by stack overflow, has become the most common security vulnerability. Security problems are common. As early as 1988, Morris, a graduate student in the Computer Science Department at Cornell University, used a Unix fingered program overflow vulnerability to write a malicious program and spread it to other machines, as a result, 6000 servers on the Internet were paralyzed, accounting for 10% of the total number at that time. There are countless overflow vulnerabilities in various operating systems. To prevent the buffer overflow vulnerability from being exploited by attackers as much as possible, today's compiler designers have begun to protect the stack at the compiler level. There are already several compiler stack protection implementations, the most famous of which is
Stackguard and stack-smashing protection (SSP, also known as propolice ).

Compiler stack protection principle

We know that when attackers exploit the stack overflow vulnerability, they usually destroy the current function stack. For example, when an attacker exploits the function stack overflow vulnerability in Listing 1, the typical situation is that the attacker attempts to allow the program to write data that exceeds the length of the array to the name array, until the return address in the function stack is overwritten, when this function is returned, It redirects to the malicious code injected by the attacker or the shellcode for execution (for the principle of overflow attacks, see the principle and Countermeasures of buffer overflow attacks in Linux). Function stack becomes a graph after the overflow attack
As shown in figure 2, compared with the previous overflow (Figure 1), we can see that the EBP in the original Stack has been overwritten by the overflow string, that is, the function Stack has been damaged.

Listing 1. Code that may have an Overflow Vulnerability

int vulFunc() {    char name[10];    //…    return 0;}

Figure 1. Function stack before overflow

Figure 2. Function stack after Overflow

If such damages can be detected during runtime, the function Stack may be protected. Most of the current stack protection implementations use the "Canaries"-based detection technology to detect such damages.

"Canaries" probe:

To detect the damage to the function stack, You need to modify the structure of the function stack and insert a canary word between the buffer zone and control information (such as EBP. In this way, when the buffer zone overflows, Canary word will be overwritten before the returned address is overwritten. By checking whether the canary word value is modified, you can determine whether an overflow attack has occurred.

Common canary word:

  • Terminator canaries
  • The vast majority of overflow vulnerabilities are caused by C string processing functions that do not perform array out-of-bounds checks, and these strings all use null as the final character. It is natural to select characters such as null, Cr, lf as canary word. For example, if canary word is 0x000aff0d, in order to prevent overflow detection, attackers need to include 0x000aff0d in the overflow string and accurately calculate the location of canaries so that canaries does not seem to have been changed. However, 0x00 in 0x000aff0d will end the replication of strcpy () to prevent the return address from being overwritten. While
    0x0a will end gets () reading. The inserted Terminator canaries creates a lot of trouble for attackers.
  • Random canaries
  • Such canaries are randomly generated. In addition, such random numbers cannot be read by attackers. This random number is generated during program initialization and then stored in a memory page that is not hidden into the virtual address space. In this way, when attackers attempt to access the memory that saves random numbers through pointers, the segment fault will be triggered. However, because the copy of the random number will eventually be saved as a canary word in the function stack, attackers may still obtain the canary word value through the function stack.
  • Random XOR canaries
  • This Canaries is obtained by a random number and all the control information in the function stack, and the return address is obtained through an exclusive or operation. In this way, canaries in the function stack, or any control information or return address, can be detected.

Currently, Major compiler stack protection implementations, such as stack guard and stack-smashing protection (SSP) All use canaries detection as the main protection technology, but Canaries has different production methods. The following uses GCC as an example to briefly introduce the application of stack protection technology in GCC.

Back to Top

Stack protection implementation in GCC

Stack guard is the first implementation of stack protection using canaries detection. It was released as an extension of GCC in 1997. The first version of stack Guard uses 0x00000000 as canary word. Although many people suggest incorporating stack guard into GCC to provide stack protection as part of GCC. But in fact, GCC 3.x does not implement any stack protection. The stack protection implemented by gcc4.1 is not stack guard, but stack-smashing.
Protection (SSP, also known as propolice ).

SSP has been improved and improved based on Stack guard. It is developed and maintained by Hiroaki rtoh, an IBM engineer. Compared with Stack guard, SSP protects the returned address of the function and the EBP information in the stack. SSP also places the arrays in local variables in the high address of the function stack, while other variables in the low address. This makes it more difficult to modify other variables (such as a function pointer) by overflow an array.

Three stack protection-related compilation options in GCC 4.1


Enable stack protection, but only insert protection code for functions with Char arrays in local variables.


Enable stack protection and insert protection code for all functions.


Disable stack protection.

Canaries detection in GCC

The following uses an example to analyze the code generated by GCC stack protection. Use the-fstack-Protector option and-fno-Stack-Protector option to compile the code in Listing 2 to obtain the executable file demo_sp (-fstack-protector ), demo_nosp (-fno-Stack-protector ).

Listing 2. Demo. c

int main() {    int i;    char buffer[64];    i = 1;    buffer[0] = 'a';    return 0;}

Then we use GDB to disassemble demo_sp and deno_nosp respectively.

Listing 3. demo_nosp assembly code

(gdb) disas mainDump of assembler code for function main:0x08048344 <main+0>:    lea    0x4(%esp),%ecx0x08048348 <main+4>:    and    $0xfffffff0,%esp0x0804834b <main+7>:    pushl  0xfffffffc(%ecx)0x0804834e <main+10>:   push   %ebp0x0804834f <main+11>:   mov    %esp,%ebp0x08048351 <main+13>:   push   %ecx0x08048352 <main+14>:   sub    $0x50,%esp0x08048355 <main+17>:   movl   $0x1,0xfffffff8(%ebp)0x0804835c <main+24>:   movb   $0x61,0xffffffb8(%ebp)0x08048360 <main+28>:   mov    $0x0,%eax0x08048365 <main+33>:   add    $0x50,%esp0x08048368 <main+36>:   pop    %ecx0x08048369 <main+37>:   pop    %ebp0x0804836a <main+38>:   lea    0xfffffffc(%ecx),%esp0x0804836d <main+41>:   ret    End of assembler dump.

Listing 4. demo_sp assembly code

(gdb) disas mainDump of assembler code for function main:0x08048394 <main+0>:    lea    0x4(%esp),%ecx0x08048398 <main+4>:    and    $0xfffffff0,%esp0x0804839b <main+7>:    pushl  0xfffffffc(%ecx)0x0804839e <main+10>:   push   %ebp0x0804839f <main+11>:   mov    %esp,%ebp0x080483a1 <main+13>:   push   %ecx0x080483a2 <main+14>:   sub    $0x54,%esp0x080483a5 <main+17>:   mov    %gs:0x14,%eax0x080483ab <main+23>:   mov    %eax,0xfffffff8(%ebp)0x080483ae <main+26>:   xor    %eax,%eax0x080483b0 <main+28>:   movl   $0x1,0xffffffb4(%ebp)0x080483b7 <main+35>:   movb   $0x61,0xffffffb8(%ebp)0x080483bb <main+39>:   mov    $0x0,%eax0x080483c0 <main+44>:   mov    0xfffffff8(%ebp),%edx0x080483c3 <main+47>:   xor    %gs:0x14,%edx0x080483ca <main+54>:   je     0x80483d1 <main+61>0x080483cc <main+56>:   call   0x80482fc <__stack_chk_fail@plt>0x080483d1 <main+61>:   add    $0x54,%esp0x080483d4 <main+64>:   pop    %ecx0x080483d5 <main+65>:   pop    %ebp0x080483d6 <main+66>:   lea    0xfffffffc(%ecx),%esp0x080483d9 <main+69>:   ret    End of assembler dump.

The command with the address 0x08048344 in demo_nosp assembly code saves ESP + 4 to ECx. In this case, the return address is saved in the memory directed by ESP. The instruction with address 0x0804834b points to the memory pressure stack of the ecx-4, because the ESP + 4 has been previously stored in ECx, so after the instruction is executed, the content that the original ESP points to will be pressed to the stack, that is, the returned address is pushed to the stack again. The and command at 0x08048348 alignment the heap top in 16 bytes. Commands from 0x0804834e to 0x08048352 save the old EBP and set a new stack box for the function. When the function is complete, 0x08048360
Put the return value in the mov command at eax and restore the original EBP and ESP. It is not hard to see that the demo_nosp assembly code does not have any code to check and protect the stack.

Compare the demo_sp compiled with the-fstack-Protector option with the demo_nosp assembly code without stack protection. The most significant difference between the demo_sp and demo_nosp is that three statements are added before the function is actually executed:

0x080483a5 <main+17>:   mov    %gs:0x14,%eax0x080483ab <main+23>:   mov    %eax,0xfffffff8(%ebp)0x080483ae <main+26>:   xor    %eax,%eax

Four more statements are added before the function returns:

0x080483c0 <main+44>:   mov    0xfffffff8(%ebp),%edx0x080483c3 <main+47>:   xor    %gs:0x14,%edx0x080483ca <main+54>:   je     0x80483d1 <main+61>0x080483cc <main+56>:   call   0x80482fc <__stack_chk_fail@plt>

These extra statements are the key to SSP stack protection. With these codes, a canary is inserted in the function stack box, the canary is used to check whether the function stack is damaged.

% GS: 0x14 is stored as a random number, 0x080483a5 to 0x080483ae at the three statements put this random number into the position of the stack [EBP-8. The function returns four statements from 0x080483c0 to 0x080483cc, then removes the canary stored in the [EBP-8] in the stack and compares it with the random number in % GS: 0x14. If not, it indicates that the function has exceeded during execution, and the function stack box has been damaged. At this time, the program jumps to _ stack_chk_fail and outputs an error message to stop running. If they are equal, the function returns normally.

Adjust the sequence of local variables

The above code exposes the implementation of canary in GCC. Observe the assembly code of demo_sp and demo_nosp carefully. It is not difficult to find that there is another subtle difference between the two: In the semo_sp program with stack protection enabled, the order of local variables is reorganized.

In the program, movl $0x1, 0xffffff ** (% EBP) corresponds to I = 1;

Movb $0x61, 0xffffff ** (% EBP) corresponds to buffer [0] = 'a ';

In demo_nosp, the address of variable I is 0xfffffff8 (% EBP), and the address of buffer [0] Is 0xffffffb8 (% EBP ). It can be seen that in demo_nosp, before variable I is in the buffer array, the order of variables in memory is the same as that defined in the Code, as shown in Figure 3 on the left. In demo_sp, the address of variable I is 0xffffffb4 (% EBP), and the address of buffer [0] Is 0xffffffb8 (% EBP), that is, the buffer array is moved to the front of variable I, see figure 3 right.

Figure 3. Function stack before and after variable order adjustment

The organization of local variables in demo_sp is helpful for defending against some overflow attacks. If the array is behind other variables (left in figure 3), the return address cannot be modified even if it is protected by canary, attackers may also modify other local variables (such as int I in this example) through an overflow array ). When other modified local variables are a function pointer, attackers may exploit this overflow to overwrite the function pointer with the shellcode address to launch attacks. However, if you use the method on the Right of Figure 3 to organize the stack, it will bring great difficulties to such overflow attacks.

Back to Top

GCC stack protection effect

We have analyzed the stack protection in GCC from the implementation perspective. The following uses a small program overflow_test.c to verify the actual effect of GCC stack protection.

Listing 5. overflow_test.c

#include <stdio.h>#include <stdlib.h>char shellcode[] =    "\xeb\x1f\x5e\x89\x76\x08\x31\xc0\x88\x46\x07\x89\x46\x0c\xb0\x0b"    "\x89\xf3\x8d\x4e\x08\x8d\x56\x0c\xcd\x80\x31\xdb\x89\xd8\x40\xcd"    "\x80\xe8\xdc\xff\xff\xff/bin/sh";int test(){    int i;    unsigned int stack[10];    char my_str[16];    printf("addr of shellcode in decimal: %d\n", &shellcode);    for (i = 0; i < 10; i++)        stack[i] = 0;    while (1) {        printf("index of item to fill: (-1 to quit): ");        scanf("%d",&i);        if (i == -1) {            break;        }        printf("value of item[%d]:", i);        scanf("%d",&stack[i]);    }    return 0;}int main(){    test();    printf("Overflow Failed\n");    return 0;}

This program is not an actual vulnerability program, nor an attack program. It is only a test program that verifies GCC stack protection by simulating overflow attacks. It first prints the shellcode address, then accepts user input, assigns values to the elements specified in the stack array, and does not check the array boundary.

Disable stack protection and compile programs

aktoon@aktoon-thinkpad:~/SCAD/overflow_test$ gcc –fno-stack-protector -o overflow_test ./overflow_test.c

When stack protection is disabled, stack [12] points to the location where the returned address is stored in the stack. Fill in the shellcode address in stack [10], stack [11], and stack [12] to simulate General overflow attacks:

aktoon@aktoon-thinkpad:~/SCAD/overflow_test$ ./overflow_test addr of shellcode in decimal: 134518560index of item to fill: (-1 to quit): 10value of item[11]: 134518560index of item to fill: (-1 to quit): 11value of item[11]: 134518560index of item to fill: (-1 to quit): 12value of item[12]:134518560index of item to fill: (-1 to quit): -1$ ps  PID TTY          TIME CMD15035 pts/4    00:00:00 bash29757 pts/4    00:00:00 sh29858 pts/4    00:00:00 ps

The program is successfully overflows and then runs shellcode to obtain a shell. Because stack protection is not enabled, overflow is successful.

Enable stack protection and compile and run the program again.

aktoon@aktoon-thinkpad:~/SCAD/overflow_test$ gcc –fno-stack-protector -o overflow_test ./overflow_test.c

Through GDB disassembly, it is difficult to figure out that after stack protection is enabled, the returned address is located at stack [17], while canary is located at stack [16. In stack [10], stack [11]… Fill in the shellcode address in stack [17] to simulate overflow attacks:

aktoon@aktoon-thinkpad:~/SCAD/overflow_test$ ./overflow_test addr of shellcode in decimal: 134518688index of item to fill: (-1 to quit): 10value of item[11]: 134518688index of item to fill: (-1 to quit): 11value of item[11]: 134518688index of item to fill: (-1 to quit): 12value of item[11]: 134518688index of item to fill: (-1 to quit): 13value of item[11]: 134518688index of item to fill: (-1 to quit): 14value of item[11]: 134518688index of item to fill: (-1 to quit): 15value of item[11]: 134518688index of item to fill: (-1 to quit): 16value of item[12]: 134518688index of item to fill: (-1 to quit): 17value of item[12]: 134518688index of item to fill: (-1 to quit): -1Overflow Failed*** stack smashing detected ***: ./overflow_test terminatedAborted

The overflow attack failed. The "Stack smashing detected" prompt indicates that the overflow was detected. According to the previous analysis of the stack protection code generated by GCC, the failure should be caused by a canary change. Through disassembly and computation, we know that the return address is in stack [17], while canary is in stack [16]. Next, try to bypass canary and modify the return address only.

aktoon@aktoon-thinkpad:~/SCAD/overflow_test$ ./overflow_test addr of shellcode in decimal: 134518688index of item to fill: (-1 to quit): 17value of item[17]:134518688index of item to fill: (-1 to quit): -1$ ls *.cbypass.c  exe.c  exp1.c  of_demo.c  overflow.c  overflow_test.c  toto.c  vul1.c$

This time, only stack [17] is overwritten with the shellcode address. Because canary is not modified, the modification of the returned address is not detected, and shellcode is successfully executed. Similarly, even if we do not modify the return address of the function, the program will be suspended as long as canary is modified (stack [16.

aktoon@aktoon-thinkpad:~/SCAD/overflow_test$ ./overflow_test addr of shellcode in decimal: 134518688index of item to fill: (-1 to quit): 16value of item[16]:134518688index of item to fill: (-1 to quit): -1Overflow Failed*** stack smashing detected ***: ./overflow_test terminatedAborted

In the test above, we can see that the protective code inserted by the compiler prevents the General overflow attack. In fact, the current compiler stack protection technology does make Stack Overflow attacks more difficult.

Back to Top

Limitations of GCC stack protection

In the above example, we found that if attackers can purchase a bypass canary, they may still successfully launch an overflow attack. In addition, there are some other methods that can break through the protection of the compiler. Of course, these methods require more skills and are more difficult to apply. The following describes how to break through the compiler stack protection.

The canary detection method only protects the control information (Canary word, EBP) and return address in the function heap, and does not protect local variables. Overwriting certain local variables through overflow may also cause overflow attacks. In addition, both stack guard and SSp provide Stack Overflow Protection and cannot defend against Stack Overflow attacks.

In some cases, attackers can also use function parameters to initiate overflow attacks. The following example is used to illustrate the attack principles.

LIST 6. Vulnerability code vul. c

int func(char *msg) {    char buf[80];    strcpy(buf,msg);    strcpy(msg,buf);}int main(int argv, char** argc) {    func(argc[1]);}

Shows the stack box of the func function during runtime.

Figure 4. Function stack of func

Through strcpy (BUF, MSG), we can overflow the Buf array until the msg parameter is overwritten. Next, strcpy (MSG, Buf) writes Buf content to the memory to which MSG points. Because the content of MSG has been controlled in the first overflow, we can write any data to any unprotected memory through the above two steps. Although Canaries has been damaged in the preceding two steps, this does not affect the overflow attack, because the Canaries check is only performed before the function returns. By constructing a suitable overflow string, we can modify
Got (Global Offset Table ). If we modify the _ exit () entry in got through an overflow string and point it to our shellcode, after checking that canary is modified before the function returns, an error is prompted and _ exit () is called to stop the program. At this time, the _ exit () point to our shellcode, so the program will not exit and the shellcode will be executed, thus achieving the purpose of an overflow attack.

The preceding example shows how to use parameters to prevent overflow attacks. In addition, because the return address is located based on EBP, even if we cannot modify the return address, if we can modify the EBP value, the location where the returned address is stored is modified, it is equivalent to indirectly modifying the return address. It can be seen that the stack protection of GCC is not omnipotent. It still has some limitations and cannot completely prevent stack overflow attacks. Although we may still have some skills to break through the stack protection of the compiler, these skills are usually restricted by many conditions and are difficult to apply in practice.

Back to Top


This article introduces the Canaries detection-based stack protection technology adopted by the compiler, and uses GCC as an example to demonstrate the implementation method and actual effect of SSP. Finally, I briefly introduced some methods to break through compiler protection. Although attackers can still use some techniques to break through compiler protection, the stack protection mechanism added by the compiler does cause great difficulties for overflow attacks. This article only discusses how to defend against overflow attacks from the perspective of compilers. To really prevent Stack Overflow attacks, it is still not enough to start with the compiler alone, and a sound system security policy is also very important. In addition, good programming habits, using libc with array out-of-bounds check also plays an important role in preventing overflow attacks.


  • Please refer to the official propolice Website: /.
  • Refer to

    Buffer overflows: attacks and defenses for the vulnerability of the decade.
  • Refer to
    Principles and Countermeasures of buffer overflow attacks in Linux
  • Find Linux developers (including
    For more information, see us.
    The most popular articles and tutorials.
  • Refer to all
    Linux skills and
    Linux tutorial.

About the author

He wenlei, a student from the School of Information Security Engineering, Shanghai Jiao Tong University, is interested in Linux and security technologies. Currently, I have an internship in the storage R & D department of IBM China System Technology Lab.

Contact Us

The content source of this page is from Internet, which doesn't represent Alibaba Cloud's opinion; products and services mentioned on that page don't have any relationship with Alibaba Cloud. If the content of the page makes you feel confusing, please write us an email, we will handle the problem within 5 days after receiving your email.

If you find any instances of plagiarism from the community, please send an email to: and provide relevant evidence. A staff member will contact you within 5 working days.

A Free Trial That Lets You Build Big!

Start building with 50+ products and up to 12 months usage for Elastic Compute Service

  • Sales Support

    1 on 1 presale consultation

  • After-Sales Support

    24/7 Technical Support 6 Free Tickets per Quarter Faster Response

  • Alibaba Cloud offers highly flexible support services tailored to meet your exact needs.