Windows Buffer Overflow and Data Execution Protection (DEP)

Introduction to buffer overflow and Data Execution Protection Dep

First look at a buffer overflow C ++ instance program, the Code is as follows (compiled in vc6.0 ):

// By morewindows # include <windows. h> # include <conio. h> # include <stdio. h> # include <process. h> # include <string. h> void Foo (const char * input) {char Buf [4]; // Buf occupies 4 bytes, the last 4 bytes are EBP, And the last 4 bytes are return addresses. Strcpy (BUF, input); // overwrite the return address with the input string, and execute the bar () function} void bar (void) {printf ("Augh! This program have been hacked by morewindows! \ N "); getch (); exit (0); // exit the program directly because the EBP has been destroyed. otherwise, an error dialog box will pop up.} int main (INT argc, char * argv []) {printf ("Address of main = % P \ n", main ); printf ("Address of Foo = % P \ n", foo); printf ("Address of bar = % P \ n", bar); // construct a string, fill in the first eight characters, and then return the address of a bar () function. Char szbuf [50] = "12341234"; DWORD * pbaraddress = (DWORD *) & szbuf [8]; * pbaraddress = (DWORD) bar; Foo (szbuf ); return 0 ;}

The program running result is as follows:

It can be seen that the bar () function is not called in the program and only the Foo () function is called. However, due to the improper handling of the Foo () function, the Foo () function is modified after Buffer Overflow () the return address of the function causes the program to execute the bar () function that should not be executed. Of course, the actual overflow attack is much more complicated than this example, but the same principle is that the program executes the code that should not be executed by tampering with the return address of the function.

 

Dep principle of Data Execution Protection

The root cause of the analysis of buffer overflow attacks is that modern computers do not clearly distinguish data and code from this first-day defect. At present, it is basically impossible to design a new computer architecture, we can only rely on forward-compatible patching to reduce the damage caused by overflow. Data Execution Protection (DEP) is a natural defect in computer obfuscation of data and code.

The basic principle of DEP is to mark the Memory Page of the data as unexecutable. When the program overflows and is successfully transferred to shellcode (note 1), the program will attempt to execute the command on the data page, in this case, the CPU throws an exception instead of executing malicious commands. 1.

Figure 1 Dep Working Principle

Dep is mainly used to prevent data pages (such as default heap pages, various stack pages, and memory pool pages) from executing code. Microsoft started to provide this technical support from Windows XP SP2, which is divided into software Dep and hardware Dep (hardware-enforced dep) based on different implementation mechanisms ).

The software DEP has nothing to do with the CPU hardware. In essence, Windows uses software simulation to implement DEP, which provides some protection for the operating system.

Hardware DEP is the true dep. Hardware Dep requires CPU support. Both AMD and Intel have designed the dep. AMD calls it no-Execute page-protection (nx ), intel is called execute disable bit (xd). The functions and working principles of the two are essentially the same.

The operating system specifies that the Code cannot be executed from the memory by setting the NX/XD attribute mark on the Memory Page. To implement this function, you need to add a special identification space (NX/xd) to the page table in the memory to identify whether commands can be executed on the page. When the flag is set to 0, the command can be executed on this page. If it is set to 1, the command cannot be executed on this page.

 

Set Data Execution Protection in the system

You can use the following method to check whether the CPU supports hardware Dep and right-click the "my computer" à "properties" à "advanced" tab on the desktop. On the "advanced" tab page, under "performance", click "set" to open the "performance options" page. Click the "Data Execution Protection" tab. on this page, you can check whether your computer's CPU supports dep. If the CPU does not support hardware DEP, the following message is displayed at the bottom of the page: "Your computer's processor does not support hardware-based dep. However, Windows can use the DEP software to protect against certain types of attacks. ", 2 shows.

Figure 2 Dep option page in Windows XP

Depending on the startup parameters, DEP can work in four states:

(1) optin: by default, only Dep protection is applied to components and services in windows, but not to other programs. However, you can use the application compatibility tool (Act, application compatibility Toolkit) enable DEP for the selected program. Dep is automatically applied to programs compiled by the/nxcompat option under Vista. this mode can be dynamically disabled by applications. It is mostly used in common user operating systems, such as Windows XP, Windows Vista, and Windows 7.

(2) optout: Enables DEP for all programs and services not included in the exclusion list. You can manually specify programs and services that do not enable Dep protection in the exclusion list. This mode can be dynamically disabled by applications. It is mostly used for server version operating systems, such as Windows Server 2003 and Windows Server 2008.

(3) alwayson: Enable Dep protection for all processes. There is no sorting list. In this mode, DEP cannot be disabled, currently, only 64-bit operating systems work in alwayson mode.

(4) alwaysoff: DEP is disabled for all processes. In this mode, DEP cannot be dynamically enabled. This mode is generally used only in a specific scenario, for example, DEP interferes with the normal operation of the program.

You can switch the optin and optout modes by switching the check boxes in Figure 2. You can also modify the value of the/noexecute startup Item in C: boot. ini to control the working mode of dep. As shown in 3, the operating mode of Dep on the XP operating system is optin.

Figure 3 default startup status of DEP in Windows XP

 

Dep protection using data in programming

After introducing the working principle and status of DEP, let's look at a program connection option closely related to Dep:/nxcompat. /Nxcompat is a link option introduced in Visual Studio 2005 and later versions. It is enabled by default. Visual Studio 2008 (vs 9.0 ), in the menu, choose "project"> "bottom Project Properties"> "linker"> "advanced" to view the Data Execution Protection (DEP. 4:

Figure 4/nxcompat compilation options in vs 2008

Therefore, if you use vs2008 to compile the above sample code, the bar () function will not be executed when the program runs the buffer overflow function, but a prompt box will pop up directly, as shown in Figure 5:

Figure 5 A program generated in vs2008 automatically defends against buffer attacks

 

What are the benefits of programs compiled by/nxcompat? Based on the previous introduction, we know that Dep generally works in the optin state in the operating system of the user edition. In this case, DEP only protects the core processes of the system, but does not protect common programs. Although users can add them by themselves through tools, this increases the security threshold virtually, So Microsoft has released the/nxcompat compilation option. Dep protection is automatically enabled for programs compiled by/nxcompat on Windows Vista and later operating systems. It can be seen that it is very easy to use Data Execution Protection in programming to enhance program security.

 

Dep limitations

Dep improves the memory management mechanism for the source of overflow attacks. By setting the Memory Page to an unexecutable state, the Shellcode Execution in the stack is blocked. This base-paid Mechanism brings unprecedented challenges to buffer overflow. This is also the most powerful protection mechanism we have encountered so far. Can it completely prevent buffer overflow attacks? The answer is no. Like the security mechanism described earlier, DEP also has its own limitations:

First, hardware Dep requires CPU support, but not all CPUs provide hardware Dep support. In some old CPUs, DEP cannot work.

Secondly, due to compatibility, Windows cannot enable Dep protection for all processes; otherwise, exceptions may occur. For example, some third-party plug-in DLL cannot confirm whether it supports DEP, so it is hard to enable Dep protection for programs involving these DLL. In addition, programs using ATL 7.1 or earlier versions need to generate executable code on the data page. In this case, DEP protection cannot be enabled; otherwise, exceptions may occur.

Again, the/nxcompat compilation option, or the image_dllcharacteristics_nx_compat setting, is only valid for Windows Vista and later systems. In previous systems, such as Windows XP SP3, this setting is ignored. That is to say, even programs that use this link option do not automatically enable Dep protection on some operating systems.

Finally, when Dep works in the two most important statuses optin and optout, DEP can be dynamically disabled and enabled, this indicates that the operating system provides some API functions to control the DEP status. Unfortunately, there are no restrictions on the calling of these API functions in the early operating system. All processes can call these API functions, which poses a great security risk, it also provides a path for attackers who break through dep.

 

 

 

Note 1. shellcode refers to a piece of code used to send to the server to exploit a specific vulnerability (or to fill in data), which is generally used to obtain permissions. In addition, shellcode is usually sent to the attacked service as data. Shellcode is the core of overflow programs and worms.

 

 

Reprinted please indicate the source, original address: http://blog.csdn.net/morewindows/article/details/6887136