Malware anti-detection technology introduction: Analysis of Anti-debugging technology (1)

In the previous article, we will introduce anti-simulation technologies commonly used by malware to readers. In this article, we will introduce various anti-Debugging techniques used by malware to impede reverse engineering, so as to help readers better understand these technologies, this enables more effective dynamic detection and analysis of malware.

I. Anti-debugging technology

Anti-debugging is a common anti-detection technique, because malware always tries to monitor its own code to detect whether it is being debugged. To do this, malware can check whether a breakpoint is set for its code, or detect the debugger directly by calling the system.

1. breakpoint

To detect whether a breakpoint is set for the code, the malware can find the command operation code 0xcc. The debugger uses this command to gain control of the malware at the breakpoint. This causes a SIGTRAP. If the malware Code itself creates a separate handler, the malware can also set a pseudo breakpoint. In this way, malware can continue to execute its commands when a breakpoint is set.

Malware can also try to overwrite breakpoints. For example, some viruses use reverse decryption loops to overwrite breakpoints in viruses. On the contrary, other viruses use the Hamming code to correct their own code. The Hamming Code allows the program to detect and modify errors, but here it enables the virus to detect and clear breakpoints in its code.

2. Calculate the checksum

Malware can also calculate its own checksum. If the checksum changes, the virus will assume that it is being debugged and its code has been placed into a breakpoint. VAMPiRE is an anti-debugging tool that can be used to escape breakpoint detection. VaMPiRE maintains a breakpoint table in the memory, which records all breakpoints that have been set. This program consists of a page troubleshooting Program (PFH), a general protection troubleshooting Program (GPFH), a single-step processing program and a framework API. When a breakpoint is triggered, the control is either sent to the PFH (processing the breakpoint set in the code, data, or memory ing I/O ), or pass it to GPFH to process the legacy I/O breakpoints ). A single-step processing program is used to store breakpoints so that they can be used multiple times.

3. Check the debugger

In Linux, the debugger has a simple method. You only need to call Ptrace, because Ptrace cannot be called more than twice consecutively for a specific process. In Windows, if the program is currently in the debugging status, the system calls isDebuggerPresent and returns 1; otherwise, 0. This system calls a simple check for a flag. This flag is set to 1 when the debugger is running. This check can be completed directly through the second byte of the process environment block. The following code shows this technology:

mov eax, fs:[30h]move eax, byte [eax+2]test eax, eax jne @DdebuggerDetected

In the above Code, eax is set to PEB (process environment block), then the second byte of PEB is accessed, and the content of this byte is moved into eax. Check whether eax is zero to complete this check. If it is zero, the debugger does not exist. Otherwise, a debugger exists.

If a process is created by a pre-running debugger, the system sets some flags for the heap operation routine in ntdll. dll. These FLG_HEAP_ENABLE_TAIL_CHECK, FLG_HEAP_ENABLE_FREE_CHECK, and FLG_HEAP_VALIDATE_PARAMETERS. We can check the flag using the following code:

mov eax, fs:[30h]mov eax, [eax+68h]and eax, 0x70test eax, eaxjne @DebuggerDetected

In the above Code, we still access PEB, and then add the PEB address plus the offset of 68h to the starting position of these marks used by the heap operation routine, by checking these labels, You can see whether a debugger exists.

Check the signs such as ForceFlags In the heap header to check whether a debugger is running, as shown below:

mov eax, fs:[30h]mov eax, [eax+18h] ;process heapmov eax, [eax+10h] ;heap flagstest eax, eaxjne @DebuggerDetected

The code above shows us how to access the heap and heap flag of a process through the offset of PEB. By checking this content, we can see whether the Force flag has been set to 1 in advance by the currently running debugger.

Another method to detect the debugger is to use the NtQueryInformationProcess system call. We can set ProcessInformationClass to 7 to call this function. This will reference ProcessDebugPort. if the process is being debugged, this function will return-1. The sample code is as follows.

push 0 push 4 push offset isdebugged push 7 ;ProcessDebugPort push -1 call NtQueryInformationProcess test eax, eax jne @ExitError cmp isdebugged, 0 jne @DebuggerDetected

In this example, the NtQueryInformationProcess parameter is first pushed into the stack. These parameters are described as follows: the first is that the handle is 0 in this example), and the second is that the length of process information is 4 bytes in this example ), next, the Process Information category is 7 in this example, indicating ProcessDebugPort), and the next is a variable used to return information about whether a debugger exists. If the value is non-zero, it indicates that the process is running in a debugger; otherwise, it indicates that everything is normal. The last parameter is the return length. The returned value after NtQueryInformationProcess is called using these parameters is in isdebugged. Then, test whether the returned value is 0.

In addition, there are other methods to check the debugger, such as checking whether the device list contains the name of the debugger and whether the registry key for the debugger exists, and scan the memory to check whether the Code contains the debugger.

Another method similar to EPO is to notify the PE Loader to reference the program's entry point through the Thread Local Memory (TLS) Table item in the PE Header. This causes the code in TLS to be executed first, instead of reading the program's entry point. Therefore, TLS can complete the detection required for anti-debugging at the startup of the program. During TLS startup, the virus can be started before the debugger starts, because some debuggers are cut in at the main entry point of the program.