Malware anti-Detection Technology Introduction

In this article, we will introduce various anti-Debugging techniques used by malware to impede reverse engineering, so as to help readers better understand these technologies, this enables more effective dynamic detection and analysis of malware.
I. Anti-debugging technology

Anti-debugging is a common anti-detection technique, because malware always tries to monitor its own code to detect whether it is being debugged. To do this, malware can check whether a breakpoint is set for its code, or detect the debugger directly by calling the system.

1. breakpoint

To detect whether a breakpoint is set for the code, the malware can search for the command operation code 0xcc (the debugger uses this command to gain control of the malware at the breakpoint), which may cause a SIGTRAP. If the malware Code itself creates a separate handler, the malware can also set a pseudo breakpoint. In this way, malware can continue to execute its commands when a breakpoint is set.

Malware can also try to overwrite breakpoints. For example, some viruses use reverse decryption loops to overwrite breakpoints in viruses. On the contrary, other viruses use the Hamming code to correct their own code. The Hamming Code allows the program to detect and modify errors, but here it enables the virus to detect and clear breakpoints in its code.

2. Calculate the checksum

Malware can also calculate its own checksum. If the checksum changes, the virus will assume that it is being debugged and its code has been placed into a breakpoint. VAMPiRE is an anti-debugging tool that can be used to escape breakpoint detection. VaMPiRE maintains a breakpoint table in the memory, which records all breakpoints that have been set. This program consists of a page troubleshooting Program (PFH), a general protection troubleshooting Program (GPFH), a single-step processing program and a framework API. When a breakpoint is triggered, the control is either sent to the PFH (processing the breakpoint set in the code, data, or memory ing I/O ), or pass it to GPFH (processing the legacy I/O breakpoint ). A single-step processing program is used to store breakpoints so that they can be used multiple times.

3. Check the debugger

In Linux, the debugger has a simple method. You only need to call Ptrace, because Ptrace cannot be called more than twice consecutively for a specific process. In Windows, if the program is currently in the debugging status, the system calls isDebuggerPresent and returns 1; otherwise, 0. This system calls a simple check for a flag. This flag is set to 1 when the debugger is running. This check can be completed directly through the second byte of the process environment block. The following code shows this technology:

Mov eax, fs: [30 h]

Move eax, byte [eax + 2]

Test eax, eax

Jne @ DdebuggerDetected

In the above Code, eax is set to PEB (process environment block), then the second byte of PEB is accessed, and the content of this byte is moved into eax. Check whether eax is zero to complete this check. If it is zero, the debugger does not exist. Otherwise, a debugger exists.

If a process is created by a pre-running debugger, the system sets some flags for the heap operation routine in ntdll. dll. These FLG_HEAP_ENABLE_TAIL_CHECK, FLG_HEAP_ENABLE_FREE_CHECK, and FLG_HEAP_VALIDATE_PARAMETERS. We can check the flag using the following code:

Mov eax, fs: [30 h]

Mov eax, [eax + 68 h]

And eax, 0x70

Test eax, eax

Jne @ DebuggerDetected

In the above Code, we still access PEB, and then add the PEB address plus the offset of 68h to the starting position of these marks used by the heap operation routine, by checking these labels, You can see whether a debugger exists.

Check the signs such as ForceFlags In the heap header to check whether a debugger is running, as shown below:

Mov eax, fs: [30 h]

Mov eax, [eax + 18 h]; process heap

Mov eax, [eax + 10 h]; heap flags

Test eax, eax

Jne @ DebuggerDetected

The code above shows us how to access the heap and heap flag of a process through the offset of PEB. By checking this content, we can see whether the Force flag has been set to 1 in advance by the currently running debugger.

Another method to detect the debugger is to use the NtQueryInformationProcess system call. We can set ProcessInformationClass to 7 to call this function. This will reference ProcessDebugPort. if the process is being debugged, this function will return-1. The sample code is as follows.

Push 0 push 4 push offset isdebuggedpush 7; ProcessDebugPortpush-1 call NtQueryInformationProcesstest eax, eaxjne @ ExitErrorcmp isdebugged, 0jne @ DebuggerDetected

In this example, the NtQueryInformationProcess parameter is first pushed into the stack. These parameters are described as follows: the first is the handle (0 in this example), and the second is the length of Process Information (4 bytes in this example ), next is the process information category (ProcessDebugPort in this example). The next is a variable used to return information about whether a debugger exists. If the value is non-zero, it indicates that the process is running in a debugger; otherwise, it indicates that everything is normal. The last parameter is the return length. The returned value after NtQueryInformationProcess is called using these parameters is in isdebugged. Then, test whether the returned value is 0.

In addition, there are other methods to check the debugger, such as checking whether the device list contains the name of the debugger and whether the registry key for the debugger exists, and scan the memory to check whether the Code contains the debugger.

Another method similar to EPO is to notify the PE Loader to reference the program's entry point through the Thread Local Memory (TLS) Table item in the PE Header. This causes the code in TLS to be executed first, instead of reading the program's entry point. Therefore, TLS can complete the detection required for anti-debugging at the startup of the program. During TLS startup, the virus can be started before the debugger starts, because some debuggers are cut in at the main entry point of the program.

4. Probe single-step execution

Malware can also detect the debugger by performing a single-step inspection. To detect single-step execution, we can put a value into the stack pointer and check whether the value is still there. If the value is there, this means that the Code is being executed in a single step. When the debugger executes a process in a single step, when it obtains control, it needs to push some commands into the stack and release them out of the stack before executing the next command. Therefore, if this value is still there, it means that other running processes are using the stack. The following sample code shows how malware detects the single-step execution through the stack status:

Mov bp, sp; select Stack pointer

Push ax; Push ax to stack

Pop ax; select this value from the stack

Cmp word ptr [bp-2], ax; comparison with values in the stack

Jne debug; if different, the debugger is found.

As described above, a value is pushed into the stack and then popped up. If a debugger exists, the value at the position-2 of the stack pointer will be different from the value in the pop-up stack. In this case, you can take appropriate actions.

5. Detect speed attenuation during running

By observing whether the program is slowing down during running, malicious code can detect the debugger. If the program slows down significantly during running, it is likely that the Code is being executed in a single step. Therefore, if the timestamps of the two calls are very different, then the malware needs to take corresponding actions. Linux tracking tool kit LTTng/LTTV tracks viruses by observing the slowdown problem. When the LTTng/LTTV tracing program is running, it does not need to add breakpoints or perform any analysis. In addition, it uses a lockless re-import mechanism, which means that it will not lock any Linux kernel code, even if the kernel code is required by the tracked program, so it will not cause the program to be tracked to slow down and wait.

6. Command prefetch

If the malicious code tampered with the next instruction in the instruction sequence and the new instruction was executed, a debugger is running. This is caused by command prefetch: if the new command is prefetch, it means that there are other programs in the process of execution. Otherwise, the original commands are prefetch and executed.

7. Self-modified code

Malware can also allow other code to be modified by itself (other code can be modified by itself). Such an example is HDSpoof. The malware first starts some exception handling routines and then eliminates them during running. In this way, if any fault occurs, the running process throws an exception and the virus stops running. In addition, it sometimes cleans up or adds exception handling routines to tamper with the exception handling routine during running. The following is the code for HDSpoof to clear all exception handling routines (except the default exception handling routine.

Exception handlers before:

0x77f79bb8 ntdll. dll: executehandler2 @ 20 + 0x003a0x0041adc9 hdspoof.exe + 0x0001adc90x77e94809 _ effect_handler3

Exception handlers after:

0x77e94809 _ effect_handler3

0x41b770: 8b44240c mov eax, dword ptr [esp + 0xc] 0x41b774: 33c9 xor ecx, ecx 0x41b776: 334804 xor ecx, dword ptr [eax + 0x4] 0x41b779: 334808 xor ecx, dword ptr [eax + 0x8] 0x41b77c: 33480c xor ecx, dword ptr [eax + 0xc] 0x41b77f: 334810 xor ecx, dword ptr [eax + 0x10] 0x41b782: 8b642408 mov esp, dword ptr [esp + 0x8] 0x41b786: 648f0500000000 pop dword ptr fs: [0x0]

The following is the code for creating a new exception handling program for HDSpoof.

0x41f52b: add dword ptr [esp], 0x9ca

0x41f532: push dword ptr [dword ptr fs: [0x0]

0x41f539: mov dword ptr fs: [0x0], esp

8. Overwrite debugging program information

Some malware uses various technologies to overwrite debugging information, which can lead to abnormal functions of the debugger or virus itself. By capturing and interrupting INT 1 and INT 3 (INT 3 is the operation code 0xCC used by the debugger), malware may also cause the debugger to lose its context. This does not affect normal running viruses. Another option is to hook various interruptions and call other interruptions to indirectly run virus code.

The following code uses the Tequila virus to hook INT 1:

New_interrupt_one:

Push bp

Mov bp, sp

Cs cmp B [0a], 1; masm mod. needed

Je 0506; masm mod. needed

Cmp w [bp + 4], 09b4

Ja 050b; masm mod. needed

Push ax

Push es

Les ax, [bp + 2]

Cs mov w [09a0], ax; masm mod. needed

Cs mov w [09a2], es; masm mod. needed