Experience of Reverse Analysis in locating Algorithms

When we started to reverse analyze a function of a M image processing software, we found that reverse engineering was a little tricky. After more than a month of hard work, we finally got it done. I hope to encounter problems because I can learn new things in this way. In fact, I have encountered many problems and learned a lot of experience.

Previously, debugging started when the reverse was directly thrown to OD, but now we know that running in a virtual machine can be completed in less than an hour or two, because such a large software can be taken directly in a snapshot of the virtual machine, just like a game archive, this can avoid registration and activation of software running each time, avoid repeating previous analyses, and avoid dynamic changes in memory addresses, which has many benefits.

One of the difficulties in analyzing algorithms is to locate the Core algorithms. It is more difficult for software with software protection. The anti-debugging of this software I analyzed only has one isdebugpresent, And the OD plug-in can be easily bypassed. Fortunately, there is no virtualization protection, or the results may be different.

Since it is an image processing function, there are two ideas: one is to find the core algorithm by tracking image data, and the other is to find the core algorithm through several parameters that affect the algorithm. At the beginning, I chose the Core Algorithm for image data location tracking. I set a breakpoint on createfile and readfile, but I didn't find it. Then I changed the method and found the core algorithm through positioning parameters, all the way to trace the parameters read from the resource file, but after several passes, they are lost and set various breakpoints, but they cannot be broken. Finally, I used the method of tracking image data, assisted by the software processmonitor, or the readfile breakpoint. I found the image data and tracked the image data all the way, you can only get the image data that stores the processing results in one place. You cannot find the source image data and set various breakpoints in the main thread. When I almost gave up, I tried to break the breakpoint in every sub-thread and finally broke down.

This involves multi-threaded resource sharing. The stack of each sub-thread in multiple threads is private and the stack is public. However, there can be private heaps and the registers are private! I had a misunderstanding about this problem before. How can a set of registers be private? It turns out that each sub-thread has its own set of register copies. The registers change every time the thread is switched, and the hardware breakpoint is the address that stores the data in the drx register, this drx register should be a copy of the registers of the current thread. Therefore, this hardware breakpoint is only valid in the current thread. When other threads access the target, this hardware breakpoint will not be triggered, this is why I failed to set breakpoints in the main thread.

The thread that is disconnected has the longest running time among all threads (except the main thread. However, you still cannot find the original image data. The processing algorithm is executed in the Child thread. By setting a breakpoint on the result image data, the algorithm is pushed forward step by step until all the Code involved in the calculation is analyzed. The core algorithm found, the analysis algorithm is relatively simple, but involves a lot of SSE extended instructions, the use of xmm0-xmm7 registers, and OD does not show the content of these registers, analysis is very painful, although windbg can display the content of these registers, it is equally painful to use windbg for analysis. Fortunately, Daniel helped write a plug-in. It feels much better to use the plug-in. You can ask me if you need it.

Experience of Reverse Analysis in locating Algorithms