Some Ideas about web Scanners

5up3rh3iblog

Recently, I saw an article on the Web Ranger blog :《WEB Application Security Scan Product OverviewIt lists some free web scanners and their features at home and abroad... from the product recommendation perspective, we can see that the market competition for the web scanning platform is still very large.

In terms of implementation technology, they are all based on url crawling. Basically, many of them extract URLs and parameters through regular expressions, but some extract URLs and parameters are not the same, it is rarely extracted directly. It is generally extracted through a local proxy [similar to a proxy man-in-the-middle attack. Such crawling lags behind the web era, such as complicated ajax, flash applications, and js obfuscation...

Then we may be able to find new models or frameworks to break through... during the discussion in the hi group, I asked chuangyu's ic chief, whose crawling tool [http://www.scanv.com] broke through the above, they said that ajax js obfuscation is basically acceptable. They applied multiple policies. Specifically, js/ajax uses js virtual machines to parse and extract URLs... awesome stuff !!

In fact, aside from the Rules library in the web scanner, the crawling url and the number of parameters determine the success or failure. Black box testing is a very important process in the sdl process. The effect of black box testing also has a lot to do with this volume! When discussing these issues in the hi group, kEvin # 80sec.com mentioned a very good method to extract URLs through the web server logs, this is a very quick and effective method [for get submitted parameters]... however, there are also many drawbacks. For example, it is difficult to process logs with large traffic volumes, post parameters are not recorded, and cannot be tested if the function is not used by anyone. This url is extracted from the URLs and parameters that the user normally uses the web application function, instead of crawling a page by using the simple crawling tool, crawls many invalid URLs. This method reminds me of one of my ideas a few years ago:

"Perform security testing while functional testing (or normal use"

Whether it's ajax, flash, or anything else, you will certainly click through interactive hype in function testing or normal users, then you will capture packets, local agents, or the browser plug-in and other methods to extract the corresponding http package [including url and parameters] for security testing. You may ask, does this also depend on the tester's access and specific features used? Yes, no error! But what if there are "many" people working together? In this way, a popular concept is proposed: "Cloud ". Specifically, you can install a small plug-in either in the product testing department of the company or in the PCs of all employees [or there are other ideas such as extracting from the gateway], used to extract URLs and parameters for security testing during user browsing or operations. This greatly accelerates the sdl process and saves costs.

Of course, the above idea can also be combined with the current product methods.