Web scanning crawler Optimization

Web scanning crawler Optimization

0x01 background

The company needs to develop automated scanning tools. Currently, tools on the market cannot detect services, so they can only be developed by themselves. Hot or not, there is a problem. crawlers need to write them by themselves.

I have previously written related crawlers, but they are either semi-finished products or spam code... Many of them cannot be directly referenced. Therefore, with a powerful KPI assessment, we force ourselves to refactor the code. It can be written in Python.

0x02 Problems

It is easy to write a multi-threaded crawler to complete the whole site scan. However, the problem arises and how to improve the efficiency. When you carefully observe the scan link, you will find that it crawls many links, all of which are highly repetitive links, such as the following:

These links are static pages, such as news pages. The page architecture is similar. We know this situation, but crawlers do not understand it. It only follows the rules we specify, so the crawler efficiency is reduced.

As I expected, some static pages can be crawled less, and we can improve the overall efficiency by reducing the number of captured pages. For example:

Http://www.xxx.com/news/ziyue/2014/1029/61069.html

Http://www.xxx.com/news/ziyue/2014/1029/61070.html

Http://www.xxx.com/news/ziyue/2014/1029/61071.html

Among the three URLs, we only need to capture one as a typical one, which can fully meet our needs and do not need to capture all the URLs. As we all know, this is a pseudo-static generation. So the question is, how should we implement this rule? The partner may have the idea: "How do you know that such static pages must have the same architecture ?" Okay, I'm not sure, but I have a way to determine. Now let's take a look at the above URL Splitting:

Http://www.xxx.com/This is host

/News/ziyue/2014/1029/is a specific directory or document category.

61069. html: this is a specific page.

After the above analysis, my friends have a new question: "No installation, how do you know that people's URLs must follow this standard ?" Well, I have sorted out the URL combination rules I found. Currently, many URLs are composed of the following methods.

1) Static Page:

Http: // [host]/xxx/[int | string).html

2) rewrite type:

Http: // [host]/xxx/[string | int]

3) Directory type:

Http: // [host]/xxx/Catalog/

4) unfixed type:

Http: // [host]/xxx/file. [asp | php | aspx | do | jsp]? [String | int] = [string | int]

0x03 wonderful experiments

After finding these rules, we should consider how to improve the crawler quality and reduce the repetition rate. Then we have the following wonderful experiments:

1) create a rule first:

2) then reference and test these rules.

The running result is as follows:

Indeed, I expected the results .... This article is just a reference, and the code may be slightly frustrated. Let's take a shot.

Before lab]

After lab]