Search engine spider algorithm and spider program architecture-reference knowledge

Search engine spider algorithm and spider program architecture

A web spider is an image name. Comparing the Internet to a Spider Network, a spider is a web crawler. The web spider searches for webpages through links. It reads the content of a webpage from a certain page of the website (usually the homepage, find other links in the web page, and then find the next web page through these links. This continues until all the web pages of the website are crawled. If the whole Internet is regarded as a website, then the network spider can use this principle to capture all webpages on the Internet.

For search engines, it is almost impossible to capture all the web pages on the Internet. From the data published currently, the largest search engine can only capture about 40% of the total number of web pages. One of the reasons is the bottleneck of crawling technology. It is impossible to traverse all webpages, and many webpages cannot be found from links of other webpages; another reason is the storage technology and processing technology. If the average size of each page is 20 KB (including images), the size of 10 billion web pages is 100 × GB, even if it can be stored, there are problems with downloading (according to a machine download 20 k per second, it takes 340 machines to download continuously for one year to download all the web pages ). At the same time, because the data volume is too large, it will also affect the efficiency when providing search. Therefore, the web spider of many search engines only crawls important webpages. The main reason for evaluating the importance of a Web page is the link depth of a Web page.

When crawling a webpage, the network spider generally has two policies: breadth first and depth first (as shown in ). Breadth First means that the network Spider first crawls all links on the starting webpage, and then selects a link webpage, continue to capture all links on this webpage. This is the most commonly used method, because it allows the network Spider to process in parallel and increase its crawling speed. Depth first means that the network spider will track a link from the start page, a link, and a link. After processing this line, it will be transferred to the next start page, link ). This method has the advantage that the network spider is easier to design. The difference between the two policies will be clearer.

Because it is impossible to capture all the web pages, some network spider sets the access layers for some less important websites. For example, in a, a is the starting web page, which belongs to layer 0, B, c, d, e, f belongs to layer 1st, g, h belongs to layer 2nd, And I belongs to layer 3rd. If the number of access layers set by the Network spider is 2, the web page I will not be accessed. This also allows some websites to search for some webpages on the search engine, while others cannot. For website designers, the flat website structure design helps search engines capture more webpages.

Network spider often encounters encryption data and webpage permissions when accessing Website webpages. Some webpages can be accessed only with membership permissions. Of course, the website owner can make the web spider not to crawl through the protocol, but for some websites that sell reports, they want the search engine to find their reports, but it cannot be viewed by the searcher for free. In this way, you need to provide the corresponding user name and password for the network spider. The web spider can crawl These webpages through the given permissions to provide search. When a searcher clicks to view the webpage, the searcher also needs to provide corresponding permission verification.
2. Website and network spider

Network spider needs to capture web pages, different from general access, if the control is not good, it will cause the website server to be overloaded. Each network spider has its own name. When crawling a webpage, it will indicate its identity to the website. The Network Spider sends a request when capturing a webpage. The request contains a User-Agent field to identify the network spider. For example, Google's spider is identified as googlebot, Baidu's spider is identified as baidusp, and Yahoo's spider is identified as Inktomi slurp. If there is access log records on the website, the website administrator will be able to know which search engine's network spider has come over, when has it come over, and how much data has been read. If a website administrator finds a spider has a problem, contact the website administrator with the ID of the spider.
To define which directories are not accessible to Spider, or which directories are not accessible to certain network spider. For example, if you do not want the executable file directories and temporary file directories of some websites to be searched by the search engine, the website administrator can define these directories as denied access directories. The robots.txt syntax is very simple. For example, if there are no restrictions on the directory, you can use the following two lines to describe the directory:

User-Agent :*

Disallow:

Of course, robots.txt is only a protocol. If the designers of the network spider do not follow this protocol, the website administrator cannot prevent the network spider from accessing some pages, but the general network spider will follow these protocols, in addition, the website administrator can also use other methods to reject web page crawling by the Network spider.

When downloading a webpage, the web spider identifies the HTML code of the webpage. Meta is included in the Code. Through these identifiers, you can tell the network spider whether the web page needs to be captured, and tell the network spider whether the links on the web page need to be tracked. For example, this indicates that the webpage does not need to be crawled, but links in the webpage need to be tracked.

This section describes the syntax of robots.txt and the meta tag syntax in detail in the previous article "methods for prohibiting search engines from indexing.

Currently, most websites want the search engine to capture the web pages of their websites more comprehensively, because this allows more visitors to find the website through the search engine. In order to make the webpage of this website more fully captured, the website administrator can create a website map, that is, site map. The multiple network spiderwill use the sitemap.htm file as the portal for crawling a website webpage. The website administrator can put all links on the webpage in this file, then, the network spider can easily crawl the entire website to avoid missing some webpages and reduce the burden on the website server. (Google provides website administrators with sitemap of XML)
3. Content Extraction by network spider

The search engine creates a web index and processes text files. For web spider, the captured webpage contains various formats, including HTML, images, Doc, PDF, multimedia, dynamic webpages, and other formats. After these files are captured, you need to extract the text information from these files. Accurately extracting the information of these documents plays an important role in the search accuracy of the search engine, and affects the correct tracking of other links by the Network spider.

For documents such as Doc and PDF, the vendor provides corresponding text extraction interfaces for documents generated by software provided by professional vendors. The Network spider can easily extract text and other file-related information by calling the interfaces of these plug-ins.
HTML and other documents are different. html has its own syntax. Different command identifiers are used to represent different fonts, colors, locations, and other la s, such, these identifiers must be filtered out when extracting text information. It is not difficult to filter identifiers, because these identifiers have certain rules, as long as the corresponding information is obtained according to different identifiers. However, when identifying such information, You Need To synchronously record many layout information, such as the text font size, whether it is the title, whether it is bold display, whether it is a page keyword, etc, this information helps calculate the importance of words on the web page. In addition to the title and body of an HTML webpage, there are many ad links and public channel links) it has nothing to do with text. When extracting Web content, you also need to filter these useless links ). For example, a website has a "Product Introduction" channel, because the navigation bar is available on every webpage of the website. If you do not filter the navigation bar link, when searching for "Product Introduction, each page in the website will be searched, which will undoubtedly bring a lot of junk information. To filter these links, you need to calculate a large number of Web Page Structure rules, extract some commonalities, and filter them in a unified manner. For important websites with special results, you still need to process them individually. This requires the design of the network Spider to be scalable.
4. Network Spider Program Architecture
Construct a network spider using ASP
So how to use ASP to build a network spider? The answer is: Internet transfer control (ITC ). The control provided by Microsoft enables you to access Internet resources through ASP programs. You can use ITC to search Web pages, access the FTP server, and even send mail titles. In this article, we will focus on the Web page searching function.

Several defects must be described first. First, Asp has no permission to access the Windows registry, which makes the constants and data values that some ITC normally stores unavailable. Generally, you can solve this problem by setting ITC as "do not use the default value", which requires you to specify the value for each operation.
Another more serious problem is the License book. Because ASP does not have the function to call License Manager (a function in Windows that can ensure the legitimate use of components and controls), when License Manager checks the key password of the current component, and compare it with the Windows registry, if they are found to be different, the component will not work. Therefore, if you want to configure ITC to another computer without a required key, it will cause the ITC to crash. One solution is to bind ITC to another VB component. The VB component copies the ITC path and tool for configuration. This job is troublesome, but unfortunately it is essential.

The following are some examples:

You can use the following code to create an ITC:

Set inet1 = Createobject ("inetctls. Inet ")
Inet1.protocol = 4 'HTTP
Inet1.accesstype = 1 'direct connection to Internet
Inet1.requesttimeout = 60' in seconds
Inet1.url = strurl
Strhtml = inet1.openurl 'Grab HTML page
Strhtml now stores the HTML content of the entire page pointed to by strurl. To create a regular network Spider, you only need to call the instr () function to check whether the string you are looking for is in the current position. You can also follow the href tag to parse the current URL, set it to the properties of the internet control, and then continue to open another page. The best way to view all links is to use recursion.

It should be noted that although this method is easy to implement, it is not very accurate and powerful. Today, many search engines can perform additional logic checks, such as calculating the number of repetitions of a phrase on a page and the approximate degree of related words, some can even be used to determine the relationship between the searched CIDR Block and the context.
Construct a network spider using VB
Spider Program website levels and their working principles:
No. Website level parent No.

1 http://www.netfox.cn/1 0

2 http://www.sina.com.cn/2 1

3 http://www.cnnic.cn/2 1

4 http://www.baidu.cn/3 2

5 http://www.yahoo.cn/3 2

The spider program first extracts all links from level 1 (http://www.netfox.cn/) and records all links to the database (or a large array ), the links of these websites are marked as Level 2;

After you record all level 2 to the database, start from Level 2 to the first (here, the website with the serial number 2) Link) start to extract all links under it to the database, and mark these links as level 3; then, record all links of the website whose level is 2 to the database, and mark their level as Level 3;

After Level 3 records all the databases, it starts to extract the links of the first website in the order of Level 3, and so on!

Note: The program should keep a pointer to record the serial number currently being operated! In addition, you can add a parent number field to record the inheritance relationship between them!

Level 1 indicates the network seed. Here we place the network seed on the first layer. You can set one or more network seed as needed, in fact, we can clearly see through this level chart that low-level URLs are high-level network seeds. That is to say, as long as there is one or several network seeds, we can find more network seeds through their links. Only in this way can our spider run forever!

Level 2 is the link captured by Level 1 (Network seed );

Level 3 is the link captured through level 2 );

And so on!

Key code of the Spider Program:

Here we use VB to implement the core part of the code, of course, you can easily switch to other language code. For the sake of simplicity, we will not operate on the database here. We will create a two-dimensional array to store our website !!

Dim web () '// create an array

Dim pointer '// creates a pointer to record the current seed

Dim id' // create a sequence number to record the number of the website in the current region

Dim layer '// create a level to record the level of the currently running seed

Dim running '// sets whether the flag is running,

 

Private function newworkseed_set () as Boolean '// used to set the network seed. For demonstration convenience, we put the seed in the array,

'// Of course, you can also directly store them in the database as needed

Web (0, 0) = 1' // serial number

Web () = "http://www.netfox.cn/" '// website

Web (2, 0) = 1' // hierarchy

Web (3, 0) = 0' // parent number, 0 indicates the original network Seed

Web () = "NEFU network"

'// Of course, multiple original network seeds can be set here

Web (0, 1) = 1

Web (1, 1) = "http://www.aspfaq.cn /"

Web (2, 1) = 1

Web (3, 1) = 0

Web () = "ASP technology site"

'// After the network seed is set, the SN after the start of the seed number is recorded. Here two seed are set, so Id = 2

Id = 2

End Function

 

Private sub spider_work () '// spider Worker Program, captures the website and records it to an array
'// You can put them in the database as needed

Dim

For each a in webbrowser. Document. All

If ucase (A. tagname) = "A" then

If isvalidweb (A. href) then

Id = ID + 1

Web (0, ID) = id' // record the serial number of the current website

Web (1, ID) = A. href '// record the current website

Web (2, ID) = Layer '// record the current website level

If Web (2, pointer) <> layer Then layer = Layer + 1' // when the pointer level is different from the current level

'// Indicates that the level has been increased.

Web (3, ID) = pointer '// record the parent Number of the current website

Web (4, ID) = A. innertext '// record the name of the current website

End if

End if

Next

Pointer = pointer + 1

Webbrowser. navigate web (1, pointer-1) '// after the current seed is captured, the system automatically jumps to the next seed.

If running = false then' // if running is not enabled, exit

Exit sub

End if

End sub

 

Private function spider_init () as Boolean '// Spider Program initialization Function

Pointer = 1' // the pointer is set to 1, indicating that the task starts from the first sequence number.

Id = 2' // The serial number is set to 2, and records can be read later

Layer = 0' // The layer is set to 0, indicating that the spider runs for the first time.

'// The above pointer, serial number, and level can be recorded and read later

If isvalidweb (Web (1, pointer-1) then' // checks whether the seed is correct. If the initialization is successful, otherwise it fails.

Running = true

Spider_init = true

Webbrowser. navigate web (1, pointer-1)

Else

Running = false

Spider_init = false

Exit Function

End if

End Function

 

Private sub webbrowser_documentcomplete (byval Pdisp as object, URL as variant) '// webbrowser Control

Call spider_work ()

End sub

 

Private function isvalidweb (_ href) as Boolean '// determines whether the function is a CN Domain Name Function

'// You can use this function to capture specified websites or data.

If instr (_ href, "http: // www.")> 0 and instr (_ href, ". CN/")> 0 and Len (_ href) <60 then

Isvalidweb = true

Else

Isvalidweb = false

End if

End Function

 

Private sub initcommand_click () '// init initialization Command Control

If spider_init () then

Msgbox "SPIDER initialization successful and started running"

Else

Msgbox "SPIDER initialization failed"

End if

End private

 

Private sub stopcommand_click () '// stop Command Control

Running = false' // stop running

End private

 

Private sub runcommand_click () '// run the Command Control

Running = true '// continue running

Call spider_work () '// spider runs the main program

End private

Specific Network spider
Relatively, a specific network spider is more complicated. As we mentioned earlier, a specific network spider will search for a specific part of a page, so it is required to know the relevant information in advance. Let's take a look at the following HTML:

<HTML>
<Head>
<Title> my news page </title>
<Meta name = "keywords" content = "News, headlines">
<Meta name = "descr iption" content = "the current news headlines.">
</Head>
<Body bgcolor = "# ffffff" text = "#000000" link = "# ff3300" vlink = "# cc0000" alink = "# 0000ff">
<P> <! -- Put headlines here -->
<A href = "/news/8094.asp _ fcksavedurl =" "/news/8094.asp""> Stocks prices fall </a> <a href = "/news/8095.asp"> new movies today </a> <a href = "/news/8096.asp"> bush and & nb sp; gore to debate tonight </a> <a href = "/news/8097.asp"> fall TV lineup </a>
<! -- End headlines -->
</Body>
</Html>

On this page, we only care about what is located between the "put headlines here" and "End headlines. You can set a function to return only the search results in the region:

Function gettext (strtext, strstarttag, strendtag)
Dim intstart
Intstart = instr (1, strtext, strstarttag, vbtextcompare)
If intstart then
Intstart = intstart + Len (strstarttag)
Intend = instr (intstart + 1, strtext, strendtag, vbtextcompar E)
Gettext = mid (strtext, intstart + 1, intend-intstart & n bsp;-1)
Else
Gettext = ""
End if
End Function

Based on the example of building the ITC control above, you can easily put the "<! -- Put headlines here --> "and" <! -- End headlines --> "is transmitted as a parameter to gettext.

Note that the start and end tags are not necessarily actual HTML-specific tags-they can be any text delimiters you want to use. In general, it is not easy to find a good HTML tag to define the search area. You can only use easy-to-use tags-for example, your first and last tags can be as follows:

Strstarttag = "/TD> <font face =" Arial "size = & Q uot; 2"> <p> <B> <u>"
Strendtag = "<p> </TD> </tr> <TD> <ums> & quo T;

You must be sure to search for unique HTML pages so that you can get what you need accurately. You can also search by link in the text you return. However, if you do not know the format of those pages, your web spider will not return.
Source: http://www.zhiseo.com/Article/1248.html