Compile reliable multi-threaded spider programs

Compile reliable multi-threaded spider programs

Thursday, 24. August 2006, 05:52:14

Technology

[This topic is used for discussion with friends in the QQ group [17371752] "search engine, data, and spider 〕

1. What does the Spider Program look like?

Spider programs are one of the most critical background programs in search engines. They must be very reliable and can run for a long time without regular maintenance. But I often see that many of my friends always put a lot of energy into the interface when they started to develop spider programs. I personally think This is unnecessary. We should pay attention to the reliability and efficiency of this program.
(Figure 1. Coolgobot, the first Spider Program to joke)

(Figure 2. Spider programs with simple interfaces)

(Figure 3. One of the most effective spider programs in the Webus System)

2. Structure of the Spider Program(Figure 4. Spider Program structure)

2.1 use a queue to transmit Url data
In Figure 4, Url Reader reads Url data to a queue, and then the download thread retrieves the Url from the queue and downloads it. The advantage of using queues is that it is easy to control the process and convenient to implement multithreading.

2.2 Two multithreading Solutions
There are two multithreading solutions for Spider programs
A. Preemptive Multithreading
In this method, only one thread is working at the beginning. When the thread downloads a page, Parser immediately extracts the new Url List from the page and then extracts the new Url List for each Url, the program starts a new thread to download until the maximum number of threads is reached.
B. Cyclical Multithreading
In this way, N threads are enabled at the beginning, and each thread downloads the Url from the Queue. After one thread is completed, the next thread is taken. Until there is no new Url, the thread enters the Sleep state.
Among the two solutions, I prefer B, because using this solution will make the program structure clearer and the functional module division more reasonable. So the multithreading that I will talk about later is a cyclical multi-threaded solution.

2.3 Save the temporary page data as a file
When many of my friends started to write spider programs, they put the downloaded page data in a Queue to pursue high efficiency, waiting for the analysis thread to take and analyze the data one by one. This method can make the entire program run, but it is hard to say how long it will take. At this time, the stability of the entire system is inversely proportional to the size of the Queue. The larger the Queue, the more memory occupied, and the worse the system stability. A day or two may seem good. If it takes a long time, the program will become less healthy and eventually crash.
We may wish to save the temporary page as a file, and then pass the file name to be processed by a Queue to the analysis thread, and then the analysis thread will process the corresponding file. This processing method is much better than the above method, and the reliability of the system is greatly enhanced.

2.4 analysis thread or analysis program?
After downloading the page, we will not discuss the data that needs to be analyzed and the specific requirements. Here we will talk about the problem of "analysis thread" or "analysis program.
In fact, there is no logical difference in using the analysis part as a thread or a separate program. The two methods should achieve the same reliability and efficiency. However, as a program, it is more difficult to implement as a thread.
As a thread, data transmission between the download thread and the analysis thread is done through a Queue. As a program, we need to consider the communication between two processes, which will be more troublesome.

2.5 Url deduplication
To prevent the Spider Program from repeatedly downloading the same page, We Need To deduplicate the Url. There are two solutions for Url deduplication. A database is used to create a unique constraint for Url fields ....... This is a lazy method. It is very effective when the data volume is small, mainly because it does not require programming and is easy to implement. Another method is to use the information fingerprint method. For this part, see: http://googlechinablog.com/2006/08/blog-post.html

3. Questions to consider

3.1 No. 1 problem: OutOfMemoryException
If there are no mistakes, memory overflow is the number one problem that spider programs will encounter. It is characterized by uncertainty, suddenness, and catastrophic. To solve this problem, we must first change our programming habits. If you want your spider to survive for a hundred years, You need to adopt a controlled memory strategy. Oh, the memory is like fat, too much will eventually lead to disease. So why is this problem unstable and unexpected? This is related to WIN32 memory mechanism, for this part, see: http://my.opera.com/talkinsmile/blog/show.dml/407303

3.2 Exception management
Microsoft master said: If there is no good Exception management policy, you should not try to manage it. However, he does not want us to listen to exceptions. At least we should capture Possible exceptions. If you want your spider to work hard and never strike, check whether the program code related to the following terms has completed exception handling:

• Network
• Database
• Thread
• Disk IO
• Out of value range
• Array subscript Overflow
• What else? Too many. Let's discover it by yourself. ^_^

3.3 distributed
For those who develop spider programs, the first concern is "multithreading" and then "distributed. Distributed is a complex problem. You need to analyze the specific deployment conditions and business requirements, and select an appropriate design mode to implement distributed applications.
For spider programs, my personal favorite is the pipeline-based distributed design model. The so-called pipeline is a complete process of Url-> download-> analysis-> data submission. The idea of pipeline-based distributed design mode is to combine many of these pipelines together through a manager, if a problem occurs in the pipeline, the normal operation of other pipelines will not be affected.
You can find this part by yourself.

After finishing the lecture, you will come up with your own solutions or ideas to jointly improve the level of Spider Program Development!