I. Search engine spider
Search engine spider (SPIDER), or spider for short, is a search engine robot (robot). It is called a spider because the Internet is compared to a spider, comparing robots to web crawlers, a search engine automatically crawls web pages.
Search engine spider function: crawlers of these search engines automatically add webpages to the search engine database. Search engine spider will automatically judge the quality of webpages, determine whether to capture data based on the established program.
Search engine spider name: The following is the name of a well-known search engine in China. I will not list many search engine spider names because of their low popularity.
- Google Spider: googlebot,
- Baidu's spider: baiduspider,
- Yahoo Chinese Spider: Yahoo! Slurp China,
- MSN Spider: msnbot,
- Youdaobot,
- Sogou's spider: sogou web spider
To view the access records of search engine spider, You need to view the access log files through the website. For more information, consult the server provider.
If you want a search engine spider to crawl webpages every day, you need to keep the original articles updated and exchange links on a regular basis.
If you do not want a search engine spider to capture your webpage, you can set robots.txt to prohibit crawling.
Ii. Principles of search engines
The technical problems that need to be solved by search engines are divided into: Spider Program, Classification Indexing, Dictionary, sorting algorithm factors, database indexing and optimization, and database structure-spider. Currently, spider can be implemented using C or PHP. Baidu's spider is mostly made of C. C can also support connection interfaces of multiple databases, and C runs more efficiently than PHP, and C can control the underlying layer. Although C is so good, I still want to use PHP to do it. I must save more time and cannot learn C again. If you need to use C for high efficiency in the future, and then use C for it, the database can remain unchanged, and MySQL can be connected to C. PHP has advantages and disadvantages. It should not be a problem for Spider. The biggest problem is that it may be slow.
1. Capture webpagesThe possible problem with web page capturing is how to record the capture order, capture failure or timeout, and when to update the capture next time. The search engine database is new at the beginning and does not have any Web site. Therefore, you need to add a large number of Web sites.
Here, we can use the for loop statement to automatically cycle by English letters. Of course, website construction is not only in English, but also in numbers. If it is still loop capturing, it is estimated that many of them will fail. The captured code needs to analyze whether the encoding type is UTF-8 or gb2312. my search engine only wants to capture Simplified Chinese.
If the capture times out, the record will be crawled about ten days later. If the capture times out three times in a row, t will be taken out of the database.
2. Create an indexIndexing is a tough issue. Baidu and Google can use their own server clusters to build distributed servers. I don't have that many servers. So I want to change the method. Create a static page. I learned before that it would take about 0.2 seconds to enter a relatively uncommon word in Baidu and Google. Generally, it only takes 0.1 seconds to enter a common word. In addition, it takes much less time to repeat a word for the second time. Most of this is the impact of the index. If the index is stored in the memory, the read speed will be very good. I only have one server. Even if I only index the common 50 thousand query words, it is a little tired. A page must have at least 20 KB, and 50 thousand million pages are 20 KB * 50 = 1 GB. This is only the first page of the 50 thousand words. If you want to query pages, the memory is definitely not enough. If you only put the first page into the memory and query pages on pages, the speed cannot be solved. So I have prepared all static resources. Simulate a query of 50 thousand words and then generate a static page. The first page of all words is stored in memory, and the subsequent pages are stored in the hard disk. If the page can be stored in memory, this problem has been solved.
3. Word LibraryTens of millions of Chinese characters, and at least three thousand commonly used Chinese characters. The commonly used words are estimated to be 20 thousand. How can I add this dictionary? What format is used for storage? CSV files, databases, or text files? Previously, I tried to find Kingsoft's dictionary file and try to copy it directly. This method is not successful yet.
4. Sorting AlgorithmAll algorithms that affect sorting should be placed in a table and then divided into fixed algorithms, which are factors of the website itself, as well as variable algorithms, it is the words entered by the user or the elements that change due to time and season. The fixed algorithm is placed in a table to calculate the total score for each website. One part of the change algorithm is generated before, and the other part is calculated after the user inputs.
5. database indexThere are no good solutions for database indexes at present. There must be no too many indexes for one expression, which will affect the speed. 6. database structure. This is critical. It is estimated that the data library structure should be set up before the website's front-end interface comes out. You also need to leave interfaces for future upgrades, such as adding algorithm factors or changing fields to Optimize Query statements.
Reference recommendations:
Search Engine
Search engine spider
View Search Engine Principles
