Search engine spider Program 1-search engine technology

Robots.txt and Robots META tags
Ping Wensheng 2003-10-29
As we know, search engines all have their own "search ROBOTS" and use these ROBOTS to link on the web page over the network (generally http and src links) constantly crawl data to build your own database.
For website administrators and content providers, there are sometimes some website content that they do not want to be crawled by ROBOTS. To solve this problem, The ROBOTS development community provides two methods: robots.txt and The Robots META tag.

I. robots.txt
1. What is robots.txt? Robots.txt is a plain text file that declares that the website does not want to be accessed by robots. In this way, some or all of the content of the website can be not indexed by the search engine, or the specified search engine can only include the specified content.
When a robot searches for a site, it first checks that the site root directory contains robots.txt. If it finds it, the search robot determines the access range based on the content in the file. If the file does not exist, the search robot crawls along the link.
Robots.txt must be placed in the root directory of a site and all file names must be in lowercase.
Website URL
URL of robots.txt
Http://www.w3.org/
Http://www.w3.org/robots.txt
Http://www.w3.org: 80/
Http://www.w3.org: 80/robots.txt.
Http://www.w3.org: 1234/
Http://www.w3.org: 1234/robots.txt
Http://w3.org/
Http://w3.org/robots.txt
2. robots.txt syntax
“Robots.txt "the file contains one or more records separated by blank rows (with CR, CR/NL, or NL as The Terminator). The format of each record is as follows:
"<Field >:< optionalspace> <value> <optionalspace>".
In this file, you can use # for annotation. The usage is the same as that in UNIX. The record in this file usually starts with one or more lines of User-agent, followed by several Disallow lines. The details are as follows:
User-agent:
The value of this item is used by the role to search for the name of the engine robot. in the robots.txt file, if there are multiple User-agent records, it indicates that multiple robots will be restricted by this protocol. For this file, there must be at least one User-agent record. If the value of this item is set to *, the agreement applies to all robots. In the robot robots.txt file, only one record such as "User-agent: *" can exist.

Disallow:
The value of this item is used to describe a URL that you do not want to access. This URL can be a complete or partial path, any URL starting with Disallow will not be accessed by the robot. For example, "Disallow:/help" for/help. phpl and/help/index. phpl does not allow access by search engines, while "Disallow:/help/" allows robot access to/help. phpl, but cannot access/help/index. phpl.