Search engine spider man (ROBOTS) secret

As we know, search engines all have their own "search ROBOTS" and use these ROBOTS to link on the web page over the network (generally http and src links) constantly crawl data to build your own database.

For website administrators and content providers, there are sometimes some website content that they do not want to be crawled by ROBOTS. To solve this problem, The ROBOTS development community provides two methods: robots.txt and The Robots META tag.

One worker robots.txt

1. What is robots.txt?

Robots.txt is a plain text file that declares that the website does not want to be accessed by robots. In this way, some or all of the content of the website can be not indexed by the search engine, or the specified search engine can only include the specified content.

When a robot searches for a site, it first checks that the site root directory contains robots.txt. If it finds it, the search robot determines the access range based on the content in the file. If the file does not exist, the search robot crawls along the link.

Robots.txt must be placed in the root directory of a site and all file names must be in lowercase.

The URL of robots.txt corresponding to the website URL
 
Http://www.w3.org/Http://www.w3.org/robots.txt
 
Http://www.w3.org: 80/Http://www.w3.org: 80/robots.txt.
 
Http://www.w3.org: 1234/Http://www.w3.org: 1234/robots.txt
 
Http://w3.org/Http://w3.org/robots.txt

2. robots.txt syntax


The "robots.txt" file contains one or more records separated by empty rows (with CR, CR/NL, or NL as The Terminator). The format of each record is as follows:

"<Field >:< optionalspace> <value> <optionalspace> ".

In this file, you can use # for annotation. The usage is the same as that in UNIX. The record in this file usually starts with one or more lines of User-agent, followed by several Disallow lines. The details are as follows:

User-agent:

The value of this item is used to describe the name of the search engine robot. In the "robots.txt" file, if multiple User-agent records indicate that multiple robots are restricted by this protocol, for this file, there must be at least one User-agent record. If this parameter is set to *, the protocol is valid for all machines. In the "robots.txt" file, only one record such as "User-agent: *" can exist.

Disallow:

The value of this item is used to describe a URL that you do not want to access. This URL can be a complete or partial path, any URL starting with Disallow will not be accessed by the robot. For example, "Disallow:/help" does not allow access by search engines to/help.html and/help/index.html, while "Disallow:/help/" allows robot access to/help.html, but cannot access/help/index.html.

If any Disallow record is blank, it means that all parts of the website are allowed to be accessed. At least one Disallow record is required in the "/robots.txt" file. If "/robots.txt" is an empty file, the website is open to all search engine robots.</