Name of a common search engine robot robots

The search engine uses a program robot (also called Spider) to automatically access webpages on the Internet and obtain webpage information.

 

You can create a pure robot file robots.txt on your website, which declares that the website does not want to be accessed by the robot. In this way, some or all of the content of the website will not be indexed by the search engine, or the specified search engine only contains the specified content.

Robots.txt (in lower case) is an ASCII text file stored in the root directory of a website. It often tells the Web search engine's roaming BOT (also called a web spider ), the content on this website cannot be obtained by the search engine's roaming server and can be obtained by the roaming server. For a few systems, the urlis sensitive, and the file names of robots.txt should be in lowercase. Robots.txt should be placed under the root directory of the website. If you want to define the behavior of the search engine's roaming folder sub-directory separately, you can merge your settings to the robots.txt in the root directory or use the robots metadata.