Use of robots.txt and robots meta tags

We know that search engines have their own "search bots", and they build their own databases by constantly crawling information on the web along the pages of links (typically HTTP and src links).
for Web site managers and content providers, sometimes there will be some site content, do not want to be crawled by robots and open. To solve this problem, the robots development community offers two options: one is robots.txt and the other is the robots meta tag.

First, robots.txt
1, what is robots.txt?
Robots.txt is a plain text file, by declaring in this file that the site does not want to be accessed by robots, so that some or all of the content of the site may not be included in the search engine, or the designated search engine only contains the specified content.
When a search robot visits a site, it will first check the site root directory exists robots.txt, if found, the search robot will follow the contents of the file to determine the scope of access, if the file does not exist, then the search robot to crawl along the link.
Robots.txt must be placed under the root of a site, and the file name must be all lowercase.
W3.org
Http://www.w3.org/robots.txt
w3.org:80
Http://www.w3.org:80/robots.txt
w3.org:1234
Http://www.w3.org:1234/robots.txt
w3.org/
Http://w3.org/robots.txt
2, the robots.txt grammar

The "robots.txt" file contains one or more records that are separated by a blank line (with CR,CR/NL, or NL as The Terminator), and each record is formatted as follows:
"<field>:<optionalspace><value><optionalspace>".

You can use # to annotate in this file, using the same method as in Unix. The records in this file usually start with one or more lines of user-agent, followed by a number of disallow lines, as detailed below: