How to generate robots.txt?

Hosts file.

Robots.txt must be placed in the root directory of the website and the file name must be in lowercase. The following is an example:

# Robots.txt file start

# Exclude files from all robots:

User-Agent :*
Disallow:/security/
Disallow:/admin/
Disallow:/admin.htm

# End robots.txt File

Forbidden access statement:
Disallow :/
User-Agent: badbat

Note all # numbers for easy reading.
User-AgentIt is a search engine spider, followed by a "*" sign, indicating that all Spider access is allowed. You can also create an empty robots. TX file.
DisallowYes indicates that directory or file capturing is not allowed,/admin.htm indicates a file,/admin/indicates a site directory, and indicates the scope of file capturing is prohibited.
Disallow :/Yes indicates that all search engine access is prohibited.
User-Agent: badbatYes indicates that access to a search engine is prohibited.