Robotsuses the robots.txt file to block or delete webpages

These two lines are considered as an entry in the file. You can add any number of entries as needed. You can add multiple Disallow rows and multiple User-agents to an entry.

Create a robots.txt file

The simplest robots.txt file uses two rules:

User-agent: a roaming bot that applies the following rules

Disallow: the URL to intercept

These two lines are considered as an entry in the file. You can add any number of entries as needed. You can add multiple Disallow rows and multiple User-agents to an entry.

Each part of the robots.txt file is independent, rather than built on the previous part. For example:

User-agent :*

Disallow:/folder1/

User-Agent: Googlebot

Disallow:/folder2/

In this example, only the URLs that match/folder2/are not crawled by Googlebot.