Learn more about robots

Absrtact: In-depth understanding of the robots search engine using the spider program to automatically sample the Web page and get information on the Internet. Spider when visiting a Web site, headman page checks the root domain of the site for a plain text called robots.txt, which is used to refer to the

Learn more about robots

Search engines use the spider program to automatically sample Web pages on the Internet and get web information. Spider when you visit a Web site, headman page checks to see if there is a plain text called robots.txt under the root domain name of the site, which is used to specify spider crawl area on your site. You can create a robots.txt on your site and declare in the file which directories in the Web site are not included in the search engine. (robots.txt files should be placed in the site root directory.) ）

1. The use of robots

User: Define the search engine name, such as: Baidu (Baiduspider), Google (Googlebot), 360 (360Spider) and so on. * Symbol for all search engines

Disallow: Not allowed to be included

Allow: Allowed to be included

Case：

User: * (Declaration prohibits all search engines from crawling with the following content)

disallow:/blog/(prohibit all pages under the website blog column.) For example:/blog/123.html.

Disallow:/api (for example:/apifsdfds/123.html will also be shielded.) ）

allow:/blog/12/(generally do not need to write Allow, and we in the prohibited directory below, we need to let search engines included, then we can use Allow, for example, we have banned the front of the blog column of all the pages, and we would like to/12/under the column is included, This time we need to pass allow to allow the search engine to be included. ）

Disallow:*?* (as long as you have a question mark in your path, the path will be blocked.) ）

disallow:/*.php$ (means to block out all the paths at the end of PHP.) ）

How to use a robots to promote rankings

A, shielding dead link

Many times, our site will produce dead links, such as: Web site revision, article deletion, etc., are likely to cause the site to appear 404 pages, then we need to pass the robots to screen these dead links. There are two ways to view dead chains:

1. Through the website log query.

We can download the Web site log to view our 404 pages (the site log can be to the space business, Space Chamber of Commerce to provide download channel), we use TXT open the site log, through the search "404", if your site has 404 pages, and was visited, then this time will be the site log to record.

2. Baidu Webmaster Platform Query

We first need to verify the Baidu Webmaster platform (how to verify that do not say, the most basic. After verification, we through Baidu Webmaster Tool dead Chain tool, you can clearly see that our site has any dead links.

Second, shielding the worthless page

Many sites have some worthless pages, such as SEO Guidance Center of the user's Personal center page, this personal center page will not produce value on the site, in Baidu above the show will not bring weight to the site. So we need to use robots to block such pages. Another example: a corporate website Contact us page, this page will not bring weight to the site, users will not go to search "Contact us" to find the products he wants. So we need to screen this page at the same time.

Third, shielding repeat path

When your site appears repeat path, Baidu will be our site for direct drop points, such as: a site did pseudo static, while his dynamic page can also access, this time will appear duplicate path. So we need to shield these repeat paths through the robots.

Summary: Perhaps a lot of people do not know, many details are likely to be your site is not ranked reasons, and these small details are often overlooked. At the same time hope that webmaster in doing site optimization when a lot of attention to some details of the site.