The site is particularly bad? Robots text file to add site map, station optimization best choice!

The site is particularly bad? Robots text file to add site map, station optimization best choice!

Before we discuss this topic, let's take a look at the concept and production of robots:

Robots Text concept: This file is mainly for the search engine to see, the search engine when entering the site, First access to the FTP server robots.txt text, you need to set in the text recommended to search engines to see the page or directory, as well as reject the search engine indexed pages or directories (reject this is what we often do).

Important point: After the robots text is created, it must be placed in the root directory of the FTP. (Similar to: http://domain.com/robots.txt)

Let's take a look at the production of robots.txt text:

1) hand-made (not difficult):

1. User-agent: * (* wildcard character/representative of all search engines)
Disallow:/(negative prefix/not allowed)
Allow:/(Allowed)

2. User-agent:baiduspider (spider name/Baidu/website log query)
Disallow:/

User-agent:googlebot (spider name/Google/website log query)
Disallow:/

3. User-agent: * AB
Allow:/directory 1/Directory 2.html (allows access to the page in Directory 2)
Disallow:/catalogue 1/

Example:
User-agent: * A invite B to a home to play
Disallow:/a/offended B
Allow:/directory 1/Directory 2.html (allows access to the page in Directory 2)

4. Other explanations:

Disallow:/?* do not crawl to the beginning of the page or a folder
Disallow:/*?* Do not crawl dynamic pages within the site

Special note: At present, Baidu's algorithm has been very astute, no longer for the dynamic, static things worry, Baidu Official guide clear inside, dynamic and static pages, are the same included.

Here's what we're going to focus on today:

Let's look at an example:
User-agent: *
Disallow:/Directory 1/1.html
Disallow:/Catalogue 3/
Disallow:/Catalogue 5/

Sitemap:/sitemap.xml (Recommended map)
Sitemap:/sitemap.html (Recommended map)

Add the site map to the robots.txt text, can promote the better collection of Web pages, imagine, search engine first look at the robots.txt text file, you put the site map directly inside, let the search engine to click, to see, believe than any method to stimulate the spider is better!

The site is very poor, I can't analyze the reason, may wish to see robots.txt text whether this item is set.

2) automatic creation of robots.txt text files using tools (very simple):

Now there are many webmaster tools such as Web site can automatically create robots.txt text, webmaster just enter the directory or Web page can generate robots.txt text online.

Example Description: http://tool.chinaz.com/robots/

Needless to say, I believe you have seen the robots text in the importance of adding a sitemap, as to how to generate, please learn, very simple.

The site is particularly bad? Robots text file to add site map, station optimization best choice!

Article written here, have not understood can contact ER Mao, the first time for you to answer.

The site is particularly bad? Robots text file to add site map, station optimization best choice!