Google's new useragent information (News) crawler website can be controlled using robots.txt

Google's new useragent information (News) crawler website can be controlled using robots.txt

Google is constantly improving its technology to meet the needs of some news sites. The official blog of the Google administrator center is called Google newscrawler to add new useragentidentifiers. The website can pass robots.txt to control whether the content is captured by Google newscrawler, for example, robots.txt:

User-Agent: googlebot
Disallow:
User-Agent: googlebot-news
Disallow :/

This means that the website can be crawled by Google's common crawlers and the Website Cannot Be crawled by Google News crawlers. In this way, the content appears in the Google Webpage Search results, rather than in the Google news search results.

For more details, refer to the new user agent for news blog on the Google administrator center.

Latest related content:
Google caffeine search results
Google caffeine end test preparation officially launched
Matt Cutts: Caffeine will be available early next year
Influence of Google caffeine on Seo
How to enter the new search interface of the UI jazz Google?
Add this article: 365key