Google spider running JS and CSS styles on webpages

Old Concept

In my existing ideas, web crawlers, Spider, and Robot of search engines only capture the HTML code of pages, JS and CSS code inside or outside are ignored. Therefore, we often say that Ajax is not conducive to search engine crawling, but to SEO.

In our opinion, search engine crawlers are not powerful browsers after all. They cannot run JS or render CSS. Those colorful pages are just plain text streams (or text information flows containing structure tags) in the eyes of crawlers.

However, the two recent news about Google completely overturned my understanding of search engine crawlers.

News 1

A video from Google's senior engineer Matt Cutts shocked me. Matt warned us that not only the text and background are in the same color, the font size is set to 0, and the text is hidden with CSS, in addition, Google can also identify how to hide text cheating through JS.

In the video, a concealed JS Code

. Style. display

Set property

'None'

To hide the text that is only intended to be provided to the search engine but not displayed to the user. Matt said that this cheating method can no longer be applied to Google.

News 2

News 2 is even more scary. Google is said to be able to crawl Ajax content! Add a specific Identifier (that is

Domain.com/?abc

Change

Domain.com /#! Abc

), So that Googlebot can understand that the URL is an Ajax page (rather than an anchor in the page) and capture it.

You may not be very interested in Google's technological improvements, but you must have noticed the nature of the problem: Googlebot can capture Ajax content, that is, Googlebot is fully capable of running JS on the page, complete Functions!

Crawlers and browsers

If both news are true, in a sense, crawlers are getting closer and closer to browsers. This also means that the search engine crawler will capture more content (including JS and CSS files), and the website's traffic load will increase.

On the other hand, crawlers consume more resources during page capturing. The resource overhead for processing text information is far less than that for completely rendering pages and running client programs.

Therefore, I am still skeptical about the two news. Is this a smoke bullet from Google? Or fake news? If Googlebot is capable of running JS or rendering CSS, Maybe Google will enable the black/White List mechanism internally to control resource overhead to a reasonable extent?

Webmasters

If you are worried about crawler erosion of host traffic, you may consider

Robots.txt

File anti-Bot

*. Js

And

*. Css

File capture. However, we are not sure whether this operation has any adverse side effects.

Some may also worry that regular page layout sometimes requires some hidden text means, such as [CSS map 文] and [hx logo information of hidden modules. Will this be determined by Google as cheating?

I believe that for a "smart" search engine like Google, it is capable of allowing crawlers to run JavaScript and CSS, and it must be able to determine what is cheating and what is a normal layout. So I don't think the webmasters need to be alarmed. What should they do? They are not afraid of shadows. The rules are always used to constrain those "bad guys.

Therefore, for some SEOer, this seems to be a bad message. I don't think it makes much sense if they are still considering whether there are new cheating schemes. Obviously, the survival space of SEO cheating means will become smaller and smaller. At the same time, the value of website content is the real foundation of SEO.