JavaScript-How to avoid being collected sites?

For several weeks in a row, Unicom Telecom will change, each access to the IP will change, access time 0, the frequency is very high, the region is unchanged, is not someone collected my station? How to avoid it? Thank you!

Reply content:

For several weeks in a row, Unicom Telecom will change, each access to the IP will change, access time 0, the frequency is very high, the region is unchanged, is not someone collected my station? How to avoid it? Thank you!

1, the use of dynamic irregular HTML tags
2. Limit IP

and search engine is not contradictory, can use Nginx to determine whether it is a search engine crawler, guide to a fixed HTML structure inside, has done so before

It is difficult to avoid, only to say as much as possible to improve the cost of crawling.

Often, the frequency limit is added to the Nginx or program (rate limit).

Turn off the website.

By adding a masking crawl strategy to the robots.txt, you can be less captured

The key is that you want search engines to crawl you;

So it's a contradiction,

Can only do as well as possible;

You can ' t.

Upstairs did not answer to the idea, the general collection is nothing but a hard crawl, such as file_get_contents, and simulation, such as CASPERJS, I have used casperjs to write crawlers, the speed is too slow, so casperjs crawler is not very feasible, if just like File_ Get_contents, such as, the background analysis log, the load of the page does not load the corresponding CSS and JS files, all ban off
Hope to write the thief program people do not see this answer, or in addition to limit the frequency of the request is not, oh forget, I am specialized in writing thief program

CNZZ statistics, add a Google statistical comparison, the flow gap is out, there are about 10 times times, speculation should be Baidu spiders and the like, identified as users, and my exhibition is so

