Shocked that most Internet users are not people

Shocked that most Internet users are not people

Website security and content distribution company Incapsula released a data, 56% of the Web page views are contributed by the crawler robot.

Crawler robots are divided into these categories because of their different functions and purposes:

Search engine crawlers: Search engine crawlers, which are able to index web pages so that people can find the content of the corresponding page in the search box. Google uses this crawler to organize information around the world.

RSS Bots: The subscription crawler, can be aggregated from the site crawl content, fresh fruit, today's headlines and web news client Use this kind of crawler.

scrapers: web crawler, is generally stolen content, email address and reverse crack pricing model, it can play a role in e-commerce sites.

impersonator: Humanoid camouflage crawler, it can disguise as a search engine or browser, to avoid being found by the site. It can collect marketing intelligence, launch DDoS attacks, consume bandwidth, and even paralyze the site.

HackingTools: A hack tool that can steal information, embed rogue software, disrupt web content, and even hijack websites and servers.

spammers: spam sending tool that can harass ordinary visitors, post unrelated content or phishing links. It can also load excessive links, let the website into the search engine blacklist and "disappear" from the Internet.

In 56 of the visits, the malicious crawler robot accounted for 29%, a bona fide account of 27%. With the decrease of RSS crawlers, the proportion of bona fide reptiles is even smaller.

Most sites have a crawl ratio of 63% to 80%, and the smaller the percentage of web crawler accesses. Search engine crawler is the main reason for this phenomenon, it is almost non-discriminatory to small sites and large sites, and on average each site day by Google's search engine crawler access 187 times.

The humanoid camouflage crawler is growing fast and it is the only reptile that has continued to grow in the last 3 years. In the above mentioned Google search engine crawler, on average, every 24 visits will have a disguise crawler visit. of these disguised reptiles, 25.16% from the United States, China accounted for 15.61%, is the second largest source country.

RSS crawlers are gradually fading. The older generation of RSS tools, such as Google Reader and fresh fruit, are already dying.

Incapsula's data comes from 20,000 websites with at least 10 visits a day, and it has compiled 15 billion access data for the past 90 days to produce this result.

What we see, data security, bandwidth consumption, and ad browsing are all linked to bots, and they reshape the way we work and live.

Shocked that most Internet users are not people