I'm Wang zebin, a web spider.

I am a web spider who crawls tirelessly every day in every corner of the Internet. In foreign countries, humans call us Web Spider. Of course, there are also some human beings without artistic cells. We are called crawlers. Although I have been in this world for a long time, we already have a very large family. In fact, our common work is very simple, that is, to go to every website on the internet and take a snapshot of their pages for them to be transported back. In this way, humans can query what they need through the search engine. Although our snapshot technology is not as professional as Miss Guan Xi, we are definitely more diligent than him and will go to various websites anytime to see if there is any content update.

First of all, we have to thank the Internet. It is with the Internet that we have a warm home, so that we can settle in and cultivate our lives. The second thing I would like to thank is the search engine, such as Gu Ge and Bai ge. When I hear their names, my heart is always awe-inspiring. They are the nvwa In the spider world, the gods in our hearts, and we will always thank their eight generations of ancestor.

 

When we were born, we were actually very pure and naive. We had our ideals, our beliefs, and we wanted to be a pure spider and a noble spider, A spider is a little different from low-level taste. Later, the greed of humans broke part of us. Human programmers complain all day: they get up earlier than a chicken, sleep later than a dog, do worse than a donkey, and eat worse than a pig. But why don't they think about it? We're much more tired than them. We work 7*24 hours a day, no matter what we eat, drink, or work, it is still at risk of being blocked by the network administrator. However, to be honest, people liked us very much at the beginning. When we took photos on various small websites, we were very popular. That guy's scene was really spectacular. The more content we crawl back, the more links indexed by the search engine, and the higher the ranking of small websites. As the traffic of small websites grows, one day, it became a big website. Then, humans took the PV or PR of the website to venture capital and said, "You see how big our traffic is, should you invest some money ". As soon as I went there, I got the silver. So, small websites don't welcome us. At that time, every time we visit the website, we eat a full Chinese banquet. When we leave, the website will also say: always come to the door.

Suddenly one day, the old brother and the old brother of the hundred second brother had a fight. Old Brother Gu said: "I am the boss, I am from miguo, some of them are silver, and I have more content than you have included !". Bai Er said: "I tmd is the boss. You are a foreign product. I know Chinese better ". As a result, the two bosses began to fight, and you sent 10 spider, and I sent 100; you sent 1000 spider, and I sent 10000 spider. Pot afraid of pot ah. As a result, the website suffers, and the website is full of spider, which consumes a lot of bandwidth and human access fails. A few days ago, csdn had been crawled, and cnblogs had been crawled several times. Websites like and fear spider. When there are too many spider s, they will be knocked to death. Besides, I am a big website now, and the traffic is surging. Don't join in with your spider. I rely on it. Humans are like this. When you use your words, you can just give it up.

It doesn't matter if there are several search engines in the world. The key is that many aggregated websites are emerging. This kind of sub-aggregation websites are pure second-channel traffickers who have never produced any content. They send a group of spider every day to collect content from various major websites. A website with a bit of conscience retains the address of the original website and contributes a few external links to the original website. A website without conscience is changed to its own, so for such websites, it is really hateful. The Spider of their family directly decides.

Humans have different qualities. We also have spider. High-quality spider, when crawling to someone else's website, his face is tagged: I am a spider (that is, in the http header, the User-agent: Robot similar words ). In this example, when the network worker is welcome, the robots.txt file can be configured in webserver ). A low-quality spider is dressed like a common spider, but it is just a result of diligence. Nowadays, many search engines are no longer qualified.

Our spider can be smart now. When we crawl webpages, we will also pay attention to strategies. Sometimes I will give priority to breadth, and sometimes I will give priority to depth. Some website designers are very bad. They have designed traps on the web page. As soon as we climb in, we cannot turn out and die in the trap. Later, our spider was able to learn and never fight independently. Once trapped, other associates would come and pull the team out of the trap. We also have to learn to constantly change the IP address because the network administrator has to change the IP address. If there is nothing in our house, we just need no vest. In other cases, we are too greedy. If there is delicious food on the website, we want to have a good time and stay away. This has indeed affected the access of customers. We have shortcomings, but if we correct them, will you still block our IP addresses? The great man said: it is a good comrade. If there is a mistake, it will be changed. If it is changed to another crime, it will be done with great efforts. Websites, do you mean this?