Learning reptiles for a short period of time, so decided to learn their own reptiles learned, written code and read the document recorded, right when the summary. The first time to write these, if there are errors, please give more advice. First we need to understand what a reptile is. According to the definition of Baidu Encyclopedia, "web crawler (also known as Web spider, Network robot, in the middle of the foaf community, more often called the Web Chaser), is a certain rules, automatically crawl the World Wide Web information program or script. "In simple terms, reptiles like to manually download Web information, such as fiction, songs. Certainly will have the question, since this, why still go to knock Code, direct manual processing not good? That is to say, but if the download is not a two novels, a song two, need to be the entire site of the information to crawl down, this time the manual operation seems a little overwhelmed. To learn what a simple crawler should do, we simply comb:
- First we need to be familiar with a programming language, here we take Python as an example, we need a simple understanding of its syntax, master the use of lists, dictionaries and other data structures, for loop, file read storage and other operations, these will be you write the crawler in the acquisition of information, processing information, storage information needs to use the knowledge. Do not introduce, basic grammar can refer to the Novice Tutorial Python basic tutorial
- Familiar with the basic knowledge of the web, do not need to be proficient, but need to have an understanding. Here is just a brief introduction, want to know more can see the document link I attached to the introduction
URL: (Universal Resource Locator is a Uniform Resource Locator) The URL is a concise representation of where and how access to resources can be obtained from the Internet. The URL provides an abstract recognition method for the location of the resource, and uses this method to locate the resource, allowing the system to perform various operations on resources (any object that can be accessed on the Internet, including file directories, files, documents, images, and any other form of data), such as storage, updates, Replace and find its properties. The general form of the URL is the Access form of the <url >://< host >:< port >/< path > The Access form of the left <url > mainly File Transfer Protocol (FTP), Hypertext Transfer Protocol (HTTP), etc. , the common form is HTTP, which is described below. < host > One is required,< ports > and < paths > can sometimes be omitted. Example: http://baidu.com
http: (Hypertext Transfer Protocol is a Hypertext Transfer Protocol) HTTP is a simple request-response protocol that typically runs on TCP, which specifies the messages that the client may send to the server, as well as the resulting response. We can simply understand the concept of the contract and the package (Get/post) details of the Novice tutorial HTTP Tutorial html: (Hypertext Markup Lanhguage is Hypertext Markup Language) HTML is a production of the World Wide Web page standard language, It eliminates the barriers to the exchange of computer information. HTML defines a number of "tags" for typography, and the various tags embedded in the World Wide Web page form an HTML document. The pages we are crawling are basically HTML pages. So for some of the tags in the HTML need to know, but also to control the layout of the Web page style CSS, easy to parse the page later. Beginner Tutorial HTML tutorial, beginner tutorial CSS Tutorial
- Mastering the packet-capture analysis tool
One way is through Google or Firefox's own developer tools, F12 or right mouse button check, can quickly and conveniently locate the page element location another way is to grab the packet analysis tool Fiddler, I am not very often used, the general browser F12+f5 done, About the use of Fiddler can be Baidu.
The expression used to extract the information from the Web page may feel a bit cumbersome, but after learning a wildcard pattern, you will feel that it is actually quite useful. For the regular tutorial, we recommend this: regular expression 30-minute introductory tutorial, in addition to regular expression, if directly through the code to test the feasibility of regular expressions, cost time and trouble, recommend a website: regular expression online test
- The last is the Python library required by some crawlers, you can install directly on the command line via Pip, with Pycharm can be installed in setting.
1.requests used to request the Web page module, and the same role of Urllib, but I am more accustomed to requests requests Chinese document 2. BeautifulSoup is used to parse the Web page of the module, in the small crawler has a powerful role, and easy to learn. With requests, you can easily crawl and parse Web pages. The BeautifulSoup4.4.0 document looks a bit more, but it's good to get started with a few small projects. Of course, the crawler can not only these simple content, the real difficulty lies in the various anti-crawling measures to do the strategy, and this, but also in the follow-up through a simple practical way to introduce to you, whether it is a document or video, is only we understand the premise of crawler knowledge, really let us grasp, only actual combat, By writing your own code, I believe you can quickly master the basic use of reptiles. Well, the first time to say so much, I hope that we have a lot of advice, learn together.
Python crawler Basics and pre-preparation
