Python crawler Simple Notes

Source: Internet
Author: User

There are a lot of very useful libraries built into the Python2.7, which are located in the/usr/lib/python2.7 on my computer.

Write a basic crawler to use the library has urllib, URLLIB2, Cookielib, Sgmllib and re, the following is a brief introduction of the next few files of information and related functions-specific really is suggested to read the source, the information on the Internet rather than its intuitive ( But first it's probably always good to know, except Sgmllib. Let's look at a piece of code first.

Opener = Urllib2.build_opener (urllib2. Httpcookieprocessor (Cookiejar))

Urllib2.install_opener (opener)

Response = Urllib2.urlopen (URL)

urllib2.py This file comment is the most conscientious, basically read the next comment and browse the file the main class and def can understand the usage. According to the note, it is an extensible library for opening URLs using a variety of protocols. The main class is:

Handler--each Handler implements a particular protocol or option. It is not a specific class, but a collection of handler that have their own special functions, such as Proxyhandler in Def Build_opener Default_classes recorded, HttpHandler (each of which is a class). Each handler is to use a specific URL protocol to open the URL or handle the URL of some relevant content, such as the above httpcookieprocessor, can be passed through the cookiejar of the previous cookie, and have HTTP_ The request and Http_response functions are rounded into the request or response. These classes are inherited from the class Basehandler.

Class Openerdirector--the Openerdirector sets up the User Agent as the Python-urllib client and manages a collection of Handler objects.

In its __init__ function can be clearly seen self.addheaders = [(' User-agent ', client_version)] a sentence, visible it is now ready to construct a request (just prepare). It then has the add_handler (self, handler) method to add handler to it. In simple terms, the individual thinks it is used to tell Urllib2 (as the Python-urllib client) How to handle the URL passed in as a parameter. The Open methods under this class are discussed later.

def build_opener (*handlers) instantiates a Openerdirector class and handles it a little bit and then return

The Def install_opener function is used to convert incoming Openerdirector objects to global variables using global, so that urllib2 open URLs can be opened in a set-up manner

def Urlopen First look at its related source code:

(URL, Data=none, timeout=socket._global_default_timeout):

Global _opener

If _opener is None:

_opener = Build_opener ()

Return _opener.open (URL, data, timeout)

Then the related description of the returned page results is that It opens the URL and returns the results as File-like object. The URL and the data that the request might use are all in the parameter list, and there's the last line of code, which is where the real send request and return page information is. In this way Openerdirector's open () has req = Request (FullUrl, data), which shows that the request was built here. Then open () calls _open () again, and _open () calls the _call_chain () ... These functions will also be a layer of calls down, and here is not a rip. We have to think about Urlopen () Here is enough for the crawler. When you look at the lower levels later, I'll write another text.

There are other things that are not to be said and feel unnecessary, such as Class Request, which is an object, which encapsulates the state of a Request.

urllib.py for a specific URL request is still directly with urllib2.py, it has part of the content is urllib.py to import past, feel that there is more convenient to use Although there are a lot of things here that are useless on the urllib2.py, there is always the feeling that they can be replaced in other, more understandable ways. In addition, there is always something puzzling about this file (such as popping out of a row Ftpcache = {} I'm starting to suspect life.

So here's just a character set of the file that handles the problem. For example, it has UrlEncode (), which can encode str url ()--of course, encoding and then integrating the data into the request. In addition, there are also tobyte () such processing methods. These are scattered, to use when you look at the source code to check Google is good.

In addition, it has a fun statement such as if sys.platform = = ' Darwin ' and elif os.name = = ' nt '. This paper, can be said to understand the basic level of processing, be sure to read.

cookielib.py I don't know much about cookies, it's just a little bit more important to say

Class Cookie is just for a cookie operation, and also __init__ __str__ __repr__ these operations,

There is also a direct manipulation of its specific properties, such as

Set_nonstandard_attr (self, Name, value):

Self._rest[name] = value

The class Cookiejar notes are clear, Collection of HTTP cookies. The source of the cookie is path (there seems to be a way to support stripping from the request). Not very sure). The main methods are _cookies_for_domain and _cookies_for_request.

In this way, the domain and path properties of the cookie are mentioned. Domain is a valid domain for a cookie object and may not be a request-generated access domain. Cookies cannot be accessed directly across domains, but there are certain ways to break this limit.

Path indicates the directory where the cookie resides

In addition, some pages seem to be attached to a specific entry page of the cookie information to be effective (at least I understand the crawler code for my school curriculum information).

sgmllib.py read this file when it was very painful, completely do not know what to say. In this case, personal advice is to go online to check the file for what, probably how to use, tally structure and then back to see, then will feel much clearer.

This document is mainly a sgmlparser. It decomposes the contents of HTML documents into useful fragments according to the labels, and processes them according to the characteristics of each fragment, and interprets each fragment according to the order appearing in the original document.

When defining what to do when parsing hypertext, create a Sgmlparser subclass and define the behavior of a particular tag (such as Def START_TD is a description of what the subclass will do when it encounters <td>, and Def end_td resolves to </td> behavior), as in between the two ... between tags is passed to the parser by calling Self.handle_data (). It also resolves custom tags. It is said that when parsing large documents, performance is better than regular. I didn't test this.

re.py is used in Python to denote the regular. The usual is Reg = Re.complie (' xxx ') to build a regular, and then Objs = reg.findall (content) in the content to find all the contents of the Reg-compliant. It is then possible for the for-obj in OBJS to traverse the processing.

In addition, if it is necessary to set UTF-8 encoding, it is best to remember the import sys, then reload (SYS), and then set sys.setdefaultencoding ("Utf-8")

And then go straight to the simplest example.

url = "Http://xxxxxx"

Cookiejar = Cookielib. Cookiejar () # There are checks below, no cookies it will be one of their own

Opener = Urllib2.build_opener (urllib2. Httpcookieprocessor (Cookiejar))

Urllib2.install_opener (opener)

Response = Urllib2.urlopen (URL)

Content = Response.read ()

Content = Content.decode ("GB18030")

Print content

Crawl down and use SGML or re to find the part you're interested in. Unfortunately I am not a cross-border, write a troublesome point must first look at the example.

Python crawler Simple Notes

Contact Us

The content source of this page is from Internet, which doesn't represent Alibaba Cloud's opinion; products and services mentioned on that page don't have any relationship with Alibaba Cloud. If the content of the page makes you feel confusing, please write us an email, we will handle the problem within 5 days after receiving your email.

If you find any instances of plagiarism from the community, please send an email to: info-contact@alibabacloud.com and provide relevant evidence. A staff member will contact you within 5 working days.

A Free Trial That Lets You Build Big!

Start building with 50+ products and up to 12 months usage for Elastic Compute Service

  • Sales Support

    1 on 1 presale consultation

  • After-Sales Support

    24/7 Technical Support 6 Free Tickets per Quarter Faster Response

  • Alibaba Cloud offers highly flexible support services tailored to meet your exact needs.