Python crawler BeautifulSoup

Simply put, Beautiful soup is a library of Python, and the main function is to fetch data from a Web page.

Beautiful soup automatically converts the input document to Unicode encoding, and the output document is converted to UTF-8 encoding.

Beautiful soup transforms complex HTML documents into a complex tree structure, where each node is a Python object, and all objects can be summed up in 4 types: tag,navigablestring beautifulsoup Comment

What is 1.Tag? Popular point is a label in HTML

Using soup spike signatures to easily get the contents of these tags, is it more convenient to feel than regular expressions? Soup.title get the title tag, SOUP.P gets the first P tag in the document, to get all the tags, you have to use the Find_all function. The Find_all function returns a sequence that can be looped, in turn, to get the thought of something.

For Tag, it has two important properties, name and Attrs.

2.NavigableString

Now that we've got the content of the tag, the question is, what do we do to get the text inside the tag? Very simple, use. String to

3.BeautifulSoup

The BeautifulSoup object represents the entire contents of a document. Most of the time, it can be used as a tag object, a special tag

4. Comment

The Comment object is a special type of navigablestring object, but the content of the output still does not include the comment symbol

5. Traverse the document Tree

Direct child nodes:

. contents

Tag's. Content property can output the child nodes of the tag as a list. can be obtained using [num]. Use contents to traverse the tree backwards, using the parent to traverse the tree forward

. Children

It does not return a list, but we can get all the child nodes through the traversal. We print the output. Children look, you can see that it's a list builder object.

You can use list to convert it to a list. Of course you can use the For statement to traverse the child inside.

All descendant nodes:

6. Search the document tree

The most common is the Find_all () function

7.CSS Selector

When writing CSS, the tag name does not have any decoration, the class name is added before the ID name plus #

Here we can also use a similar method to filter elements, the method used is Soup.select (), the return type is a list

1. Search by tag name

2. Find by class name

3. Find by ID

4. Combination Lookup

5. Property Lookup

Reference: http://blog.csdn.net/freeking101/article/details/60886125

Python crawler BeautifulSoup