Use BEAUTIFULSOUP+URLIB2 to install, crawl, and parse Web pages in Python, and parse shtml

Source: Internet
Author: User

Installing Beautiful Soup?

If you are using a new version of Debain or Ubuntu, you can install it through the system's package management:

$ apt-get Install PYTHON-BS4

easy_install   or     to install. The name of the package is   This package is compatible with Python2 and Python3.

$ Easy_install BEAUTIFULSOUP4

$ pip Install Beautifulsoup4

beautifulsoup beautiful Soup3   release, because many projects are still using BS3, so     package is still valid. But if you're writing a new project, you should install the    )

If you do not have the installation Easy_install or Pip , then you can also Download BS4 's source code , and then install it through setup.py.

$ Python setup.py Install

If none of the above installation methods work, Beautiful Soup's release protocol allows you to package BS4 code in your project so that you don't need to install it.

Authors develop beautiful Soup in versions Python2.7 and Python3.2, theoretically beautiful Soup should work correctly in all current Python versions

Installing the parser?

Beautiful Soup supports the HTML parser in the Python standard library and also supports some third-party parsers, one of which is lxml depending on the operating system, you can choose from the following methods to install lxml:

$ apt-get Install Python-lxml

$ Easy_install lxml

$ pip Install lxml

Another alternative parser is the pure Python implementation Html5lib , the html5lib is parsed in the same way as the browser, and you can choose the following methods to install Html5lib:

$ apt-get Install Python-html5lib

$ Easy_install Html5lib

$ pip Install Html5lib

Parse the General Web page (HTML) directly:

From BS4 import Beautifulsoup,beautifulstonesoup

Import Urllib2

Import Html5lib

Url_header = "XXXXXX"

Webpage = Urllib2.urlopen (url_header). Read ()

Soup = BeautifulSoup (webpage)

Print soup.prettify ()


However, when parsing the sHTML Web page, the BeautifulSoup module supports the interpreter lxml,html5lib and htmlparse three kinds, only html5lib support parsing shtml, so when the BeautifulSoup object is generated, To add a parameter: Soup = BeautifulSoup (webpage, "Html5lib"), otherwise, the contents of the sHTML </body> tag cannot be parsed when parsing the,<body> page.

For more information, please refer to: BeautifulSoup 4.2.0 official file http://www.crummy.com/software/BeautifulSoup/bs4/doc/index.zh.html


Use BEAUTIFULSOUP+URLIB2 to install, crawl, and parse Web pages in Python, and parse shtml

Related Article

Contact Us

The content source of this page is from Internet, which doesn't represent Alibaba Cloud's opinion; products and services mentioned on that page don't have any relationship with Alibaba Cloud. If the content of the page makes you feel confusing, please write us an email, we will handle the problem within 5 days after receiving your email.

If you find any instances of plagiarism from the community, please send an email to: info-contact@alibabacloud.com and provide relevant evidence. A staff member will contact you within 5 working days.

A Free Trial That Lets You Build Big!

Start building with 50+ products and up to 12 months usage for Elastic Compute Service

  • Sales Support

    1 on 1 presale consultation

  • After-Sales Support

    24/7 Technical Support 6 Free Tickets per Quarter Faster Response

  • Alibaba Cloud offers highly flexible support services tailored to meet your exact needs.