Python urllib2 simple parsing of HTML pages

First, URLLIB2 easy to get HTML page

   # !/usr/bin/env python   # -*-coding:utf-8-*-      Import urllib2    = Urllib2.urlopen ('http://www.baidu.com');    = Response.read ();    Print HTML

A few lines of code will be able to get the HTML page, the next situation of HTML parsing work.

Imagination is beautiful, the actual operation of the problem. Baidu does not prohibit the robot crawl can normally crawl to the page, but for example: https://b.ishadow.tech/is not allowed to crawl the robot, simple simulation browser header information is not.

And then want to find a crawler on GitHub to test it, so found the https://github.com/scrapy/scrapy, it seems to compare the mouth.

Follow the readme installed, the installation failed, carefully read the document.

Install Scrapy

The official recommended installation in Python's virtual environment is described below (https://doc.scrapy.org/en/latest/intro/install.html#intro-using-virtualenv):

inch  Do  Install scrapy within a so-called "virtual environment" (VIRTUALENV). Virtualenvs allow you to do conflict with already-installed Python system packages (whichinstallsu do and the likes).

Then decide to install a Python virtual environment with the following command:

sudo Install virtualenv

View Basic usage

$virtualenv-husage:virtualenv [OPTIONS] Dest_dir

You just need to virtualenv the target directory.

So create a new virtual environment

 in ~/e27/bin/pythoninstalling setuptools, pip, Wheel ... done.

Enabling the Environment

$source./bin/activate

Note the current directory will be identified after the switch environment is successful, as follows

?  E27 source./bin/Activate (E27)?  

Exit the Environment

$deactivate

Get started now, install crawlers in virtual environments (https://github.com/scrapy/scrapy)

Install Scrapy

After approximately three minutes of installation, the installation of altogether directly in the global environment has failed. After success the shell output is as follows:

......
Successfully built lxml pydispatcher Twisted pycparserinstalling collected Packages:lxml, Pydispatcher, Zope.interface, Constantly, incremental, attrs, Automat, Twisted, IPAddress, Asn1crypto, Enum34, Idna, Pycparser, Cffi, cryptography, PYOP Enssl, Queuelib, W3lib, Cssselect, Parsel, PYASN1, PYASN1-modules, service-identity, scrapysuccessfully installed Automat-0.5.0pydispatcher-2.0.5twisted-17.1.0asn1crypto-0.22.0attrs-16.3.0cffi-1.10.0constantly-15.1.0cryptography-1.8.1cssselect-1.0.1enum34-1.1.6idna-2.5incremental-16.10.1ipaddress-1.0. -lxml-3.7.3parsel-1.1.0pyopenssl-16.2.0Pyasn1-0.2.3pyasn1-modules-0.0.8pycparser-2.17queuelib-1.4.2scrapy-1.3.3service-identity-16.0.0w3lib-1.17.0zope.interface-4.3.3

Try a simple connection after installing the scrapy.

(E27)?  ' http://quotes.toscrape.com/page/1/ '

Get a bunch of results as follows

 .-Geneva- -  A: ,: the[Scrapy.core.engine] Info:spider opened .-Geneva- -  A: ,: +[Scrapy.core.engine] Debug:crawled ( $) <get http://quotes.toscrape.com/page/1/> (Referer:none)[s] Available scrapy objects:[s] scrapy scrapy module (contains Scrapy. Request, Scrapy. Selector, etc) [s] crawler<scrapy.crawler.crawlerObjectAt0x1100bab50>[s] Item {}[s] request<get http://quotes.toscrape.com/page/1/>[s] Response < $http//quotes.toscrape.com/page/1/>[s] Settings <scrapy.settings.settingsObjectAt0x1100baad0>[s] Spider<defaultspider'default'At0x11037ebd0>[s] useful shortcuts:[s] Fetch (url[, redirect=True]) Fetch URL and update local objects (by default, redirects is followed) [s] Fetch (req) fetch a scrapy.  Request and update local objects [s] shelp () Shell Help (print this help) [s] View (response) View response inchA browser

Prove that it works, and then try connecting: https://b.ishadow.tech/

(E27)?  ' https://b.ishadow.tech/ '  

The results are as follows:

 .-Geneva- -  A:Ten: +[Scrapy.middleware] info:enabled item pipelines:[] .-Geneva- -  A:Ten: +[scrapy.extensions.telnet] debug:telnet console listening on127.0.0.1:6023 .-Geneva- -  A:Ten: +[Scrapy.core.engine] Info:spider opened .-Geneva- -  A: One: $[Scrapy.downloadermiddlewares.retry] debug:retrying <get https://b.ishadow.tech/> (failed 1 times): TCP connection timed out:60:operation timed out. .-Geneva- -  A: A:Wuyi[Scrapy.downloadermiddlewares.retry] debug:retrying <get https://b.ishadow.tech/> (Failed 2 times): TCP connection timed out:60:operation timed out. .-Geneva- -  A: -: -[Scrapy.downloadermiddlewares.retry] debug:gave up retrying <get https://b.ishadow.tech/> (Failed 3 times): TCP connection timed out:60:operation timed out.Traceback (most recent call Last):

Crawled out of time, it seems to be recognized as a robot crawl content is rejected (of course, the site through the browser is accessible), the bad my brother! Here you have guessed my true purpose, no, please open my crawl to get the connection to see it.

No crawl, no time. It's good to study this reptile. Document Connection: https://doc.scrapy.org/en/latest/intro/overview.html

Then slowly study how to break through the blockade.

Python urllib2 simple parsing of HTML pages