Capture and analyze webpage content using python curl and lxml

Curl is a powerful URL syntax client that supports DICT, FILE, FTP, FTPS, Gopher, HTTP, HTTPS, IMAP, IMAPS, LDAP, LDAPS, POP3, POP3S, RTMP, RTSP, SCP, SFTP, SMTP, SMTPS, and other protocols.

Lxml is a very useful xml module in python and supports html processing. Although html is a subset of xml, some html webpages are not properly written. For example, if you write <option> but do not write </option>, in this case, an error is returned when you use an xml tool for analysis. However, html tools are generally compatible with these non-standard situations.

Using urljoin of urlparse can help you deal with relative path problems.

If your python lacks curl or lxml modules, you can go to pypi to find them.

= 8 === ht_doc.xpath (I urlparse. urljoin (base_url, I. get (lxml. etree. tostring (I, encoding =