Python crawler learning-Get web page source code, python crawler web page source code

Source: Internet
Author: User

Python crawler learning-Get web page source code, python crawler web page source code

By chance, I can see a topic about crawlers. <what are the cool, interesting, and useful things that crawlers can use?>, I am very curious and think that writing crawlers is a very high thing, so I am interested in crawlers.

I will not talk about the definition of web crawlers. If you do not know, click here. => Baidu encyclopedia web crawlers and Wikipedia Web Crawlers

Many programming languages can write web crawlers, but each has its own advantages and disadvantages. Here I choose to write crawlers in Python, because Python is a very suitable language for programming crawlers, the amount of code used to implement crawlers is much less than that of other languages, and the python language encapsulates modules such as network programming very well, its language features make many programmers willing to use it to write programs. In order to learn crawlers, I have been familiar with the Python language and are constantly learning to combine it with crawlers to facilitate crawlers. The version I learned and used is Python3.

To learn about web crawlers, you must first understand some basic knowledge:

The first crawler I learned is to crawl the source code of the webpage. Do not think that obtaining web page source code is a very small and simple program. It is the foundation of crawlers and is crucial. The following is the code that I understand and implement. If there is something wrong with it, point it out so that you can learn and improve it.

1 #-*-coding: UTF-8-*-# Set the encoding type to utf-82 import requests # import related request module 3 4 url = 'HTTP: // www.jianshu.com/'# the URL of the webpage to be obtained (the homepage of the simplified book) 5 response = requests. get (url) # get the webpage connection status code 6 content = response through get () in requests. text # retrieve webpage information from the returned status code through text 7 print (content) # output source code to the console

 

Contact Us

The content source of this page is from Internet, which doesn't represent Alibaba Cloud's opinion; products and services mentioned on that page don't have any relationship with Alibaba Cloud. If the content of the page makes you feel confusing, please write us an email, we will handle the problem within 5 days after receiving your email.

If you find any instances of plagiarism from the community, please send an email to: info-contact@alibabacloud.com and provide relevant evidence. A staff member will contact you within 5 working days.

A Free Trial That Lets You Build Big!

Start building with 50+ products and up to 12 months usage for Elastic Compute Service

  • Sales Support

    1 on 1 presale consultation

  • After-Sales Support

    24/7 Technical Support 6 Free Tickets per Quarter Faster Response

  • Alibaba Cloud offers highly flexible support services tailored to meet your exact needs.