Python crawler learning-Get web page source code, python crawler web page source code
By chance, I can see a topic about crawlers. <what are the cool, interesting, and useful things that crawlers can use?>, I am very curious and think that writing crawlers is a very high thing, so I am interested in crawlers.
I will not talk about the definition of web crawlers. If you do not know, click here. => Baidu encyclopedia web crawlers and Wikipedia Web Crawlers
Many programming languages can write web crawlers, but each has its own advantages and disadvantages. Here I choose to write crawlers in Python, because Python is a very suitable language for programming crawlers, the amount of code used to implement crawlers is much less than that of other languages, and the python language encapsulates modules such as network programming very well, its language features make many programmers willing to use it to write programs. In order to learn crawlers, I have been familiar with the Python language and are constantly learning to combine it with crawlers to facilitate crawlers. The version I learned and used is Python3.
To learn about web crawlers, you must first understand some basic knowledge:
The first crawler I learned is to crawl the source code of the webpage. Do not think that obtaining web page source code is a very small and simple program. It is the foundation of crawlers and is crucial. The following is the code that I understand and implement. If there is something wrong with it, point it out so that you can learn and improve it.
1 #-*-coding: UTF-8-*-# Set the encoding type to utf-82 import requests # import related request module 3 4 url = 'HTTP: // www.jianshu.com/'# the URL of the webpage to be obtained (the homepage of the simplified book) 5 response = requests. get (url) # get the webpage connection status code 6 content = response through get () in requests. text # retrieve webpage information from the returned status code through text 7 print (content) # output source code to the console