Introduced
This article mainly introduces how to crawl the course information of the Wheat College (this reptile is still a single thread crawler), before beginning to introduce, first look at the result sketch
What, are you ready to go? Let's start by opening the web site of the Wheat Institute and finding the full course information of the Wheat Academy, as follows
This time to page, watch the changes in the Web site, first of all, the first page of the URL is http://www.maiziedu.com/course/list/, the second page into a http://www.maiziedu.com/course/list/ all-all/0-2/, the third page has become http://www.maiziedu.com/course/list/all-all/0-3/, you can see that every time you flip a page, the number after 0 will increase by 1, then someone will think, take the first page? We try to put http://www.maiziedu.com/course/list/all-all/0-1/into the browser's address bar, we found that we can open the first column, it is good to do, we only need to use Re.sub () You can easily get to the content of any page. After getting to the URL link, the following is to get the source code of the Web page, first right click to view the review or check the elements, you can see the following interface
Find the location of the course, you can easily use the regular expression will we need to extract the content, as to how to extract, it depends on your own, try to find their own laws to have a greater harvest. If you really do not know how to extract, then continue down, look at my source code
Actual source code
# Coding=utf-8 Import re import requests import sys reload (SYS) sys.setdefaultencoding ("UTF8") class spider (): def __init__ (self): Print starts crawling content ... def changepage (self, URL, total_page): nowpage = Int (Re.search ('/0-(\d+)/', URL, re. S). Group (1)) PageGroup = [] for I in range (Nowpage, Total_page + 1): link = re.sub ('/0-(\d+)/', '/0-%s/ '% i, URL, re. S) pagegroup.append (link) return PageGroup def getsource (self, url): html = requests.get (URL) return h Tml.text def getclasses (self, source): Classes = Re.search (' <ul class= ' zy_course_list ' > (. *?) </ul> ', source, re. S). Group (1) Return classes def geteach (self, classes): Eachclasses = Re.findall (' <li> (. *?) </li> ', classes, re. S) return eachclasses def getinfo (self, eachclass): info = {} info[' title ' = Re.search (' <a title= ' (. *?) "', EAC Hclass, Re. S). Group (1) info[' People ' = Re.search (' <p class= ' Color99 ' > (. *?) </p> ', eachclass, Re. S). Group (1) Return info def saveinfo (self, classinfo): F = open (' Info.txt ', ' a ') for each in CLASSINFO:F.WR Itelines (' title: ' + each[' title '] + ' \ n ') f.writelines (' People: ' + each[' people '] + ' \ n ') F.close () If _ _name__ = = ' __main__ ': ClassInfo = [] url = ' http://www.maiziedu.com/course/list/all-all/0-1/' Maizispider = SP Ider () all_links = Maizispider.changepage (URL, to each in all_links:htmlsources = Maizispider.getsource (E ACH) classes = maizispider.getclasses (htmlsources) eachclasses = Maizispider.geteach (classes) for each in
Eachclasses:info = Maizispider.getinfo (each) classinfo.append (info) maizispider.saveinfo (ClassInfo)
The above code is not difficult to understand, the basic is the use of regular expressions, and then directly run to see the beginning of our screenshot content, because this is a single-threaded crawler, so the running speed feel a bit slow, next will continue to update the multithreaded crawler.
At the request of the small partners, please attach the installation and simple example of the requests Reptile Library below
First install the PIP Package Management tool and download the get-pip.py. There are both Python2 and Python3 installed on my machine.
Install Pip to Python2:
python get-pip.py
Install to Python3:
python3 get-pip.py
After the PIP installation is complete, install the requests library to open the Python crawler learning.
Install requests
pip3 install requests
The python3,python2 I use can be used directly with PIP install requests.
Examples of Getting Started
Import requests
Html=requests.get ("http://gupowang.baijia.baidu.com/article/283878")
html.encoding= ' Utf-8 '
print (html.text)
The first line introduces the requests library, the second line uses the requests get method to obtain the Web page source code, the third line sets the encoding format, and the fourth line of text output.
Save the retrieved Web page source code in a text file:
Import requests
import OS
html=requests.get ("http://gupowang.baijia.baidu.com/article/283878")
Html_ File=open ("News.txt", "W")
html.encoding= ' Utf-8 '
print (html.text,file=html_file)