Python Crawler Learning Notes single-threaded crawler _python

Source: Internet
Author: User

Introduced

This article mainly introduces how to crawl the course information of the Wheat College (this reptile is still a single thread crawler), before beginning to introduce, first look at the result sketch

What, are you ready to go? Let's start by opening the web site of the Wheat Institute and finding the full course information of the Wheat Academy, as follows

This time to page, watch the changes in the Web site, first of all, the first page of the URL is http://www.maiziedu.com/course/list/, the second page into a http://www.maiziedu.com/course/list/ all-all/0-2/, the third page has become http://www.maiziedu.com/course/list/all-all/0-3/, you can see that every time you flip a page, the number after 0 will increase by 1, then someone will think, take the first page? We try to put http://www.maiziedu.com/course/list/all-all/0-1/into the browser's address bar, we found that we can open the first column, it is good to do, we only need to use Re.sub () You can easily get to the content of any page. After getting to the URL link, the following is to get the source code of the Web page, first right click to view the review or check the elements, you can see the following interface

Find the location of the course, you can easily use the regular expression will we need to extract the content, as to how to extract, it depends on your own, try to find their own laws to have a greater harvest. If you really do not know how to extract, then continue down, look at my source code

Actual source code

 # Coding=utf-8 Import re import requests import sys reload (SYS) sys.setdefaultencoding ("UTF8") class spider (): def __init__ (self): Print starts crawling content ... def changepage (self, URL, total_page): nowpage = Int (Re.search ('/0-(\d+)/', URL, re. S). Group (1)) PageGroup = [] for I in range (Nowpage, Total_page + 1): link = re.sub ('/0-(\d+)/', '/0-%s/ '% i, URL, re. S) pagegroup.append (link) return PageGroup def getsource (self, url): html = requests.get (URL) return h Tml.text def getclasses (self, source): Classes = Re.search (' <ul class= ' zy_course_list ' > (. *?) </ul> ', source, re. S). Group (1) Return classes def geteach (self, classes): Eachclasses = Re.findall (' <li> (. *?) </li> ', classes, re. S) return eachclasses def getinfo (self, eachclass): info = {} info[' title ' = Re.search (' <a title= ' (. *?) "', EAC Hclass, Re. S). Group (1) info[' People ' = Re.search (' <p class= ' Color99 ' > (. *?) </p> ', eachclass, Re. S). Group (1) Return info def saveinfo (self, classinfo): F = open (' Info.txt ', ' a ') for each in CLASSINFO:F.WR Itelines (' title: ' + each[' title '] + ' \ n ') f.writelines (' People: ' + each[' people '] + ' \ n ') F.close () If _ _name__ = = ' __main__ ': ClassInfo = [] url = ' http://www.maiziedu.com/course/list/all-all/0-1/' Maizispider = SP Ider () all_links = Maizispider.changepage (URL, to each in all_links:htmlsources = Maizispider.getsource (E  ACH) classes = maizispider.getclasses (htmlsources) eachclasses = Maizispider.geteach (classes) for each in


 Eachclasses:info = Maizispider.getinfo (each) classinfo.append (info) maizispider.saveinfo (ClassInfo)

The above code is not difficult to understand, the basic is the use of regular expressions, and then directly run to see the beginning of our screenshot content, because this is a single-threaded crawler, so the running speed feel a bit slow, next will continue to update the multithreaded crawler.

At the request of the small partners, please attach the installation and simple example of the requests Reptile Library below

First install the PIP Package Management tool and download the get-pip.py. There are both Python2 and Python3 installed on my machine.

Install Pip to Python2:

python get-pip.py

Install to Python3:

python3 get-pip.py

After the PIP installation is complete, install the requests library to open the Python crawler learning.

Install requests

pip3 install requests

The python3,python2 I use can be used directly with PIP install requests.

Examples of Getting Started

Import requests

Html=requests.get ("http://gupowang.baijia.baidu.com/article/283878")
html.encoding= ' Utf-8 '
print (html.text)

The first line introduces the requests library, the second line uses the requests get method to obtain the Web page source code, the third line sets the encoding format, and the fourth line of text output.
Save the retrieved Web page source code in a text file:

Import requests
import OS

html=requests.get ("http://gupowang.baijia.baidu.com/article/283878")
Html_ File=open ("News.txt", "W")
html.encoding= ' Utf-8 '
print (html.text,file=html_file)

Contact Us

The content source of this page is from Internet, which doesn't represent Alibaba Cloud's opinion; products and services mentioned on that page don't have any relationship with Alibaba Cloud. If the content of the page makes you feel confusing, please write us an email, we will handle the problem within 5 days after receiving your email.

If you find any instances of plagiarism from the community, please send an email to: info-contact@alibabacloud.com and provide relevant evidence. A staff member will contact you within 5 working days.

A Free Trial That Lets You Build Big!

Start building with 50+ products and up to 12 months usage for Elastic Compute Service

  • Sales Support

    1 on 1 presale consultation

  • After-Sales Support

    24/7 Technical Support 6 Free Tickets per Quarter Faster Response

  • Alibaba Cloud offers highly flexible support services tailored to meet your exact needs.