Realizing geek College curriculum crawler based on requests

Background

The purpose of this paper is to complete the actual work performed by instructors in the "Python single-threaded crawler" of the Geek's college course.

Development environment

Operating system: Windows 10

Python:python 2.7

Ide:pycharm 4.5.3

Installing requests

The cmd Command Prompt window can be installed by tapping the following line command.

PIP Install requests

Implementation code

#Coding=utf-8__author__='Scaleworld'ImportRequestsImportReImportsysreload (SYS) sys.setdefaultencoding ("Utf-8")classSpider:def __init__(self):Print 'Start Crawling Geek College Course info ... '    #Get Source code    defgetsource (self, URL): HTML=requests.get (URL)returnHtml.text#get information about each course block    defgetlessons (self, source): Lessons= Re.findall ('deg= "0" > (. *?) </li>', source, re. S)returnLessons#access to course information such as course name, course description, course time, course level, number of learners    defgetlessoninfo (Self, lesson): Info={} info['title'] = Re.search ('
', lesson, Re. S). Group (2). Strip () info['desc'] = Re.search ('<p style= "height:0px; opacity:0; display:none;" > (. *?) </p>', lesson, re. S). Group (1). Strip () Timeandlevel= Re.findall ('<em> (. *?) </em>', lesson, re. S) info[' Time'] = Timeandlevel[0].strip (). Replace ("\ n",""). Replace ("    ","") info[' Level'] = timeandlevel[1].strip () info['Learnnumber'] = Re.search ('"Learn-number" > (. *?) </em>', lesson, Re. S). Group (1). Strip ()returnInfo#Save course information to file LessionInfos.txt    defSavelessioninfos (Self, Lessoninfos):#' W ': Write-only, overwriting previously written content        #you can also use ' a ': Append to the end of the file        #If the file does not exist, the file is created automaticallyf = open ('LessionInfos.txt','W') I=0 foreachinchlessoninfos:i+ = 1F.writelines ('Section'+ STR (i) +'course: \ n') F.writelines ('Title:'+ each['title'] +'\ n') F.writelines ('desc:'+ each['desc'] +'\ n') F.writelines ('Time :'+ each[' Time'] +'\ n') F.writelines ('Level :'+ each[' Level'] +'\ n') F.writelines ('Learnnumber:'+ each['Learnnumber'] +'\ n') F.close ()if __name__=='__main__':    #define an array of course informationLessoninfos = []    #Course Information Page URLURL ='http://www.jikexueyuan.com/course/'    #instanced CrawlersSpider =Spider ()#take [1,21] and 1 to 20 pages of course information     forIinchRange (1, 21):        #building a paging URLPageurl = URL +'? pagenum='+Str (i)Print 'Working with page:'+Pageurl Source=Spider.getsource (pageurl) Lessons=spider.getlessons (source) forLessoninchLessons:lessoninfo=spider.getlessoninfo (Lesson) lessoninfos.append (lessoninfo)#print ' title: ' +lessoninfo.get (' title ')            #print ' desc: ' +lessoninfo.get (' desc ')            #print ' Time: ' +lessoninfo.get (' time ')            #print ' Level: ' +lessoninfo.get (' level ')            #print ' Learnnumber: ' +lessoninfo.get (' Learnnumber ')        Print 'has been processed'+ str (lessons.__len__()) +'Course information. '    Print 'Geek College Course information crawl complete, saving course information ... 'Spider.savelessioninfos (Lessoninfos)Print 'The Geek College course information is saved. '

Sample Crawl Results

1th Course:
Title:wordpress Theme Template Files
Desc: This course mainly introduces the template files that make up WordPress theme, the reference of template files and the hierarchical system of template files.
Time:3 hours 36 minutes
Level: Intermediate
learnnumber:2215 People Study

2nd Course:
Title:android Development Code
Desc: This course is about the overall design architecture style and specification as well as the infrastructure, code naming and writing specifications, and so on when building projects in Android development.
Time:3 hours 41 minutes
Level: Beginner
learnnumber:2874 People Study

......

Conclusion

Geek College is a very good learning site, you can click this link to get a free 30-day VIP benefits.

Realizing geek College curriculum crawler based on requests