0 Python Basics-Crawler Practice Summary

Web crawler is a program or script that automatically crawls world wide Web information according to certain rules.

Crawler main problem: 1.http request 2. Parsing HTML source Code 3. Counter-crawl mechanism.

I think the crawler is very interesting, just to see someone to share a little reptile tutorial: https://zhuanlan.zhihu.com/p/20410446 immediately learn!

Main steps:

1, follow the tutorial download python, configure environment variables, learn to use the PIP command, install development ide:pycharm

2. Learn to use Python to send requests for pages

3, use the Chrome Developer tool to observe the page structure features, using BeautifulSoup parsing page

4. Save the page to a local file

Major problems encountered:

1.python basic syntax: variables, functions, loops, exceptions, conditional statements, creating directories, writing files.

2.python indentation is important, and indentation determines the grouping and layering of statements, especially when looping.

3. Encoding format: From code editing, to Web content, Chinese file names, there is no coding format problem.

4.beautifulsoup use.

5. Failure of the crawl rule, re-analysis of the invalid page, re-select page features.

Practice, use crawlers to get the questions on the Web page (automatically grab the next page) code:

#Encoding=utf8#sets the encoding format of the edit source py file to UTF8Importrequests, SYS, CHARDET, OS, time, random, time fromBs4Importbeautifulsoupreload (SYS)#you have to reloadSys.setdefaultencoding ("UTF8")PrintSys.getdefaultencoding (), sys.getfilesystemencoding ()#UTF8 Mbcs:mbcs (multi-bytechactactersystem, or multibyte character system) it is a type of encoding, not a name for a particular encodingPath = OS.GETCWD ()#gets the directory where the current file residesNewPath = Os.path.join (Path,"Computer")if  notOs.path.isdir (NewPath): Os.mkdir (NewPath)#New FolderDestFile = Unicode (NewPath +"/topic. docx"," Utf-8) #Save As Word can, but after the subsequent use of office editing, save always need to be saved as; Unicode (), the name of the Chinese name will not become garbled#The most common simulation browser, camouflage headersheaders = {    'user-agent':'mozilla/5.0 (Macintosh; Intel Mac OS X 10_11_2) applewebkit/537.36 (khtml, like Gecko) chrome/47.0.2526.80 safari/537.36'}defdownloadhtml (URL): HTML= Requests.get (URL, headers=headers) Content=html.content Contenten= Chardet.detect (content). Get ("encoding","Utf-8")    #Print Contenten #GB2312    Try: Trancon= Content.decode (Contenten). Encode (sys.getdefaultencoding ())#Convert Web content encoding format; eliminate Chinese garbled characters    exceptException:returnContent#Why is there a small number of page exceptions with encoding conversions?     #Print Trancon    Else:        returnTrancondefparsehtml (URL):#print URL, "Now"Content =downloadhtml (URL) contenten= Chardet.detect (content). Get ("encoding","Utf-8") Soup= BeautifulSoup (Content,"Html.parser")#Soup.name [Document] BeautifulSoup object represents the entire contents of a document    #Find Next Page URLTheul = Soup.find ("ul", {"class":"Con_updown"}) Theli= Theul.find ("Li") href= Theli.find ("a"). Get ("href") PreUrl=Noneifhref:PrintHref"Next"PREURL=href#Find what you needTopics = []    Try: Divcon= Soup.find ("Div", attrs={"class":"CON_NR"})        ifdivcon:subjects= Divcon.find_all ("P")#the __len__ property is not an integer, but instead: Method-wrapper ' __len__ ' of ResultSet objectindex = 0#What's the other way to find the first number with the index logo?              forResinchsubjects:#Skip what you don't want to read in the guide line                ifindex = = 0 andRes.string = =""Guide"": Index= 1#Jump out of the loop and add 1 .                    Continue  #Skip GuideTOPIC = Res.string#res has child tags and text, it will return none                iftopic:#just leave plain text as needed, save to file                    Try: Parsed= Topic.decode (Contenten). Encode ("UTF8")                    exceptException:topics.append ("This page is decoded incorrectly, please check it yourself:"+ URL +"\ n")#'%d '%index str (index) number to string                         Break                    Else: Topics.append (parsed+"\ n") Index= index + 1Topics.append ("\ n")        Else: Topics.append ("This page to find the wrong questions, please see for yourself:"+ URL +"\ n")    exceptException:topics.append ("this page resolves the error, please check it yourself:"+ URL +"\ n") FP= Open (DestFile,'a')#a append writefp.writelines (topics) fp.close ()returnPREURL#execute the entry for the. py fileif __name__=='__main__': I= 0#record How many pages are processedNext ="http://xxxxx/1.html" #Start Page    Print "Start time:", Time.strftime ('%y-%m-%d%h:%m:%s', Time.localtime (Time.time ()))#print time, see how long it's been running    PrintNext"Start"     whileNext andI < 1000: Next=parsehtml (next) I= i + 1#stime = Random.randint (3, 8) #随机整数 [3,8)        #Time.sleep (stime) # Rest: Anti-crawl    Print "End Time:", Time.strftime ('%y-%m-%d%h:%m:%s', Time.localtime (Time.time ()))Print "i ="I"URL:", next FP= Open (DestFile,'a')#a append writeFp.writelines (["LastPage:"+ STR (next) +"\ n","Total:"+ STR (i) +"\ n"])#None and Number: cannot and string with + splicingFp.close ()

0 Python Basics-Crawler Practice Summary