Python crawler----article crawler (reasonably handle the \n\t\r in a string ... ）

  Importurllib.requestImportReImportTimenum=input ("Enter Date (20150101000):")defopenpage (URL): HTML=urllib.request.urlopen (URL) page=html.read (). Decode ('gb2312')        returnpagedefgetpassage (page): Passage= Re.findall (r'<p class= "MsoNormal" align= "Left" > ([\s\s]*) </FONT>', str (page)) Passage1=re.sub ("</?\w+[^>]*>","", str (passage)) Passage2=passage1.replace ('\\r','\ r'). Replace ('\\n','\ n'). Replace ('\\t','\ t'). Replace (']',"'). Replace ('[',"'). Replace ('&nbsp;','   ')    Print(Passage2) with open (load,'a', encoding='Utf-8') as F:f.write ("-----------------------------"+"Date"+STR (date) +"---------------------------------\ n"+passage2+"----------------------------------------------------\ n") forIinchRange (1,32): Date=int (num) +Int (i)Print(date) Load="c:/users/home/desktop/new text document. txt"URL=("Http://www.hbuas.edu.cn/news/xyxw/news_"+STR (date) +". htm")
　　
Try: Page=openpage (URL) getpassage (page)Print("Section"+str (i) +"number an article,----downloaded")    except:        Print("Section"+str (i) +"no article. ") Time.sleep (2)

Wrote a crawler crawling the school News Network,

Mainly involves the writing of re regular urllib.request files

A lot of code that affects aesthetics is often returned when you crawl an article

As follows:

Optimization:

Two times regular

Passage = Re.findall (R ' <p align= "left" > ([\s\s]*) </FONT> ', str (page))       #第一次匹配字段    passage1=re.sub ( "</?\w+[^>]*>", "", str (passage) # The second time the HTML tag is removed

Replace

Passage2=passage1.replace ('\\r ', ' \ R '). Replace (' \\n ', ' \ n '). Replace (' \\t ', ' \ t '). Replace ('] ', '). "). Replace ('   ', '   )

The effect is as follows:

over!

Python crawler----article crawler (reasonably handle the \n\t\r in a string ... ）