A small crawler that keeps searching from Baidu

I learned python on the third day. I want to write something to commemorate it. So I wrote crawler for a long time, but it is not a good worm. I can only talk about the keywords of the web page and store them locally, but I think it's basically a bug.
The beautifulsoup library is used in this article to process HTML document analysis. Because I only extracted the keyword of the title, it can be replaced by a regular expression. Another library is jieba, this library is used for Chinese word segmentation. Another library is chardet, which is used to judge character encoding. I wanted to be multi-threaded, but I thought I was confused and gave up.
? 1. [Code] [Python] Code
# Coding: UTF-8
Import re
Import urllib
Import urllib2
Import sys
Import time
Import queue
Import thread
Import threading
Import jieba
Import chardet
From beautifulsoup import beautifulsoup as BS
 
 
Deep = 1000
Lock = threading. Lock ()
Path = "C: \ test \\"
Urlqueue = queue. Queue ()
Def pachong ():
Url = 'HTTP: // www.baidu.com'
Return URL
 
Def getpageurl (HTML ):
Reurl = Re. Compile (R' <\ s * [AA] {1} \ s + [^>] *? [HH] [RR] [EE] [ff] \ s * = \ s * [\ "\ ']? ([^> \ "\ '] +) [\" \']?. *?> ')
URLs = reurl. findall (HTML)
For URL in URLs:
If Len (URL)> 10:
If Url. Find ('javascript ') =-1:
Urlqueue. Put (URL)
 
Def getcontents (URL ):
Try:
Url = urllib2.quote (URL. Split ('#') [0]. encode ('utf-8'), safe = "%/: = &?~ # +! $,; '@ () * [] ")
Req = urllib2.urlopen (URL)
Res = Req. Read ()
Code = chardet. Detect (RES) ['encoding']
# Printhttp: // www.huiyi8.com/donghua /?
# Print codeflash Material
Res = res. Decode (STR (CODE), 'ignore ')
Res = res. encode ('gb2312', 'ignore ')
Code = chardet. Detect (RES) ['encoding']
# Print Code
# Print res
Return res
Failed t urllib2.httperror, E:
Print E. Code
Return none
Failed t urllib2.urlerror, E:
Print STR (E)
Return none

Def writetofile (HTML, URL ):
Fp = file (path + STR (Time. Time () + '.html ', 'w ')
FP. Write (HTML)
FP. Close ()


Def getkeywords (HTML ):
Code = chardet. Detect (HTML) ['encoding']
If code = 'iso-8859-2 ':
Html. Decode ('gbk', 'ignore'). encode ('gb2312', 'ignore ')
Code = chardet. Detect (HTML) ['encoding']
Soup = BS (HTML, fromencoding = "gb2312 ")
Titletag = soup. Title
Titlekeywords = titletag. Contents [0]
Cutwords (titlekeywords)

Def cutwords (contents ):
Print Contents
Res = jieba. cut_for_search (contents)
Res = '\ n'. Join (RES)
Print res
Res = res. encode ('gb2312 ')
KEYWORDS = file (path + 'cutkeywors.txt ', 'A ')
Keywords. Write (RES)
Keywords. Close ()

Def start ():

While urlqueue. Empty () = false:
Url = urlqueue. Get ()
Html = getcontents (URL)
Getpageurl (HTML)
Getkeywords (HTML)
# Writetofile (HTML, URL)
 

If _ name _ = '_ main __':
Starturl = pachong ()
Urlqueue. Put (starturl)
Start ()


?

A small crawler that keeps searching from Baidu