Python web page capture program (continued)

Continue to the previous topic. The webpage captured this time is in the Tianya forum, "view the world in a geographical environment"

1. Obtain the website address: Use a regular expression to obtain the URLs of each post.

Link = 'HTTP: // www.tianya.cn/publicforum/content/worldlook/1/223829.shtml'
Html = urllib2.urlopen (link). read ()
M = re. search (r'name = \ 'idarticleslist \ 'value = \ S *> ', html)
IDs = re. findall (R' [0-9] + ', m. group (0 ))

For ID in IDs:
Url = "http://www.tianya.cn/publicforum/content/worldlook/1/%s.shtml" % ID

2. Download the webpage: Previously, the webpage was downloaded and processed. This process takes a long time and sometimes the download may fail. Download the webpage to the specified directory and check whether the same name exists before the download.

Htmldir = R'. \ html \\'

Filename = htmldir + url. split ('/') [-1]
If (not OS. path. exists (filename) or OS. path. getsize (filename) = 0:
Print 'downloading' + filename + '\ N'
Html = urlRead (url)
If len (html)> 0:
F = open (filename, 'w ')
F. write (html)
F. close ()

3. After the download, analyze the webpage content. Before the analysis, process the webpage to remove parts that cannot be processed by htmlparser. In essence, the webpage is intercepted and cannot be replaced by a string.

Txts = re. split (R' <div class = "respond" id = "adsp_content_replybox_frame_1"> ', html)
Txt = txts [0]
Txt = re. sub ('\ xcb \ xce \ xcc \ xe5',' \ xcb \ xce \ xcc \ xe5 \ '', txt)

4. Extracting the body of a post is still a regular htmlparser method. However, this method is slow and can also adopt regular expressions, but it is not adaptable. The text also uses

This post contains a large number of images, which are saved in the form of

Class DocParser (HTMLParser. HTMLParser ):
Def _ init _ (self, pool ):
Self. pool = pool
Self. startread = 0
Self. pre = 0
HTMLParser. HTMLParser. _ init _ (self)
Self.doc =''
Def handle_starttag (self, tag, attrs ):
If tag = 'span ':
For (name, value) in attrs:
If name = 'value' and value = '000000 ':
Self. pre = 1
If tag = "div" and self. pre = 1:
For (name, value) in attrs:
If name = 'class' and value = 'post ':
Self. startread = 1
If tag = 'img 'and self. startread = 1:

For (name, value) in attrs:
If name = 'original ':

Imgname = value. split ('/') [-4] + value. split ('/') [-3] \
+ Value. split ('/') [-2] + value. split ('/') [-1]
Self.doc + = ' \ n' % imgname
If not OS. path. exists (htmldir + imgname ):
Self. pool. add_task (getImg, value, htmldir)

Def handle_endtag (self, tag ):
If tag = 'div 'and self. startread = 1:
Self.doc + = '\ n \ N'
Self. pre = 0
Self. startread = 0
Def handle_data (self, data ):
If self. startread:
Self.doc + = data
Self.doc + = '\ N'

5. Crack external links: Implemented by setting Referer

Preurl = 'HTTP: // www.tianya.cn /'

Req = urllib2.Request (url)

Req. add_header ('Referer', preurl)

6. Improve urlopen robustness and set the number of retries and timeout wait.

The transformed urlopen is as follows:

Def urlRead (url ):
Fails = 0
Rs =''
Preurl = 'HTTP: // www.tianya.cn /'
While True:
Try:

If fails >=100:
Print 'failed to read' + url
Break
# Set Referer to avoid anti-leech Protection
Req = urllib2.Request (url)
Req. add_header ('Referer', preurl)
Response = urllib2.urlopen (req, timeout = 30)
Length = response.info () ['content-length']
Rs = response. read ()
If len (rs) = length:
Continue

Failed t Exception:
Fails + = 1
Time. sleep (10)
Else:
Break

Return rs

7. Simple multi-thread download: I have tried stackless before, but I still use the threadpool class,

From Queue import Queue
From threading import Thread

Class Worker (Thread ):
"Thread executing tasks from a given tasks queue """
Def _ init _ (self, tasks ):
Thread. _ init _ (self)
Self. tasks = tasks
Self. daemon = True
Self. start ()

Def run (self ):
While True:
Func, args, kargs = self. tasks. get ()
Try: func (* args, ** kargs)
Failed t Exception, e: print e
Self. tasks. task_done ()

Class ThreadPool:
"Pool of threads consuming tasks from a queue """
Def _ init _ (self, num_threads ):
Self. tasks = Queue (num_threads)
For _ in range (num_threads): Worker (self. tasks)

Def add_task (self, func, * args, ** kargs ):
"Add a task to the queue """
Self. tasks. put (func, args, kargs ))

Def wait_completion (self ):
"Wait for completion of all the tasks in the queue """
Self. tasks. join ()

Use Time:

Pool = ThreadPool (200)

For ID in IDs:
Url = "http://www.tianya.cn/publicforum/content/worldlook/1/%s.shtml" % ID
Pool. add_task (getHtml, url, htmldir)

Pool. wait_completion ()

8. output to pdf. If reportlab is used, note the following:

Reference Font:

Reportlab. rl_config.warnOnMissingFontGlyphs = 0

Using metrics. registerFont (TTFont ('yahei', 'msyh. ttf '))
Using metrics. registerFont (TTFont ('yaheibd ', 'msyhbd. ttf '))
Fonts. addMapping ('yahei', 0, 0, 'yahei ')
Fonts. addMapping ('yahei', 0, 1, 'yahei ')
Fonts. addMapping ('yaheibd ', 1, 0, 'yaheibd ')
Fonts. addMapping ('yaheibd ', 1, 1, 'yaheibd ')
Stylesheet = getSampleStyleSheet ()
NormalStyle = copy. deepcopy (stylesheet ['normal'])
NormalStyle. fontName = 'yahei'

Next is the Chinese line feed:

NormalStyle. wordWrap = 'cjk'

After the left indent of Chinese characters is set with a bug, the first line is moved to the right, not a few words less than the first of other lines.

Finally, embed the image: Get the length and width of the image, and then set the zoom ratio,

Def get_image (path ):
Width = 439
Height = 685
Img = utils. ImageReader (path)
Iw, ih = img. getSize ()

If iw> width or ih> height:
Rw = float (iw)/float (width)
Rh = float (ih)/float (height)
If rw> rh:
Return Image (path, width, round (ih/rw ))
Else:
Return Image (path, round (iw/rh), height)
Else:
Return Image (path)

Conclusion: python is a powerful tool for processing web pages, but I have wasted a lot of time coding. It does not support Chinese encoding in htmlparser and re, and I don't know whether there are any good methods for developers.

Python is called a "glue language", but I am always confused about how to use it with other java such as C.

Multi-threaded download still has many problems. You must pay attention to it when using it for reference.