Learn Python by example: crawling the body of a Web page with Python

This method is based on the text density method, the original idea originates from hit's "General Web page body extraction algorithm based on the row block distribution function", this article makes some minor changes based on this.

Conventions:

This article is based on the Web page does not peer to the statistics, therefore, assume that the content of the page is not compressed, that is, the Web page has a normal line-wrapping.

Some news pages may be short on the text content of the news, but embedded in a video file, so I will give the video a higher weight; the same applies to the picture, where there is a deficiency, it should be based on the size of the image display to determine the weight, but this method does not achieve this.

Because of ads, navigating these non-body content usually appears as hyperlinks, so the text will give the hyperlink a text weight of zero.

This assumes that the content of the body is continuous and does not contain non-body content, so in fact, extracting the body content is to find out where the body content begins and ends.

Steps:

First clear the page css,javascript, comments, meta,ins the contents of these tags, clear blank lines.

Calculates the processed values for each row (1)

Calculates the starting end position of the largest positive substring of each line of text that is drawn above

The second step is to explain:

For each row, we need to calculate a numeric value that is calculated as follows:

A picture label IMG, equivalent to a text with a length of 50 characters (given weight), x1,

A video tag embed, equivalent to a 1000-character length of text, x2

The text length of all linked label A in a row is X3,

Text length of other labels X4

Number of each row = * X1 The number of occurrences + $ * x2 The number of occurrences + x4–8

Description, 8 because we are going to calculate a maximum positive substring, so we subtract a positive number, and as for how big the numbers should be, I think it's up to the experience.

Full code

#coding: Utf-8import redef remove_js_css (content): "" "Remove the JavaScript and the stylesheet and the comment con Tent (<script>....</script> <style>....</style> <!--xxx--) "" "R = Re.compile (r" " <script.*?</script> ", Re. I|re. M|re. s) s = r.sub (", content) R = Re.compile (r" <style.*?</style> ", re. I|re. M|re. s) s = R.sub (", s) r = Re.compile (r" <!--. *?--> "", Re. I|re. M|re. s) s = R.sub (", s) r = Re.compile (r" <meta.*?> "", Re. I|re. M|re. s) s = R.sub (", s) r = Re.compile (r" <ins.*?</ins> "", Re. I|re. M|re. s) s = R.sub (', s) return sdef remove_empty_line (content): "" "Remove multi Space" "" R = Re.compile (r "" ^\s+$ "", Re. M|re. s) s = r.sub (", content) R = Re.compile (r" "\n+", re. M|re. s) s = r.sub (' \ n ', s) return sdef Remove_any_tag (s): s = re.sub (r "<[^>]+>" "," ', s) return S.strip () def remove_any_tag_but_a (s): Text = Re.findall (R ' <a[^r][^>]*> (. *?) </a> ", S,re. I|re. S|re.    s) text_b = Remove_any_tag (s) return Len (". Join (text)), Len (text_b) def remove_image (s,n=50): image = ' a ' * n R = Re.compile (R '  ', re. I|re. M|re. s) s = r.sub (image,s) return sdef Remove_video (s,n=1000): Video = ' a ' * n r = re.compile (R ' ' <embed.*?&gt ; "", Re. I|re. M|re.    s) s = r.sub (video,s) return sdef Sum_max (values): Cur_max = values[0] Glo_max = -999999 Left,right = 0,0 For Index,value in Enumerate (values): Cur_max + = value if (Cur_max > Glo_max): Glo_max =        Cur_max right = Index elif (Cur_max < 0): Cur_max = 0 for I in range (right,-1,-1): Glo_max-= Values[i] If ABS (Glo_max < 0.00001): left = I break return left,right+1 def method_1 (content, k=1): If not content:return none,none,none,none tmp = content.split (' \ n ') Group_va Lue = [] for I In range (0,len (TMP), k): group = ' \ n '. Join (tmp[i:i+k]) group = remove_image (group) group = Remove_vi Deo (Group) text_a,text_b= remove_any_tag_but_a (group) temp = (text_b-text_a)-8 Group_value.app End (temp) Left,right = Sum_max (group_value) return left,right, Len (' \ n '. Join (Tmp[:left])), Len (' \ n '. Join (Tmp[:rig    HT]) def extract (content): Content = Remove_empty_line (remove_js_css (content)) Left,right,x,y = method_1 (content) Return ' \ n '. Join (Content.split (' \ n ') [left:right])

The code starts with the last function call.

Learn Python by example: crawling the body of a Web page with Python