Getting started with python: web page data capture and python crawling

Getting started with python: web page data capture and python crawling

This is good. Just getting started.

1. Use feedparser:
Tip: use Universal Feed Parser to control RSS
Http://www.ibm.com/developerworks/cn/xml/x-tipufp.html
Please visit feedparser.org to learn more about the Universal Feed Parser, which also includes some download materials and documents.

Feedparser:
Http://code.google.com/p/feedparser/downloads/list

2. In addition, you need to add the UTF-8 bom header to the file and use python to write hexadecimal characters:
Http://linux.byexamples.com/archives/478/python-writing-binary-file/
Python writes hexadecimal characters
File. write ("\ x5F \ x9D \ x3E ")
File. close ()

3. to debug the file, change the open mode to w.

Python code

1. Import urllib
2. Import sys
3. Import re
4. From feedparser import _ getCharacterEncoding as enc
6. Class TagParser:
7. Def _ init _ (self, value ):
8. Self. value = value
9. Def get (self, start, end ):
10. Regx = re. compile (R' <'+ start + R'. *?>. * </'+ End + R'> ')
11. Return re. findall (regx, self. value)
13. If _ name _ = "_ main __":
14. Baseurl = "http://data.book.163.com/book/section/000BAfLU/000BAfLU"
15. F = open ("test_01.txt", "w ")
16. F. write ("\ xef \ xbb \ xbf ")
17. # For ndx in range (0, 56 ):
18. For ndx in range (0, 1 ):
19. Url = baseurl + str (ndx) + ". html"
20. Print "get content from" + url
21. Src = urllib. urlopen (url)
22. Text = src. read ()
24. F1 = open ("tmp _" + str (ndx) + ". txt", "w ")
25. F1.write (text)
26. F1.close ()
28. Encoding = enc (src. headers, text) [0]
30. Tp = TagParser (text)
32. Title = tp. get ('h1 class = "f26s tC" ', 'h1 ')
33. Article = tp. get ('P class = "ti2em" ', 'P ')
35. T = re. sub (R' </. +> ',' \ n', title [0])
36. T = re. sub (R' <. +> ',' \ n', t)
37. Data = t
39. C = ""
40. For p in article:
41. Pt = re. sub (R' </p> ',' \ n', p)
42. C + = pt
43. C = re. sub (R' <. +> ',' \ n', c)
44. Data + = c
45. Data = data. decode (encoding)
46. F. write (data. encode ('utf-8', 'ignore '))
48. F. close ()

How can I extract the main content of a captured webpage using python?

Use beautiful soup
There are too many specific code. Check the link.
Reference: www.crummy.com/...h.html

Python captures webpage data

S = WAF webpage content
Try the following four: [try one by one] (by the way, are you from Minhang tech?
S = s. decode ("utf8 ")
S = s. decode ("gbk ")
S = s. encode ("utf8 ")
S = s. encode ("gbk ")