Python web crawler and Information extraction--5. Information organization and extraction method

1. Three forms of information markers

(1) XML (extensible Markup Language) Extensible tagged Voice

<name> ... </name> tagged with content
<name/> label with no content
<!‐‐‐‐> notes

(2) JSON (Javsscript Object Notation) has a type of key-value pair Key:value

"Key": "Value"
"Key": ["value1", "value2"] multi-value with [,] organization
"Key": {"subkey": "Subvalue"} key value pair nested {,}

(3) Yaml (Yaml Ain ' t Markup Language) untyped key-value pair Key:value

Indent Expression owning relationship

-Express side-by-side relationships

| Expression of the whole block of data # represents a comment

Key:value
Key: #Comment
‐value1
‐value2
Key:
Subkey:subvalue

2. Compare

Xml

The earliest universal Information Markup language, extensibility is good, but cumbersome

Information interaction and delivery on the Internet

Json

Information has a type, suitable for program processing (JS), more concise than XML

Mobile app Cloud and node information Communication, no annotations

Yaml

No type of information, the highest proportion of text information, good readability

Various types of system configuration files, annotated easy to read

3. General methods of information extraction

Method One: Complete parsing information in the form of markup, and then extract the key information

Advantages: Accurate information analysis
Disadvantage: The extraction process is cumbersome and slow

Method Two: Ignore the Mark form, search the key information directly

Advantages: The extraction process is simple and fast
Disadvantage: The accuracy of extraction results is related to information content

Fusion method: Combining form parsing and searching method to extract key information

Need tag parser and text lookup function

4. HTML content Lookup method based on BS4 library

<>.find_all (name, Attrs, recursive, string, **kwargs)
Returns a list type that stores the results of a lookup

? Name: Retrieves a string for the label name

? Attrs: Retrieving strings for Tag property values, labeling attribute retrieval
? Recursive: Whether to retrieve all descendants, default True
? String: Retrieving strings for the string range in <>...</>

<tag> (..) equivalent to <tag>.find_all (..)
Soup (..) Equivalent to Soup.find_all (..)

Extension methods:

<>.find () Search and return only one result, with the. Find_all () parameter
<>.find_parents () Search in ancestor node, return list type, same. Find_all () parameter
<>.find_parent () Returns a result in the ancestor node with the. Find () parameter
<>.find_next_siblings () Searches in subsequent parallel nodes, returns the list type, and the same. Find_all () parameter
<>.find_next_sibling () Returns a result in subsequent parallel nodes, with the. Find () parameter
<>.find_previous_siblings () searches in a sequential parallel node, returns the list type, and the same. Find_all () parameter
<>.find_previous_sibling () Returns a result in a sequential parallel node with the. Find () parameter

4. Chinese University Ranking crawler example

2. #CrawUnivRankingB. py
4. Import Requests
6. From bs4 import beautifulsoup
8. Import BS4
12. def gethtmltext (URL):
14. try:
16. r = requests.get (URL, timeout=+)
18. r.raise_for_status ()
20. r.encoding = r.apparent_encoding
22. return r.text
24. except:
26. return ""
30. def fillunivlist (ulist, HTML):
32. Soup = beautifulsoup (HTML, "Html.parser")
34. for tr in soup.find (' tbody '). Children:
36. if isinstance(tr, bs4.element.Tag):
38. TDS = TR (' TD ')
40. ulist.append ([tds[0].string, tds[1].string, tds[3].string])
44. def printunivlist (ulist, num):
46. tplt = "{0:^10}\t{1:{3}^10}\t{2:^10}"
48. print(tplt. Format("Rank","School name","Total score",chr(12288 )))
50. for i in range(num):
52. u=Ulist[i]
54. print(tplt. Format(u[0],u[1],u[2],chr(12288 )))
58. def Main ():
60. uinfo = []
62. URL = ' https://www.zuihaodaxue.cn/zuihaodaxuepaiming2016.html '
64. html = gethtmltext (URL)
66. fillunivlist (uinfo, HTML)
68. printunivlist (Uinfo, ) # univs
70. Main ()

Python web crawler and Information extraction--5. Information organization and extraction method