Python implements simple HTML table Parsing

Python implements simple HTML table Parsing

This example describes how to implement simple HTML table parsing in Python. Share it with you for your reference. The specific analysis is as follows:

Libxml2dom is dependent on here. Make sure to install libxml2dom first! Import to your step and call the parse_tables () function.

1. source = a string containing the source code you can pass in just the table or the entire page code

2. headers = a list of ints OR a list of strings

If the headers are ints this is for tables with no header, just list the 0 based index of the rows in which you want to extract data.

If the headers are strings this is for tables with header columns (with the tags) it will pull the information from the specified columns

3. The 0 based index of the table in the source code. If there are multiple tables and the table you want to parse is the third table in the code then pass in the number 2 here

It will return a list of lists. each inner list will contain the parsed information.

The Code is as follows:

?

1

2

3

4

5

6

7

8

9

10

11

12

13

14

15

16

17

18

19

20

21

22

23

24

25

26

27

28

29

30

31

32

33

34

35

36

37

38

39

40

41

42

43

44

45

46

47

48

49

50

51

52

53

54

55

56

57

58

59

60

61

62

63

64

65

66

67

68

69

70

71

72

73

74

75

76

77

78

79

80

81

82

83

84

85

86

87

88

89

90

91

92

93

94

95

96

97

98

99

100

101

102

103

104

105

106

107

108

109

110

111

112

113

114

115

116

117

118 # The goal of table parser is to get specific information from specific

# Columns in a table.

# Input: source code from a typical website

# Arguments: a list of headers the user wants to return

# Output: A list of lists of the data in each row

Import libxml2dom

Def parse_tables (source, headers, table_index ):

"Parse_tables (string source, list headers, table_index)

Headers may be a list of strings if the table has headers defined or

Headers may be a list of ints if no headers defined this will get data

From the rows index.

This method returns a list of lists

"""

# Determine if the headers list is strings or ints and make sure they

# Are all the same type

J = 0

Print 'printing headers: ', headers

# Route to the correct function

# If the header type is int

If type (headers [0]) = type (1 ):

# Run no_header function

Return no_header (source, headers, table_index)

# If the header type is string

Elif type (headers [0]) = type ('A '):

# Run the header_given function

Return header_given (source, headers, table_index)

Else:

# Return none if the headers aren't correct

Return None

# This function takes in the source code of the whole page a string list

# Headers and the index number of the table on the page. It returns a list

# Lists with the scraped information

Def header_given (source, headers, table_index ):

# Initiate a list to hole the return list

Return_list = []

# Initiate a list to hold the index numbers of the data in the rows

Header_index = []

# Get a document object out of the source code

Doc = libxml2dom. parseString (source, html = 1)

# Get the tables from the document

Tables = doc. getElementsByTagName ('table ')

Try:

# Try to get focue on the desired table

Main_table = tables [table_index]

Except t:

# If the table doesn' t exits then return an error

Return ['The table index was not found ']

# Get a list of headers in the table

Table_headers = main_table.getElementsByTagName ('th ')

# Need a sentry value for the header loop

Loop_sentry = 0

# Loop through each header looking for matches

For header in table_headers:

# If the header is in the desired headers list

If header. textContent in headers:

# Add it to the header_index

Header_index.append (loop_sentry)

# Add one to the loop_sentry

Loop_sentry + = 1

# Get the rows from the table

Rows = main_table.getElementsByTagName ('tr ')

# Sentry value detecting if the first row is being viewed

Row_sentry = 0

# Loop through the rows in the table, skipping the first row

For row in rows:

# If row_sentry is 0 this is our first row

If row_sentry = 0:

# Make the row_sentry not 0

Row_sentry = 1337

Continue

# Get all cells from the current row

Cells = row. getElementsByTagName ('td ')

# Initiate a list to append into the return_list

Cell_list = []

# Iterate through all of the header index's

For I in header_index:

# Append the cells text content to the cell_list

Cell_list.append (cells [I]. textContent)

# Append the cell_list to the return_list

Return_list.append (cell_list)

# Return the return_list

Return return_list

# This function takes in the source code of the whole page an int list

# Headers indicating the index number of the needed item and the index number

# Of the table on the page. It returns a list of lists with the scraped info

Def no_header (source, headers, table_index ):

# Initiate a list to hold the return list

Return_list = []

# Get a document object out of the source code

Doc = libxml2dom. parseString (source, html = 1)

# Get the tables from document

Tables = doc. getElementsByTagName ('table ')

Try:

# Try to get focus on the desired table

Main_table = tables [table_index]

Except t:

# If the table doesn' t exits then return an error

Return ['The table index was not found ']

# Get all of the rows out of the main_table

Rows = main_table.getElementsByTagName ('tr ')

# Loop through each row

For row in rows:

# Get all cells from the current row

Cells = row. getElementsByTagName ('td ')

# Initiate a list to append into the return_list

Cell_list = []

# Loop through the list of desired headers

For I in headers:

Try:

# Try to add text from the cell into the cell_list

Cell_list.append (cells [I]. textContent)

Except t:

# If there is an error usually an index error just continue

Continue

# Append the data scraped into the return_list

Return_list.append (cell_list)

# Return the return list

Return return_list

I hope this article will help you with Python programming.