Python crawler--regular expression of several methods of parsing web pages

1. Regular expressions

A regular expression is a special sequence of characters that can help you easily check whether a string matches a pattern.

The RE module enables the Python language to have all the regular expression functionality.

Re.match function

Re.match attempts to match a pattern from the starting position of the string, and if the match is not successful, match () returns none.

Import Re Print (Re.match ('www'www.runoob.com'). span ())  # match at start position Print (Re.match ('com'www.runoob.com'))         # Do not match at start position

Results:

(0, 3) None

Importreline="Cats is smarter than dogs"Matchobj= Re.match (r'(. *) is (. *?). *', line)ifMatchobj:Print("Matchobj.group ():", Matchobj.group ())Print("Matchobj.group (1):", Matchobj.group (1))   Print("Matchobj.group (2):", Matchobj.group (2))Else:   Print("No match!!")

Results:

Matchobj.group ():  Cats is smarter than Dogsmatchobj.group (1):  catsmatchobj.group (2):  Smarter

R ' (. *) is (. *). * ', r means raw string, pure string, group (0), is the overall result of the matching regular expression, group (1) lists the first bracket matching part, Group (2) lists the second bracket matching part.

Re.search method

Re.search scans the entire string and returns the first successful match.

Re.match matches only the beginning of the string, if the string does not begin to conform to the regular expression, the match fails, the function returns none, and Re.search matches the entire string until a match is found.

Importreline="Cats is smarter than dogs"; Matchobj= Re.match (r'Dogs', line, re. m|Re. I)ifMatchobj:Print("match--matchobj.group ():", Matchobj.group ())Else:   Print("No match!!") Matchobj= Re.search (r'Dogs', line, re. m|Re. I)ifMatchobj:Print("Search--Matchobj.group ():", Matchobj.group ())Else:   Print("No match!!")

Results:

-Matchobj.group ():  dogs

Re.findall method

FindAll is able to find the matching result and return it as a list.

ImportRequestsImportrelink="http://www.sohu.com/"Headers= {'user-agent':'mozilla/5.0 (Windows; U Windows NT 6.1; En-us; rv:1.9.1.6) gecko/20091201 firefox/3.5.6'}r= Requests.get (link, headers=headers) HTML=r.texttitle_list= Re.findall ('href= ". *?". <strong> (. *?) </strong>', HTML)Print(title_list)

[' News ', ' finance ', ' Sports ', ' real estate ', ' entertainment ', ' cars ', ' fashion ', ' technology ', ' gourmet ', ' constellation ', ' email ', ' map ', ' thousand sails ', ' swim ']

Grab the main title of Sohu.

Python crawler--regular expression of several methods of parsing web pages