Use Python regular expression to extract sites from search results

Recently, we need to export the site address of the Google search results. Then, the regular expression in python is used to extract the search results.

This involves several issues to be solved:

1. Obtain the search result text

To get more addresses, I used Google's advanced search function,Each pageDisplay 100 results.

After obtaining the displayed result, you can view the source code and keep it as a text file.Search Result text

 

2. analyze how to extract site information

First, analyze the obtained page to see how the site information can be extracted.

I use the built-in IE8Development tools (F12 will pop up)InProbeFunction to view the special format of the content you want to care about

We can see that the site I need is in the tag <cite> </CITE>, so can I use a regular expression to extract the text in it?

3. Write a regular expression to get the site address

The next step is to write the expression. I wrote it using python3.2 for ease of use (~ _~)

The Code is as follows,First, save the search result page to E:/t3.txt and execute the following code:

Import re

P = Re. Compile (R' <cite> ([^ <> \/]. + ?) </CITE> ')

F = open ("E:/t3.txt", encoding = 'utf-8 ')

Content = f. Read ()

Print ("\ n". Join (P. findall (content )))

Run the following command:

Check that all the site addresses are obtained.