Python multi-thread crawling Google search link webpage

1) urllib2 + BeautifulSoup capture Goolge Search links

Recently, participating projects need to process Google search results. I have learned Python tools related to webpage processing. In practical applications, urllib2 and beautifulsoup are used to capture webpages. However, when capturing google search results, it is found that if the source code of the google search results page is directly processed, you will get a lot of "dirty" links.

The search result is "titanic james:

 

In the figure, the red mark is not required, and the blue mark is to be captured.

These "Dirty links" can be filtered out by rule filtering, but the complexity of the program is high. Just as I was frowning about filtering rules. I was reminded that google should provide related APIs to make it easy.

(2) Google Web Search API + Multithreading

 

This document provides an example of searching using Python:

[Python]
Import simplejson

# The request also provided des the userip parameter which provides the end
# User's IP address. Doing so will help distinguish this legitimate
# Server-side traffic from traffic which doesn' t come from an end-user.
Url = ('https: // ajax.googleapis.com/ajax/services/search/web'
'? V = 1.0 & q = Paris % 20 Hilton & userip = USERS-IP-ADDRESS ')

Request = urllib2.Request (
Url, None, {'Referer':/* Enter the URL of your site here */})
Response = urllib2.urlopen (request)

# Process the JSON string.
Results = simplejson. load (response)
# Now have some fun with the results...

Import simplejson
 
# The request also provided des the userip parameter which provides the end
# User's IP address. Doing so will help distinguish this legitimate
# Server-side traffic from traffic which doesn' t come from an end-user.
Url = ('https: // ajax.googleapis.com/ajax/services/search/web'
'? V = 1.0 & q = Paris % 20 Hilton & userip = USERS-IP-ADDRESS ')
 
Request = urllib2.Request (
Url, None, {'Referer':/* Enter the URL of your site here */})
Response = urllib2.urlopen (request)
 
# Process the JSON string.
Results = simplejson. load (response)
# Now have some fun with the results...

In actual applications, many web pages of google may need to be crawled. Therefore, multithreading is required to share the capture task. For more information about using the google web search api, see here (Standard URL Arguments is introduced here ). In addition, note that the rsz parameter in the url must be values below 8 (including 8). If the value is greater than 8, an error is returned!

(3) code implementation

Code implementation still has problems, but it can run and has poor robustness and needs to be improved. I hope all the experts will point out the errors (beginner Python) and I am very grateful.

[Python]
#-*-Coding: UTF-8 -*-
Import urllib2, urllib
Import simplejson
Import OS, time, threading
 
Import common, html_filter
# Input the keywords
Keywords = raw_input ('Enter the keywords :')
 
# Define rnum_perpage, pages
Rnum_perpage = 8
Pages = 8
 
# Defining thread Functions
Def thread_scratch (url, rnum_perpage, page ):
Url_set = []
Try:
Request = urllib2.Request (url, None, {'Referer': 'http: // www.sina.com '})
Response = urllib2.urlopen (request)
# Process the JSON string.
Results = simplejson. load (response)
Info = results ['responsedata'] ['result']
Except t Exception, e:
Print 'error occured'
Print e
Else:
For minfo in info:
Url_set.append (minfo ['url'])
Print minfo ['url']
# Processing links
I = 0
For u in url_set:
Try:
Request_url = urllib2.Request (u, None, {'Referer': 'http: // www.sina.com '})
Request_url.add_header (
'User-agent ',
'Csc'
)
Response_data = urllib2.urlopen (request_url). read ()
# Filter files
# Content_data = html_filter.filter_tags (response_data)
# Writing files
Filenum = I + page
Filename = dir_name + '/related_html _' + str (filenum)
Print 'write start: related_html _ '+ str (filenum)
F = open (filename, 'W + ',-1)
F. write (response_data)
# Print content_data
F. close ()
Print 'write down: related_html _ '+ str (filenum)
Except t Exception, e:
Print 'error occured 2'
Print e
I = I + 1
Return
 
# Creating folders
Dir_name = 'related _ html _ '+ urllib. quote (keywords)
If OS. path. exists (dir_name ):
Print 'existfile'
Common. delete_dir_or_file (dir_name)
OS. makedirs (dir_name)
 
# Webpage capture
Print 'start to scratch web pages :'
For x in range (pages ):
Print "page: % s" % (x + 1)
Page = x * rnum_perpage
Url = ('https: // ajax.googleapis.com/ajax/services/search/web'
'? V = 1.0 & q = % s & rsz = % s & start = % s') % (urllib. quote (keywords), rnum_perpage, page)
Print url
T = threading. Thread (target = thread_scratch, args = (url, rnum_perpage, page ))
T. start ()
 
# The main thread waits for the sub-thread to capture
Main_thread = threading. currentThread ()
For t in threading. enumerate ():
If t is main_thread:
Continue
T. join ()