Two code examples for implementing multi-thread collection in python: Multiple-thread in python

Two code examples for implementing multi-thread collection in python: Multiple-thread in python

Code 1:

#! /Usr/bin/python #-*-coding: UTF-8-*-# encoding = UTF-8 import threadingimport Queueimport sysimport urllib2import reimport MySQLdb # Database variable setting # DB_HOST = '100. 0.0.1 'db _ USER = "XXXX" DB_PASSWD = "XXXXXXXX" DB_NAME = "xxxx" # variable setting # THREAD_LIMIT = 3 jobs = Queue. queue (5) singlelock = threading. lock () info = Queue. queue () def workerbee (inputlist): for x in xrange (THREAD_LIMIT): print 'thead {0} started. '. forma T (x) t = spider () t. start () for I in inputlist: try: jobs. put (I, block = True, timeout = 5) handle T: singlelock. acquire () print "The queue is full! "Singlelock. release () # Wait for the threads to finish singlelock. acquire () # Acquire the lock so we can print "Waiting for threads to finish. "singlelock. release () # Release the lock jobs. join () # This command waits for all threads to finish. # while not jobs. empty (): # print jobs. get () def getTitle (url, time = 10): response = urllib2.urlopen (url, timeout = time) html = response. read () response. c Lose () reg = R' <title> (.*?) </Title> 'title = re. compile (reg ). findall (html) # title = title [0]. decode ('gb2312', 'replace '). encode ('utf-8') title = title [0] return title class spider (threading. thread): def run (self): while 1: try: job = jobs. get (True, 1) singlelock. acquire () title = getTitle (job [1]) info. put ([job [0], title], block = True, timeout = 5) # print 'this {0} is {1 }'. format (job [1], title) singlelock. release () jobs. task_done () Partition T: break; if _ name _ = '_ main _': con = None urls = [] try: con = MySQLdb. connect (DB_HOST, DB_USER, DB_PASSWD, DB_NAME) cur = con. cursor () cur.exe cute ('select id, url FROM 'table _ name' WHERE 'status' = 0 LIMIT 10') rows = cur. fetchall () for row in rows: # print row urls. append ([row [0], row [1]) workerbee (urls) while not info. empty (): print info. get () finally: if con: con. close ()

Code 2:

#! /Usr/bin/python #-*-coding: UTF-8-*-# encoding = UTF-8 # Filename: robot. py import threading, Queue, sys, urllib2, re ## variable setting # THREAD_LIMIT = 3 # set the number of threads jobs = Queue. queue (5) # Set the Queue length singlelock = threading. lock () # Set a thread Lock to avoid repeated calls of urls = ['HTTP: // callback ',' Bytes 3322.shtml ', 'HTTP: // response -04-26/1402703292. shtml ', 'HTTP: // response Defdef wor Kerbee (inputlist): for x in xrange (THREAD_LIMIT): print 'thead {0} started. '. format (x) t = spider () t. start () for I in inputlist: try: jobs. put (I, block = True, timeout = 5) handle T: singlelock. acquire () print "The queue is full! "Singlelock. release () # Wait for the threads to finish singlelock. acquire () # Acquire the lock so we can print "Waiting for threads to finish. "singlelock. release () # Release the lock jobs. join () # This command waits for all threads to finish. # while not jobs. empty (): # print jobs. get () def getTitle (url, time = 10): response = urllib2.urlopen (url, timeout = time) html = response. read () response. c Lose () reg = R' <title> (.*?) </Title> 'title = re. compile (reg ). findall (html) title = title [0]. decode ('gb2312', 'replace '). encode ('utf-8') return title class spider (threading. thread): def run (self): while 1: try: job = jobs. get (True, 1) singlelock. acquire () title = getTitle (job) print 'this {0} is {1 }'. format (job, title) singlelock. release () jobs. task_done () Worker T: break; if _ name _ = '_ main _': workerbee (urls)

Python enables multi-threaded access to the web site and performs operations

I would like to ask what you mean by "accessing a website", is it to download an object? Or do I need to submit some forms or other access methods? What is multithreading used? Is it because multiple threads download an object at the same time so that this object can be downloaded faster? Or what other purposes?
It is easy for python to access the web. Several Functions of urllib can be called as soon as possible, but it is not clear what you want to do, so I don't feel how to answer you.

Python Multithreading

It calls fun () separately, for example, the code in the fun function is inserted in both threads. However, these two threads share the variables of the fun function. To avoid code exceptions, we recommend that you first understand thread communication before writing code.