How to extract the url list on the page using python

This article describes how to extract the url list from a page by using python, and describes how to operate on page elements by using Python, for more information about how to extract the url list on the page using python, see the following example. Share it with you for your reference. The specific implementation method is as follows:

from bs4 import BeautifulSoupimport time,re,urllib2t=time.time()websiteurls={}def scanpage(url):  websiteurl=url  t=time.time()  n=0  html=urllib2.urlopen(websiteurl).read()  soup=BeautifulSoup(html)  pageurls=[]  Upageurls={}  pageurls=soup.find_all("a",href=True)  for links in pageurls:    if websiteurl in links.get("href") and links.get("href") not in Upageurls and links.get("href") not in websiteurls:      Upageurls[links.get("href")]=0  for links in Upageurls.keys():    try:      urllib2.urlopen(links).getcode()    except:      print "connect failed"    else:      t2=time.time()      Upageurls[links]=urllib2.urlopen(links).getcode()      print n,      print links,      print Upageurls[links]      t1=time.time()      print t1-t2    n+=1  print ("total is "+repr(n)+" links")  print time.time()-tscanpage("http://news.163.com/")

I hope this article will help you with Python programming.