Python crawlers crawl webpage table data and python crawlers webpage tables

Python crawlers crawl webpage table data and python crawlers webpage tables

Use python to crawl webpage table data for your reference. The details are as follows:

From bs4 import BeautifulSoup import requests import csv import bs4 # Check the url address def check_link (url): try: r = requests. get (url) r. raise_for_status () r. encoding = r. apparent_encoding return r. text failed T: print ('the server cannot be connected !!! ') # Crawl resource def get_contents (ulist, rurl): soup = BeautifulSoup (rurl, 'lxml') trs = soup. find_all ('tr ') for tr in trs: ui = [] for td in tr: ui. append (td. string) ulist. append (ui) # Save resource def save_contents (urlist): with open ("D:/top ranking list of Chinese enterprises. csv", 'w') as f: writer = csv. writer (f) writer. writerow (['ranking top 2016 Chinese enterprises in 500 ']) for I in range (len (urlist): writer. writerow ([urlist [I] [1], urlist [I] [3], urlist [I] [5]) def main (): urli = [] url = "http://www.maigoo.com/news/463071.html" rs = check_link (url) get_contents (urli, rs) save_contents (urli) main ()

The above is all the content of this article. I hope it will be helpful for your learning and support for helping customers.