Python crawlers crawl webpage table data and python crawlers webpage tables

Source: Internet
Author: User

Python crawlers crawl webpage table data and python crawlers webpage tables

Use python to crawl webpage table data for your reference. The details are as follows:

From bs4 import BeautifulSoup import requests import csv import bs4 # Check the url address def check_link (url): try: r = requests. get (url) r. raise_for_status () r. encoding = r. apparent_encoding return r. text failed T: print ('the server cannot be connected !!! ') # Crawl resource def get_contents (ulist, rurl): soup = BeautifulSoup (rurl, 'lxml') trs = soup. find_all ('tr ') for tr in trs: ui = [] for td in tr: ui. append (td. string) ulist. append (ui) # Save resource def save_contents (urlist): with open ("D:/top ranking list of Chinese enterprises. csv", 'w') as f: writer = csv. writer (f) writer. writerow (['ranking top 2016 Chinese enterprises in 500 ']) for I in range (len (urlist): writer. writerow ([urlist [I] [1], urlist [I] [3], urlist [I] [5]) def main (): urli = [] url = "http://www.maigoo.com/news/463071.html" rs = check_link (url) get_contents (urli, rs) save_contents (urli) main ()

The above is all the content of this article. I hope it will be helpful for your learning and support for helping customers.

Related Article

Contact Us

The content source of this page is from Internet, which doesn't represent Alibaba Cloud's opinion; products and services mentioned on that page don't have any relationship with Alibaba Cloud. If the content of the page makes you feel confusing, please write us an email, we will handle the problem within 5 days after receiving your email.

If you find any instances of plagiarism from the community, please send an email to: info-contact@alibabacloud.com and provide relevant evidence. A staff member will contact you within 5 working days.

A Free Trial That Lets You Build Big!

Start building with 50+ products and up to 12 months usage for Elastic Compute Service

  • Sales Support

    1 on 1 presale consultation

  • After-Sales Support

    24/7 Technical Support 6 Free Tickets per Quarter Faster Response

  • Alibaba Cloud offers highly flexible support services tailored to meet your exact needs.