URLLIB2 and BeautifulSoup crawl data save in Python MongoDB

Source: Internet
Author: User
Tags datetime html page mongodb split

Beautiful soup is a Python library that parses HTML and XML, and it can parse files in the way you like, and find and modify the parse tree. It can handle nonstandard tags well and generate parse trees (parse tree). It provides simple and commonly used navigation (navigating), searches, and modifies the parse tree operation.

Use the URLLIB2 and BS4 modules to crawl HTML page data, respectively, for title, content, stock name, stock ID, publication time, and number of onlookers.


The code is as follows

Import time
From BS4 import BeautifulSoup
Import Urllib2
Import Pymongo
Import re
Import datetime

def update ():

datas = {}

connection = Pymongo. Connection (' ', 27017)


db = Connection.test_hq


For I in Soup.find_all ("div", class_= "item"):

datas[' _id '] = str (i.h2.a[' href ']). Split ('/') [ -1].split ('. ') [0]


datas[' title ' = I.h2.get_text ()


url2 = i.h2.a[' href ']


HTML2 = Urllib2.urlopen (URL2)

html_doc2 = Html2.read ()

soup2 = BeautifulSoup (HTML_DOC2)

datas[' content ' = Soup2.find (attrs={"name": "description"}) [' Content ']


stock_name = []

stock_id = []

for name in Re.findall (U "[u4e00-u9fa5]+", I.find (class_= "Stocks"). Get_text ()):

stock_name.append (name)

#获取影响股票名称, an array of ways to save the corresponding stock ID number, MONGO support array insertion

datas[' stock_name '] = stock_name

for ID in Re.findall ("d+", I.find (class_= "Stocks"). Get_text ():

stock_id.append (ID)


datas[' stock_id '] = stock_id

datas[' update_time ' = Datetime.datetime.strptime (Re.search ("w+.*w+", I.find (class_= "FL Date"). Span.get_text (). Group (), '%y-%m-%d%h:%m ')-Datetime.timedelta (hours=8)

#获取发布时间, converted to MONGO time format

datas[' onlooker ' = Int (Re.search ("d+", I.find (class_= "Icons Ic-wg"). Get_text ()). Group ())


Db.test.save (datas)


Def get_data ():
    title = str (soup.h2.a[' href ']). Split ('/') [-1] . Split ('. ') [0]
    with open (' Update.txt ', ' R ') as F:
         time = F.readline ()
    if  title = = Time:
         print ' Currently no update ', title
   & nbsp;    with open (' Update.txt ', ' W ') as F:
             F.write (title)
        Update ()
While True:
    if __name__ = = ' __main__ ':
         url = ' http://www.ipython.me/qingbao/'
        html = Urllib2.urlopen (URL)
        html_doc = Html.read ()
        soup = BeautifulSoup (html_doc)
   & nbsp;    get_data ()
        time.sleep ()
 # Refresh once every 30 seconds

Related Article

E-Commerce Solutions

Leverage the same tools powering the Alibaba Ecosystem

Learn more >

Apsara Conference 2019

The Rise of Data Intelligence, September 25th - 27th, Hangzhou, China

Learn more >

Alibaba Cloud Free Trial

Learn and experience the power of Alibaba Cloud with a free trial worth $300-1200 USD

Learn more >

Contact Us

The content source of this page is from Internet, which doesn't represent Alibaba Cloud's opinion; products and services mentioned on that page don't have any relationship with Alibaba Cloud. If the content of the page makes you feel confusing, please write us an email, we will handle the problem within 5 days after receiving your email.

If you find any instances of plagiarism from the community, please send an email to: info-contact@alibabacloud.com and provide relevant evidence. A staff member will contact you within 5 working days.