URLLIB2 and BeautifulSoup crawl data save in Python MongoDB

Source: Internet
Author: User
Tags datetime html page mongodb split

Beautiful soup is a Python library that parses HTML and XML, and it can parse files in the way you like, and find and modify the parse tree. It can handle nonstandard tags well and generate parse trees (parse tree). It provides simple and commonly used navigation (navigating), searches, and modifies the parse tree operation.

Use the URLLIB2 and BS4 modules to crawl HTML page data, respectively, for title, content, stock name, stock ID, publication time, and number of onlookers.

Example:

The code is as follows

##-coding:utf-8-##
Import time
From BS4 import BeautifulSoup
Import Urllib2
Import Pymongo
Import re
Import datetime

def update ():


datas = {}


connection = Pymongo. Connection (' 192.168.1.2 ', 27017)


#连接mongodb


db = Connection.test_hq


#创建或连接test_hq库


For I in Soup.find_all ("div", class_= "item"):


datas[' _id '] = str (i.h2.a[' href ']). Split ('/') [ -1].split ('. ') [0]


#获取html页面名称为id号


datas[' title ' = I.h2.get_text ()


#获取标题


url2 = i.h2.a[' href ']


#获取标题内容url地址


HTML2 = Urllib2.urlopen (URL2)


html_doc2 = Html2.read ()


soup2 = BeautifulSoup (HTML_DOC2)


datas[' content ' = Soup2.find (attrs={"name": "description"}) [' Content ']


#获取文章内容


stock_name = []


stock_id = []


for name in Re.findall (U "[u4e00-u9fa5]+", I.find (class_= "Stocks"). Get_text ()):


stock_name.append (name)


#获取影响股票名称, an array of ways to save the corresponding stock ID number, MONGO support array insertion


datas[' stock_name '] = stock_name


for ID in Re.findall ("d+", I.find (class_= "Stocks"). Get_text ():


stock_id.append (ID)


#获取影响股票id


datas[' stock_id '] = stock_id


datas[' update_time ' = Datetime.datetime.strptime (Re.search ("w+.*w+", I.find (class_= "FL Date"). Span.get_text (). Group (), '%y-%m-%d%h:%m ')-Datetime.timedelta (hours=8)


#获取发布时间, converted to MONGO time format


datas[' onlooker ' = Int (Re.search ("d+", I.find (class_= "Icons Ic-wg"). Get_text ()). Group ())


#获取围观数


Db.test.save (datas)


#插入数据库

Def get_data ():
    
    title = str (soup.h2.a[' href ']). Split ('/') [-1] . Split ('. ') [0]
    #获取html页面名称做更新判断
    with open (' Update.txt ', ' R ') as F:
         time = F.readline ()
    if  title = = Time:
         print ' Currently no update ', title
    else:
   & nbsp;    with open (' Update.txt ', ' W ') as F:
             F.write (title)
        Update ()
      &NBSP
While True:
    if __name__ = = ' __main__ ':
         url = ' http://www.ipython.me/qingbao/'
        html = Urllib2.urlopen (URL)
        html_doc = Html.read ()
        soup = BeautifulSoup (html_doc)
   & nbsp;    get_data ()
        time.sleep ()
 # Refresh once every 30 seconds

Related Article

E-Commerce Solutions

Leverage the same tools powering the Alibaba Ecosystem

Learn more >

Apsara Conference 2019

The Rise of Data Intelligence, September 25th - 27th, Hangzhou, China

Learn more >

Alibaba Cloud Free Trial

Learn and experience the power of Alibaba Cloud with a free trial worth $300-1200 USD

Learn more >

Contact Us

The content source of this page is from Internet, which doesn't represent Alibaba Cloud's opinion; products and services mentioned on that page don't have any relationship with Alibaba Cloud. If the content of the page makes you feel confusing, please write us an email, we will handle the problem within 5 days after receiving your email.

If you find any instances of plagiarism from the community, please send an email to: info-contact@alibabacloud.com and provide relevant evidence. A staff member will contact you within 5 working days.