Crawl Sina Weibo data into MongoDB to avoid repetitive insertion of microblogging data

Def getmydatalist ():
#id这个key
key = str (U ' ID '). decode (' Utf-8 ')
#存储旧数据的id列表
Old_ids = []
#存储新微博的列表
EXTR_WB = []
#从MongoDB上获取的数据
Old_datalist = Weibodata.find ()
For-old in old_datalist:
Old_ids.append (Old[key])

#从微博上抓取新数据
data = Client.statuses.home_timeline.get ()
New_datalist = data.statuses
For new in New_datalist:
If New[key] not in Old_ids:
Extr_wb.append (New)
Weibodata.insert (Extr_wb,save=true)

Ideas:

1. First get the old Weibo data from MongoDB and put all the Weibo data ID into a list old_ids

2. Crawl new data from Weibo and traverse to determine if the Weibo ID is old_ids, then add the microblog to a list that holds the newly inserted data EXTR_WB

3. Perform BULK INSERT INTO MongoDB