貓眼電影爬取(三)：requests+pyquery，並將資料存放區到mysql資料庫

標籤：from   size   str   參數   exe   apple   sql資料庫   pass   text   

還是以貓眼電影為例，這次用pyquery庫進行爬取

1.簡單demo，看看如何使用pyquery提取資訊，並將提取到的資料進行組合

# coding: utf-8# author: hmkimport requestsfrom pyquery import PyQuery as pqurl = ‘http://maoyan.com/board/4‘header = {"Accept": "text/html,application/xhtml+xml,application/xml;q=0.9,image/webp,*/*;q=0.8",              "Accept-Encoding": "gzip, deflate, sdch",              "Accept-Language": "zh-CN,zh;q=0.8",              "Cache-Control": "max-age=0",              "Connection": "keep-alive",              "Host": "maoyan.com",              "Referer": "http://maoyan.com/board",              "Upgrade-Insecure-Requests": "1",              "User-Agent": "Mozilla/5.0 (Windows NT 6.1; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/49.0.2623.75 Safari/537.36"}r = requests.get(url, headers=header)r.encoding = r.apparent_encodinghtml = r.textprint(type(html))doc = pq(html)# print((doc(‘dd‘).find(‘.board-index‘)))# print(doc(‘.name‘).text())# print(doc(‘.releasetime‘).text())# print(doc(‘dd‘).find(‘.integer‘).text()+doc(‘.fraction‘).text())list = []for t in doc(‘dd‘):   index = pq(t).find(‘.board-index‘).text()   print(index)   movie = pq(t).find(‘.name‘).text()   print(movie)   time = pq(t).find(‘.releasetime‘).text()   print(time)   score = pq(t).find(‘.integer‘).text() + pq(t).find(‘.fraction‘).text()   print(score)   list.append([index, movie, time, score])print(list)

 

2.正式代碼

# coding: utf-8# author: hmkimport requestsfrom pyquery import PyQuery as pqimport pymysql.cursorsdef get_html(url, header):    try:         r = requests.get(url=url, headers=header)         r.encoding = r.apparent_encoding         return r.text    except:        return Nonedef get_data(html, list_data):    doc = pq(html)    for t in doc(‘dd‘):        index = pq(t).find(‘.board-index‘).text()        print(index)        movie = pq(t).find(‘.name‘).text()        print(movie)        time = pq(t).find(‘.releasetime‘).text()        print(time)        score = pq(t).find(‘.integer‘).text() + pq(t).find(‘.fraction‘).text()        print(score)        list_data.append([index, movie, time, score])def write_sql(data):    conn = pymysql.connect(host=‘localhost‘,                           user=‘root‘,                           password=‘123456‘,                           db=‘test‘,                           charset=‘utf8‘)    cur = conn.cursor()    for i in data:        """這裡的data參數是指正則匹配並處理後的列表資料(是一個大列表，包含所有電影資訊，每個電影資訊都存在各自的一個列表中；        對大列表進行迭代，提取每組電影資訊，這樣提取到的每組電影資訊都是一個小列表，然後就可以把每組電影資訊寫入資料庫了)"""        movie = i  # 每組電影資訊，這裡可以看做是準備插入資料庫的每組電影資料        sql = "insert into maoyan_movie(ranking,movie,release_time,score) values(%s, %s, %s, %s)"  # sql插入語句        try:            cur.execute(sql, movie)  # 執行sql語句，movie即是指要插入資料庫的資料            conn.commit()  # 插入完成後，不要忘記提交操作            print(‘匯入成功‘)        except:            print(‘匯入失敗‘)    cur.close()  # 關閉遊標    conn.close()  # 關閉串連def main():    start_url = ‘http://maoyan.com/board/4‘    depth = 10  # 爬取深度(翻頁)    header = {"Accept": "text/html,application/xhtml+xml,application/xml;q=0.9,image/webp,*/*;q=0.8",              "Accept-Encoding": "gzip, deflate, sdch",              "Accept-Language": "zh-CN,zh;q=0.8",              "Cache-Control": "max-age=0",              "Connection": "keep-alive",              "Host": "maoyan.com",              "Referer": "http://maoyan.com/board",              "Upgrade-Insecure-Requests": "1",              "User-Agent": "Mozilla/5.0 (Windows NT 6.1; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/49.0.2623.75 Safari/537.36"}    for i in range(depth):        url = start_url + ‘?offset=‘ + str(10 * i)        html = get_html(url, header)        list_data = []        get_data(html, list_data)        write_sql(list_data)        # print(list_data)if __name__ == "__main__":    main()

其實就這個例子來說，使用pyquery來提取資訊是最簡單省事的了，直接使用css選取器就可以把想要的資料拿到

貓眼電影爬取(三)：requests+pyquery，並將資料存放區到mysql資料庫