python爬蟲---->github上python的項目

標籤：toc   beautiful   imp   基礎功   連結   search   connect   tin   .exe   

　　這裡面通過爬蟲github上的一些start比較高的python項目來學習一下BeautifulSoup和pymysql的使用。我一直以為山是水的故事，雲是風的故事，你是我的故事，可是卻不知道，我是不是你的故事。

 

github的python爬蟲

爬蟲的需求：爬取github上有關python的優質項目，以下是測試案例，並沒有爬取很多資料。

一、實現基礎功能的爬蟲版本

這個案例可以學習到關於pymysql的批量插入、使用BeautifulSoup解析html資料以及requests庫的get請求資料的知識。至於pymysql的一些使用，可以參考部落格：python架構---->pymysql的使用

import requestsimport pymysql.cursorsfrom bs4 import BeautifulSoupdef get_effect_data(data):    results = list()    soup = BeautifulSoup(data, ‘html.parser‘)    projects = soup.find_all(‘div‘, class_=‘repo-list-item‘)    for project in projects:        writer_project = project.find(‘a‘, attrs={‘class‘: ‘v-align-middle‘})[‘href‘].strip()        project_language = project.find(‘div‘, attrs={‘class‘: ‘d-table-cell col-2 text-gray pt-2‘}).get_text().strip()        project_starts = project.find(‘a‘, attrs={‘class‘: ‘muted-link‘}).get_text().strip()        update_desc = project.find(‘p‘, attrs={‘class‘: ‘f6 text-gray mb-0 mt-2‘}).get_text().strip()        result = (writer_project.split(‘/‘)[1], writer_project.split(‘/‘)[2], project_language, project_starts, update_desc)        results.append(result)    return resultsdef get_response_data(page):    request_url = ‘https://github.com/search‘    params = {‘o‘: ‘desc‘, ‘q‘: ‘python‘, ‘s‘: ‘stars‘, ‘type‘: ‘Repositories‘, ‘p‘: page}    resp = requests.get(request_url, params)    return resp.textdef insert_datas(data):    connection = pymysql.connect(host=‘localhost‘,                                 user=‘root‘,                                 password=‘root‘,                                 db=‘test‘,                                 charset=‘utf8mb4‘,                                 cursorclass=pymysql.cursors.DictCursor)    try:        with connection.cursor() as cursor:            sql = ‘insert into project_info(project_writer, project_name, project_language, project_starts, update_desc) VALUES (%s, %s, %s, %s, %s)‘            cursor.executemany(sql, data)            connection.commit()    except:        connection.close()if __name__ == ‘__main__‘:    total_page = 2 # 爬蟲資料的總頁數    datas = list()    for page in range(total_page):        res_data = get_response_data(page + 1)        data = get_effect_data(res_data)        datas += data    insert_datas(datas)

運行完之後，可以在資料庫中看到如下的資料：

11	tensorflow	tensorflow	C++	78.7k	Updated Nov 22, 2017
12	robbyrussell	oh-my-zsh	Shell	62.2k	Updated Nov 21, 2017
13	vinta	awesome-python	Python	41.4k	Updated Nov 20, 2017
14	jakubroztocil	httpie	Python	32.7k	Updated Nov 18, 2017
15	nvbn	thefuck	Python	32.2k	Updated Nov 17, 2017
16	pallets	flask	Python	31.1k	Updated Nov 15, 2017
17	django	django	Python	29.8k	Updated Nov 22, 2017
18	requests	requests	Python	28.7k	Updated Nov 21, 2017
19	blueimp	jQuery-File-Upload	JavaScript	27.9k	Updated Nov 20, 2017
20	ansible	ansible	Python	26.8k	Updated Nov 22, 2017
21	justjavac	free-programming-books-zh_CN	JavaScript	24.7k	Updated Nov 16, 2017
22	scrapy	scrapy	Python	24k	Updated Nov 22, 2017
23	scikit-learn	scikit-learn	Python	23.1k	Updated Nov 22, 2017
24	fchollet	keras	Python	22k	Updated Nov 21, 2017
25	donnemartin	system-design-primer	Python	21k	Updated Nov 20, 2017
26	certbot	certbot	Python	20.1k	Updated Nov 20, 2017
27	aymericdamien	TensorFlow-Examples	Jupyter Notebook	18.1k	Updated Nov 8, 2017
28	tornadoweb	tornado	Python	14.6k	Updated Nov 17, 2017
29	python	cpython	Python	14.4k	Updated Nov 22, 2017
30	reddit	reddit	Python	14.2k	Updated Oct 17, 2017

 

友情連結

 

python爬蟲---->github上python的項目