python實現多線程抓取知乎使用者_python

需要用到的包：

beautifulsoup4
html5lib
image
requests
redis
PyMySQL

pip安裝所有依賴包：

pip install \Image \requests \beautifulsoup4 \html5lib \redis \PyMySQL

運行環境需要支援中文

測試回合環境python3.5，不保證其他運行環境能完美運行

需要安裝mysql和redis

配置 config.ini 檔案，設定好mysql和redis，並且填寫你的知乎帳號

向資料庫匯入 init.sql

Run

開始抓取資料: python get_user.py
查看抓取數量: python check_redis.py

效果

總體思路

1.首先是類比登陸知乎，利用儲存登陸的cookie資訊
2.抓取知乎頁面的html代碼，留待下一步繼續進行分析提取資訊
3.分析提取頁面中使用者的個人化url，放入redis（這裡特別說明一下redis的思路用法，將提取到的使用者的個人化url放入redis的一個名為already_get_user的hash table，表示已抓取的使用者，對於已抓取過的使用者判斷是否存在於already_get_user以去除重複抓取，同時將個人化url放入user_queue的隊列中，需要抓取新使用者時pop隊列擷取新的使用者）
4.擷取使用者的關注列表和粉絲列表，繼續插入到redis
5.從redis的user_queue隊列中擷取新使用者繼續重複步驟3

類比登陸知乎

首先是登陸，登陸功能作為一個包封裝了在login裡面，方便整合調用

header部分，這裡Connection最好設為close，不然可能會碰到max retireve exceed的錯誤
原因在於普通的串連是keep-alive的但是卻又沒有關閉

# http請求的headerheaders = {  "User-Agent": "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/53.0.2785.143 Safari/537.36",  "Accept": "text/html,application/xhtml+xml,application/xml;q=0.9,image/webp,*/*;q=0.8",  "Host": "www.zhihu.com",  "Referer": "https://www.zhihu.com/",  "Origin": "https://www.zhihu.com/",  "Upgrade-Insecure-Requests": "1",  "Content-Type": "application/x-www-form-urlencoded; charset=UTF-8",  "Pragma": "no-cache",  "Accept-Encoding": "gzip, deflate, br",  'Connection': 'close'}# 驗證是否登陸def check_login(self):  check_url = 'https://www.zhihu.com/settings/profile'  try:    login_check = self.__session.get(check_url, headers=self.headers, timeout=35)  except Exception as err:    print(traceback.print_exc())    print(err)    print("驗證登陸失敗，請檢查網路")    sys.exit()  print("驗證登陸的http status code為：" + str(login_check.status_code))  if int(login_check.status_code) == 200:    return True  else:    return False

進入首頁查看http狀態代碼來驗證是否登陸，200為已經登陸，一般304就是被重新導向所以就是沒有登陸

# 擷取驗證碼def get_captcha(self):  t = str(time.time() * 1000)  captcha_url = 'http://www.zhihu.com/captcha.gif?r=' + t + "&type=login"  r = self.__session.get(captcha_url, headers=self.headers, timeout=35)  with open('captcha.jpg', 'wb') as f:    f.write(r.content)    f.close()    # 用pillow 的 Image 顯示驗證碼    # 如果沒有安裝 pillow 到原始碼所在的目錄去找到驗證碼然後手動輸入  '''try:    im = Image.open('captcha.jpg')    im.show()    im.close()  except:'''  print(u'請到 %s 目錄找到captcha.jpg 手動輸入' % os.path.abspath('captcha.jpg'))  captcha = input("請輸入驗證碼\n>")  return captcha

擷取驗證碼的方法。當登入次數太多有可能會要求輸入驗證碼，這裡實現這個功能

# 擷取xsrfdef get_xsrf(self):  index_url = 'http://www.zhihu.com'  # 擷取登入時需要用到的_xsrf  try:    index_page = self.__session.get(index_url, headers=self.headers, timeout=35)  except:    print('擷取知乎頁面失敗，請檢查網路連接')    sys.exit()  html = index_page.text  # 這裡的_xsrf 返回的是一個list  BS = BeautifulSoup(html, 'html.parser')  xsrf_input = BS.find(attrs={'name': '_xsrf'})  pattern = r'value=\"(.*?)\"'  print(xsrf_input)  self.__xsrf = re.findall(pattern, str(xsrf_input))  return self.__xsrf[0]

擷取xsrf，為什麼要擷取xsrf呢，因為xsrf是一種防止跨站攻擊的手段，具體介紹可以看這裡csrf
在擷取到xsrf之後把xsrf存入cookie當中，並且在調用api的時候帶上xsrf作為頭部，不然的話知乎會返回403

# 進行類比登陸def do_login(self):  try:    # 類比登陸    if self.check_login():      print('您已經登入')      return    else:      if self.config.get("zhihu_account", "username") and self.config.get("zhihu_account", "password"):        self.username = self.config.get("zhihu_account", "username")        self.password = self.config.get("zhihu_account", "password")      else:        self.username = input('請輸入你的使用者名稱\n> ')        self.password = input("請輸入你的密碼\n> ")  except Exception as err:    print(traceback.print_exc())    print(err)    sys.exit()  if re.match(r"^1\d{10}$", self.username):    print("手機登陸\n")    post_url = 'http://www.zhihu.com/login/phone_num'    postdata = {      '_xsrf': self.get_xsrf(),      'password': self.password,      'remember_me': 'true',      'phone_num': self.username,    }  else:    print("郵箱登陸\n")    post_url = 'http://www.zhihu.com/login/email'    postdata = {      '_xsrf': self.get_xsrf(),      'password': self.password,      'remember_me': 'true',      'email': self.username,    }  try:    login_page = self.__session.post(post_url, postdata, headers=self.headers, timeout=35)    login_text = json.loads(login_page.text.encode('latin-1').decode('unicode-escape'))    print(postdata)    print(login_text)    # 需要輸入驗證碼 r = 0為登陸成功碼    if login_text['r'] == 1:      sys.exit()  except:    postdata['captcha'] = self.get_captcha()    login_page = self.__session.post(post_url, postdata, headers=self.headers, timeout=35)    print(json.loads(login_page.text.encode('latin-1').decode('unicode-escape')))  # 儲存登陸cookie  self.__session.cookies.save()

這個就是核心的登陸功能啦，非常關鍵的就是用到了requests庫，非常方便的儲存到session
我們這裡全域都是用單例模式，統一使用同一個requests.session對象進行訪問功能，保持登入狀態的一致性

最後主要調用登陸的代碼為

# 建立login對象lo = login.login.Login(self.session)# 類比登陸if lo.check_login():  print('您已經登入')else:  if self.config.get("zhihu_account", "username") and self.config.get("zhihu_account", "username"):    username = self.config.get("zhihu_account", "username")    password = self.config.get("zhihu_account", "password")  else:    username = input('請輸入你的使用者名稱\n> ')    password = input("請輸入你的密碼\n> ")  lo.do_login(username, password)

知乎類比登陸到此就完成啦

知乎使用者抓取

def __init__(self, threadID=1, name=''):  # 多線程  print("線程" + str(threadID) + "初始化")  threading.Thread.__init__(self)  self.threadID = threadID  self.name = name  try:    print("線程" + str(threadID) + "初始化成功")  except Exception as err:    print(err)    print("線程" + str(threadID) + "開啟失敗")  self.threadLock = threading.Lock()  # 擷取配置  self.config = configparser.ConfigParser()  self.config.read("config.ini")  # 初始化session  requests.adapters.DEFAULT_RETRIES = 5  self.session = requests.Session()  self.session.cookies = cookielib.LWPCookieJar(filename='cookie')  self.session.keep_alive = False  try:    self.session.cookies.load(ignore_discard=True)  except:    print('Cookie 未能載入')  finally:    pass  # 建立login對象  lo = Login(self.session)  lo.do_login()  # 初始化redis串連  try:    redis_host = self.config.get("redis", "host")    redis_port = self.config.get("redis", "port")    self.redis_con = redis.Redis(host=redis_host, port=redis_port, db=0)    # 重新整理redis庫    # self.redis_con.flushdb()  except:    print("請安裝redis或檢查redis串連配置")    sys.exit()  # 初始化資料庫連接  try:    db_host = self.config.get("db", "host")    db_port = int(self.config.get("db", "port"))    db_user = self.config.get("db", "user")    db_pass = self.config.get("db", "password")    db_db = self.config.get("db", "db")    db_charset = self.config.get("db", "charset")    self.db = pymysql.connect(host=db_host, port=db_port, user=db_user, passwd=db_pass, db=db_db,                 charset=db_charset)    self.db_cursor = self.db.cursor()  except:    print("請檢查資料庫配置")    sys.exit()  # 初始化系統設定  self.max_queue_len = int(self.config.get("sys", "max_queue_len"))

這個是get_user.py的建構函式，主要功能就是初始化mysql串連、redis串連、驗證登陸、產生全域的session對象、匯入系統配置、開啟多線程。

# 擷取首頁htmldef get_index_page(self):  index_url = 'https://www.zhihu.com/'  try:    index_html = self.session.get(index_url, headers=self.headers, timeout=35)  except Exception as err:    # 出現異常重試    print("擷取頁面失敗，正在重試......")    print(err)    traceback.print_exc()    return None  finally:    pass  return index_html.text# 擷取單個使用者詳情頁面def get_user_page(self, name_url):  user_page_url = 'https://www.zhihu.com' + str(name_url) + '/about'  try:    index_html = self.session.get(user_page_url, headers=self.headers, timeout=35)  except Exception as err:    # 出現異常重試    print("失敗name_url：" + str(name_url) + "擷取頁面失敗，放棄該使用者")    print(err)    traceback.print_exc()    return None  finally:    pass  return index_html.text# 擷取粉絲頁面def get_follower_page(self, name_url):  user_page_url = 'https://www.zhihu.com' + str(name_url) + '/followers'  try:    index_html = self.session.get(user_page_url, headers=self.headers, timeout=35)  except Exception as err:    # 出現異常重試    print("失敗name_url：" + str(name_url) + "擷取頁面失敗，放棄該使用者")    print(err)    traceback.print_exc()    return None  finally:    pass  return index_html.textdef get_following_page(self, name_url):  user_page_url = 'https://www.zhihu.com' + str(name_url) + '/followers'  try:    index_html = self.session.get(user_page_url, headers=self.headers, timeout=35)  except Exception as err:    # 出現異常重試    print("失敗name_url：" + str(name_url) + "擷取頁面失敗，放棄該使用者")    print(err)    traceback.print_exc()    return None  finally:    pass  return index_html.text# 擷取首頁上的使用者列表，存入redisdef get_index_page_user(self):  index_html = self.get_index_page()  if not index_html:    return  BS = BeautifulSoup(index_html, "html.parser")  self.get_xsrf(index_html)  user_a = BS.find_all("a", class_="author-link") # 擷取使用者的a標籤  for a in user_a:    if a:      self.add_wait_user(a.get('href'))    else:      continue

這一部分的代碼就是用於抓取各個頁面的html代碼

# 加入帶抓取使用者隊列，先用redis判斷是否已被抓取過def add_wait_user(self, name_url):  # 判斷是否已抓取  self.threadLock.acquire()  if not self.redis_con.hexists('already_get_user', name_url):    self.counter += 1    print(name_url + " 排入佇列")    self.redis_con.hset('already_get_user', name_url, 1)    self.redis_con.lpush('user_queue', name_url)    print("添加使用者 " + name_url + "到隊列")  self.threadLock.release()# 擷取頁面出錯移出redisdef del_already_user(self, name_url):  self.threadLock.acquire()  if not self.redis_con.hexists('already_get_user', name_url):    self.counter -= 1    self.redis_con.hdel('already_get_user', name_url)  self.threadLock.release()

使用者加入redis的操作，在資料庫插入出錯時我們調用del_already_user刪除插入出錯的使用者

# 分析粉絲頁面擷取使用者的所有粉絲使用者# @param follower_page get_follower_page()中擷取到的頁面，這裡擷取使用者hash_id請求粉絲介面擷取粉絲資訊def get_all_follower(self, name_url):  follower_page = self.get_follower_page(name_url)  # 判斷是否擷取到頁面  if not follower_page:    return  BS = BeautifulSoup(follower_page, 'html.parser')  # 擷取粉絲數量  follower_num = int(BS.find('span', text='粉絲').find_parent().find('strong').get_text())  # 擷取使用者的hash_id  hash_id = \    json.loads(BS.select("#zh-profile-follows-list")[0].select(".zh-general-list")[0].get('data-init'))[      'params'][      'hash_id']  # 擷取粉絲列表  self.get_xsrf(follower_page) # 擷取xsrf  post_url = 'https://www.zhihu.com/node/ProfileFollowersListV2'  # 開始擷取所有的粉絲 math.ceil(follower_num/20)*20  for i in range(0, math.ceil(follower_num / 20) * 20, 20):    post_data = {      'method': 'next',      'params': json.dumps({"offset": i, "order_by": "created", "hash_id": hash_id})    }    try:      j = self.session.post(post_url, params=post_data, headers=self.headers, timeout=35).text.encode(        'latin-1').decode(        'unicode-escape')      pattern = re.compile(r"class=\"zm-item-link-avatar\"[^\"]*\"([^\"]*)", re.DOTALL)      j = pattern.findall(j)      for user in j:        user = user.replace('\\', '')        self.add_wait_user(user) # 儲存到redis    except Exception as err:      print("擷取正在關註失敗")      print(err)      traceback.print_exc()      pass# 擷取正在關注列表def get_all_following(self, name_url):  following_page = self.get_following_page(name_url)  # 判斷是否擷取到頁面  if not following_page:    return  BS = BeautifulSoup(following_page, 'html.parser')  # 擷取粉絲數量  following_num = int(BS.find('span', text='關注了').find_parent().find('strong').get_text())  # 擷取使用者的hash_id  hash_id = \    json.loads(BS.select("#zh-profile-follows-list")[0].select(".zh-general-list")[0].get('data-init'))[      'params'][      'hash_id']  # 擷取粉絲列表  self.get_xsrf(following_page) # 擷取xsrf  post_url = 'https://www.zhihu.com/node/ProfileFolloweesListV2'  # 開始擷取所有的粉絲 math.ceil(follower_num/20)*20  for i in range(0, math.ceil(following_num / 20) * 20, 20):    post_data = {      'method': 'next',      'params': json.dumps({"offset": i, "order_by": "created", "hash_id": hash_id})    }    try:      j = self.session.post(post_url, params=post_data, headers=self.headers, timeout=35).text.encode(        'latin-1').decode(        'unicode-escape')      pattern = re.compile(r"class=\"zm-item-link-avatar\"[^\"]*\"([^\"]*)", re.DOTALL)      j = pattern.findall(j)      for user in j:        user = user.replace('\\', '')        self.add_wait_user(user) # 儲存到redis    except Exception as err:      print("擷取正在關註失敗")      print(err)      traceback.print_exc()      pass

調用知乎的API，擷取所有的關注使用者列表和粉絲使用者列表，遞迴擷取使用者
這裡需要注意的是頭部要記得帶上xsrf不然會拋出403

# 分析about頁面，擷取使用者詳細資料def get_user_info(self, name_url):  about_page = self.get_user_page(name_url)  # 判斷是否擷取到頁面  if not about_page:    print("擷取使用者詳情頁面失敗，跳過，name_url：" + name_url)    return  self.get_xsrf(about_page)  BS = BeautifulSoup(about_page, 'html.parser')  # 擷取頁面的具體資料  try:    nickname = BS.find("a", class_="name").get_text() if BS.find("a", class_="name") else ''    user_type = name_url[1:name_url.index('/', 1)]    self_domain = name_url[name_url.index('/', 1) + 1:]    gender = 2 if BS.find("i", class_="icon icon-profile-female") else (1 if BS.find("i", class_="icon icon-profile-male") else 3)    follower_num = int(BS.find('span', text='粉絲').find_parent().find('strong').get_text())    following_num = int(BS.find('span', text='關注了').find_parent().find('strong').get_text())    agree_num = int(re.findall(r'<strong>(.*)</strong>.*贊同', about_page)[0])    appreciate_num = int(re.findall(r'<strong>(.*)</strong>.*感謝', about_page)[0])    star_num = int(re.findall(r'<strong>(.*)</strong>.*收藏', about_page)[0])    share_num = int(re.findall(r'<strong>(.*)</strong>.*分享', about_page)[0])    browse_num = int(BS.find_all("span", class_="zg-gray-normal")[2].find("strong").get_text())    trade = BS.find("span", class_="business item").get('title') if BS.find("span",                                       class_="business item") else ''    company = BS.find("span", class_="employment item").get('title') if BS.find("span",                                         class_="employment item") else ''    school = BS.find("span", class_="education item").get('title') if BS.find("span",                                        class_="education item") else ''    major = BS.find("span", class_="education-extra item").get('title') if BS.find("span",                                           class_="education-extra item") else ''    job = BS.find("span", class_="position item").get_text() if BS.find("span",                                      class_="position item") else ''    location = BS.find("span", class_="location item").get('title') if BS.find("span",                                         class_="location item") else ''    description = BS.find("div", class_="bio ellipsis").get('title') if BS.find("div",                                          class_="bio ellipsis") else ''    ask_num = int(BS.find_all("a", class_='item')[1].find("span").get_text()) if \      BS.find_all("a", class_='item')[        1] else int(0)    answer_num = int(BS.find_all("a", class_='item')[2].find("span").get_text()) if \      BS.find_all("a", class_='item')[        2] else int(0)    article_num = int(BS.find_all("a", class_='item')[3].find("span").get_text()) if \      BS.find_all("a", class_='item')[3] else int(0)    collect_num = int(BS.find_all("a", class_='item')[4].find("span").get_text()) if \      BS.find_all("a", class_='item')[4] else int(0)    public_edit_num = int(BS.find_all("a", class_='item')[5].find("span").get_text()) if \      BS.find_all("a", class_='item')[5] else int(0)    replace_data = \      (pymysql.escape_string(name_url), nickname, self_domain, user_type,       gender, follower_num, following_num, agree_num, appreciate_num, star_num, share_num, browse_num,       trade, company, school, major, job, location, pymysql.escape_string(description),       ask_num, answer_num, article_num, collect_num, public_edit_num)    replace_sql = '''REPLACE INTO           user(url,nickname,self_domain,user_type,           gender, follower,following,agree_num,appreciate_num,star_num,share_num,browse_num,           trade,company,school,major,job,location,description,           ask_num,answer_num,article_num,collect_num,public_edit_num)           VALUES(%s,%s,%s,%s,           %s,%s,%s,%s,%s,%s,%s,%s,           %s,%s,%s,%s,%s,%s,%s,           %s,%s,%s,%s,%s)'''    try:      print("擷取到資料：")      print(replace_data)      self.db_cursor.execute(replace_sql, replace_data)      self.db.commit()    except Exception as err:      print("插入資料庫出錯")      print("擷取到資料：")      print(replace_data)      print("插入語句：" + self.db_cursor._last_executed)      self.db.rollback()      print(err)      traceback.print_exc()  except Exception as err:    print("擷取資料出錯，跳過使用者")    self.redis_con.hdel("already_get_user", name_url)    self.del_already_user(name_url)    print(err)    traceback.print_exc()    pass

最後，到使用者的about頁面，分析頁面元素，利用正則或者beatifulsoup分析抓取頁面的資料
這裡我們SQL語句用REPLACE INTO而不用INSERT INTO，這樣可以很好的防止資料重複問題

# 開始抓取使用者，程式總入口def entrance(self):  while 1:    if int(self.redis_con.llen("user_queue")) < 1:      self.get_index_page_user()    else:      # 出隊列擷取使用者name_url redis取出的是byte，要decode成utf-8      name_url = str(self.redis_con.rpop("user_queue").decode('utf-8'))      print("正在處理name_url：" + name_url)      self.get_user_info(name_url)      if int(self.redis_con.llen("user_queue")) <= int(self.max_queue_len):        self.get_all_follower(name_url)        self.get_all_following(name_url)    self.session.cookies.save()def run(self):  print(self.name + " is running")  self.entrance()

最後，入口

if __name__ == '__main__':  login = GetUser(999, "登陸線程")  threads = []  for i in range(0, 4):    m = GetUser(i, "thread" + str(i))    threads.append(m)  for i in range(0, 4):    threads[i].start()  for i in range(0, 4):    threads[i].join()

這裡就是多線程的開啟，需要開啟多少個線程就把4換成多少就可以了

Docker

嫌麻煩的可以參考一下我用docker簡單的搭建一個基礎環境：

mysql和redis都是官方鏡像

docker run --name mysql -itd mysql:latestdocker run --name redis -itd mysql:latest

再利用docker-compose運行python鏡像，我的python的docker-compose.yml：

python: container_name: python build: . ports:  - "84:80" external_links:  - memcache:memcache  - mysql:mysql  - redis:redis volumes:  - /docker_containers/python/www:/var/www/html tty: true stdin_open: true extra_hosts:  - "python:192.168.102.140" environment:  PYTHONIOENCODING: utf-8

最後附上原始碼： GITHUB https://github.com/kong36088/ZhihuSpider

本站下載地址： http://xiazai.jb51.net/201612/yuanma/ZhihuSpider(jb51.net).zip