Python crawler scrapy Login know

Let's take a look at the basic notation for using scrapy to simulate logins:

Note: We often debug the code when the basic use of Chrome browser, but I just because of the use of Google Browser (it is always logged on without prompting me to use a verification code, misleading me that I do not need to login to verify code, in fact, the login must be verified code), here you can try several browsers, Be sure to find a browser debugging that prompts you to enter a verification code.

1, we log in, prompted us to enter a verification code, when the code pops up before there will be a request, we open the request, it is clear that the type is login, verification code is undoubtedly, even if the request because of the name, you should also know that this is the verification code request, or open this Code request URL, this.

　　

Verification code of the picture, miserable, this how the whole. Don't worry.

　　

2, the verification code prompts us to hit the written font, this ... , reptiles and anti-reptiles are endless torment. This is obviously the information of the picture above.

　　

3, wit me, found the verification code request parameters inside there are three parameters, R is a 13-digit number, type is login with, Lang very suspicious, change it, the CN to him to en.

　　

The

Code is as follows:

Import jsonimport scrapyimport timefrom PIL import imageclass zhihuloginspider (scrapy.  Spider): name = ' Zhihu_login ' allowed_domains = [' zhihu.com '] start_urls = [' https://www.zhihu.com/'] Header = {' user-agent ': ' mozilla/5.0 (Windows NT 10.0; Win64; x64) applewebkit/537.36 (khtml, "like Gecko) chrome/62.0.3202.94 safari/537.3 6 ',} def parse (self, Response): #主页爬取的具体内容 print (Response.text) def start_requests (self): ' "1, first construct and crawl login need to submit the verification code: return:" "t = str (int (time.time () *)) Captcha_url = ' htt ps://www.zhihu.com/captcha.gif?r={0}&type=login&lang=en '. Format (t) return [Scrapy.        Request (Url=captcha_url, Headers=self.header, Callback=self.parser_captcha)] def parser_captcha (self, Response):        "1, according to the verification code returned by the Start_requests method, save it to local 2, open the downloaded Verification Code 3, here is the need to manually input, here can access code platform:P Aram Response: : return: '' With open (' captcha.jpg ', ' WB ') as F:f.write (Response.body) f.close () Try:  im = Image.open (' captcha.jpg ') im.show () im.close () Except:pass captcha = Input ("Please enter your verification >") return scrapy. Formrequest (url= ' https://www.zhihu.com/#signin ', Headers=self.header, Callback=self.login, meta={' captcha ': CA Ptcha}) def login (self, response): XSRF = Response.xpath ("//input[@name = ' _xsrf ']/@value"). Extract_first ( If XSRF is None:return ' post_url = ' https://www.zhihu.com/login/phone_num ' Post_data = {"_XSRF": xsrf, "phone_num": ' Your account name ', ' Password ': ' Your account password ', ' captcha ': res ponse.meta[' Captcha '} return [Scrapy. Formrequest (Url=post_url, Formdata=post_data, Headers=self.header, Callback=self.check_login)] # Verify that the return is successful def Check_ Login (self, response): JS = json.loads (reSponse.text) Print (JS) if ' msg ' in JS and js[' msg '] = = ' Login succeeded ': for URL in Self.start_urls: Print (URL) yield scrapy. Request (Url=url, Headers=self.header, dont_filter=true) else:print ("Login failed, please check!!! ")

　　

Python crawler scrapy Login know