Scrapy Shell Debug Return 403 return to []

Source: Internet
Author: User

Question: Sometimes it is convenient to debug with a scrapy shell, but sometimes using the Scrapy shell returns 403, such as the following:

c:\users\fendo>scrapy shell https://book.douban.com/subject/26805083/   2017-04-17  15:18:53 [scrapy.utils.log] info: scrapy 1.3.3 started  (Bot: scrapybot)    2017-04-17 15:18:53 [scrapy.utils.log] info: overridden settings: {' Dupefilter_class ':  ' scrapy.dupefilters.BaseDupeFilter ',  ' logstats_interval ': 0}   2017-04-17 15:18:53 [scrapy.middleware] info: enabled extensions:   [' Scrapy.extensions.corestats.CoreStats ',     ' scrapy.extensions.telnet.TelnetConsole ']   2017-04-17 15:18:54 [scrapy.middleware] info: enabled downloader middlewares:    [' Scrapy.downloadermiddlewares.httpauth.HttpAuthMiddleware ',     ' Scrapy.downloadermiddlewares.downloadtimeout.DownloadTimeoutMiddleware ',     ' Scrapy.downloadermiddlewares.defaultheaders.DefaultHeadErsmiddleware ',     ' scrapy.downloadermiddlewares.useragent.UserAgentMiddleware ',     ' Scrapy.downloadermiddlewares.retry.RetryMiddleware ',     ' Scrapy.downloadermiddlewares.redirect.MetaRefreshMiddleware ',     ' Scrapy.downloadermiddlewares.httpcompression.HttpCompressionMiddleware ',     ' Scrapy.downloadermiddlewares.redirect.RedirectMiddleware ',     ' Scrapy.downloadermiddlewares.cookies.CookiesMiddleware ',     ' Scrapy.downloadermiddlewares.stats.DownloaderStats ']   2017-04-17 15:18:54 [ scrapy.middleware] info: enabled spider middlewares:   [' Scrapy.spidermiddlewares.httperror.HttpErrorMiddleware ',     ' Scrapy.spidermiddlewares.offsite.OffsiteMiddleware ',     ' Scrapy.spidermiddlewares.referer.RefererMiddleware ',     ' Scrapy.spidermiddlewares.urllength.UrlLengthMiddleware ',     ' scrapy.Spidermiddlewares.depth.DepthMiddleware ']   2017-04-17 15:18:54 [scrapy.middleware]  info: enabled item pipelines:   []   2017-04-17 15:18:54 [ scrapy.extensions.telnet] debug: telnet console listening on 127.0.0.1:6024    2017-04-17 15:18:54 [scrapy.core.engine] info: spider opened   2017-04-17 15:18:54 [scrapy.core.engine] debug: crawled  (403)  <GET https ://book.douban.com/subject/26805083/>  (referer: none)    2017-04-17 15:18:54 [ traitlets] debug: using default logger   2017-04-17 15:18:54 [traitlets ] debug: using default logger   [s] available scrapy objects:    [s]   scrapy     scrapy module  (contains  Scrapy. Request, scrapY.SELECTOR,&NBSP;ETC)    [s]   crawler    < scrapy.crawler.crawler object at 0x000001e696fbad68>   [S]   item        {}   [s]   request     <GET https://book.douban.com/subject/26805083/>   [s]   response    <403 https://book.douban.com/subject/26805083/>   [s]    settings   <scrapy.settings.settings object at 0x000001e6993c7b70>    [s]   spider     <defaultspider  ' default '  at  0x1e69964d1d0>   [s] useful shortcuts:   [S]   fetch (url[,  Redirect=true])  Fetch URL and update local objects  (by default,  redirects are followed)    [S]   fetch (req)                    fetch a scrapy. request and update local objects   [S]   shelp ()             Shell help  (PRINT&NBSP;THIS&NBSP;HELP)    [s ]   view (response)     View response in a browser    IN&NBSP;[1]:

Answer:


(1): The first method is to add-s user_agent= ' mozilla/5.0 ' on the command

(2): The second method is to modify the User-agent default value of Scrapy


Find Python: default_settings.py files under the installation directory, such as my F:\Software\Python36\Lib\site-packages\scrapy\settings\default_ settings.py


Put


[HTML] view plain copy user_agent = ' scrapy/%s (+http://scrapy.org) '% import_module (' scrapy '). __version__

To


[HTML]   View plain  copy user_agent =  ' mozilla/5.0  (windows nt  5.1; rv:5.0)  gecko/20100101 firefox/5.0 '     

Related Article

Contact Us

The content source of this page is from Internet, which doesn't represent Alibaba Cloud's opinion; products and services mentioned on that page don't have any relationship with Alibaba Cloud. If the content of the page makes you feel confusing, please write us an email, we will handle the problem within 5 days after receiving your email.

If you find any instances of plagiarism from the community, please send an email to: info-contact@alibabacloud.com and provide relevant evidence. A staff member will contact you within 5 working days.

A Free Trial That Lets You Build Big!

Start building with 50+ products and up to 12 months usage for Elastic Compute Service

  • Sales Support

    1 on 1 presale consultation

  • After-Sales Support

    24/7 Technical Support 6 Free Tickets per Quarter Faster Response

  • Alibaba Cloud offers highly flexible support services tailored to meet your exact needs.