[Python] garbled: Python crawl script

Source: Internet
Author: User

Reference:

Http://www.zhxl.me/1409.html

solutions that are garbled when crawling Web pages using Python urllib2 a layman solves the process of garbled, crashing, seeking help, resolving and rising experience when crawling Web pages using URLLIB2. This kind of problem, after all, seems to be a very small hole, but it takes a lot of time, it is worth recording. The process is as follows: Target: Crawl http://sports.sina.com.cn/g/premierleague/index.shtmlCode:123456# Coding:u8import Urllib2url="http://sports.sina.com.cn/g/premierleague/index.shtml"Response=urllib2.urlopen (URL) HTML=response.read () print HTML output: w?83′? L/J.uv? [W5;:s 煝 {7l! Zp8'-y?=epus?; __zj::] K?EY?KV%ibuvy "*')? S.JT> "ttzk+!x*) ld2i,kuu?/kxjjkhi U0n2}jus?" >!pj^[?g'O^=nq?7n|57yy'\ulj=9t,g/t0?7'^o|v}>8=7?! Tp?? Gfs?zd~ ' muc%u2\ f? Vq?7~2~?le=}m}xwo}us'>?*zps:7o?~чb=HK!S?INQR} @TsY |, #b \d+#[email protected]? [((tgp,a$o/EXP) onga\ ' Z4eL7? Vn+? ER FT '&W?bvf{[email protected][Email protected] [Ib_? Cz'!4o1c,?hy b0w (? Z? V5-? X) 5{ekvxýn (Ppuck?? J (v3{z! loop+lp%wpl!\=! @XD8? jpt,w+ #we ~? {[Email protected]_y+ijp;^,= (h:nxh| ar]-| bkq<?+}.? Lt.) [Email protected]&6dógsr^=/[email protected] #m} uzh.v80_) $.1WHS*zqjñ|? {nipa±a#? l<SA%^YG2*\FXJHQH_FBK (C%cbkwaherb 8w6<? K @.k*[k|^_¹BV;,PU]24YBW? Cm3 '>5#FzFG-%? W0a{t?#[email protected]?*:* 6?t&[email protected]%c?h|? y$hhgv3s$ (Y) [email protected] (. tk?6k (E;OP1?:D 6wо?fo&zqZ3Z>0mc{?i#.tp?u-u-t38x wt2h!. >9; Tvkrj_$yabz?6.? I\YK:¬S#LHSXZB=inse/fuad4h3lnho0t^ "j*]yfrmy!? -#I (Yvaρ@[email protected]@ (GX) i-z$lnx,vg^~CE/Qiu &jz=АUDY__\FGA} ... First think of coding problems, referring to the "also talk about Python Chinese encoding processing" a article, I feel basic understanding of how it happened, supposedly1isinstance (HTML, str)==true and the encoding of the page is determined to be GBK, then1Html.decode ('GBK'). Encode ('Utf-8') can decode the machine code in GBK and then re--8code, you can get the correct text. But you get the hint: unicodedecodeerror: ' GBK ' codec can ' t decode bytesinchPosition1-2: illegal multibyte sequence after V2ex help, as well as repeatedly toss a bit of what was found to be gzip garbled, so try to extract by Zlib12Import zlibhtml=zlib.decompress (HTML) but get the following error Zlib.error:Error-3  whiledecompressing data:incorrect Header check helpless, had to use gzip library and Stringio library detour to solve123import gzip, stringiohtml= gzip. Gzipfile (Fileobj=stringio.stringio (HTML), mode="R") HTML= Html.read (). Decode ('GBK'). Encode ('utf-8 ')Finally got the right content and the right coding ~ ^^The problem is solved here, but for not directly use concise zlib library Express very unwilling, after all, according to the Python document Gzip library is also called zlib to extract, why not directly with zlib? Kung Fu is not a conscientious, finally found the answer on the StackOverflow. So the final code is as follows:12345678Request=Urllib2. Request (URL) request.add_header ('accept-encoding','gzip') Opener=Urllib2.build_opener () response= Opener.open (request) HTML =response.read () gzipped= Response.headers.Get('content-encoding')ifgzipped:html= zlib.decompress (HTML, -+zlib. max_wbits) print HTML code in the request header by default to accept Gzip, the server will first return to the Gzip page, which greatly reduce the size of the data flow, the vast majority of servers are supported gzip. The response header is also added to the unexpected case,-Encoding "Data stream will not be decompressed. This seems to be completed, but there are still a lot of accidents, beyond the scope of this article, this is not involved. PostScript, later learned that this is a very common pit, out of the consideration of the prevention of crawling, some sites have taken a variety of measures. For example, if the Accept is not specified-encoding's request will also return gzip content, verifying the user-agent and Referer of the request Header and even cookies. For crawling interested can continue to read the "crawl Web pages with Python considerations," Although the Web crawl is a very mature area, but the layman faces a lot of unknown challenges, only to read more and more accumulation just good.

[python] garbled: Python crawl script

Contact Us

The content source of this page is from Internet, which doesn't represent Alibaba Cloud's opinion; products and services mentioned on that page don't have any relationship with Alibaba Cloud. If the content of the page makes you feel confusing, please write us an email, we will handle the problem within 5 days after receiving your email.

If you find any instances of plagiarism from the community, please send an email to: info-contact@alibabacloud.com and provide relevant evidence. A staff member will contact you within 5 working days.

A Free Trial That Lets You Build Big!

Start building with 50+ products and up to 12 months usage for Elastic Compute Service

  • Sales Support

    1 on 1 presale consultation

  • After-Sales Support

    24/7 Technical Support 6 Free Tickets per Quarter Faster Response

  • Alibaba Cloud offers highly flexible support services tailored to meet your exact needs.