Unicodeencodeerror: ' GBK ' codec can ' t encode character U ' \u200e ' in position 43:illegal Multib

Problem

When executing the code, prompt the above error, the source code is as follows:

1 #Download the novel ...2 defDownload_stoy (crawl_list,header):3 4     #Create a file stream to read each chapter into memory5With open ('e:\ tomb Test22.txt','W') as F:6          forEach_urlinchcrawl_list:7             #sometimes access to a webpage will always be unresponsive, the program will be stuck there, I let him 0.6 seconds after the automatic timeout and throw an exception8              whileTrue:9                 Try:TenRequest = Urllib.request.Request (Url=each_url, headers=header) OneWith Urllib.request.urlopen (Request, timeout=0.6) as response: Ahtml = Response.read (). Decode ('Utf-8') -                          Break -                 except: the                     #for crawled exceptions, let the program stop for 1.1 seconds, and then iterate over the link again, exit the loop when the access succeeds -Time.sleep (1.1) -  -             #Match article title +Title_req = Re.compile (r'') -             #matches the article content, the content has the line break, therefore makes the flags=re. S Re. s means matching across rows +             #content_req = re.compile (R ' <div class = "Content-body" > (. +) </div> ', re. S) AContent_req = Re.compile (r'<p> (. *?) </p>', Re. S) at             #"<div[^>]+>.+?<div> (. +?) </div></div> ", Re. I -             #content_req = re.compile (R ' <div[^>]+>.+?<div> (. +?) </div></div> ', Re. S) -             #Get title -title =Title_req.findall (HTML) [0] -             #Get content -Content_test =content_req.findall (HTML) in             Print('Crawl Chapters >'+title) -F.write (title +'\ n') to             #print (content_test) +              foreachinchcontent_test: -                 #sift through unwanted HTML elements theSTR1 = Each.replace ('&ldquo;',' ') *STR2 = Str1.replace ('&hellip;',' ') $STR3 = Str2.replace ('&rdquo;',' ')Panax NotoginsengF.write (STR3 +'\ n')

"Resolution Process"

1. Reconfirm its encoding format, which is indeed utf-8;

2. This problem is very strange is that itself called UTF-8 to decode, but the decoding error is prompted GBK, rather than UTF-8 related decoding error.

3. Another post was found, trying to add the Ignore attribute when decoding, but not resolved. The second explanation provided in the article is that intuition is not the reason. Continue looking for other posts.

https://www.crifan.com/unicodeencodeerror_gbk_codec_can_not_encode_character_in_position_illegal_multibyte_sequence/

4. Find another one,

Http://www.jb51.net/article/64816.htm

As prompted, add encoding= ' Utf-8 ' When the file is opened, that is,

    With open ('e:\ Tomb Raider test22.txt'w', encoding='  Utf-8') as F:

Problem solving.

Reference

"Summary" errors in the coding and decoding of common characters in Python 2.x and their solutions

Unicodeencodeerror: ' GBK ' codec can ' t encode character U ' \u200e ' in position 43:illegal Multib