About Python document read UTF-8 encoded file issues

Recently received a small project, read each line of the target file URL, and request the URL, to get the desired data.

#-*-coding:utf-8-*-classIpurlmanager (object):def __init__(self): Self.newipurls=set ()#Self.oldipurls = set ()    defIs_has_ipurl (self):returnLen (self.newipurls)! =0defGet_ipurl (self):ifLen (self.newipurls)! =0:new_ipurl=Self.newipurls.pop ()#Self.oldipurls.add (New_ipurl)            returnNew_ipurlElse:            returnNonedefDownload_ipurl (self,destpath):Try: F= Open (DestPath,'R') Iter_f=iter (f) Lines=0 forIpurlinchIter_f:lines= lines + 1Self.newipurls.add ((Ipurl.rstrip ('\ r \ n'))            #How many line IP URLs are read by the log record            #Print Lines        finally:            iff:f.close ()

At first glance code writes no problem, each URL is added into the newipurls set set. However, during the request process, the following error occurs after Requests.get:

Raise Invalidschema ("No connection adapters were found for '%s '"% URL)

It was later found that each time the first line of the URL request failed. It then prints the URL of the print request. No abnormalities were found. Then look at the root, OK, print newipurls set set.

Sure enough, the problem is here.

Strange, why is this URL preceded by the default \XEF\XBB\XBF these characters?

Read some information on the Internet, originally in the Python file object ReadLine and ReadLines program, for some UTF-8 encoded files, the beginning will be added to the BOM to indicate the encoding method.

What is a BOM?

The so-called BOM, full name is the byte order Mark, it is a Unicode character, usually appears at the beginning of the text, used to identify the byte order (Big/little Endian), In addition, you can identify the encoding (UTF-8/16/32).

In fact, if you have UltraEdit tool can be found in the Save as a file, you can save as UTF-8 and UTF-8 no BOM files.

If the file has a UTF-8 format, the beginning of the file will be incremented by three bytes \xef\xbb\xbf.

How to check if the file is UTF-8 with BOM?

ImportCodecsdefDownload_ipurl (self,destpath):Try: F= Open (DestPath,'R') Iter_f=iter (f) Lines=0 forIpurlinchIter_f:lines= lines + 1ifIpurl[0:3] = =codecs. BOM_UTF8:self.newipurls.add ((Ipurl.rstrip ('\ r \ n'). Lstrip ('\XEF\XBB\XBF'))        #Print Self.newipurls        #How many line IP URLs are read by the log record        #Print Lines    finally:        iff:f.close ()

Refer to the Codecs module to determine if the first three bytes are Bom_utf8. If yes, the \XEF\XBB\XBF byte is rejected.

In fact, we can eliminate the BOM byte by other means, I write a bit rough.

About Python document read UTF-8 encoded file issues