Python small script for url decoding and Chinese parsing (pythonurldecoder)

This article mainly introduces the python code for url decoding and Chinese parsing. For more information, see The code is as follows:

#-*-Coding: utf8 -*-
#! Python
Print (repr ("test alarm, xxxx is a big pig ". decode ("UTF8 "). encode ("GBK ")). replace ("\ x", "% "))

Note that the first decode ("UTF8") must be the same as the file declaration encoding.

The first touch on this issue came from a Javascript puzzle game. The following is a prompt for a certain level:

The first few links are simple and simple ~~ This is just a simple string deformation .....

Followed by a long string starting with % 5Cu4e0b % 5Cu4e00% 5Cu5173% 5Cu7684.
This kind of thing was often seen in the browser's address bar before, that is, it never knows how to convert it into something that can be understood,
Google it online, combined with python url decoding and unicode decoding, the solution is as follows:

The code is as follows:

Import urllib failed = "% failed % 5Cu7684% 5Cu9875% 5Cu9762% failed % 5Cu662f % 5Cx20% 5Cx69% 5Cx32% 5Cx6a % 5Cx62% 5Cx6a % 5Cx33% 5Cx69 5Cx34% 5Cx62% 5Cx62% 5Cx35% 5Cx34% 5Cx62% 5Cx35% 5Cx32% 5Cx69% 5Cx62% 5Cx33% 5Cx2e % 5Cx68% 5Cx74% 5Cx6d"
Print urllib. unquote (escaped_str). decode ('unicode-escape ')

Recently, I have become interested in the Chinese vocabulary in gfwlist in firefox's autoproxy plug-in (students who have used proxies, you know). However, these URLs are all url encoded, such:

The code is as follows:

Import urllib
Import re
With open ("listfile", "r") as f:
For url_str in f:
Match = re. compile ("(% \ w {2}) {3,})"). findall (url_str)
# The Chinese character url encoding style is: Percent + 2 hexadecimal numbers, repeated three times

If match! = None:
# If the matching is successful, the extracted part is converted to Chinese
For trans in match:
Print urllib. unquote (trans [0]),

However, this script still has some disadvantages. some Chinese characters in the list file still cannot be decoded normally. for example, the following lines of test code

The code is as follows:

Import urllib
A = "http://zh.wikipedia.org/wiki/%BD%F0%B6"
B = "http://zh.wikipedia.org/wiki/%E9%97%A8"
De = urllib. unquote
Print de (a), de (B)

The output result is that the former can be decoded correctly, but the latter cannot. I personally think the cause may be related to the big5 encoding. if anyone knows the solution, please let me know ~

The following is a supplement:

De (a). decode ("gbk", "ignore ")
De (B). decode ("utf8", "ignore ")

In this way, you can obtain the unicode character strings of these strings.

The unquote you use is not a decoder. you need to decode and encode as necessary. I have been using utf8 for my tacit consideration. I think you probably use gbk, so the latter's solution was your loss. I guess it would be a tiring thing to guess. if we all use utf8, it would be nice, but some people have downloaded gb.

Http://yac163.svn.sourceforge.net/viewvc/yac163/trunk/yac163-nox/Pic.py? Revision = 198 & view = markup

Test me in this very old code #102-147 lines to each decode and encode used (...," Ignore ").

The code is as follows:

Def strdecode (string, charset = None ):
If isinstance (string, unicode ):
Return string
If charset:
Try:
Return string. decode (charset)
Except t UnicodeDecodeError:
Return _ strdecode (string)
Else:
Return _ strdecode (string)

Def _ strdecode (string ):
Try:

Return string. decode ('utf8 ')
Except t UnicodeDecodeError:
Try:
Return string. decode ('gb2312 ')
Except t UnicodeDecodeError:
Try:

Return string. decode ('gbk ')
Except t UnicodeDecodeError:
Return string. decode ('gb18030 ')

Def strencode (string, charset = None ):
If isinstance (string, str ):
Return string
If charset:
Try:
Return string. encode (charset)
Except t UnicodeEncodeError:
Return _ strencode (string)
Else:
Return _ strencode (string)
Def _ strencode (string ):

Try:
Return string. encode ('utf8 ')
Except t UnicodeEncodeError:
Try:
Return string. encode ('gb2312 ')
Except t UnicodeEncodeError:
Try:
Return string. encode ('gbk ')
Except t UnicodeEncodeError:
Return string. encode ('gb18030 ')