Golang string encoding conversion GBK turn UTF8

Source: Internet
Author: User
Tags cdata translate function

Problem Description:
Need to tune out a third-party interface, the interface returned by the encoding format is GBK, but Golang built only recognize UTF8, so you need to convert the GBK string to UTF8, in order to perform subsequent data processing.

Problem handling:
Introduction to open source package go get Github.com/axgle/mahonia
The open source protocol for this package is BSD, so you can safely reference and use

Mahonia Source Interpretation:

Convertstring converts a string from d ' s encoding to UTF-8.
Func (d Decoder) convertstring (s string) string {
    bytes: = []byte (s)
    runes: = Make ([]rune, Len (s))
    Destpos: = 0

    for Len (bytes) > 0 {
        c, size, Status: = d (bytes)

        if status = = state_only {
            bytes = bytes[size:]
            C Ontinue
        }

        If status = = No_room {
            c = 0xfffd
            size = Len (bytes)
            status = Invalid_char
        }

        bytes = bytes[size:]
        Runes[destpos] = c
        destpos++
    }

    return string (runes[:d Estpos])
}

       thought to call the above function to complete the conversion, and found that the Chinese part of the returned string is still garbled. After checking the data online, it is found that a translate function needs to be called.

Translate enables a Decoder to implement Go-charset ' s Translator interface. 
    Func (d Decoder) Translate (data []byte, EOF bool) (n int, CDATA []byte, err error) {CDATA = make ([]byte, Len (data) +1) Destpos: = 0 for n < len (data) {rune, size, Status: = D (data[n:]) switch Status {C ASE State_only:n + = size Continue case no_room:if!eof {RE
            Turn N, cdata[:d Estpos], nil} rune = 0xfffd n = len (data) Default: n + = size} If Rune <, {if Destpos >= len (CDATA) {CDATA = Doubl  Elength (CDATA)} Cdata[destpos] = Byte (rune) destpos++} else {if Destpos+utf8. Runelen (rune) > len (CDATA) {CDATA = Doublelength (CDATA)} Destpos + = UTF8. Encoderune (Cdata[destpos:], Rune)}} reTurn N, cdata[:d Estpos], nil} 

The internal encoding conversion code is not analyzed, looked at the next not how to understand ... As soon as a take doctrine, look at the function of the parameter and return value, directly called. found that the return value has three parameters, N should be the capacity or length of the string, err I find the return nil anyway, so these two parameters for me I do not care, only need to get the second return parameter []byte on the line.

As a result, the conversion function is encapsulated as follows

SRC is the string to be converted, Srccode is the encoding format to be converted, Targetcode is the encoding format to convert
func converttobyte (src string, srccode string, Targetcode string) []byte {
    Srccoder: = Mahonia. Newdecoder (Srccode)
    Srcresult: = srccoder.convertstring (src)
    tagcoder: = Mahonia. Newdecoder (Targetcode)
    _, CDATA, _: = Tagcoder.translate ([]byte (Srcresult), true)
    return CDATA
}

Use

Response = util. Converttobyte (String (response), "GBK", "UTF8")
    err = json. Unmarshal (response, &result)
    if err! = Nil {
        logger. Error ("Error:", err)
        return result, err
    }
    return Result,nil

       accomplished

Contact Us

The content source of this page is from Internet, which doesn't represent Alibaba Cloud's opinion; products and services mentioned on that page don't have any relationship with Alibaba Cloud. If the content of the page makes you feel confusing, please write us an email, we will handle the problem within 5 days after receiving your email.

If you find any instances of plagiarism from the community, please send an email to: info-contact@alibabacloud.com and provide relevant evidence. A staff member will contact you within 5 working days.

A Free Trial That Lets You Build Big!

Start building with 50+ products and up to 12 months usage for Elastic Compute Service

  • Sales Support

    1 on 1 presale consultation

  • After-Sales Support

    24/7 Technical Support 6 Free Tickets per Quarter Faster Response

  • Alibaba Cloud offers highly flexible support services tailored to meet your exact needs.