Golang's pit HTTP read large files must be read out

This is a creation in Article, where the information may have evolved or changed.

First on the code

 PackageMainImport (    "FMT"    "Net/http")Func main() {resp,Err:=http.Get("Http://mirrors.ustc.edu.cn/opensuse/distribution/12.3/iso/openSUSE-12.3-GNOME-Live-i686.iso")    ifErr!= Nil {Panic(Err)    }FMT.Println("RESP Code",resp.StatusCode)resp.Body.Close() //Note that RESP is not read here. Body, and resp. Body has about 700MB not read}

What do you think? Got Stuck?!

If your internet speed is fast enough, you will find that after a few 10 seconds, the program automatically exits, but if you are unfortunately a small water pipe, you will find stuck ...

What is the reason?

HTTP packets reuse the connection by default, and reusing the connection requires reading the data from the previous connection

Code Snippets (NET/HTTP/TRANSFER.GO)

func(b*Body) Close()Error{    ifb.closed{        return Nil    }defer func() {b.closed= true    }()    ifb.HDR== Nil &&b.closing{        return Nil    }        ifb.Res!= Nil &&b.Res.Requestbodylimithit{        return Nil    }    ///The problem is here, read the whole body!!.    if _,Err:=io.Copy(Ioutil.Discard,b);Err!= Nil {        returnErr}    return Nil}

How to solve it?

By the logic of the code snippet above, you need to return nil in advance to avoid being read

    // b.hdr 总是为nil,因为从不设置    // 那b.closing什么时候为true呢?    if b.hdr == nil && b.closing {        return nil    }

Read the source, B.closing relies on the Transferreader close value

And the close value of Transferreader is determined by the Shouldclose method.

//The header here is resp.func shouldclose(Major,Minorint,HeaderHeader) BOOL {    ifMajor< 1 {        return true    } Else ifMajor== 1 &&Minor== 0 {        if !Strings.Contains(Strings.ToLower(Header.Get("Connection")), "Keep-alive") {            return true        }        return false    } Else {        //Todo:should split on commas, toss surrounding white space,        //And check each field.        ifStrings.ToLower(Header.Get("Connection")) == "Close" {Header.Del("Connection")            return true        }    }    return false}

Since the RESP header cannot be modified before the code, modify the Req header so that the server always returns Connection:close

Final code

 PackageMainImport (    "FMT"    "Net/http")Func main() {req, _ :=http.newrequest("GET", "Http://mirrors.ustc.edu.cn/opensuse/distribution/12.3/iso/openSUSE-12.3-GNOME-Live-i686.iso", Nil)req.Header.Set("Connection", "Close")resp,Err:=http.defaultclient. Do(req)    ifErr!= Nil {Panic(Err)    }FMT.Println("RESP Code",resp.StatusCode)resp.Body.Close()}

One months did not write blog, poor mood + body scrub ~_~ Hey, eventful March

April 5, 2013 update

Coocood reminds me that go1.1 has a new API to complete this egg-crushing thing.

http.DefaultTransport.(*http.Transport).CancelRequest(req)

If it's not the default Defaulttransport, find your own set of transport.