Go Standard library Anatomy 1 (transport HTTP request bearer)

This is a creation in Article, where the information may have evolved or changed.

Send HTTP requests using the Golang Net/http Library, and finally the roundtrip method that calls transport

type RoundTripper interface {    RoundTrip(*Request) (*Response, error)}

RoundTrip executes a single HTTP transaction, returning the Response for the request req.(roundtrip represents an HTTP transaction that returns a response to a request)
Plainly, you give it a request, it gives you a response

Next we look at his implementation, corresponding to the source file net/http/transport.go , I feel here is the essence of the HTTP package inside, go inside a struct is like a class, transport this kind of long

type Transport struct {    idleMu     sync.Mutex    wantIdle   bool // user has requested to close all idle conns    idleConn   map[connectMethodKey][]*persistConn    idleConnCh map[connectMethodKey]chan *persistConn    reqMu       sync.Mutex    reqCanceler map[*Request]func()    altMu    sync.RWMutex    altProto map[string]RoundTripper // nil or map of URI scheme => RoundTripper    //Dial获取一个tcp 连接，也就是net.Conn结构，你就记住可以往里面写request    //然后从里面搞到response就行了    Dial func(network, addr string) (net.Conn, error)}

Space limit, HTTPS and agent-related I ignored, two for, is to map idleConn idleConnCh idleConn save from Connectmethodkey (representing different protocols different host, that is, different requests) to persistconn mapping, c5/> is used to send persistent connections between multiple Goroutine in concurrent HTTP requests, that is, these persistent connections can be reused, and your HTTP request is used by some to persistConn channel send to other HTTP requests using this persistConn And then we find transport the RoundTrip method

func (t *Transport) RoundTrip(req *Request) (resp *Response, err error) {    ...    pconn, err := t.getConn(req, cm)    if err != nil {        t.setReqCanceler(req, nil)        req.closeBody()        return nil, err    }    return pconn.roundTrip(treq)}

Before facing the input error handling part we ignore, in fact, 2 steps, first get a TCP long connection, so-called TCP long connection is three times the handshake to establish a connection, close but always keep the reuse (save the environment) and then call this persistent connection Persistconn The roundtrip method of this struct

We follow the first step.

Func (t *transport) getconn (req *request, CM Connectmethod) (*persistconn, error) {if PC: = t.getidleconn (cm); pc! = N  Il {//Set request Canceler to some non-nil function so we//can detect whether it is cleared between now  And when//we enter roundtrip T.setreqcanceler (req, func () {}) return PC, nil} type Dialres    struct {PC *persistconn err error} Dialc: = Make (chan dialres)//define a channel to send Persistconn Prependingdial: = prependingdial postpendingdial: = postpendingdial handlependingdial: = Func () {if prePend                Ingdial! = nil {prependingdial ()} go func () {if V: = <-dialc; V.err = = Nil {            T.putidleconn (V.PC)} if postpendingdial! = nil {postpendingdial () }} ()} CANCELC: = Make (chan struct{}) T.setreqcanceler (req, func () {Close (CANCELC)})//initiates a Goroutine, this goroutine wonTake the inside call Dialconn get//Persistconn, then send to the above established channel Dialc inside, go func () {pc, err: = T.dialconn (cm) Dialc <-dialres{pc, Err}} () Idleconnch: = t.getidleconnch (cm) Select {Case V: = <-dialc://di ALC Our dial method first got through the Dialc channel sent over the return v.pc, V.err case PC: = <-idleconnch://Here on behalf of the other HTTP request used up the returned per Sistconn is sent by idleconnch this//channel to the Handlependingdial () return PC, nil case <-req. Cancel:handlependingdial () return nil, errors. New ("Net/http:request canceled while waiting for connection") case <-cancelc:handlependingdial () ret Urn nil, errors. New ("Net/http:request canceled while waiting for Connection")}}

The code in this is very fastidious, the above code inside I also commented, defined a Send persistConn channel dialc , started a goroutine , this goroutine gets inside the call to get dialConn persistConn , and then sent to the dialc inside, the main process goroutine in selectlisten to a plurality channel of inside, see which channel inside first send over persistConn , use which, then return .

Note here is that the idleConnCh channel is sent to the other HTTP request is used up persistConn , if from the channel inside, dialc This channel is also waiting for the hair, can not be wasted, through handlePendingDial this method to the dialc channel inside the persistConnalso sent to idleConnCh , waiting for follow-up to other HTTP requests to use.

Also, the reader can flip the code, each new persistconn to the TCP connection in the input stream, and the output stream with BR ( br *bufio.Reader ), and BW ( bw *bufio.Writer ) wrapped a bit, to bw write to the TCP input stream, read the output stream is also read through the BR, and started the read loop and write loop

pconn.br = bufio.NewReader(noteEOFReader{pconn.conn, &pconn.sawEOF})pconn.bw = bufio.NewWriter(pconn.conn)go pconn.readLoop()go pconn.writeLoop()

We follow the second step pconn.roundTrip to call this persistent connection to persistconn the struct's roundTrip method.
Look at persistConn this struct first.

Type persistconn struct {t *transport cacheKey connectmethodkey Conn Net. Conn tlsstate *tls. ConnectionState BR *bufio. Reader//From the TCP output stream read saweof bool//Whether we ' ve seen EOF from Conn; Owned by Readloop bw *bufio.                                      Writer//write to TCP input stream Reqch Chan Requestandchan//main goroutine to Channnel inside write, read loop from                                                                       Channnel inside Accept Writech Chan writerequest//main goroutine go to Channnel inside write Write loops receive Closech Chan struct{} from the channel inside the channel Writeerrch cha that notifies off TCP connections N Error lk Sync.    Mutex//Guards following fields numexpectedresponses int closed BOOL//Whether Conn have been closed broken BOOL//An error had happened on this connection;    Marked broken so it's not reused.   canceled BOOL//Whether this conn is broken due a cancelrequest Mutateheaderfunc is a optional func to modify extra//headers on each outbound request before it ' s written. (The//original Request given to roundtrip are not modified) Mutateheaderfunc func (Header)}

Inside is a variety of channel, with a superb, you have to understand, I paint here

There are three goroutine, respectively, with three circles, and the channel is represented by an arrow

There are two channel writeRequest andrequestAndChan

type writeRequest struct {    req *transportRequest    ch  chan<- error}

The Lord Goroutine to writerequest inside, writing loops from the writerequest inside to accept

type responseAndError struct {    res *Response    err error}type requestAndChan struct {    req *Request    ch  chan responseAndError    addedGzip bool}

The main goroutine is written inside the Requestandchan, and the reading cycle is received from the Requestandchan.

Note that the channel here is a two-way channel, that is, the channel's struct has an Chan type of field, such as the reqch chan requestAndChan Requestandchan inside ch chan responseAndError .

This is a very bull fork, the main goroutine sent through the reqch Requestandchan to read the loop, and then read the loop to get response after Requestandchan Inside the channel Responseanderror response back to the main goroutine, so I drew a two-way arrow.

Let's look at the code, and I understand that it's actually the process of three goroutine working through the channel.

Main loop:

Func (PC *persistconn) roundtrip (req *transportrequest) (resp *response, err error) {... Ignore/Write the request concurrently with waiting for a response,//In case the server decides to reply before read    ing our full/request body. Writeerrch: = Make (chan error, 1) Pc.writech <-writerequest{req, writeerrch}//Send request to write loop Resc: = Made (ch An responseanderror, 1) pc.reqch <-requestandchan{req. Request, Resc, requestedgzip}//Sent to read cycle Var re responseanderror var Respheadertimer <-chan time. Time Cancelchan: = req. Request.CancelWaitResponse:for {Select {case err: = <-writeerrch:if Isnetwriteerror (err {//write loop report error through this channel select {Case re = <-resc:pc. Close () break Waitresponse case <-time.                After (Time.millisecond)://Fall through. }} If Err ! = Nil {re = Responseanderror{nil, err} pc.close () Break Waitresponse } if D: = Pc.t.responseheadertimeout; D > 0 {timer: = time. Newtimer (d) Defer timer. Stop ()//prevent leaks Respheadertimer = timer. C} case <-pc.closech://If the long connection is hung up, here the channel has data, enter in this case to process SE                    lect {case re = <-resc:if fn: = testhookpersistconnclosedgotres; fn! = nil { fn ()} default:re = responseanderror{err:errclosed} if pc.i Scanceled () {re = responseanderror{err:errrequestcanceled}}} b Reak waitresponse Case <-respheadertimer:pc.close () Re = Responseanderror{err:errtimeout } Break Waitresponse//If timeout, the channel here has data,Break off for loop case re = <-resc:break waitresponse//Get response to read loop, break off for loop CA Se <-cancelchan:pc.t.cancelrequest (req. Request) Cancelchan = nil}} if Re.err! = nil {Pc.t.setreqcanceler (req. Request, Nil)} return re.res, Re.err}

This code is basically doing three things.

• Master Goroutine->requestandchan-read cycle goroutine

• Main Goroutine->writerequest-> Write loop goroutine

• The main goroutine through select to listen to the data on each channel, such as request cancellation, timeout, long connection hung, write out the wrong, read out the wrong, are the other goroutine sent over, and the same as the interruption, and then the corresponding treatment, the above also mentioned, Some of the channel is the channel that the main goroutine through the channel sends to other Goroutine's struct, for example case err := <-writeErrCh:case re = <-resc:

Read Loop code:

 func (PC *persistconn) Readloop () {... Ignore Alive: = True for alive {... Ignore RC: = <-pc.reqch var resp *response if Err = = Nil {resp, err = Readresponse (pc.br, Rc.req) If Err = = Nil && resp. StatusCode = = {//100 Continue The initial request has been accepted, the customer shall continue to send the request for its//remainder RESP, err = Readresponse (pc.br, rc.req)//Read the data in pc.br (TCP output stream), here the code in response//parse StatusCode, header field, turn into Standard in-memory response type//HTTP in TCP data flow, head and body separated by/r/n/r/n, each header//field separated by/r/n} } if resp! = nil {resp. TLS = pc.tlsstate} ... Ignore//Above some logical behavior of some HTTP protocols, rc.ch <-RESPONSEANDERROR{RESP, err}//Return the read response to Main Goroutine. Ignore//Ignore section, handle cancel req Interrupt, send idleconnch to return PC (persistent connection) to persistent connection pool (map) pc.close ()} 

Irrelevant code ignores that this code is mostly doing a thing

The Read loop Goroutine accepts the request () sent by the main Goroutine via the channel Requestandchan rc := <-pc.reqch and reads the response from the TCP output stream, then deserializes into the struct, and finally passes the channel Return to main goroutine ( rc.ch <- responseAndError{resp, err} )

func (pc *persistConn) writeLoop() {    for {        select {        case wr := <-pc.writech:   //接受主goroutine的 request            if pc.isBroken() {                wr.ch <- errors.New("http: can't write HTTP request on broken connection")                continue            }            err := wr.req.Request.write(pc.bw, pc.isProxy, wr.req.extra)   //写入tcp输入流            if err == nil {                err = pc.bw.Flush()            }            if err != nil {                pc.markBroken()                wr.req.Request.closeBody()            }            pc.writeErrCh <- err             wr.ch <- err         //  出错的时候返给主goroutineto         case <-pc.closech:            return        }    }}

The write loop is much simpler, the request of the master Gouroutine in the Select channel is then written to the TCP input stream, and the channel notifies the caller if something goes wrong.

Overall look down, the process is very simple, but there is a lot of code to learn, such as high concurrent request how to reuse TCP connection, here is the practice of connection pooling, if using multiple goroutine to cooperate with each other to complete an HTTP request, how to notify the caller interrupt error when an error occurs, Code style also has a lot to learn from the place.

I'm going to write a series that thoroughly analyzes the highlights of the Go standard library and shares it with everyone.