Go language Introduction "six": Source Learning-net/http

This is a creation in Article, where the information may have evolved or changed.

package net/httpIt is one of the main applications of the Go Language Web Application Foundation, from which can learn a lot of the above mentioned Io, as well as not mentioned a series of basic package, such as sync package Knowledge, code volume is relatively more, is a repository of source learning. This article starts with an HTTP server and explains how go implements an HTTP protocol server.

Mainly related to the following source files:
Net/net.go
Net/server.go
Net/http.go
Net/transfer.go
Sync/pool.go
Sync/mutex.go

0. Intro: Starting with the simplest HTTP server

func main() {    http.HandleFunc("/hi", hi)    http.ListenAndServe(":9999", nil)    fmt.Printf("hello, world\n")}func hi(res http.ResponseWriter, req *http.Request) {    fmt.Fprintf(res, "hi")}

The above is the simplest server code, running after listening to the 9999 port, in the browser http://localhost:9999 can see the return of Hi, then start to analyze the Net/http module.

1.Handler: Starting from routing

First to analyze http.HandleFunc("/hi", hi) this sentence, see the Source code discovery:

func HandleFunc(pattern string, handler func(ResponseWriter, *Request)) {    DefaultServeMux.HandleFunc(pattern, handler)}

First we learned that the definition of handler is like this func(ResponseWriter, *Request) . This definition is very important, let me mention it first.
And then DefaultServeMux you see, this class comes from ServeMux an instance of the struct, and the latter is a "router" role, which is ServeMux used to match the requested address and allocate the appropriate handler to complete the business logic, as described later in the request processing process.
In complete terms, we should first define our own and ServeMux assign routes to him, like this:

mux := http.NewServeMux()mux.HandleFunc("/", func(w http.ResponseWriter, req *http.Request) {    fmt.Fprintf(w, "Welcome to the home page!")})http.ListenAndServe(":9999", mux)

1. Generate a router
2. Registering a route with the router
3. Establish the underlying connection and provide services by routers and service addresses

The previous shorthand method simply omits the process of establishing a route, and actually uses the system's own defaultservemux as a router.

2. A cursory glance at the net package: Everything is based on net. Conn

Next see the http.ListenAndServe(":9999", nil) code source.

func ListenAndServe(addr string, handler Handler) error {    server := &Server{Addr: addr, Handler: handler}    return server.ListenAndServe()}

First, a server object is generated and its Listenandserve method is called. The server object, as its name implies, encapsulates all the information related to providing a Web service and is a more important class.

// A Server defines parameters for running an HTTP server.// The zero value for Server is a valid configuration.type Server struct {    Addr         string        // TCP address to listen on, ":http" if empty    Handler      Handler       // handler to invoke, http.DefaultServeMux if nil    ReadTimeout  time.Duration // maximum duration before timing out read of the request    WriteTimeout time.Duration // maximum duration before timing out write of the response    TLSConfig    *tls.Config   // optional TLS config, used by ListenAndServeTLS    MaxHeaderBytes int    TLSNextProto map[string]func(*Server, *tls.Conn, Handler)    ConnState func(net.Conn, ConnState)    ErrorLog *log.Logger    disableKeepAlives int32     // accessed atomically.    nextProtoOnce     sync.Once // guards setupHTTP2_* init    nextProtoErr      error     // result of http2.ConfigureServer if used}

1.handler is the router (actually the router itself as handler, which has many handler registered), see handler definition:

type Handler interface {    ServeHTTP(ResponseWriter, *Request)}

is almost the same as the previously registered function.
2. The ErrorLog default stdErr is as output, and other logger forms can be provided.
3. Other configuration and HTTPS,HTTP2 support, aside.

Initializing a server must be an address (port) and a route, and others can follow the default values. After generating the good server, enter the Listenandserve, the source code is mainly:

ln, err := net.Listen("tcp", addr)return srv.Serve(tcpKeepAliveListener{ln.(*net.TCPListener)})

It is important to first call the underlying net module to listen for the address, and the returned LN is a listener type with three methods:

• Accept () (Conn, error)
• Close () Error
• ADDR () Addr

We do not touch the net module, as long as we know that LN can be accept() returned by one net.Conn is enough, to get a connection context means that the client has established a channel, can fetch data, and return the results of processing to the client. The next srv.Serve() method accepts LN, where the program is divided into two layers: ln is responsible for the underlying establishment of the connection, read and write, shut down, and the server is responsible for data processing.

Add a description of net. Conn, this Conn is different from the latter to talk about the Server.conn, is relatively low-level, there

• Read (b []byte) (n int, err error)
• Write (b []byte) (n int, err error)

Two methods, also means that IO is implemented. Reader, Io. Writer interface.

3. Go back to server: Set up a server to gracefully handle concurrency with Goroutine

Then the previous said, after the establishment of the good ln, with the Tcpkeepalivelistener type simple packaging, as a parameter to srv. The Serve () method, which is important, is worth releasing all the code:

Serve accepts incoming connections on the Listener L, creating a//new service Goroutine for each. The service Goroutines read requests and//then call SRV. Handler to reply to them.////for HTTP/2 support, SRV. Tlsconfig should is initialized to the//provided listener ' s TLS Config before calling Serve. if//srv. Tlsconfig is non-nil and doesn ' t include the string "H2" in//Config.nextprotos, HTTP/2 support is not enabled.////Serve Always returns a Non-nil Error.func (SRV *server) Serve (l net. Listener) Error {defer l.close () if fn: = Testhookserverserve; fn! = nil {fn (SRV, L)} var Tempdelay Time. Duration//How long-to-sleep on accept failure If err: = Srv.setuphttp2_serve (); Err! = Nil {return err}//Todo:allow changing base context?    Can ' t imagine concrete//use cases yet. Basectx: = Context. Background () CTX: = Context. Withvalue (Basectx, Servercontextkey, srv) CTX = context. Withvalue (CTX, Localaddrcontextkey, L.addr ()) for {       RW, E: = L.accept () if E! = nil {if NE, ok: = E. (NET. ERROR); Ok && ne. Temporary () {if Tempdelay = = 0 {tempdelay = 5 * time. Millisecond} else {Tempdelay *= 2} if Max: = 1 * time.s Econd; Tempdelay > Max {tempdelay = max} srv.logf ("Http:accept error:%v; Retrying in%v ", E, Tempdelay) time. Sleep (Tempdelay) Continue} return e} tempdelay = 0 c: = srv.ne Wconn (rw) c.setstate (C.RWC, statenew)//Before Serve can return go C.serve (CTX)}}

Analysis:

A) The first is the context of this type

This type is more wonderful, its role is a map, in the form of key,value set some background variables, using the method iscontext.WithValue(parentCtx,key,value)

b) then enter a for Infinite loop,

L.accept () blocks until it gets to one net.Conn , and then srv.newConn(rw) sets the server.conn state to statenew by establishing one (which belongs to the private variable, not exposed externally).

c) Start a goroutine to handle this connection

Called go c.serve(ctx) . It can be seen from here that the concurrency model of the go language is different from the single-threaded callback model of Nodejs, and also differs from the multithreaded scheme of Java, using the native Goroutine to deal with both isolation and performance. Because this does not occur in Nodejs because of exception handling problems often cause the server to hang out. At the same time, the creation cost of goroutine is much lower than the creation thread, and of course the same machine can achieve greater concurrency than the Java server.

4. From server to conn: one request all the essences are in the Conn.

Server.conn mentioned earlier, look at the source code:

A Conn represents the server side of an HTTP Connection.type conn struct {//server was the server on which the Conn    Ection arrived. immutable;    Never nil.    Server *server//RWC is the underlying network connection. This was never wrapped by other types and was the value given out//to Closenotifier callers. It is usually of type *net.    Tcpconn or//*tls.conn. RWC net. Conn//REMOTEADDR is RWC. Remoteaddr (). String ().    It is not populated synchronously//inside the Listener ' s Accept goroutine, as some implementations block.    It is populated immediately inside the (*conn). Serve Goroutine. This is the value of a Handler ' s (*request).    Remoteaddr.    Remoteaddr string//Tlsstate is the TLS connection state when using TLS.    Nil means not TLS. Tlsstate *tls.    ConnectionState//Werr is set to the first write error to RWC.    It is set via Checkconnerrorwriter{w}, where BUFW writes. Werr Error//R is Bufr ' s read source. It ' s AWrapper around RWC that provides//IO. Limitedreader-style limiting (while reading request headers)//and functionality to support Closenotifier.    See *connreader Docs.    R *connreader//BUFR reads from R.    Users of Bufr must hold MU. Bufr *bufio.    Reader//BUFW writes to Checkconnerrorwriter{c}, which populates werr on error. BUFW *bufio.    Writer//Lastmethod is the method of the very recent request//on this connection, if any.    Lastmethod string//MU guards Hijackedv, use of BUFR, (*response). Closenotifych. Mu sync.    Mutex//HIJACKEDV is whether this connection have been hijacked//by a Handler with the hijacker interface.    It is guarded by MU. HIJACKEDV BOOL}

Explain:
First, hold the server reference; hold net.Conn a reader, package from the underlying read interface, read the data from the connection, and a bufr (or front reader, with Buffering). and a corresponding synchronization lock, lock the parameter modification to itself, prevent the synchronization update error.
Then, here the MU type is the sync.Mutex role of this type is a bit like Java synchronized block (there is a Java synchronized, you can refer to my other my book "Java Multithreading you just need to look at a piece is Enough"), Mu is holding the object lock that instance. We can see that the Hijackedv property of Conn is maintained through MU to prevent the synchronization update problem. Reference conn.hijackLocked() , no longer expanded.

Keep looking at serv.Serve() the method, then the previous 3 points:

d) setState (state)

In fact, the state is maintained in the server, but it is called by Conn. There are altogether StateNew, StateActive, StateIdle, StateHijacked, StateClosed five states. Starting with new, after reading a byte and entering active, after reading and sending response, enter idle. There are two kinds of termination, the active termination of closed and the takeover: Hijack let the caller take over the connection, and after calling Hijack (), the HTTP Server library will no longer process the connection, and the management and shutdown responsibilities for that connection will be taken over by the caller. Referenceinterface Hijacker

e) C.serve (CTX)

Let's look at the conn.serve() source code first:

Serve a new Connection.func (c *conn) Serve (CTX context. Context) {c.remoteaddr = C.rwc.remoteaddr (). String () defer func () {if err: = Recover (); Err! = Nil {Const SIZE = ten << buf : = Make ([]byte, size) buf = Buf[:runtime. Stack (BUF, false)] C.SERVER.LOGF ("Http:panic serving%v:%v\n%s", c.remoteaddr, Err, BUF)} if! C.hijacked () {c.close () c.setstate (C.RWC, Stateclosed)}} () If tlsconn, OK: = C.RWC. ( *tls. Conn); OK {if d: = c.server.readtimeout; D! = 0 {c.rwc.setreaddeadline (time). Now (). ADD (d))} if D: = C.server.writetimeout; D! = 0 {c.rwc.setwritedeadline (time. Now (). ADD (d))} If Err: = Tlsconn.handshake ();        Err! = Nil {C.SERVER.LOGF ("Http:tls Handshake error from%s:%v", c.rwc.remoteaddr (), err) return } c.tlsstate = new (TLS. ConnectionState) *c.tlsstate = Tlsconn. ConnectionState () if proto: = C.tlsstate.negotiatedprotocol; VALIDNPN (proto) {if fn: = C.server.tlsnextproto[proto]; fn! = Nil {h: = Initnpnrequest{tlsconn , Serverhandler{c.server}} fn (C.server, Tlsconn, h)} return}}//HTTP    /1.x on. C.R = &AMP;CONNREADER{R:C.RWC} c.bufr = Newbufioreader (C.R) C.BUFW = Newbufiowritersize (Checkconnerrorwriter{c}, 4& LT;&LT;10) CTX, Cancelctx: = context. Withcancel (CTX) defer cancelctx () for {W, err: = C.readrequest (CTX) if c.r.remain! = c.server.initial            Readlimitsize () {//If we read any bytes off of the wire, we ' re active. C.setstate (C.RWC, stateactive)} if Err! = Nil {if Err = = Errtoolarge {//their HTTP client may or is not is//able to read the If we ' re//responding to them and hangin G up//while they ' re still WRIting their//request.                Undefined behavior. Io. WriteString (C.RWC, "http/1.1 431 Request Header fields Too large\r\ncontent-type:text/plain\r\nconnection:close\r\n\r            \n431 Request Header fields Too Large ") c.closewriteandwait () return} If err = = Io. EOF {return//don ' t reply} if neterr, OK: = Err. (NET. ERROR); OK && neterr. Timeout () {return//don ' t reply} var publicerr string if V, OK: = Err. ( BADREQUESTERROR); OK {publicerr = ":" + string (v)} io. WriteString (C.RWC, "http/1.1 request\r\ncontent-type:text/plain\r\nconnection:close\r\n\r\n400 bad Request" + PUBLICERR) return}//Expect Continue support req: = W.req if Req.expectsconti Nue () {if Req. Protoatleast (1, 1) && req.          ContentLength! = 0 {      Wrap the Body reader with one, replies on the connection req. Body = &expectcontinuereader{readcloser:req. Body, Resp:w}}} else if req. Header.get ("Expect")! = "" {w.sendexpectationfailed () return}//HTTP cannot has MU Ltiple simultaneous active requests. [*]//Until The server replies to this request, it can ' t read another,//so we might as well run the Handl        Er in this goroutine. [*] not strictly true:http pipelining.        We could let them all process//in parallel even if their responses need to be serialized. Serverhandler{c.server}.        Servehttp (W, W.req) w.cancelctx () if c.hijacked () {return} w.finishrequest () If!w.shouldreuseconnection () {if W.requestbodylimithit | | w.closedrequestbodyearly () {c.cl Osewriteandwait ()} return} c.setstate (C.RWC, StatEidle)}} 

5. From Conn to Conn. Where the SERVE:HTTP protocol is implemented, Conn becomes request and response

Above the Conn. Serve (), we focus only on the main logic:

1. Initialize BUFR and BUFW.

...c.bufr = newBufioReader(c.r)c.bufw = newBufioWriterSize(checkConnErrorWriter{c}, 4<<10)...

These two are read-write entry points, from the efficiency of consideration, is added a layer of buffering. It's worth noting that BUFW and BUFR also added a layer of sync. The pool package, which is a pooled object from the sync package, is intended to be reused and does not need to execute new allocation memory every time.

2. Next, it is important to read the data sent from the underlying client:

...w, err := c.readRequest(ctx)...

We see Readrequest definition:

func readRequest(b *bufio.Reader, deleteHostHeader bool) (req *Request, err error)
The return is (w *response, err Error), And response is an important object in the Server.go, it is the conn of the higher layer package, including Req,conn, as well as a writer, of course, this write operation is actually by the Conn, and then by the lower level net.Conn to execute. For developers, the face is basically this response, can be said to be a design pattern in the façade mode.

Also, notice that the Mu.lock () is called when the readrequest executes.

3. Most important, call the user's handler

...serverHandler{c.server}.ServeHTTP(w, w.req)

First of all, Serverhandler is just a wrapper, which is actually called c.server.Handler.ServeHTTP() . And in the initialization of the server mentioned earlier, it Handler is DefaultServeMux either user-specified ServeMux , which we call the router. In routers, user-defined routing rules are used to specifically invoke the users ' business logic methods.

A router can be seen as a map, with a routing rule (string) as key and a business method (func type) as value.

Servehttp introduced the most important two high-level package response objects and request objects (strictly speaking, response is a private type, exposed ResponseWriter , but understood from the nature of HTTP, or called response).

From the level, the two encapsulated objects are encapsulated in the bottom conn , the data sent by the client (Req.body), and the interface Reader,writer read and write.

The user's business logic then accepts the data, processes it, and returns the data. The return data is generally written directly to this w, that is, ResponseWriter in. This completes the complete process for an HTTP request.

4. Finally do some processing work

Mainly include: exception handling, resource recovery, status update. We understand, and the focus is on the main process.