The third session of the Gopherchina Congress

This is a creation in Article, where the information may have evolved or changed.

It was a good experience to attend the Gopherchina third Congress last weekend. After three years, Go's development is very hot. The meeting size from the original hundreds of people to thousands of people, there are many standing on both sides of the seat to listen to friends. The content of the conference is also from the Go itself, to the architecture, to the container and other related fields are involved, can say dry good. It is a hard work to run the General Assembly, thanks very much to Astaxie for its efforts. It also takes a lot of skill to speak a topic, and thanks to all the lecturers who participated in the conference. So, let's talk about the conference.

Go on the language

Each language has its own characteristics, Go is no exception. When learning Go, it is inevitable to bring in other language experience, causing some trouble. So going from Go is a process that the Go language developer has to go through. Tony Bai shared the go mindset from the go language perspective.

Go is about orthogonal composition of simple concepts with preference in concurrency.Go 是在偏好并发的环境下的简单概念/事物的正交组合

From this sentence, we can summarize several principles of Go development:

• The simplification of things, the logical unit is not too large; it is not a function from beginning to end, it is a large function composed of several small functions
• Orthogonal combination, the independence between the logical units, because of the independence between functions, can be reused, can also be concurrent
• The need for concurrent backgrounds, where the business can be simultaneous, not linear

These principles still require a lot of development techniques, such as the vertical combination of interfaces, such as:

type ReadWriter interface {    Reader // 这里体现了逻辑单元的简单化    Writer}

The level of acceptance logic that is more adaptable according to the interface is extended:

func ReadAll(r io.Reader) ([]byte, error)// 可以支持文件流，网络数据流等ReadAll(*os.File)ReadAll(*http.Response.Body)

Finally, I am interested in the error handling of the introverted, such as:

// *bufio.Writerfunc (b *Writer) Write(p []byte) (nn int, err error) {    if b.err != nil {        return nn, b.err  // 错误在结构内部    }    ... ...}// usagebuf := bufio.NewWriter(fd)buf.WriteString("hello, ") // 这样就不需要每行 if err != nilbuf.WriteString("gopherchina ")buf.WriteString("2017")if err := buf.Flush() ; err != nil {    return err}

As mentioned above, the vertical expansion of the interface, Francesc's share of the more in-depth talk about the interface{} use of Go skills. interface{}There are two meanings to my understanding. When there interface{} is a method, it is the definition of the behavior, such as:

type Reader interface{    Read(data []byte)(n int,err error) // 注意定义的时候写一下变量名，否则不是啥意思}

Such an interface can compensate for the lack of Go-generic. When a generic is returned as if <List,Map> it is not true, the real meaning is probably <Iterator> . In go programming, you're forced to think like this, to find out what data types are common when you need generics, to do the same thing (if it's a different behavior, isn't it better to write two functions?). ）。

interface{}Another special scenario is an empty interface, and the corresponding code is the need for type inference:

func do(v interface{}){    switch t := v.(type){        case int:            fmt.Printf("int - %d",t)        case error:            fmt.Printf("error - %s",t.Error())        default:            fmt.Printf("interface - %v",t)    }}

It is not the last resort to write code like this . Otherwise there is an increasing number of case types that need to be inferred, and code maintainability drops instantly.

Ezbuy share has a lot of micro-service-related content, selection of GRPC use, context tracing and other issues. But what I care about most is their development environment building tools Goflow . The package management mechanism of Go has been criticized. Even if vendor some third-party package dependencies are solved, it is very coarse and straightforward. At the same time, the project management Gopath has a third-party library, and its own project code, but also cause trouble for engineering practice. Goflow is like a GB + package registry from sharing. Goflow Modify the system variable Gopath to the current directory, and then download the third-party package to the vendor directory from the Intranet registry. Really very convenient, is a good solution. and the domestic network access a lot of package resources is not very smooth, the intranet has registry provide a great convenience. This set of things used inside the company I feel very good.

In addition he is the only mentioned internal sharing with the package. For example, the code structure:

--product  |  |---internal  |   |  |   |---product_get.go  |   |---product_search.go  |   |  |---product.go

The code and functionality are more clear. I will also try to use this technique later on.

The Go on the architecture

Many of the Go usage scenarios are large-scale concurrent services. The ability to withstand concurrent services is not only a language thing of Go, but also an integral architectural design. A lot of lecturers talk about the role of Go in the whole and related use from the aspect of architecture.

The seven-cow lecturer shared the application of Go in a small piece of their big data analytics system Pandora. Big Data analysis system, simple imagination will also have data acquisition, data processing, analysis and analysis of the results of the process of landing these processes. This share is the details of the components that analyze the results of the data. Again to Lenovo, landing will have what problems: data loss, transmission delay, a variety of downstream output, how to multi-instance distribution. The entire share is about the means that almost all distributed architectures will use:

• Data goes into memory queue, memory queue may dump to disk queue
• Transaction of data, guaranteed entry and exit, failure replay, outgoing to ensure correct handling
• Program shutdown restart and other special status of data offline
• A balance algorithm between multiple instances

As a whole, it is more routine to listen. The use of general means, coupled with reasonable architectural design, can carry a large number of services , which is sufficient. (Don't toss it)

TIDB lecturers share the content of the database implementation itself. Because TIDB is a distributed database, it can be imagined that a query request came, after the query optimization, the actual execution of the query operation of the process, need to ask Meta data source table structure, index structure and other information, and then go to the specific TIKV instance for data retrieval. It is possible that a query needs to fetch data from many TIKV instances, apparently concurrency logic, and Go is well suited.

What I am interested in is that TIKV supports some simple query operations. Is that some of the details of the query task can be below the tikv, such as LIMIT 10. TIKV can intelligently return a limited number of results, aggregating at the SQL layer. Otherwise each tikv to a large wave of data in the SQL layer aggregation, too wasted.

Message Queuing is a common application scenario for Go, and it's a great way to share the NSQ Message Queue transformation. A couple of interesting points were heard throughout the share:

• The data queue is read as a cursor, a part of the entire segment of the move. (NSQ's data queue has memory and file queue, but consumes to write files)
• Because it is a cursor, it is possible to read the queue concurrently. Multiple cursors read data on the same queue and seem to maintain a relatively high degree of complexity. Another angle reminds me thatringbuffer
• Concurrent read queue, also a common model for channel, 1 producer-N Consumer
• There are too many timers in the queue, and the time channel is used to manage them uniformly. It's a time wheel , actually???

Micro-service is a hot topic. A long time ago, the distributed architecture was divided into modules. With the rise of Docker, the dimensions of the module are too large, smaller business units + Docker containerized, the formation of the cluster becomes a popular way to implement MicroServices now. After microservices, business processes break down between various microservices. We need to track and peer data in each microservices running state,trace becomes a very important issue, Bilibili's Mao Jian share a large part of the chat B station in the process of service, data trace way.

Because of the diversity of business, the vast majority of trace is intrusive. The first Mao Jian is to net/rpc add the tracing context to the standard library of Go, while also using the standard library context.Context to control the flow of data. Context can do a lot of things for RPC to do with microservices. such as context.WithTimeout() the control of micro-service request Timeout, context.Value() a lot of links, users, operations related data for authentication, filtering and statistics. Again, such as load balancing, the client request in the context record server request weights, automatically do a balanced send data to the service side of the small pressure. (You can also see that the server is stateless)

After chatting, Mao Jian uses mature tools, Message Queuing Kafka, cache Redis + modified Twemproxy, distributed tracking of Google dapper, storage Hbase and ES, etc. Reasonable architecture design + mature tools can withstand large-scale business. And seven cows share to my experience is similar. Finally there is a bit of fun, B station The earliest code is based on Dedecms magic, all the code is rubbed together, almost uncontrollable, haha haha!

The next day Grab's superb talk about Go in the application of Grab also talked about the use of the context tracing under the micro-service, positioning problems. What is more noticeable in sharing is the process of code project management. Grab put the code in a Git repository, and follow the team namespace to standardize the directory structure. This simple to do the role of separation, but also to increase the transparency of the Code. Perhaps, if you want to make large-scale changes, you can easily discuss and refactor the code between the teams. In addition, the ultimate code reuse, and simple dependency management, unified version updates, but also brings the overall stability of the project. These are the benefits, but a project everyone updates the code at the same time, there will certainly be conflicting situations. The automated process of code test and review then provided a solution to the problem.

The Code collaboration tool, Phabricator, provides the great convenience for code review, automating the specification of codes, unit testing, and coverage detection. Such conflicting code will cause unit test failure, code coverage degradation and so on, through slack and so on immediately prompt to the developer. Even if the code conflicts, you can immediately follow up the changes. Moreover the code base is transparent, the other party modifies what content you can go to read, after the reference modifies own content. A set of tools for automated testing is built up, and large warehouses are more beneficial to Grab than disadvantages.

360 share the technical details of the Poseidon search platform with a lot of information that can be consulted. The use of CGO from the old C + + over to go has caused a lot of trouble and eventually chose to completely rewrite the old code with go. Visible, CGO In many cases is thankless, careful choice. In addition to the problem of panic capture across Goroutine, he embeds the error into the data structure, as in the above-mentioned processing pattern. Then the goroutine of the computational data can be panic-recover to get the error into the data. The goroutine that handles the results of the data operation can be data.error to get an error with this data operation. 360 said that their handling capacity is daily average of 100 trillion, more want to know how many machines propped up business.

The Go in the container

One of the big-star applications of Go is docker . Almost everything about Docker is developed in the Go language, such as Docker Hub, Kubernetes, ETCD, etc. My daily use of Docker is also a switchover of native multiple development environments. There is no concept of large quantities of container orchestration and routine maintenance. Listen to Docker's share more dead in the posture of rising.

Deng Hongshu introduced Kubernetes Operator let me in front of a light. Kubernetes Operator is to solve the problem of stateful service containerized. For example, ETCD dynamically configure the location of each node. Kubernetes Operator is responsible for monitoring configuration changes in a unified location, starting and stopping containers based on changes, making the entire container group state and defined. He's on the scene. Modify the Kubernetes Operator configuration in the container service in the version number, the number of container groups and other information, Kubernetes automatically start and close the container, to meet the requirements of the configuration. See these, feel intelligent operation and maintenance of progress, especially supporting services such as orchestration more and more trend of automation and intelligence, gratifying to celebrate.

Huawei Ma Daochang shares the upgraded version of DevOps ContainerOps . The development business of large companies is very complex. If the general DevOps to build the environment, the installation of a variety of supporting facilities, and then build the current business development of other business modules, but also need to develop test operations, very complex. Docker helps solve the problem of building environments and installing infrastructure. Development tests in Docker are the same as in VMs. That's the problem of building dependencies on other business modules. As I understand ContainerOps Component it, it is to solve this problem. To develop a business process, you define a Component, business input to the business output. Component uses k8s to pull up the Docker image of the other business modules needed from registry to form a complete business flow internally. The personnel are then developed and tested on this business flow. This is a lot easier to develop for large companies than DevOps. At the same time, in order to build a series of Docker images, the business itself of the module split or microservices, but also for the future of daily maintenance, deployment and expansion of saving a lot of things.

In addition, MA Dao long ppt is better understood, the process is very good to know how the Containerops based workflow is running.

VMWare's introduction to Harbor is comprehensive. From the need analysis of the Docker image distribution problem solved by Harbor, to some problems encountered in the concrete implementation are mentioned. What I care about is the worker pool in his code and the state machine. I vaguely remember the last time in order to answer the QQ group's small and medium-sized partner's question, went over the Harbor worker pool code, found very much like handling 1 Million requests per Minute with Go article design. In this way, the worker pool pattern everyone's idea is almost haha.

The Ultimate Go

Dave Cheney shares the Go's #Pragma full of black technology. Compiler directives have always been a very confusing thing to do. Unless you know a lot about the compiler itself, it's easy to make things that you can't clean up. go://nosqlitand go://noinline is relatively easy to understand compiler operation instructions. Goroutine's continuous stack model brings up the question of whether or not to split, and the new SSA backend compilation also brings the question of whether to be inline. My understanding of the implementation of Go itself is not deep, presumably only with this effect. The concrete content also needs me to carry on the thorough research.

#PragmaJust listen, don't use it.

The sharing of GF securities is very interesting. Requirements for high-frequency trading in securities business: ultra-low latency, ultra-high concurrency, ultra-high reliability and ultra-strict supervision. No matter what language development it is, there are huge challenges. What we need to do when Go faces these problems is very useful.

The first problem is the GC pauses. Go 1.8 has compressed GC STW time to < 100μs, which can be said to have solved many problems. However, from the point of view of the code, there are many things you can do to reduce GC consumption. One is the Goroutine pool. This is confirmed by the use of fasthttp, where efficiency is several times the standard library, while memory and GC and standard libraries differ little. In addition, you can control the number of object pools. Go Standard library is sync.Pool too extensive, in some cases it is very necessary to write an object pool to do fine control. Another point is the problem of variable escape. If a variable escapes to the heap, it affects the performance of the GC (the scan variable is required). So you need to learn some code tricks to avoid escaping to the heap.

It also mentions the optimization of multilevel Map. Large maps are very influential in GC performance. The strategy is to be large and small, such as multiple small maps to reduce the scanning time (Mark in Go GC mark-sweep is concurrent). Additionally, access to the MAP requires a lock to ensure concurrency security. Large Map lock granularity is too large, as if a warehouse only one door, people in the door can not go inside, until the people come out. Very much affects the concurrency of the operation. Map large and small, the equivalent of a number of doors. and multi-level Hash after the Map, the granularity is very small, a lot of doors. Even if the number of goroutine is huge and the door is much greater, the situation of access to a door is significantly reduced. That is, the process of reading data is rarely subject to lock contention.

In addition, it also shares the offload problem of network card, and uses hardware shards to send packets into large packets.

Off Topic

This gopherchina is full of content, from the Go language level, to code skills, architecture design, and container applications are involved. However, too much content, not in what areas everyone is interested. Still want to have more content in the Go language itself and in code programming. From requirements analysis to architectural design, it is a prelude to code writing. To cope with the design, my Go program becomes what it looks like. What functions does my Go program provide in order to implement the functionality? Hopefully, more lecturers can fall from the architectural level to the code level. Architectural design principles and tools are similar, but the company's business is different, the resulting architectural model is very diverse. Go, as part of the architecture, wants to make the design fit for this architecture. This design can also inspire developers to better use the Go language.

Gopherchina the contents of the previous conferences:

Https://github.com/gopherchina/conference