Finally, why Choose Go-kit

This is a creation in Article, where the information may have evolved or changed.

Objective

Work these years, has experienced two companies, respectively, participated in the design of PHP language framework and led the Golang technology stack landing work, in this process has some sentiment and summary. I would like to use the Golang technology stack I led before to state some of the problems encountered at the time, as well as analysis of problems and solutions to the problem. The main purpose is to state the Golang technology system in our team landing process, analysis of our various stages, encountered some problems, and the analysis of the problem of thinking and problem-solving methods to record, so that later students understand Golang in the team evolution process, draw relevant experience, So that in the future system design and development of less detours.

In the process of continuous evolution of the system, sometimes the selection of the framework is very casual, that can meet the current function on the line, not to its scalability and performance considerations, whether it can continue to support the development of the business-go sustainable development route, leading to the development of the business, found that the wrong choice, but it is difficult to turn. So now, I'll talk about how we choose these things.

Why do we have to move from PHP to Golang

At first, about 2015, all the business systems in the platform were made up of PHP language, on-line not long, the platform traffic began to explode, concurrency increased, then the fastest and most effective optimization means nothing to add machines and increase the number of php-fpm, but, The network model, which is limited by PHP itself, is not suitable for this high-concurrency, large-traffic scenario. In the face of this thorny problem, coupled with a limited number of staff, business tasks and other factors, so find other departments to borrow a batch of written LUA foreign aid, help to some logic simple and access to a large interface, replaced by LUA, temporarily carried over the late peak, so there is a considerable period of time, Most projects are a state of Php+lua coexistence. But because of some of the limitations of LUA itself, it is not very suitable to do some complex business logic, so in the end, the business has a strong demand for the premise, accompanied by the trend of technological development, at the end of 2015, we began to choose to turn to Golang.

How we move from PHP to Golang

Since the previous team is all PHP stack, the accumulation of Golang is not much, so in the process of PHP to Golang, faced with the transformation process will face the problem:

1 what frame to use;

2 How to Golang a PHP project in the case of heavy business tasks and extremely scarce personnel.

With what frame?

Before the team had developed a Golang framework for the internal PHP framework, it was suggested that it be used directly, and some people said to find an open source such as Beego,gin,martini such a popular framework. Because before with the internal PHP framework to do development, encountered a lot of problems, so I was still relatively exclusive use of self-research framework, the main reasons are as follows: 1 fewer documents, more loopholes, 2 people need to invest in human development and maintenance, at that time the shortage of manpower is not realistic. In addition, the community at that time the framework is more popular, but ultimately did not choose those popular framework, mainly for the following considerations: short time, the task is heavy, there is no energy to identify the merits of each frame, the application scenarios and performance. What if you use a framework that you don't have a deeper understanding of? Especially at that time the system frequently problems, under a variety of pressure conditions.

Although, I cannot choose a suitable framework in a short time, but I am still quite certain, what is our demand?

1 only do high-performance HTTP interface;

2 need a complete unit test system;

3 extensible, modular;

Based on the above three points, it can be found that the characteristics of Golang can meet these requirements. So we started to decide to use Golang to write naked.

Naked writing is not a scrawl.

Naked writing is not a scrawl. Controversial is well known, one of the advantages of the framework is to ensure the consistency of the team code style, of course, the current market in addition to the majority of Beego outside the framework, in the code style is not bound. In order to ensure the standardization and consistency of the Golang code, we have developed a set of Golang programming templates, from top to bottom: Router layer, service layer, DAO layer, and the entity layer through the three layers, as shown in Figure 1, according to the classic layered architecture and past experience. Among them, the router layer is responsible for dealing with the HTTP handler logic, request parameters and response format related processing work, service layer processing business logic, DAO layer processing data Access logic, entity layer is responsible for entities definition related logic, through the router, Service,dao these three layers. The layers are not directly coupled to each other, and the high-level modules are not directly dependent on the lower-level modules, they all depend on the defined abstraction interfaces. Booch once said: "All well-structured object-oriented architectures have a clear hierarchical definition, with each level providing a set of cohesive services out of a well-defined, controlled interface." In addition, we also maintain a common library of public components, such as: Log library, various database driver and so on.

Figure 1 Hierarchical architecture

How to Refactor

Once we have a programming template in place, we start the project refactoring work. Because the business task is heavy and the manpower is few, the basic direction of refactoring is to reconstruct the interface importance according to the business requirement. Only in this way can we ensure the reconfiguration of the system in the event of constant business demands. So, during this period, there was a fairly long period of time in the Php+golang mixed programming, mixed deployment state, which lasted for a year. Take the idea of mixed programming in the early stages of refactoring, you may encounter the same code, you need to write with Golang once again with PHP, undoubtedly increased a certain amount of work, of course, this is not to be avoided.

Finally, why should we introduce Go-kit

With the development of the business, the demand is more and more large, in order to meet the greater challenges, the team has a trend towards grpc,thrift direction, in addition we need to standardize the use of some middleware to ensure the stability of the system. So, I began to fall into a deep-thinking, but now the team great squad, business stability, left me a full of thinking and research time. This time thinking, or like the original framework of the idea of the choice, first of all, to understand what our needs are?

1 need a framework that supports http,grpc,thrift and other protocols with good extensibility;

2 The framework itself and the business code maintain a low-coupling state;

3 requires a common set of middleware, which is independent of the HTTP,GRPC and other transport protocols.

Currently the popular framework is around the HTTP protocol, including Gin,beego. After research, I found that go-kit can meet our needs. Go-kit itself is not a framework, but a set of microservices toolsets. Its design ideas and our initial Golang template formulation of the idea is also a coincidence-layered design, modular, extensible. Go-kit is shown in Architecture 2, divided into three layers: Transport layer, endpoint layer, service layer. The transport layer is responsible for the logic related to the transmission protocol Http,gprc,thrift, and the endpoint layer is responsible for the conversion of the Request/response format and the logic of the common interceptor, while the service layer focuses on the business logic. In addition to the classic layered architecture, Go-kit provides a number of common interceptors in the endpoint layer, such as log,metric,tracing,circuitbreaker,rate-limiter, to ensure the availability of business systems. They have a common feature in design: they are all unrelated to the transport protocol. In some of the previous HTTP frameworks, these interceptors are tightly coupled with the transport protocol. So, with the Gokit toolset, we can extend the transport protocol, middleware, without compromising the design of the business itself.

Figure 2 Go-kit Frame composition

How we integrate Go-kit into our existing business systems

After we find the right open source tool, how do we bring it into our business system at a lower cost? As we mentioned before, our Golang template is divided into three layers: Router,service and DAO. And Go-kit is divided into three layers, we can according to each layer of responsibility of the different to regroup, from top to bottom: Transport layer, endpoint layer, service layer, DAO layer. This can be easily integrated into the go-kit, of course, if one day for some reason, do not want to continue to use go-kit this set of things, directly remove the endpoint layer and transport layer. In the process of integration, it is important to note that the router layer in the previous code cannot contain any business logic, otherwise it cannot be integrated.

Figure 3 Evolution of the architecture

Summarize

Whether it is one of my previous article, "Talking about the Internet Business system design," said the system design, or this article stated in the framework selection. The first thing we need to make clear is: What is the demand? How to make the frame and the system have some flexibility while satisfying the requirement. The use of the classic Five design principles: a single responsibility principle, open closure principle, dependency inversion principle, interface isolation principle, for your design to provide a solid theoretical basis and direction of guidance. In addition, do not blindly follow the selection, others mouth good things, not necessarily suitable for you, only clear their own needs, to find the best for their own!

Reference books

"Driving Design for the field"

"Agile software Development Principles model and practice"