Go Kit:go in the modern Enterprise

This is a creation in Article, where the information may have evolved or changed.

Original http://peter.bourgon.org/go-kit/

The modern enterprise

When we hear the word enterprise , we often associate it with the old-school, slow, and traditional, such as IBM HP and even Red Hat, who have been a leader in the technology industry for over dozens of years. Now, some companies such as Google Amazon twitter,netflix,facebook,spotify and even SoundCloud are becoming the mainstream of the Internet industry, and these modern businesses often have the following qualities:

• Technology-oriented
• Focus on user products and user experience
• A successful, explosive growth
• 100-1000 Order of magnitude engineers
• Adopt a service-oriented architecture (SOA)

The last point is especially critical.

A service-oriented Architecture

With the exception of a few exceptions, the vast majority of modern companies have shifted to a service-oriented architecture (or Microservice) after experiencing similar technological architecture evolution:

• A giant that is rapidly developed based on RoR (or similar frameworks) in the early stages of entrepreneurship
• Peel off key modules independently and move toward SOA
• may be more finely decoupled and become microservices

There are many benefits to using SOA, but at a cost. A distributed system is naturally more complex than a single, behemoth service that implements the same functionality. Addressing and managing the complexity of these distributed system proxies usually eventually yields some class libraries and frameworks. In modern enterprises, some of the proven facts standards begin to come into view, often in the form of open source projects. I think Twitter's finagle is one of the most noteworthy, and Netflix has maintained an open-source project like this. Of course, there are many small companies open source a lot of homogeneous projects.

These class libraries are usually written in Scala and are used in conjunction with other JVM languages. Scala has a number of benefits in this area: strong expression, good performance in some scenarios. But it is not a silver bullet, from any point of view, it is too complex syntax, the tool chain is very difficult to use, and even in the language design is somewhat contradictory (functional vs procedural). This means that very few people can understand and use the language very well, which leads to the creation of some unavoidable bad patterns: copy-paste code, wrong abstraction for some problems, and repetition of the wheel.

In this big environment, go has a chance. Clear (concise) high cohesion design, developer-friendly toolchain, generate native binary code (no platform-dependent runtime), close to C's runtime efficiency makes it ideal for SOA architecture development, especially when compared to the latest programming languages and related frameworks. In addition, Go has an excellent ecosystem and third-party libraries that do not match its age, and there are many success stories to consider. But Go also lacks a mature, comprehensive distributed Service Framework (Toolkit). I think the pit is going to be filled in soon:)

What Go needs

I think Go needs a toolkit like Finagle, Gokit, to become a language for the development of modern SOA architectures (another qualified candidate language outside of Java/scala)

Gokit

In my mind, Gokit has a set of interrelated packages that serve as the framework for developing a large SOA architecture. The toolkit should be comprehensive and contain components that implement a variety of important parts of a distributed system, including some common distributed system modules in its own framework, such as Finagle, and it should integrate some of the most common infrastructure components to reduce development barriers while enhancing communication with existing systems.

I think the toolkit should be a minimal set that contains only the core components, and I'll describe each of the components I envision, but it's only the smallest set I think I'm going to be, and I'll start a discussion here to gather the ideas.

Package Metrics

Indicator modules, the client and server need to be thoroughly instrumented (metrics can be measured in real time), which means that you need to collect such things as system memory, CPU usage, GC information, and so on. This also means collecting metrics for the service layer, such as request response time, internal queue length, latency for each process (such as pipelining), and the need for metrics that include business logic, such as the throughput of certain business events, or the exception of top-level business.

Package Log

Log module, different companies have different requirements for the log, such as the log level, schema, log storage location. I think the toolkit can be based on the log library of the standard library and then provide optional modules such as log level, schema definition and enhancement, and different log backend, log back end can support output stdin/stdout, can also output to Syslog, NSQ, Kafka This kind of system.

The log module of the toolkit should be able to serve all types of data in log form, not just the diagnostic information of the application.

Package Server

The server package, which is probably the largest and most important component. Ideally, we should be able to implement a service directly in the form of a Go interface. The server package internally needs to encapsulate some of the common requirements of the service, such as service monitoring, request tracing, connection pooling management, compression, throttling, and so on. For any of these function points, you also need to provide an interface to implement a pluggable strategy. The server library also needs to integrate the service Discovery module while running well in different transport modes. can refer to the implementation is: Fingle, Karyon and so on.

Package Client

The client package is a natural companion to the server package, and it should encapsulate many common patterns such as the client's speed limit, break-out protection, connection pooling, and so on, like a server package. It should also integrate with the Service Discovery module and work properly in different transports ways. A reference to the implementation is: Finagle, Ribbon and so on. As you can see, the client and server packages should be able to share code, such as connection pooling, speed limit, request tracing, etc.

Service Discovery (Services discovery)

Service discovery, connecting the client and the server in a dynamic (variable) network is a huge issue. There are many ways to achieve this, from static configuration IP to dynamic topology change subscriptions, and a variety of scenarios in between. This toolkit should be compatible with your company's existing basic services, at least support static hard-coded IP addresses, and support some common open source solutions such as Consul, ETCD, Smartstack (zookeeper) and other systems.

Package Transport

Communication modules, the implementation of the service should be decoupled from the way it communicates, and a separate service should support binding many different communication formats. The communication method here refers to a generalized concept: request (Reply) data serialization (deserialization), support such as JSON/GOB/PROTOBUF, Thrift, Avro and other formats, while the request (reply) of the channel can support Http/rpc/raw TCP.

Benefits

If our distributed systems really just rely on a set of conventions, we can use them as a simplified hypothesis and build a more understandable and reliable pattern. After all, simplicity is a precondition for reliability. Further: Imagine, using these conventions, the tools to get the runtime state of the entire system. It may allow us to measure the uniformity of a heterogeneous service instance. With this information, we can create dynamic systems that enable the system to react dynamically to changes in the environment. This will enable high-level patterns like load-based auto-scale, or secure continuous deployment without human involvement.

Just the benefits are enough to make up the taxes we pay back:) It also translates these basic components into real business value. And the future is open, there are many possibilities, and we don't know what it is.

Next

When I posted this talk at FOSDEM and Google Campus London, I received a lot of enthusiastic developers with similar ideas. I was inspired and I wanted to create a working group to explore these ideas and start implementing gokit.

I was motivated to do the job because I wanted to see the go language more successful and to introduce go into the field, and I believe I would benefit greatly from it. With luck, I think we'll have an early version of the build environment available in a few months. In the beginning, I'll create a mailing list where we can start collecting requirements and ideas, and organize initial development work. I also created Gokit.io as the official website, some packages that might later be used for host Gokit. Http://gokit.io If you are interested in these things, I hope you can join the mailing discussion group!