The path to the evolution of today's headline structure

Xiahong, today's headline architect, focuses on high-performance large-scale WEB architectures, cloud computing, performance optimization, programming language theory and other directions, PHP COMMITTER,HHVM Project contributors. 2009 joined Baidu, has engaged in large-scale IDC self-operation and maintenance facilities, cloud computing platform Architecture Design, paste business performance optimization, Baidu General RPC design and optimization. 2015 join today's headlines responsible for infrastructure, system architecture design and optimization to solve the problems of system performance, reliability and operational efficiency under high traffic and concurrency.

Today we share the evolution of today's headlines, with the previous lecturers speaking a lot of specific dry goods, my share of infrastructure and architectural ideas, and our idea is to help the architecture do better iterations by providing better infrastructure.

From an architectural point of view, the pressure of the technical team is mainly from three aspects:

1. service stability . The stability of the interface to make the service more reliable;
2. iteration Speed . Iteration speed for large companies is relatively less important, large scale, survival pressure is relatively small, but relatively small and medium-sized companies, the iteration speed is to be ensured, time window is also a decision whether the success of the important factor;
3. Quality of service . Focusing on user satisfaction, it is also a particularly important topic.

Today's headlines are particularly fast, with only 4 years of history, from the number of people and the size of the growth of very fast, the stability of the availability of a large pressure, on the one hand need to quickly implement the business, but on the other hand, similar to these high-availability issues will often harass engineers: on-line hanging, operating activity of large service collapse Single-machine performance, a small service on-line to the core services to hang up ... Like these questions, how does the technical team need to respond better?

In addition to my understanding of the evolution of architecture, companies in different stages will face various pressures. Small company pressure may be the business is not up, the QPS is very low, to do optimization there is no environment and conditions; When the company is big, the server may not be a problem, but you need to constantly consider tuning and coping with access pressures, improving infrastructure to provide a more stable development environment. So the evolution of architecture is a process that lasts and has no end.

Why is there so much pressure on today's headlines? Today's headline growth rate is relatively fast, it can be seen that the company has been 4 years, 2014-2016 each year is DAU doubling. This business challenge is very large, scale up, our original architecture is difficult to do linear expansion, some can linearly expand the service, the problem is more, business growth is too fast, the back-end pressure is larger.

A brief history of the headline architecture: three historical stages

How does the structure of today's headlines evolve?

Never a perfect architecture can always be supported, the architecture is dynamic system, real-time change, because of quantitative changes in the quality of change, different stages require different architectures .

When do you need to make an architectural transformation? When suddenly found that more and more system problems, frequent accidents or alarm particularly much, communication efficiency and so on, it is possible that your architecture problems.

There is a problem with the software architecture, which has a relatively long period of change. Structure of the mode of thinking set down, with the growth of business, the burden is getting bigger. People who have done the infrastructure have this experience: it's easy to have a good idea, but there's a lot of difficulty in doing a great software. The technical transformation is long, in the year unit. So this time only allows the architecture to iterate a little faster. Finally, do not attempt to do a particularly perfect architecture, we just need to maintain the agile evolution is good.

The architecture will inevitably deteriorate.

Headline first stage: three-storey structure

When the headlines started today, it was a simple WEB application, a database, and a business implementation. The first advantage of the headline is the recommendation engine, and another set of data mining and offline computing. The online service is relatively clear in the front-end mode, and the three layer is done. When the business starts up, there's nothing wrong with the increased level of access that can be solved.

The second stage of the headline: Split

Very similar to most of the company's architectural evolution history, when the last version encountered some performance problems, the simplest to do some of the split. In the process of optimization, the piece is too heavy to be split from the code. , A, B and C are different business, just start the code is together, the process of evolution, iterative one year or two years of product, heterogeneous disassembly is actually quite painful.

The structure of the previous era, basically did not consider too many people or the size of the development, just beginning there is no special person to do the architecture optimization, a lot of people are flapping in business, the function points added. For example, the recommended effect is not good, strengthen the recommendation, each piece has no special person to consider the overall structure to how to organize planning.

By the end of last year, every quarter of the budget, to the second month the machine ran out. At peak times there are 60% to 70% pressures, and there are two problems: the first, some of which are performance decay, and the other, which is too much pressure on the business.

The architecture team needs to find ways to get faster, even if there is an access problem, the pressure, the machine is not enough, but also to ensure that our services. Business has been fast forward, the burden is relatively heavy, the cost of transformation is relatively high. Based on these problems, we discuss the next stage of thinking, do micro-services.

The third stage of the headline: micro-Services

At present, our idea is to make a new architecture through micro-service mode. By splitting into subsystems, large applications are split into small applications, and abstract generic layers are used for code reuse.

(click image to zoom in small)

The hierarchical comparison of the system is typical. We focus on infrastructure to improve rapid iteration, disaster recovery, and a range of tasks through our infrastructure, and we hope that business teams can do business iterations and architectural tweaks faster.

Micro-Service Architecture

Micro-service We think the most critical three points

1. Decoupling, a service relies on the concept of another service, module, or sub-service.
2. Light weight, reduce maintenance personnel costs.
3. Easy to manage.

The key to micro-service in reality is autonomy. Although microservices are self-contained, they also need to have a hierarchy. For example, the service you provide is provided by the outside company, the services provided by Weibo, you can not request the microblogging service for you to make changes. MicroServices need to have boundaries, at the corporate level, not to be too independent, too independent will increase the cost of communication. Infrastructure and specifications should be reused.

What is the real micro-service?

• The architecture has to be landed into something that is figurative. Micro-Service has a development framework, do business students do not need to care about disaster, do not need to do such a thing repeatedly, how to deploy this thing, they do not care;
• Need to have a process normative to constrain. The overall optimization can be done with the specification;
• The manifestation of microservices is to provide a platform or tools.

The present and future of the headline service

Finally, we will introduce the idea of service in front of the today's headlines is how to carry out? How to provide services to all business team developers?

The main services of the headlines are as follows:

• regulations . What does the specification do? Deploying RPC, how does one service tune up another service? Innovation I think there is no problem, but you have to consider the cost to other people, this specification still need to have, this can do global control. Service stability and unity, you have to consider it to bring real advantages, high performance is a point, but local priority will be better;
• laying the groundwork . With the specification, the service begins to actually fall into the ground. For example, the base library, the Ngnix, Redis, MySQL these libraries packaged together to do something. Development framework, you do not have to focus on data optimization;
• Progressive . First, detach and iterate, and optimize the service in.
• Everything is a service , 4th, and other companies or teams slightly different places, our idea is that everything is a service, each node is an abstraction attributed to a specific service. Storage is really a service, but it is not just to provide API or provide functionality, but also need to include the quality of service, need others to use is relatively simple;
• platform-based . The final landing is a platform-based thing. How is our framework designed, and how does the service combine?

Primary specification: Everything is a service

• Resources are limited: on-demand applications, need to apply and authorize;
• Simple to use: Developers only need to focus on the business;
• There is only one way to locate: using global resources to locate;
• Finally, each service has an owner (owner), a partial engineering architecture aspect, and my specifications must be executable.

Our specifications

• Must have the global center, the service unified registration to consul;
• The service has a unique label, name: {product line}. {subsystem}.  Module P.S.M, the company has many departments, we do not want to communicate between departments have differences, so need to have a global plan to trace it;
• Business services use Thrift to describe interfaces and must pass standard parameters. If the data is weakly described, there is no strong constraint, and the data on the client may be typed incorrectly;
• RPC uses a unified convergence library;
• Nginx, Redis, MC, MySQL, etc are all services

Service Registration

Our services are unified using loader or wrapper scripts to start, and the specific startup is determined by the business.

Service startup will have a name, the app registered to the service inside, there seems to be some constraints, database MySQL can be started? Can Redis do that?

Start, the service mode does not have to pipe, with the same framework, a new specification, easy to migrate existing services, but this is not a particularly strong specification, considering the cost of migration. light specification, easy to migrate.

Service Center

Service Center has service information, together with what kind of service, others relatively simple tune this service is OK. The quality of service that this service provides, the owner can manage this information. Redis goes to service, load balance, service a project, and connect the service up.

Service Relationship and authorization

There is a key concept between services: Service authorization. In general we have a service, through the IP can be connected to it. The database has a user name authentication, and IP authorization can also be granted. However, many service restrictions on the intranet are relatively small, not all services are authorized certification. We want to record and execute the relationship between the services and the global topology.

A service provider interface, we can be authorized by the owner, other services authorized to access it.

Description information: What does this service look like? What is the maximum QPS? By describing the information to find the problem, the user information Service can not hold up, refused, the resources to other services above, you could do more things. and the computer room information can be placed in this area.

Service Authorization Certification Ideas:

• Based on service labeling, important services add more authentication methods;
• Collaborative authentication, the client itself assists the authentication.

Give an example of a Thrift. There are two dashed lines on both sides, the service center has a strong ability to expand the level, to it to basic licensing information, can I transfer this service? The default is yes, is a Thrift package, I know who you are, do your own strategy, service pack brought over. Request to bring up, analysis call is not a problem, this is part of the specification. The development of the students is not concerned about how the framework to do this.

Another way to call the service center is to reject you. The QPS is under pressure and can't support you. One benefit is that you can avoid wasting resources and, in addition, virtualize the Docker link. Previous ideas by IP authorization, each IP control, providing similar to anonymous services, according to the node's IP to do. Now use Docker to take a logo is not very good to do, in the network layer is not too good to do, in the intranet environment has certain credibility, I consciously tell you, I am who, and then call.

MySQL is currently doing a scenario see, unlike the Redis requirement to bring on who you are, call MySQL needs to put callers who are brought up. An important database, definitely do security authorization, I just said under normal circumstances. These kinds of ways are superimposed to do, bring the original information, Redis brought over, do a weighted check.

Redis does not work on the protocol layer, and MySQL adds the above information in the call without affecting semantics. Our server provides the HTTP interface to provide this information in the HTTP header for authorization authentication.

There is an authorization relationship, and all services constitute a topological relationship of the full service. A service is pre-authorized to tune it. If the real topological relationship on the wired, it can do alarm optimization. Redis Alarm, MySQL alarm, there is such a topology, will improve the speed of the problem tracing.

We have such topological information, know the global meta-information of the service, we can better do the service change impact assessment and alarm and so on optimization.

RPC Development Framework

We have developed an RPC framework ourselves. The development framework will help us develop the code, and this is something that many people are doing. Its main features include:

• Rapid development: code generation;
• Service discovery: understanding of service;
• Observability (observability): Logid, pprof, admin port;
• Disaster tolerance downgrade: business downgrade switch;
• Overload protection: Circuit breaker, frequency control;
• Multi-lingual Support: Python/go

For example, observability is that all services can expose the internal state, which has a very good advantage, after the service, the default analysis of the internal port or service port, service on-line and platform. Based on the topological relationship, the service state is automatically analyzed, and even the performance analysis is done, so that developers can naturally acquire these abilities without caring for these things.

There are disaster-tolerant downgrades and overload protection, and we have a platform to manage relationships and downgrades so you can focus more on your business.

Below are the approximate modules, which are modular, rather than embedded in the frame, to make our maintenance costs even lower.

Front service is the embodiment of self-government, compared with Docker, we will also do containerized development. Just to run the service in the container is not enough, the service system to open up, our ideas to achieve openness, to achieve our "attitude" of the private cloud, the infrastructure of this piece let us do the platform, the business department only care about business.

We are currently at this stage, doing a service refactoring, building our private cloud. The front frame,

Continue to iterate.

Finally, how do we plan and virtualize the PaaS platform?

We achieve it through the three tier, unified management through the PaaS platform. Provides generic SaaS services while providing a common App execution engine. At the bottom is the IaaS layer.

IaaS manages all the machines, integrates the public cloud, headlines there are some hot events to promote the national push, on the network bandwidth is high, we rely on the public cloud, which type of computing resources need, unified abstraction together. Infrastructure combined with services, such as logging, monitoring and other functions, the business does not need to pay attention to the details of the infrastructure to enjoy the ability to provide.

Q&aq: How do you think about the increase in the cost of splitting services into microservices?

Xiahong: I built a database in the past and ran straight up. Once the entire library upgrade, now only need to upgrade a small portion of the business compared to a simple scale smaller when the single service is really low cost. When your business increases the number of machines, the monomer service will become a bottleneck, but if the micro-services are standardized, can be automated tools, platform to manage, cannot rely on people to manage, so the cost is reduced.

Q: The service is running in the container, using the consul, its own container and IP and other information to register to consul, update your authorized ACL?

Xiahong: This is really a thought, we used the consul is to center, but also added a layer. If you need to control the access and security of the microservices, the container nodes also have levels, such as I will be divided into small clusters, the physical layer is isolated, in this way to achieve security. It is not enough to be consul alone.

What is Q:RPC service discovery? is RPC implemented by itself?

Xiahong: Service discovery is CONSUL;RPC is the implementation of their own on the basis of Thrift, service calls also implemented a fuse mechanism.

Q: Why not open source when choosing the model, we are the whole platform of the architecture to do micro-service transformation, want to choose the service architecture, you do this piece of your own, we in the open source and do the choice between ourselves. Can you give me an example?

Xiahong: Look at the scene, you're not doing anything now. You can consider a variety of open source solutions, we also have some of our own special scenarios, open source and internal services to integrate, need to consider some of the cost of integration and our own maintenance costs. Many times the open source project will be based on universality, will consider the characteristics of more, the code will be relatively complex, some features we do not use, we have to do the transformation, the overall is not complex;

Authorization standards are also self-made, based on the service identity, the server inside, and does not consider the scene interconnection.

Q: Now if we do micro-service platform transformation, business system development model is not a relatively big change? Our platform from the development model to the design will be changed, you have been modified, what experience do you have?

Xiahong: We have not reformed until now, this transformation is very difficult. You first set a general direction, because a lot of things are related to the problem of communication, to promote the problem, you need to communicate the direction of good, to achieve agreement, to how to transform, less mobile, or you put most of the functions to achieve good, only need to do a small migration, reduce the cost of migration.

Http://h2ex.com/1235?utm_source=tuicool&utm_medium=referral

The path to the evolution of today's headline structure