Google and ebay micro-service experience

Excerpt from: http://www.infoq.com/cn/articles/ecosystems-of-microservices

Diversified (polyglot) micro-service is the ultimate game

• Large-scale systems and diversified micro-services will eventually be implicated. Among them, pluralism means to write together in multiple languages.
  • ebay, which was created in 1995, experienced 3 architectural changes.
    • Ebay first adopted the Perl program that the founders had spent two days writing.
    • After that, it starts using a program written by 3.4 million lines of C + + code.
    • Then, ebay uses a distributed system written in Java.
    • Today's ebay still uses a lot of Java, and also uses a lot of multi-code to write a variety of micro-services.
  • The schema change process for Twitter is similar.
    • First, Twitter was using Ruby on Rails.
    • After that, it started using JavaScript and rails at the front end, and the latter used a lot of Scala code.
    • Currently, Twitter is also dependent on a diversified micro-service.
  • Amazon's schema changes are similarly similar.
    • First, it uses a C + + program.
    • Then, Java and Scala are used in two languages.
    • Currently, Amazon also relies on a diversified micro-service.

Ecosystem of services

• How does a large-scale ecosystem with a diversified micro-service operate?
  • Ebay and Google use hundreds of to thousands of separate services to work together.
  • Today's large-scale systems are in the form of graphs rather than hierarchical or multiple connections to form services.
  • interdependence between services.
  • Only older large-scale systems are organized in a highly integrated manner.

How to create a service ecosystem

• Currently well-functioning systems are the product of constant change. For example, Google has never had a centralized design of the system. The current system is purely adapted to the evolution of the times and technological developments.
• Mutation and natural selection. When a new problem arises, the engineer usually chooses to use the existing product or service to solve it. Therefore, a service can avoid the fate of being eliminated only if it provides value continuously and is used continuously.
• These large-scale systems employ a bottom-up design approach.

• Take some of the services used in Google App Enginer (GAE) as an example:

  • Cloud datastore nesting relies on the following services: Megastore (a structured database) →bigtable (a structured service at the cluster level) →colossus (a clustered file system) →borg (Cluster management framework).
  • The hierarchical relationship is very clear. Each layer adds something that doesn't exist in the next layer.
  • It is designed with a bottom-up approach. The first is to build the Colossus file system and finally the cloud Datastore.

• This is not an architect's architecture. In Goolge, most of the technical decisions are taken by small teams at their own point of view, rather than the global perspective.

• ebay in the 2004 was diametrically opposed to it. At the time, ebay adopted a form of architectural review to make decisions on all large-scale projects. The result is:

  • Usually the censor is involved when the project cannot be changed.
  • Centralized decision-making quickly became a bottleneck. Its biggest impact is to cast a veto at the last minute.

• Instead, ebay then put the idea of an experienced engineer on the review board and then executed the projects that many small teams would use. Encapsulate these ideas as a library, a service, or some guide that employees can use themselves, rather than making decisions at the last minute.

How to define standards without architects ' change

• Standardization can be done without centralized control.
  • Standardization tends to arise in communication between different services or architectures.
• Communications are typically standardized to:
  • Network protocol. Google uses the stubby protocol, and ebay uses the rest protocol.
  • Data format. Google uses protocol Buffer, while ebay uses JSON.
  • interface Syntax Standard. Google uses protocol Buffer, which is JSON by its own syntax.
• Common modules that are typically standardized in a framework include:
  • Source control.
  • Configuration management.
  • Cluster Manager.
  • Monitoring System.
  • Warning System.
  • Diagnostic tools
  • All components that can be detached from the traditional.
• In a transformative environment, standards are enforced: coding, submission, code review, and code retrieval.
• The best way to encourage best practices is to speak in real code. This is not a top-down review or avant-garde design, but someone who can write the code to get the job done as soon as possible.
• Encouragement is done through the way the team provides the library.
• Encouragement is also done through the services that the engineer relies on in support of the X protocol or the Y protocol.
• Google is well known for its code: any line of code that is under source control is reviewed by at least one additional coder. This is a good way for you to communicate your experience.
• In most cases, every engineer in Google can retrieve the entire code base. The code base can provide a good reference when programmers plan how to implement a thing. In the case of 10,000 engineers who are constantly working, what a programmer wants to do may have been done by someone else ahead of time. This makes it possible for a good project to be propagated through the code base. Of course, errors in the code can also be propagated.
• In order to encourage generic projects and standardize habits, it is necessary to make the right things simple and to do the wrong things a lot harder.
• Services are independent of each other.
  • At Google, there is no standardization implementation within the service. For the outside, the service is a black box.
  • There are habits and common libraries, but there are no programming language requirements. The four most common languages are: C + +, Go, Java, and Python. Many different services are also written in a variety of languages.
  • There is no standardization work for frameworks or persistence mechanisms.
• In a mature service ecosystem, standardization should be the radian of the graph, not the node itself. Defines a common shape, rather than a generic implementation.

Create a new service

• New services are created only when their use is confirmed.
• Typically, a feature is designed for a particular usage scenario. The feature is then found to be generic and effective.
  • Form a team, and then the service expands to its own separate unit.
  • This happens only when a feature is successful and applies to multiple usage scenarios.
• These architectures are developed through a pragmatic approach. No one can command that a service should be added.
  • Google File system supports search engines. Distributed file systems are obviously easier to become generic.
  • BigTable initially supported the search engine, but it was not widely used.
  • Megastore was designed to be used as a storage mechanism for Google Apps.
  • Gae was initially built by a group of engineers to help design the site.
  • Gmail comes from a project that is often used internally by Google, and is then promoted to external use.

Deprecation of outdated services

• What if a service is no longer being used?
  • Techniques that can be redefined can be reused.
  • Engineers can be expelled or assigned to other groups.
  • Google Wave is not a successful market case, but some of these technologies are applied to Google Apps. For example, apps that support multi-person editing documents come from wave.
  • Typically, the core service undergoes multiple upgrades, and the old version is discarded. That's what happens within Google.

Build a service

• What happens when a service owner builds a system of massively diversified microservices?
• Performing a good service in a large-scale framework should be:
  • Purpose single. It should have a well-defined interface.
  • Modularity and independence. The so-called microservices.
  • You should not share a persistent layer.

What is the goal of the service owner?

• Meet customer needs. Provide the necessary functionality at the right level of quality, while meeting the performance of the Protocol, maintaining stability and reliability, and continuous improvement services.
• Meet demand at the lowest cost and effort.
  • The objective is motivated by encouraging the use of the universal framework.
  • Each team has limited resources, so you should use common tools, processes, components, and services.
  • It also encourages good operational behavior. The building and deployment of automated services.
  • It also encourages efficient use of resources.

What is the responsibility of the service owner?

• Who builds who runs.
  • Typically, a team, especially a small team, is responsible for the design, development, deployment, and eventual destruction of the service.
  • No additional OPS team.
  • The team has the freedom to choose their own technology, methods and work environment.
  • The team is responsible for their choices.
• Serves as a limited context.
  • The cognitive load of a team is limited.
  • There is no need to understand other services in the ecosystem.
  • A team needs to have a deep understanding of its services and the services they depend on.
  • This means that the size of the team can be small and flexible. A typical team is 3-5 people. (Like the U.S. Marines, a squad consists of 4 soldiers.) ）
  • Such a small team size means that communication within the team can be very smooth.
  • The Conway rule can play a very good advantage. Small teams often build small components.

What are the relationships between different services?

• Even within the same company, the relationship between different services is like the relationship between the vendor and the customer.
• Try to be friendly and cooperative, but be cautious about the relationship between services.
• Be sure to be aware of your affiliation.
• Be sure to identify each individual's division of labor. Define an explicit interface and maintain it well.
• Customers can choose whether or not to use the service as an effective means of adjusting incentives. Encourage services to evolve in the right direction through our customers. This is also one of the ways in which new services are built.
• Define SLAs. Because the service provider provides a certain guarantee to the customer, the customer can rely on the service.
• The team using the service will be charged for the service.
  • Paying for services can bring financial incentives. It can stimulate the efficient use of resources by providers and customers.
  • When things are free, people don't always know how to cherish them.
  • For example, an internal customer used Gae for free and used a lot of resources. Requiring them to use resources more efficiently is proving ineffective. However, a week after the introduction of the charging mechanism, they will soon be able to reduce the resource consumption of Gae 90% by a simple 1-2-point modification.
  • Teams that don't use gae don't know how to cherish them. They have their own areas of concern, so optimizing the use of Gae is not in their target list. However, it turns out that a more efficient architecture can lead to better response times.
  • Payment can also stimulate service providers to ensure service quality. Otherwise, an internal customer may turn to other services. Pay directly to bring good development and management projects. Code review is an example. Google's ultra-large-scale design and testing system is another example. Google runs millions of automated tests every day. Once the code is received by the repository, the receive test for all relevant code is run. This method can help the small team to maintain the service quality very well.
  • A rebate model can be a good incentive for small iterations to change. Small changes are easier to understand. The effect of code changes is non-linear. 1000 lines of code change the risk of not only 100 lines of code brought about 10 times times the risk, but about 100 times times.
• Maintains full forward/reverse compatibility of interfaces.
  • Never break the client code.
  • This means maintaining multiple interface versions. In some cases, it means maintaining multiple deployments: one for the new version and the other for the old version.
  • Because of small incremental changes, the model interface is not usually modified.
• has an explicit discard policy. The service provider proactively upgrades all users from version N to version n+1.

Operation of large-scale services

• As a service provider, what is the service in a large-scale system that operates a diversified micro-service?
  • Predictable performance is the basic requirement.
    • Large-scale services are prone to performance changes.
    • The predictability of performance is far more important than average performance.
    • There is no guarantee that the low latency of performance is not a low latency at all.
    • When services provide predictable performance, it is easier for customers to program for services.
    • When a service uses many other services to do the work, the service's performance is determined by the most deferred services.
    • Example of a service with an average delay of 1ms and a maximum delay of 0.001% for a probability of 1s:
    • Calling the service once means that the 0.01% probability time is delayed.
    • Then, for a large-scale service that Google uses for 5000 machines, the delay time probability is 50%.
    • For example, a memory cache-related problem is traced to a redistribution event of the underlying data structure with a probability of one out of 10,000. However, when there is a delay in large jitter, the problem arises at a higher level. However, the underlying details are important for large-scale systems.
  • Depth of reliability.
    • Service outages are generally caused by human actions, not hardware or software errors.
    • Be sure to address the failures of machines, clusters, and data centers.
    • Load balancing and flow control when using other services
    • Supports fast rollback operations.
    • Incremental deployment.
    • Do not deploy to all machines at once. Select a system, deploy the new version of the software, and observe how the system works.
    • If it works well, expand the size of the deployed machine to 10%, then 20%, and finally all the machines.
    • If there is a problem deploying to 50%, you should be able to roll back the operation.
    • Ebay leverages feature options to decouple code deployment and feature deployment. Typically, you first close the feature, then deploy the code, and then choose to turn the feature on or off. This allows the code to be deployed correctly before the new feature is opened. At the same time, this also means that if a new feature has bugs, performance issues, or business problems, the feature can be turned off without deploying new code.
  • There may be more warnings, but there will never be more monitoring.

Anti-pattern Service

• Super Service:
  • In fact, the customer wants a small service ecosystem.
  • The super service is difficult to clarify, expand, change, but also constructs more upstream and downstream dependencies.
• Shared persistence
  • In a hierarchical model, the service is placed at the application layer, while the persistence layer is a generic service provided to the application.
  • ebay also uses the same approach, but it doesn't work. It destroys the encapsulation of the service. The application can "black" into the service by upgrading the database. It will eventually re-introduce multiple services. The shared database does not allow loosely coupled services.
  • MicroServices prevent the problem by becoming small, isolated, and independent, and in this way to ensure healthy ecosystem growth.

Google and ebay micro-service experience