Development test environment operation and maintenance practice based on. NET micro-service architecture

For now, the most popular architecture for Internet applications is microservices, and one of the hottest research and development management is devops. MicroServices and DevOps have been heavily used, and they have been as legendary as they can be. Special call Cloud platform, through nearly two years of practice, found that it is not as simple as everyone said, we are bad news, is really water too deep, who do know. Share with you today some operational pain points and solutions for developing test environments under the +devops of microservices architectures.

The complexity of the architecture directly determines the workload of operations, the architecture is not the more complex the better, but the best. Here's a brief talk about the pros and cons of several architectures. When building applications based on. NET, the most common approach is to use ASP. The front-end, back-end all-in to a site that saves time and effort without worrying about the complexity of deploying operations. But the disadvantage is also very obvious, all content is deployed under a site, if the business complex, the system change frequency is very high, the stability is worrying, basically no solution. What is more complicated is the front-end separation: H5 (or WinForm) + WebAPI, which separates the changes in the front and back, but the backend logic is missing, there are a lot of public methods or duplicate code. More complicated is: the front-end +webapi+service, this mode, although the public services are extracted, but the granularity of deployment is still very coarse, basically will be in accordance with the business scope of multi-cluster deployment. Under the same cluster, the deployment of services if too many, assembly conflicts, service changes to restart the cluster has a large scope of the problem is still difficult to solve. Therefore, in order to isolate the change and reduce the impact on other services, the granularity of cluster will be smaller, and even evolve into a service cluster, which is the form of micro-services. These architectural patterns are summed up as horizontal, vertical two dimensions change, horizontal dimension from a type of site into a multi-class site, to address the impact of changes in Access, code to take the problem. Vertical dimensions are deployed from all applications in a single site, changing to a service cluster, isolating the impact of change. The architecture evolves from one point to two dimensions, resulting in an exponential increase in operational cost. Here is the special call Cloud Platform MicroServices architecture diagram, you can see that the overall architecture is more complex.

In order to support the national charging business, special call Shuyun currently has nearly thousands of servers, application 100 class +,webapi interface 2000+, service interface, the development test environment dozens of, only the generation environment of the daily hot update will be up to 20 times +. More than 20 hot updates, which must be unit tested, are deployed to interface testing, UI testing in a test environment that is nearly consistent with the production environment, and then regression testing in a quasi-production environment, which can eventually be published in grayscale to two data centers. When it comes to this, it's likely that you'll want to standardize the synchronization of your environment through DevOps: When CI is complete, the product updates are deployed to multiple environments for testing by CD and then released to the production environment. Yes, normally this process is fine, but the reality is very brutal. There are too many factors that cause the test environment to be inconsistent with production. Here is a brief talk about:

1. The. NET Framework cannot take Docker, and update packages are not only updates to program files, but also configured updates and SQL updates. On such a large scale, the high cost of manual updating is outrageous, and basically needs to be dedicated to do. It is easy to make mistakes by doing it manually. must be tool-based, automated, patch update must be 100% through the tool, there can be no manual intervention, otherwise there will be inconsistent in each environment .
2. The system changes almost every hour, the common increase or decrease the application, increase or decrease the service, increase or decrease the WEBAPI, these information changes will affect the system environment. As long as a program, interface, service management is not in place, the system may give you color to see. All machine information, service information, configuration information must be centrally managed and maintained, and automatically synchronized in a variety of environments. The CMDB is a necessary management system.
3. In the daily research and development, many requirements will involve the joint development of multiple product development departments, integration testing cycle is very long, and the number of test environment is limited, there are often some urgent needs without test environment embarrassing problem.
4. The test environment will perform frequent updates, and even one update will be repeated several times, which can easily lead to a mismatch between the test environment and the production environment. This causes, the program hot update has the bug, needs the urgent fallback.
5. The development test environment is open to every developer, and everyone logs on to the system Debug. You know, as long as the debug, the program has a large probability of change.
6. A business may require several or even more than 10 programs to provide services in order to function properly, one link problem, the whole system will be wrong. How to analyze and troubleshoot the problem quickly is a painful problem. This is a problem that makes a lot of people crazy.
7. 。。。

In the face of so many changes, DevOps becomes ideal and powerless. The landing of DevOps requires a balance to be found in continual improvement. Our solution is to:

• Build CMDB system to manage the deployment information of various environments. For example: cluster information, process information, service information, WEBAPI information, configuration information and so on. And the change of this information to be centralized management through the system, never allow the CMDB information and deployment information inconsistent situation.
• Build a patch system. Develop delivery package standardization management. Patch installation is done through patch authoring tools, formatted patches, and patch-installed platforms. System programs, SQL changes all through the patch platform. The patch system is a very complex system, followed by an opportunity to introduce it in detail.

• Set up the template machine environment, when the production system updated patches or changes in the CMDB information, through automated tools, the initiative to push to the template machine. Ensure real-time synchronization of production environment and test environment. Various test environments are cloned from the template machine and are synchronized with the tool and the template machine timing. Once synchronization is complete, the environment is determined to be available through automated test tools and environmental inspection tools.

• Develop a full-link monitoring system and embed points at various entrances to the system to help developers analyze system problems. The full-link monitoring system is also a very complex system, followed by the opportunity to introduce in detail. Below is the schematic diagram and operation of the full link.

To do Internet applications need to have genes, more need to have pits courage and perseverance, pits process is to climb the process of technological peaks, endless! This article is only from the aspect of the architecture to share, but all the content has been landed, not the meaning of blowing NB. Hope this technology sharing can be useful for everyone!

Development test environment operation and maintenance practice based on. NET micro-service architecture