"Turn" the Road of technical transformation: from monolithic application to micro service, my Blood and tears summary

Technical transformation is the abbreviation of technological transformation, is the transformation of technology. Technical transformation, for the company and technical personnel are very rare, more participants, less dominant. I had the privilege to dominate the 3-time OTA system of technical transformation, scale has a small, each environment and problems, although not the same, but there are routines to follow.

"Technical Innovation Road" less talk about technology, we do not pay much attention to technical details and the implementation of middleware, and focus on the process of technological transformation and thinking, the following is the share of the topic:

• System background

• Pre-work

• Technical Transformation Implementation

• Summarize

1

System background

1. Technology scale

Company

• Domestic leading business-to-business air ticketing distribution platform

• Primitive accumulation of capital, good financial and always profitable

System size

• 200+ applications

• 100+ Library, 10,000 + table

Research and development Scale

• Developers about 200 people

• The server has about 200 units

This case is a medium-sized e-commerce company, the boss from scratch, the primitive accumulation of capital, now make money in the Internet companies very few oh. The company from the 2006 several research and development personnel, to the technical transformation before the 200 research and development personnel, the business develops well, is the domestic leading service-to-air ticket distribution platform, the Internet reputation is not small, but is in the state of stuffy sound hair big wealth.

Before the company tried 2 times the system reconstruction, asked a batch of cattle, before and after 4 years. The company consumes a lot, but it all ends in failure. This case is my 2nd time technical transformation, the effect is good, the overall progress smoothly, team skill level also has one or two grades of promotion, is a relatively successful practice. In addition, because the case is too real, some UML will be mosaic, please more understanding.

2. Single-Block application

3. Major issues

• Single-block application, the merger is not merged, the disassembly is not open, the single individual volume is unreasonable, the main platform volume is too large, and others too small;

• Technology is too old: the use of technology 7 years ago, the main platform using a single block application, and the volume is too large to update the overall maintenance;

• Multi-Version Coexistence: Version confusion, only dare to add, dare not modify;

• The whole system is very fragile, the problem is many, the number of visits is a big hang;

• Management issues: Publishing difficulties, testing difficulties, modification difficulties, troubleshooting difficulties.

Pre-work

1, the structure of the establishment

Establishment of the structure Department:

• Recruit several old programmers, Waizhao architects

Culture:

• Inside out, to improve the horizons; outside the cattle, please come in. Learn about history and business

System:

• Project Management + knowledge Sharing: Jira+wiki

• Team building, technology sharing, engineer culture

2. Master Plan

Is the architecture evolved or is it designed? For entrepreneurial scenarios, entrepreneurship itself is in the unknown to look for opportunities, will not be clear to become clear, the system structure is naturally evolved, and for technical transformation or Google search and other complex engineering scenarios, the system of course must be carefully planned.

The overall structure of the company's e-commerce system, we spent 1 months, the overall planning of the project, and then the promotion of internal propaganda, so that each participant to understand their own goals and values. How do you know to stand in the right place without a map?

3. Middleware Construction

The middleware we build is:

Job/redis/center Log/Business monitoring Metrics/dashboad/debug Tools Windbg/rabbitmq/orm Tools dapper/mongodb/jetermclient/public class library J Fx/zookeeper/opentsdb/hbase/searcher tools SOLR/Meta data Management Ddm/dll management NuGet/Auto release Jenkins/microservices architecture Jsoa/

Middleware is the infrastructure of application system and the equipment and tools of application. Rural housing is a piece of brick to the upper base, the city built Big House is the first to hit the foundation, and then build the main frame, the last is the base brick, so the construction of middleware is the premise of large and medium-sized system construction.

The construction process of the above-mentioned pieces runs through the entire technological life cycle, and each middleware may take up to 1-2 months, most of them based on open source. Keep an eye on the above-mentioned order, confront current issues, and build and drive on demand quickly. Although the use of open source, but between the introduction and transformation of the process has its own set of processes: Survey + = trial + + selection +-in-depth research + demo + wiki + share promotion = Business system trial = improve and improve Scale promotion.

The construction and increase of middleware not only has little impact on current business system, but also can solve some business problems and reduce the pressure of database. It also facilitates the establishment of technical atmosphere and sharing mechanisms. A passionate, love technology research and development team, the specific implementation of technical transformation is very important.

3

Technical Transformation Implementation

1. Database transformation

When faced with 100 multi-Libraries, I think the system Architect is concerned about the database level, and build the library. The database is migrated by module as a whole, but it is not as difficult as it should be, ideally only a database link should be modified, and for table and field optimization, it can be implemented by an application architect or a technical director, either SOA or application refactoring.

How the database can be scalable, can be small and convenient to split it, consider the following:

1. From the Inside Out, a box can be a library, it can be a module, it can also be a table, according to the current scale of business and system complexity to achieve;

2. Our large entity diagram specifically: products, users, orders, billing, infrastructure. They can be a library in the early stage, there are 5 modules, the medium can be divided into 5 libraries, the latter can be divided into more bottom level of the library;

3. Naming specification: Database name: Business line abbreviation + library name; module Name: reference Big E-R map + professional glossary abbreviation; table name: module abbreviation + table name; self-increment number: Table name +id;

4. Multi-table joins can be used in the module, the connection between modules is reduced, and the connection between databases is not allowed;

5. Each database has and only one owner group, in principle only allow one team to create, other team access needs to be hierarchical control, L1 interface, L2 as read-only library, L3 for direct read and write "Writing library".

Database planning

The database is the longest and most difficult part of the whole information system's life cycle. So let time to solve the problem of time, to strengthen the design, the specific implementation process is as follows:

1. After the map, the overall schema document was promoted, we created a new batch of libraries, which were also complained by DBAs in the early days.

2. After adding the relevant library, the new table is created according to the new rules, special circumstances go to the bead approval;

3. Go to SP to correlate, let the database reduce computation, return to the storage essence;

4. The database splits, changes the table to change the field, adopts the module whole migration or the application reconstruction;

5. After a year, we went to the database and found that nearly half of the tables were in the new library without special project and driver.

Data changes

State diagram is the change of data, it is the interaction between data and behavior, the change of data will cause the change of behavior, the change of behavior will produce different data. is the domestic order status change chart, its value not only belongs to the database layer, but also lies in SOA service and core business process.

2. Service transformation

Service is a verb, is an abstraction of behavior or activity, its value lies in the reuse of business logic or behavior, the implementation process is as follows:

1. Service lists and service agreements, using Excel tables in the design phase;

2. Unified Request/response Specification;

3. Service implementation, because there is no direct visibility of business value output, it is best to work orders or projects to the ground;

4. Service governance, early in the absence of tools, using wikis to do simple management, post-use professional service governance tools.

Domain model

No domain diagram architecture is bullying, we draw the domain diagram architect is 2 old staff, not much tall, even if they did not draw UML, but our state and domain models are from their hands. In fact, the key to drawing the field map is to understand the things themselves and to know their relationship. Our field map corresponds to 5 business process one by one in the business model, including: booking process, order processing process, product supply process, financial settlement process, account management process.

Micro-Service

Our microservices Jsoa V2.0 is based on Servicestack's current version number 4.0.50, which natively supports lightweight protocols and metadata, as well as swagger, an architectural implementation of microservices. In addition, it can be expanded to implement the open API in the way of API Gateway.

What is the difference between a microservices MSA and our previous SOA and ESB?

• The ESB has a bus and intelligent pipeline management capabilities;

• SOA weakens the middle channel and the bus, strengthens the two ends;

• MicroServices MSA uses a common lightweight protocol and more web (RESTFUL).

3. Transformation of Application Architecture

What is the system? System = element + relationship. What is the application architecture? Application Architecture = Application + architecture. Application is the smallest unit of the system, application classification and application number constitute the application relationship is the application of the framework, it is conducive to application management, interaction and tracking. The application is divided into product line, subsystem and application level 3, each level is numbered 2 bits, such as 100206. Application to from the user's perspective, the user first, then there is the application function, so that the user-centric to build the system.

4, organizational structure fine-tuning

The organizational structure has no best practices and is only appropriate for its own current choices, and the following is a reflection of organizational and technical architecture alignment:

1. Art and engineering related separation: UED;

2. Software development and hardware phase separation: operation and maintenance;

3. Technology development and business development phase separation: Architecture department;

4. requirements, implementation, acceptance phase separation: Each business Line sub-product group, Development Group, test group;

5. Development is separated by business responsibility: Reservation Group, product group, order group;

6. Professional Technical Committee system: testing, product, development, rotating chair, set up the Chairman;

Summarize

1. Process Summary

1. First Step master plan: Hand map, clear route;

2. The second step of the database: Build the library to remove the library, to join the SP;

3. The third step middleware: Build on demand, first increase the usual;

4. Fourth Step service: Technical transformation = Work order, with business value output;

5. Fifth Step application: Split application, build portal, reconstruct application;

6. Sixth step organizational structure fine-tuning: the frame technology and organizational structure alignment, after technical transformation adjustment;

7. Seventh Step Hardening: Framing, automating, and managing the process of tooling such as DevOps.

2. Experience perception

• Starting from the service is wrong, starting with the database or middleware is correct. Service belongs to the advanced stage, convenient behavior reuse, is a deep-seated optimization, but too slow;

• Starting from the current problem or fault, it is very important to put out the fire and reverse analyze the dump tool.

• History should respect, early changes can not be large, can not affect the existing business too much. It is recommended to only add, build new libraries and new middleware, so there will not be too much resistance and burden;

• Generally not all reconstruction, unless the system is small, the size of the system can only be divided after the split-step reconstruction;

• Technology is not the most complex in the process of technological transformation, people and things and relations is a troublesome part of the history of the people behind the problem;

• Every time the environment and problems are different, to be prepared to take off a layer of skin mentality.

3, the overall no coup

Technical transformation is a big toss, in the company in the individual is, small change to love, big change to the killing, we should avoid the big technical transformation, but this phenomenon is more common, especially the business development of fast start-up companies. So the real master chess, should be the overall no coup, so that the right thing is easy to happen, based on the evolution of nature to realize the evolution of technology.

How can the overall no coup, the system benign and long-term development? We need two forces, one is the technology, the other is the business, if only the business, and it is easy to overwork technically, if the full technology-driven, it is easy to forget the business objectives. Therefore, they should accompany and develop together, after the implementation of the large technical transformation, after the framework and process is relatively cured, small technical reconstruction projects should exist for a long time, so as to virtuous cycle, let the system into the state of natural evolution.

5

Interactive Quiz

Question: May I ask Mr. Zhang what you would do if you had another technical improvement? What do you think your biggest gain is in the process of technical transformation you have done? What do you think is a bad thing to do?

This question is very good, in order to better answer your question, I briefly introduce my 3 times technical transformation experience. My 1th Time technical transformation is reconstruction, project from October to August 2, history 10 months, can be said to be technical success project failure. The 2nd Time Technical transformation is reconstruction, just technology, juvenile and business, the overall effect is good, can be attributed to success. The 3rd Time Technical transformation or reconstruction, both tube technology and people, and business in the high-speed development period, less resources, can be summed up as technology and business companion, technical effect generally.

If there is a chance, since it is no longer directly responsible for, it is too tired, from inside and outside each recruit several architects, and according to the above work flow and way, then grasp the technology and business relationship and resource occupancy can be. Back to specific questions:

• If one more time, I will refer to the 2nd time Technical innovation experience, that is, ppt sharing process summary;

• After the transformation of the harvest is a reborn, technology and management have greatly improved;

• Bad place: To focus more on the business, and to balance the relationship between business and technology.

Question: What are the tricks of a smooth transition when a monolithic app migrates to a microservices? How to resolve distributed transactional consistency? Are there landing tools for continuous delivery, testing, monitoring (semantic monitoring) of microservices?

1. Smooth transition technology: Face the current system problems, and constantly have value output, and then refer to the above process summary, first planning, then middleware and database, and finally services and applications;

2. Distributed transactional consistency: Use alternative scenarios, such as eventual consistency;

3. Landing tools: There is no essential difference between MSA and SOA governance, or devops/trace/metrics/, we are using servicestack/jmetrics/centerlog/jenkins/.

Problem: If the old module related complex, but also affect the existing system performance, the related developers lost, bad grooming, the transformation of risk, rewrite the boss do not agree, how to choose?

• The old module has complex correlation and affects the existing system performance: extinguishing the current fault and analyzing the dump tool in reverse;

• Related developer Churn: the introduction of middle pieces, the establishment of sharing mechanism and learning team, talk about the overall planning of technical transformation, so that everyone understand their own values and goals;

• Bad grooming, the transformation has the risk: recruit several old staff to become the application architect;

• Rewrite the boss do not agree: There is business value input, technical transformation = = Work order or project, with the help of business projects to achieve technical transformation.

"Turn" the Road of technical transformation: from monolithic application to micro service, my Blood and tears summary