The way of Dark architecture--system upgrade in time and effort

CTO Cory Von Wallenstein in Dyn's main responsibility for the project is to develop the company's IaaS platform technology for decision-making, development and the direction of innovation. His article on Wired describes the way to save time and effort to upgrade the system-Dark architecture(Dark Architecture). The following is a translation:

Growth in business is a goal pursued by any company, but growth is not just your technology infrastructure that needs to be constantly evolving in every sector of the company, and many companies have been using very primitive ways to do this, although there are very simple solutions.

And the company upgrades its infrastructure and system architecture for the following 3 reasons:

For size--to get the capacity of the present 10 times times or hundredfold

For performance--trying to remove a system bottleneck so that you can perform work more quickly

To understand decoupling-to achieve better reliability and maintainability while also striving for future expansion

The earlier approach was total negation: start with a 3-month plan, finish the plan, and the developer starts to build the new system in the hope that it will be completed in three months time. But the reality is that 3 months becomes 6 months (inevitably longer than expected), and that the business moves from the old system to the new system also faces very high risks. When the new system 100% implements all the features of the old system, legacy systems can only be discarded.

Given the lack of prioritization and other improvements in the old system during the six-month development period, there is no doubt that the business has lost its flexibility. This update is like "all" or "nothing" proposition, and until the development team completes the new system and moves all the business to the new system, the promotion of the business is essentially zero. In the face of complaints from customers, the technical team's morale has slipped and slipped. The areas where this approach needs improvement are mainly due to 3 aspects:

Bring value to customers right away-we don't have to wait until the whole new system is finished and put into use to realize value

Reduce the risk of product introductions – we need to avoid "all" or "nothing" migration plans

Keep your business flexible-you can choose priority, at least to ensure timely delivery of core value (if you can choose, we're going to be able to solve and deliver the relatively important solution as quickly as possible)

For the above solution: the way to achieve the dark architecture

The Dark Architecture (Dark architecture) is not only an idea, but also a technical solution to address scale, performance, and coupling issues, and the dark architecture not only makes hardware a success but also allows employees to have a clear mind. You can do this in the following ways:

Prioritize the workflow in the system instead of changing the components of the system

Parallel operation of the old system and the dark architecture

Send input data to both systems, collect output data, compare output results, and discard a

Using the dark architecture approach, the operation will be as follows:

1. Before contacting the code and system, optimize the data flow in the system in the following order: short boards, opportunities, business values, or other things that are proven to be meaningful to the business.

2. Consider component-related (for example, using Cassandra to replace MySQL in the original system) rather than a streaming system (for example, a wildcard query type graph rendering for customer X takes about 40 seconds, and a much faster rendering type with other graphs). This step allows you to see the drawbacks in the system, so you can focus on the bottleneck solution first, and then consider other issues.

3. If the flow optimization is used first (assuming that the function of optimization is 2% of the entire system), the fact is that the next step is not to directly build the functionality. Instead, a bracket is built for these functions, allowing the system to be computed in two different ways at the same time (in order to compare the output and record the difference).

Dark Architecture Specific Practice plan

From the actual situation, this could be a copy of a network service call (old and new) or a copy of the database interaction call (the same old one), then compare the return value and, if there is a difference, record it to a file or server or message bus.

Once the input/output bracket is in place, you can begin to set up the function. To achieve the shortest board in the system 2% function, ready, put it into the product of the dark structure. It will receive the input data of the product with the old system, although it will be discarded in the end, but in this process we can compare the output values of the old system. If the results of the output are different, record and check. This allows you to become more familiar with your new system and gain some practical experience in the process.

Once the new system is determined to work properly and has achieved desired performance or extensibility, the output is converted to the new system. In this way, the implementation of the system is only 2% of the shortest board function, other functions are inherited from the old system. This is equal to the ability to change 2%, and it takes a small amount of time to deliver the value of the entire new system.

Benefits of using Dark architecture:

Morale is high-what is more inspiring than a technician to see their results quickly put into use?

Customer affirmation--their headaches are solved.

Reduce risk-The new system is actually validated during project rewriting

Here's a look at more high-priority streams that need to be optimized--accounting for 20% of the entire system, and using the dark architecture approach, the business will quickly find its own direction:

Continuously update the stream until system 100% is migrated

Evaluate the remaining 80% functions and business priorities

So visible the powerful role of the dark architecture, in the shortest possible time to solve the system short Board also has the choice of more flexibility, you can calmly on the business priority to deal with.