Five years on the path of the Skype architect

Brief introduction

As architects and designers, we often focus on what we have at hand and seldom reflect on the past. We should know the new from the warm. I've summed up 6 experiences from my 55-month experience as the Skype architecture group. Some are technical and others are softer views of the architect. First, introduce Skype background information.

Skype background

Skype is software that allows users to make audio and video calls, and can also make regular calls and send short messages. The company was founded in 2003 and has an incredible growth curve since its inception. The company now has more than 520 million registered users, about 650 employees. These users also produce an average of 210,000 calls, about One-third of which contain video. This number is roughly 8% of the world's international calls.

Needless to say, it is also known that this traffic has a rare extensibility challenge. On Skype, end-to-end (peer to peer) technology has been used as a major weapon in dealing with similar challenges. Peer-to-peer networks (implemented in C language) are composed primarily of server-side services and Postgre database support written by C + +, combined with powerful Python scripts. Web services are built using PHP.

The rule of thumb for

technology is not applicable

In a career as a software engineer, some patterns will emerge and some rule of thumb will emerge. Obviously, you are willing to use these rules whenever and wherever you want. After all, they used to work, didn't they?

It turns out that even if you have a good hammer, don't take everything around you as a nail. In a rapidly changing modern science and technology society, the rule of thumb does not always apply. For example, let's look at how the Skype database is structured.

Traditional wisdom says never to implement business logic in a database. Why does this claim spread so widely? Most architects have similar experiences, which can cause the original database to grow in hardware like a behemoth, not run, and very difficult to maintain.

The reason for this fake Cthulhu is that the main database platform often lacks two important and erecting features: the ability to divide the database horizontally (for example, by dividing data from the data entity) and to vertically divide the database (different database entities into different databases). Of course, we can build these two features ourselves, but people outside the database management team often want to deal with similar problems. This is a means of survival for DBAs rather than a problem-solving capability. In other words, the database partitioning or queue technology often exists outside the database, so that developers need to deal with the protocol transformation, a variety of interfaces, data integration and other issues.

On Skype, these people who maintain databases happen to be important contributors to Postgre. From a very early time, they refused to view the database as a useless jar in the system architecture corner, but instead took a positive approach to learning technology to address the scalability, performance, and maintainability problems they encountered. As you might guess, these are not enough, even the best database architecture will be discarded in a rash code. Fortunately, the Skype database administrator has been in control of the need to enter the database layer from a very early time, and Skype's DBA does not relinquish control until a series of non-functional requirements, code implementations, and peer review processes are in place to ensure that the implementation code is designed to fit the database layer and other related parts.

Figure I explains how they use these tools to build a Skype database architecture.

This is composed of four layers:

The

access layer provides the ability to access the database and also handles database partitioning (piproxy) and connection pooling (Pgbouncer). and allows developers to use these features transparently. The online transaction processing layer is where OLTP databases exist. The queue layer, which is responsible for transferring data and replicating data between layers and layers. The internal server layer, which contains databases for recording, statistics, viewing, batching, and ETL purposes.

All of this is to ensure that database scalability is not a problem for developers. We put the necessary business logic as close to the data as possible, so that its most efficient work, that is, "business logic should be away from the database" rule of thumb does not apply. Of course there are problems like publishing, debugging, and unit testing, but we're not afraid of the raw database raging.

Figure I: Database layer

The same is true for schema schemas. It is important to establish a common technical glossary between engineers and to provide a proven common technical problem prescription, which should be treated with care. Skype's end-to-end network is a good example. If the problem is presented in the form of "Designing an Internet phone," in most cases, SIP is designed to implement the requirements. But if Skype implements services based on SIP, it will not bring changes to the communications industry. The early engineers of Skype were unwilling to limit themselves to how it was usually done, but to find the best possible solution they could build.

In short, slightly different organizations and skills may be necessary to establish a completely different architecture pattern application. You should always welcome these differences to your own traditional thinking challenges.

neglect of functional architecture

We rarely have the opportunity to work as a lead designer at the initial stage of the project. Most of the work is to modify existing systems, and change management becomes an important part of the architect's work. Most of our change management now focuses on the technical architecture and the effective design of the system to ensure that the design remains meaningful after the changes are implemented.

But it's not the whole story.

All technical changes come from functional changes. We rarely modify the system just for refactoring. Often there are external drivers that require the system to behave differently. This could be a new product on the market, or it could be a legal change or a better extension for people in the operations department. In any case, technical changes are often accompanied by functional changes.

So our systems and processes need to ensure that technology changes are easier, and we want this management process to be more orderly and not as messy as spaghetti for the person taking over. But what is functional change? Who will focus on the functionality of the system and ensure that changes do not make the system more cluttered?

Let me illustrate with an example.

For the past four years, there has been a strong urge to revise Skype's networked storage architecture, even if I prove that every tiny change is accompanied by pain. Selling four of products on the Internet is not complicated, and most of the time the entire system is running as usual, even if some problems are found and then solved.

That's why.

Figure 2 shows the functional components of all Skype networked storage. There are about 200. The chart is not very clear and accurate and only wants to show the functionality and complexity of the entire application system. This is the result of countless changes, additions, modifications, legal issues, fine-tuning. All of these, of course, have a reason and value.

A considerable number of architects have not carefully considered the technical changes, resulting in spaghetti-like chaos, and the application system has become dysfunctional in function without thinking. This does not mean that as a software architect, I intentionally blocked these issues from the beginning. However, if you do not care enough about the functional architecture of the system, it can lead to fragmentation of the functional architecture. The result can only be a messy technical architecture.

Figure 2: Networked Storage functional architecture

All in all, you should always keep an eye on the system features you want to maintain. Modify the technical architecture, and often maintain the functional architecture.

simple Things are effective

Simply put, anything that requires more than three sentences to explain to others will not work effectively. That's why rest can actually be applied and soap doesn't, and that's why people prefer hibernate to Java-ee beans.

PGQ[1] is an example of a slightly simpler requirement that produces good results. Message reliability is one of the major performance issues for all messaging systems. It is frustrating to flag messages for different clients as "used," and to store these messages and ensure they do not block data stores that have not yet been consumed. But when you promise to distribute each message at least once rather than just once, these headaches disappear half. This is acceptable in most cases for client applications, as long as they are allowed to freely implement the validation mechanisms they need.

Another effect of simple solutions is to make you think, and it's always good to think more. It's interesting to design interface WSDL, but how much do you really care about the nature, such as what types of objects should go into other objects and what do you want to look like? That's it.

All in all, the goal of making the system easier to use is to meet all the requirements, laws, and standards, relentlessly removing all the excess fat that is causing the system to slow down.

Non-technical perspective dangerous popular language

There are times when some people build software in such a "nice" way: invent an appealing name and paint it everywhere in PowerPoint before you know it. Unfortunately, most of these ideas are very complex and rarely practical. such as Java EE, CORBA, SOA, are not designed to solve day-to-day problems, they can sometimes work, but it is very accidental.

On Skype, we have had similar problems many times and managed them quite successfully. Although we hear that an organization has very different experiences. At some point, we see many large application developers have recently discovered that their entire engineering management system has been replaced.

Some expert told the story.

Management executives have some time on the surface to deal with specific issues, such as taking advice from certain consultants, customizing major products and fully entering the cloud and SOA decisions that will help them. So they started talking to engineering leaders, though the latter reported empty eyes. Just like Poter Comics, these are just a big bubble of snake oil. After a while, the inevitable happened, management tired of being fooled like a fool (consultants charge is very expensive), when the next step is started, or no one to solve the problem at the beginning. The company may not be able to regain its strength, even if it is rid of incompetent and dissenting people.

This is the architect's failure, really.

This story demonstrates the two-dollar nature of the architect's responsibilities: first, we need to think about these ideas, and just put what actually makes sense into the system and keep the system running. On the other hand, we cannot ignore these often meaningless terms, because the real problem may be hidden behind. The reason why it is not easy to find the root cause is that the customer's management lacks some words we can understand to express demand. Also, when a concept jumps out, it seems to have solved a problem that has plagued customers for a long time. They picked up the rope and became so powerful that they used it in the organization. Responding to these situations from a technical point of view, such as claiming that the whole thing is false, does not solve the fundamental problems of running and is not constructive. When the leader discovers that the organization has a problem and believes he has found the solution, and you refuse to implement it or even refuse to discuss it, you are out. If you don't make these buzzwords meaningful by yourself, there will be a bunch of advisers who can help you define them.

In a word, users rarely fool you, and you shouldn't fool users. You should work with users to find and solve real problems. Because trusting you, your president will have better things to do than lose some of the meaningless advertising words that make you shiver.

Architects need to work with your organization

Most people work every day to make things as good as possible. Architects work to build a great system architecture that can be infinitely scalable and modular.

Actually it's not a matter of paying us to do it.

Each system has a specific context environment. This environment includes existing technology systems, as well as skills, attitudes and corporate culture that people deal with problems. Even more importantly, all systems exist in a particular business environment. Startups are different from giant telecoms operators, and banks are different from government agencies. Clearly, no good or graceful architecture can be adapted to different business and organizational structures. Architecture needs to adapt to organizations to help them achieve their goals (or not). This often means that you need to suppress your desire to build a graceful system, because often you think the graceful system and organizational needs are two different things.

The reality is to put the concept of technical debt [2] aside and not work with debt. It may not be technologically advanced or perfect, but it can help your organization well.

This has been an important issue in Skype's environment. The primary services that our large number of users use are provided by Peer-to-peer networks. Peer-to-peer networks are very beautiful things, but not necessarily so-called "clean" or "simple". End-to-end is ridiculous for people with a traditional Web application background. It is more difficult to build, maintain, debug, line up, test and explain this, especially at this level, we are the only company that operates a peer-to-peer network. And there's always a consultant who's pressuring us to fall back to a server-based architecture like everyone else.

Technically, this pressure is understandable, and there's a bunch of reasons why it's reasonable to switch. The decision becomes difficult when you see that this change may affect our business model. For example, our users have the same order of magnitude of video traffic on video calls as YouTube. With the End-to-end architecture, Skype is not heavily invested in hardware. Changes to the End-to-end architecture are highly likely to mean the end of the free video telephony service, which means the end of the business model without subsidy fees. So no matter how I consider and whether I like to use an end-to-end architecture, it will prevail in comparison.

In short, all of your architectural decisions need to be made according to the circumstances of your organization rather than a personal preference.

Communication is important

As we've seen before, how to make a schema depends on the business function. Because the system architecture is correct or not, it is reasonable to conclude that people are interested in system architecture because of business interests. But how do people with fan towels understand the intricacies of the system developers find and how software engineers can find business functions?

The answer is very simple, communication. Both need to reach across cultural barriers and start talking. The architect's job is to translate the business strategy into technology. This means communication.

It's not easy to understand that it's difficult to get management's respect. But without respect and communication, engineers can only tolerate arbitrary technical decisions, and businesses have to deal with systems that limit their development. If there is no communication, there is no understanding, not to mention cooperation.

Figure III: Architect organization

Communication is also important to another obvious user of the architecture, that is, the developer. Without the developer's perfect implementation, the architecture cannot become the actual code for the service user, and it cannot generate value for the business. Again, trust and respect are key.

Figure III shows the general organization Chart of the Skype architect, which is a very simple association model without the necessary team or reporting hierarchy. The central part is the architect group, which primarily maintains relationships and sets common directions. Business Unit Architects (called Solution Architects, very similar to analyst roles) and Development Group Architects (known as technical architects) complement them. The former is responsible for helping the business unit to organize ideas into a technically feasible form and to provide feedback that explains the technical soundness. The latter is responsible for overseeing the development and refinement of the high-level design provided by the architect.

The architect organization provides sufficient organizational structure and coordination in different stakeholders, along with certain degrees of freedom. Of course, you need to find the right model for your organization, and whatever the solution is, you need to promote communication between your architect and your key customers.

In a word, communicate with others!

Conclusion

As you can see, these years of experience have taught me a lot. If you feel familiar and trivial, you may have had similar experiences. I hope it's better than I've ever been. Summarize the two main insights that architects need to succeed at this time and age:

no matter how you used to work, such as working for a giant like Facebook or Skype, or having a chat with your local CIO community, it should only be a starting point for helping your organization find solutions. Technical skills are prerequisites for architects. You need technical skills to get the job, but the emotional quotient and the ability to understand the organization's business define how good you are. References

[1] "SkyTools page at Pgfounry."

[2] M. Fowler, "Technical distressed," August 2004. 12

Learnings from Hep Perton as a Skype Architect

Thank Cao Yunfei for the revision of this article.

For the INFOQ Chinese site to contribute or participate in content translation work, please email to editors@cn.infoq.com. We also welcome you to join our editors and other readers in the Infoq Chinese site user discussion group.