PIT Series---high-availability architecture for silver bullets

0. Connecting

Before that article written out I think there will be a lot of different views, haha, that only represents my personal opinion ah, welcome to discuss.

First of all, I give an example, OA系统 not to say that OA must be so designed, but an exaggerated approach, in order to explain the back 完全脱离了业务场景来进行技术架构的设计就是过度设计 , not that the OA system is too simple so can not be designed, in addition, PHP low write efficiency is only a metaphor, not to belittle the world's best language , it's not necessary for many people to spray these two.

Well, the topic of today's article, the previous article on the overall structure of the design of the excessive design, this article says high availability.

1. Superstition architecture can solve high availability, but there is no silver bullet

高可用, I know that once I bring this word, no matter what you write, someone will disagree, I say I think the 高可用 pit bar.

I think a lot of people understand that a 高可用 single machine hangs off the entire service will not hang, so write code when using the idea of the cluster to write code, such as a stateless service, to ensure that the cluster when the use of stateless, single-machine failure does not affect the service, thereby achieving 高可用 the effect.

The system built up by this idea is likely to look like this, and I think a lot of people have seen this architecture pattern.

?

First of all, this architecture pattern itself is not a problem, but also really good, there are services found, there are clusters, a single machine hanging off and other machines can be used, in the search system, recommendation system, advertising system, the site backend system are in a large number of use.

Many people receive the information is the kind of architecture, then the system becomes a highly available system, think this architecture pattern is 高可用 one 银弹 .

But in fact, the system solves mainly the following two problems.

• Data synchronization, mainly public configuration of this small amount of data in the synchronization between the various machines.

• Service discovery, the addition or reduction of the machine, so that other machines can perceive the new node to join or have the old node offline.

In addition to the above two problems, the last is to solve the so-called 高可用 problem, here 所谓 two words, because I think 高可用 this kind of thing is not an architectural model can be solved, a highly available system is code-level solution, not by a few open source modules can be solved.

Some people always think that the 高可用 system has 银弹 , in various forums, the meeting to see a variety of architectures, and basically used some mature open source software, so I feel that with these can be a highly available system, I have zookeeper, then service stand-alone, service run as usual, But in fact, and eggs, zookeeper to solve the external uncontrolled factors caused by the machine hangs, such as the machine hard disk is broken, the network is broken, this factor caused the service hangs, zookeeper can solve, your code problems caused the machine to hang up, Zookeeper hanging 1000 machines also can not solve Ah, the general situation is still a hanging all hanging.

such as a distributed search system, index shards, so there is a cluster, there are 50 machines, each shard about 10 machines, and the machine can dynamically increase the reduction, cluster with zookeeper management, this 高可用 system? This is a standard search system 高可用 architecture, can only say, in code excellent premise, this system 高可用 , network problems and machine hardware problems have been more difficult to hang the whole cluster. But once the code has a small bug, or index data generated when there is a point of problem, in general, the cluster is all hung up, talk about what is high availability.

高可用No 银弹 , you see everywhere, hear, learn the various 高可用 architectures, they will only tell you how good the system architecture, with a few boxes box a few modules, and then tell you, the service in this box all kinds of emergencies can be self-adaptive, traffic peaks to the linear plus the machine can be solved, For you it is the egg, they did not tell you how awesome their code, and only in this premise 高可用 , want to rely purely on a few boxes to frame a 高可用 system, that is the PPT architect.

Real 高可用 without the tangled architecture design, only need the code of robust, robust code and the main standby system design, no other, basically a highly available system, the bank's core data processing center plus geo-disaster preparedness is such a son, you dare say he is not highly available?

So, write good code, can be highly available, learning architecture, more than just to improve the overall understanding of the system a supplement 高可用的架构不存在，存在的只有高可用的代码 .

2. A chestnut

I have seen such a system in the previous period, this is a start-up company in the background of a module, the main function is to just open the app to provide users with a personalized referral page, external access to some other system generated some data.

Data from other systems, the first access to a kafka message queue, the data comes in after a service cluster to obtain this data, different services through kafka different topic access, and then two times to process the data, to generate a structured personalized data, the resulting data stored in redisin the cluster, each app user corresponding to redis a key, the previous app call API, directly from the redis cluster to get the data back, these clusters are managed with zookeeper.

This architecture, Message Queuing is to solve the third-party data push too hard to do cache, and Redis cluster is actually to solve the front-end app high concurrency access.

I asked first, the message queue this cluster in other system modules are also in use, this is no problem, everyone needs to use, deploy a cluster also should ha.

But this redis cluster is only used here, I think there is a problem, it is necessary to do a cluster with zookeeper? Just to open the app's personalization page and use a Redis cluster? Not everyone shared resources, I think there is no need to Redis cluster, a master one holds stocks, but also easy to maintain. If you feel that the single-machine memory is not big enough, you can use redis2.0, turn on the VM function, break the limit of physical memory, Redis can also keep hot data in memory.

You say this is to solve high concurrency under high availability, if Redis hangs, but also automatically switch, so to speak, I think a system, troubleshooting hardware problems, in general, and so your service is all hung up, Redis also firm. and Redis's concurrency ability can only be described with horror, stand-alone 2, 30,000 of the QPS (data size 2,3kb around) completely no problem, a start-up company's daily user volume generally do not need to use the cluster to anti-concurrency it?

Later, I suggested that they take the Redis cluster, replaced by a single master, and I found that the so-called personalized recommendations in fact most people see the same page, it is also very good understanding, the initial lack of data, personalized recommendations out of things is not enough, Redis cluster memory utilization is actually very low, So I further suggested that they use a nginx+lua local dictionary to cache the hottest data, and a redis behind it to become a level three cache (Redis local disk, Redis memory, Nginx Local dictionary). If the volume of business comes up, it is easy to switch to a Redis cluster, and now there is no need to waste machine resources, after all, start-up companies.

Well, in the end they cast a disdain for my eyes, this structure, people can not see, in case of a sudden increase in the number of users a day how to do, and the most important thing is that they are not bad money, well, hehe.

3. Where is the highly available silver bullet?

So much nonsense, are there any high-availability silver bullets? Well, good code is the cornerstone of all high-availability architectures and silver bullets, good code coupled with a reasonable architecture is a highly available architecture, a highly available architecture is not built on the building blocks of open source software, the mature open source software solves a part of the code that should have been written by you to become more excellent.

All right, welcome to the discussion:)

PIT Series---high-availability architecture for silver bullets