Virtualization expert Xiao Li: five years of game virtualization O & M practices (1)

Virtualization expert Xiao Li: five years of game virtualization O & M practices (1)

This article is a summary of my five-year game virtualization O & M practices and mainly shares the following content:

Why game virtualization?

Suitable for/not suitable for Virtualization

How to Implement game Virtualization

Problems to be Solved during implementation

Virtualization Technology highlights during implementation

Virtualized Storage

Virtual Machine resource restrictions

What are the key points of monitoring, alarm, disaster recovery and emergency response in virtualization O & M?

Software and Hardware Selection

In the end, it is just one sentence: How to migrate the game to a virtualized environment and run it stably.

Why game virtualization?

Why is game virtualization or what virtualization brings to the game.

The first and most important point is to save costs. For example, assume that there is a game with a block, 40 servers, and 40 servers usually occupy 3-5 cabinets Based on the height. Now, assuming that virtualization can be performed at a rate of, 40 servers can be compressed to 8 servers, and 8 servers can only occupy one cabinet, in this way, we have saved at least 32 servers and 2 cabinets! The cost savings on hardware are amazing.

Second, resource isolation. An important feature of virtualization is resource isolation. It is very difficult to deploy two games on the same physical machine, because of the conflict of ports and configuration files, and even the maintenance time of the two games are different, if a game needs to restart the server during maintenance, it will affect the operation of another game. Virtualization can solve this problem well. We can deploy a game on two virtual machines on the same host without any conflict, the isolation of virtualization can effectively solve this problem.

Third, Fast deployment. From the perspective of the host machine, a virtual machine is an image file. To get another virtual machine, you only need to copy the image file. This is usually several minutes, up to ten minutes. However, we need to get a physical machine, from rack mounting, power cord insertion, and operating system installation. It takes only one hour for us to be proficient in time. This is an order of magnitude difference.

The rapid deployment of virtual machines is especially suitable for mobile game web games, because we all know that mobile game web games are characterized by frequent opening and grouping, and virtualization can meet this requirement, this is why most browser games prefer virtualization.

Suitable for/not suitable for Virtualization

So those games are suitable for virtualization. The first type is single-process games, which are mainly old client games. During Game Development, there is no concept of multi-core CPU, these games are especially suitable for virtualization.

The second is that the lifecycle enters the middle and late stages, the number of people is relatively stable, and the consumption pressure is also certain. By integrating several blocks into a single host, you can achieve the purpose of integrating resources.

Virtualization integration data case

For example, I used to play a game that uses more than 500 physical machines. The game was already in balance. In other words, it had no money and faced the fate of being completed immediately, because of the implementation of virtual machines and integration according to the ratio of, more than 500 physical servers are compressed to more than 70 host machines, greatly saving the game operation costs. This game project began to make profits again and was able to survive for a while.

The third type is mobile and web games, which have been introduced above. Especially for mobile games, the development cycle is very short, and the program does not have time and energy to optimize the system. It can only be done through virtualization.

So are all games suitable for virtualization? games with extremely high stress are not suitable for virtualization. If the stress on physical machines is already very high, it is very difficult for us to integrate the Games through virtualization, such games are not suitable for virtualization.

How to Implement game Virtualization

The implementation of game virtualization is recommended to follow the steps below to ensure smooth game migration to the virtualization platform.

First, you need to evaluate the business performance requirements, mainly to assess the pressure of the business, the performance data can be viewed on the monitoring platform, or the script can be used to capture, after obtaining the business pressure model, you can design a business virtualization solution based on the stress model.

The design of the virtualization solution is mainly about how to select the host and how to determine the virtualization ratio. Then we will build a test environment for testing.

The test consists of two parts: system comprehensive test and business stress test. The comprehensive system test mainly tests the performance bottlenecks of the host machine, virtual machine disk, and network, and obtains the limit data, so that you can know exactly when using the data. Business testing is mainly to test whether a game program can run on a virtual machine, whether the client can log on to the game normally, play games normally, and the maximum number of players that a virtual machine can load.

After the test is complete, start a small-scale deployment. Generally, start an experience server, run for 2 weeks, find a block with the least number of people, and run for 2 to 4 weeks, then, find the most popular block and run it for 2 weeks to 4 weeks. If there is no problem, migrate the game to the virtualized environment gradually until the final virtualization O & M is completed.

A painful case of the test time

At that time, we had a game that was a relatively old client game. We had a lot of stress tests on robots in the early stage, and there was no problem. We started the test of the trial server, and there was no problem, then, I ran a group with the least number of people in the production environment for a week and there was no problem. At this time, I felt that there was no problem and I was a little impatient. I suddenly came up with several groups, these block virtualization functions will be stable one week later, but on Thursday of the second week, frequent game downtime and large-scale disconnection of players began. At this time, it was very passive, and I had to solve it with a hard head. Now, in retrospect, if the blockchain of the first production environment virtualization was able to be tested for a long time, this problem would be exposed early, at least the impact is much smaller.

Problems to be Solved during game virtualization implementation

The first point is stability. Stability is especially important in the gaming industry, because stability is linked to revenue. Without stability, there will be no income and no cost savings will be significant.

The second is fast Management and Maintenance. After stability is solved, the next problem is how to quickly obtain one or more virtual machines, A management platform is often needed to quickly manage the lifecycle of virtual machines from generation, maintenance, and destruction. The management platform can use open-source products, such as oVirt and OpenStack, and can also be developed by itself.

The third is close integration with the business. It is best to run the game program when a virtual machine is generated. There are two solutions: one is to create different dedicated images for different games, in this way, when the virtual machine is generated, the game program is in it, but this solution is relatively passive when the game frequently releases the version, because the image must be updated every time the version is released. Another method is to use a common image to insert game programs and configuration files into the virtual machine when the virtual machine is generated. After the virtual machine is generated and started, the configuration is automatically completed.

The fourth point is to make monitoring alarms and respond to emergencies. Everything has two sides. After virtualization, when a fault occurs, the impact will be much greater. In a game, when it was originally on a physical machine, a physical machine failure affected at most one block. When virtualization was implemented, a host machine failure affected several game zone groups, this requires us to respond to emergencies and monitor alarms, discover and solve problems early, and even prevent problems in advance when they happen.

Virtualization Technology highlights during implementation

The detailed KVM virtualization technology will not be introduced here. It will only introduce some considerations in game virtualization. Let's take a look at the pressure model of the game business before introducing it.