On GitHub, we always say "if the website response speed is not fast enough, we should not let it go online for operation ". We previously introduced some methods to improve the website response speed in the previous article, but this is only part of the story. The MySQL database architecture affects GitHub.com performance. Let's look at our foundation.
On GitHub, we always say "if the website response speed is not fast enough, we should not let it go online for operation ". We have previously introduced some methods to improve the website response speed in this article, but this is only part of the story. The MySQL database architecture affects GitHub.com performance. Let's look at our foundation.
On GitHub, we always say "if the website response speed is not fast enough, we should not let it go online for operation ". We previously introduced some methods to improve the website response speed in the previous article, but this is only part of the story. The MySQL database architecture affects GitHub.com performance. Let's take a look at how our infrastructure team seamlessly upgraded the MySQL architecture. This happened in last August, and the result was that the speed of the GitHub website was greatly improved.
Task
Last year, we moved most of the data on GitHub to a new data center with the world's top hardware resources and network platforms. Since MySQL was used as the basis of our back-end system, we have always expected some improvements to greatly improve database performance, however, it is not a simple task to deploy a new cluster environment with all-new hardware in the data center. Therefore, we have developed a set of plans and tests, so that data can smoothly transition to the new environment.
Preparations
For a huge change in architecture like ours, data indicators need to be collected at every step of implementation. After the basic operating system is installed on the new machine, the next step is to test the performance of the new configuration. To simulate a real workload environment, we use the tcpdump tool to copy the ongoing SELECT request from the old cluster and replay it on the new cluster.
MySQL tuning is a tedious and meticulous task, like the well-known innodbBufferPoolThe size parameter often has a huge impact on MySQL performance. We must take this type of parameters into account, so we have a list of parameters, including innodbThreadConcurrency, innodbIoCapacity, and innodbBufferPoolInstances, and others.
In each test, we are very careful to change only one parameter and have a test run for at least 12 hours. We will observe the response time curve, the number of responses per second, and the parameters that may cause lower concurrency. We use the "show engine innodb status" command to print InnoDB performance information. We have observed the content in the "SEMAPHORES" section, which provides us with the STATUS information of the workload.
When we are satisfied with the running results after setting the parameters, we start to migrate one of our largest data tables to a set of independent clusters. This step serves as an early test of the entire migration process, to ensure that our core clusters have more cache pool space and provide more flexibility for failover and storage functions. This initial migration solution also introduces an interesting challenge: we must maintain multiple customer connections and direct these connections to the correct cluster.
In addition to improving the hardware performance, we also need to add some improvements to the processing process and Topology: We have added the delayed Copy Technology to back up data faster and more frequently, and more read and copy space. These functions are ready for release.
List the task list, think twice
There are millions of users using GitHub.com every day. It is impossible for us to switch actual data only when no one is using GitHub.com. We have a detailed task list for Migration:
We also planned a maintenance period and informed everyone in our blog to let users notice this.
Migration Time
At five o'clock A.M. on Saturday, Pacific time, our migration team launched a set dialogue and started data migration:
We set the GitHub website to maintain mode, post a statement on Twitter, and start to work as follows:
13 minutesTo ensure that the new cluster works properly:
Next, let GitHub.com exit from the maintenance mode, and let users all over the world know our latest status:
A large number of preliminary testing and preparation work lead us to shrink the maintenance period to the minimum.
Test Final Results
In the next few weeks, we will closely monitor GitHub.com's performance and response time. We found that after the migration, the average loading time of the website is reduced by half, and the load time is reduced by 99%.2/3:
What features have we learned?
During the migration process, we adopted a better method: migrating large data tables (mainly recording some historical data) first, empty the disk space and cache pool space of the old cluster. This step leaves us with more resources for "hot" data and isolates some connection requests from multiple clusters. This step laid the foundation for our subsequent victory, and we will use this mode for migration in the future.
Test and Test
Perform acceptance and regression tests on your applications. During the process from copying data from the old cluster to the new cluster, if the acceptance test and response status test are performed, the data obtained is inaccurate. If the data is not satisfactory, this is normal and should not be surprised, do not use the data to analyze the cause.
Cooperation strength
Major changes to the infrastructure usually involve many people. We need to work together for the same purpose as a team. Our team members are from all over the world.
Team member map:
This cooperation created a new workflow: we submitted a change (pull request), obtained real-time feedback, and checked the modified wrong commit-no telephone communication or face-to-face meetings throughout the process. When everything can provide information through URLs, communication and feedback between people in different regions will become very simple.
One year later ......
A year has passed, and we are happy to announce that this data migration was successful-MySQL performance and reliability have always been in our desired state. In addition, the new cluster allows us to further upgrade to provide better reliability and response time. I will continue to record these optimization processes.