Improve Linux kernel and scalability to adapt to enterprise environments

Article title: improve Linux kernel and scalability to adapt to the enterprise environment. Linux is a technology channel of the IT lab in China. Includes basic categories such as desktop applications, Linux system management, kernel research, embedded systems, and open source.
The first step to improve Linux performance is to quantify it. But how can we precisely quantify the performance of Linux or its equivalent system performance? In this article, members of the IBM Linux Technology Center described several benchmark program tests they performed on Linux 2.4 and 2.5 kernels at the end of last year. Based on these expert experiences, they are intended to serve readers.
　　
Currently, the Linux operating system is one of the most successful open source projects. Linux, as a Web server operating system, demonstrates its high reliability. it occupies a large share in the Web server market. These Web servers are usually low-end and intermediate-grade systems with up to four symmetric multi-processor (SMP); enterprise-level systems have more complex requirements, for example, you need more processors, I/O configurations, and large memory and bandwidth. In order to make Linux ready to enter the enterprise environment and to enter the SMP market as a commercial application, compared with commercial UNIX systems, its SMP scalability, disk and network I/O performance, scheduler, and virtual memory manager must be improved.
　　
Linux Scalability Effort (LSE) is an open source project that solves the Linux kernel problems for enterprise-level machines, the scalability of these machines is 8 or higher.
　　
The Linux Performance Team of the IBM Linux Technology Center (LTC) actively participated in the LSE activity. In addition, they aim to improve Linux kernel performance (especially for SMP scalability.
　　
This article describes the strategies and methods used by the team to measure, analyze, and improve the Linux kernel performance and scalability while focusing on platform independence. The team used a set of benchmarking procedures to complete the task. These benchmarking programs take into account various workloads, including Web services, databases, and file services. In addition, we will show you the kernel components (such as the disk I/O subsystem) that each benchmark program focuses on ).
　　
　　 Analysis Method
Here we will discuss how to quantify the Linux performance of SMP scalability. If you want to, you can directly jump to the benchmark test program result section.
　　
Policies we use to improve Linux performance and scalability include running several industry-accepted and component-level benchmarking programs, selecting suitable hardware and software, and developing program running rules for benchmark testing, set performance and scalability goals, and measure, analyze, and improve performance and scalability. These processes are described in detail in this section.
　　
Performance is defined as the raw throughput on a single processor (UP) or SMP. We separate SMP scalability (CPU) from resource scalability (for example, the number of network connections.
　　
　　 Hardware and Software
Most of this work uses the IA-32 (x86) architecture, from one to eight processors. We also studied issues related to the IA-32 and NUMA IA-64 architecture for future use of inconsistent memory access (NUMA. The hardware selection is usually based on the benchmark test program and related workloads. Software selection should be combined with IBM's Linux middleware policies and/or open source middleware. For example:
　　
Database
We use the database benchmark test program, while on the hardware, we use eight SMP systems with large disk configurations. The database software uses IBM DB2 for Linux, and the SCSI controller is IBM ServeRAID 4 H. This database is for 8-way SMP.
SMB file service
The benchmark test program is NetBench, and the hardware is a 4-way SMP system. up to 48 clients can drive the SMP server. The middleware is Samba (open source code ). SMB file service is for 4-way SMP.
Web services
The benchmark test program is SPECweb99, the hardware is 8 SMP, and has a large memory configuration, up to 32 clients. This benchmark is only used for research purposes and is not used. (For more information about this, see the benchmarking procedure section ). The Web server is Apache, which is the foundation of the ibm http server. We chose 8-way SMP to study scalability, but Apache because it supports the measurement and analysis of the next-generation posix Thread (NGPT) (see references ). In addition, it is open source code and the most popular Web server.
Linux kernel version
Which version of Linux kernel.org kernel (2.2.x, 2.4.x, or 2.5.x) is used depends on the benchmark test program, which will be further discussed in the section of the benchmark test program. To simplify management, the selected Linux distributions are Red Hat 7.1 or 7.2. Our focus is on kernel performance, not the performance of the distribution version: we replaced the Red Hat kernel with the kernel.org kernel and patch to be evaluated.
　　
　　 Running rules
During the setup of the benchmark test program, we developed runtime rules to detail how to install, configure, and run the benchmark test program, and how to explain the results. The running rules are as follows:
　　
Defines a measurement that is used to measure the performance and scalability of the benchmark test program (for example, message/second ).
Ensure that the results of the benchmark test program are suitable for measuring the performance and scalability of workload and kernel components.
Provides documented instructions that allow others to repeat this performance test.
Define the collected dataset so that you can analyze the performance and scalability of the tested System (System Under Tes, SUT) to determine the bottleneck.
　　
　　 Set goals
The performance and scalability of a benchmark test program are related to the specific SUT (hardware and software configuration. To set performance and scalability goals, you must:
　　
Baseline measurement is used to determine the performance of benchmark testing programs for baseline kernel versions. Then calculate the baseline scalability.
The initial performance analysis is used to determine the feasible direction of performance enhancement (for example, it is recommended to try the O (1) scheduler to display the summary file that is very busy with the scheduler ).
Compare the baseline results with similar published results (for example, search for the SPECweb99 public results on the same Web server similar to eight SMP on spec.org ).
　　
If no external public results are available, you can try using internal results. You can also try to compare it with other operating systems. Give competitive data and baseline, and then select the performance target of the UP and SMP machines.
　　
Finally, you can declare that the application has changed its target. For example, if you know that an application uses an asynchronous I/O method with low efficiency, you can assume that the I/O method is changed to publish this performance goal.
　　
　　 Optimization, measurement, and analysis
You need to tune hardware and software configurations before performing any measurement. Optimization and measurement are a process of continuous optimization and measurement. It involves the measurement of system components (such as CPU utilization and memory usage), and may also involve the adjustment of system hardware parameters, system resource parameters, and middleware parameters. Tuning is the first step in performance analysis. Without optimization, the measured results may lead to misunderstandings; that is, these results may not indicate kernel restrictions, but show other problems.
　　
The benchmark test program is run according to the running rules, so that performance and scalability can be measured according to the defined performance measurement. When computing the SMP scalability for a given machine, you can choose to calculate this metric value based on the performance of the UP kernel, or you can choose to calculate this metric value based on the SMP kernel (where the number of processors is set to 1-one processor) to compute it. We decided to use UP measurement to calculate SMP scalability, which can more accurately reflect the improvement of SMP kernel performance.
　　
Use the Linux kernel version determined earlier to measure the baseline. For most benchmarking programs, the UP and SMP baselines need to be measured. For a few benchmark testing programs, you only need to measure eight SMP performance because the collection of UP performance information is limited by time. Most other benchmarking programs measure the workload completed within a specific period of time, and the measurement time on the UP does not exceed the measurement time on the 8-way SMP.
　　
The first step required to analyze the performance and scalability of the SUT (tested system) is to understand the tested benchmark testing program and workload. The initial performance analysis was made for the tuning system. Sometimes, analysis reveals other modifications to tuning parameters.
　　
The analysis of SUT performance and scalability requires a set of performance tools. Our strategy is to use Open Source community (OSC) tools as much as possible. This allows us to publish the analysis data on OSC to illustrate performance and scalability bottlenecks. This also enables members in the OSC to repeat our results with tools, or understand the results after testing another application with the tool. If a dedicated performance tool is developed to better understand a specific performance bottleneck, then this dedicated performance tool is usually shared on QSC. These dedicated performance tools are usually simple tools that provide testing means for specific Linux kernel components. Our performance tools include:
　　
/Proc file system
Meminfo, slabinfo, interrupted, network status, and I/O status.
Lockmeter of SGI
SMP lock analysis
Kernel summary analyzer (kernprof) of SGI)
Time-based summary analysis, performance-based counter summary analysis, and annotated call graph (ACG) for kernel space only)
IBM tracking tools
Single-step (mtrace) for user space and system space, and simultaneous time-based and performance-based counter-based summary analysis
　　
Develop dedicated performance tools to further understand a specific aspect of the system.
　　
For example:
　　
Sstat
Collect scheduler statistics
Schedret
Determine which kernel functions are blocked due to idle time
Acgparse
Post-processing kernprof ACG
Copy the import/export tool
Determine the buffer adjustment, replication size, and CPU usage of the copy-in/out algorithm
　　
Then, the performance analysis data is used to determine the performance and scalability bottlenecks. We need to have a broad understanding of SUT and a more detailed understanding of the Linux kernel components that benchmark testing programs focus on to understand where the performance bottleneck exists. You must also understand the Linux kernel source code that causes the bottleneck. In addition, we need to work closely with the LTC Linux kernel development team and OSC (open source community) to develop patches to fix bottlenecks.
　　
　　 Exit Policy
To evaluate the Linux kernel performance, you may need to run the benchmark test program, analyze the results to determine the performance and scalability bottlenecks, and then integrate the patches into the Linux kernel to solve all these bottlenecks, finally, run the benchmark test program again, which is a cyclical process. As a member of the performance team, we need to work with the Linux kernel development team or OSC to obtain patches by searching for existing patches in OSC or developing new patches. There is a set of criteria to determine when Linux is "adequate" and can end the process.
　　
First, if we have achieved our goal and there are no outstanding Linux kernel problems to solve for a specific benchmark test program (these problems can be greatly improved)