Analysis of NUMA-based servers in Linux

This section analyzes the Linux Enterprise Application-Linux server application information of a server built based on NUMA in Linux. The following is a detailed description. NUMA (Non-Uniform Memory Access Architecture) systems are widely used in the market. Many vendors have successfully launched NUMA-based servers. This article focuses on the NUMA Technology in Linux, it mainly includes: storage management, NUMA scheduling, and user-layer APIs, and performs basic NUMA tests on the SGI Altix 350 system, which is of reference value for the study of Linux NUMA technology.

I. Introduction

With the increasing requirements of scientific computing and transaction processing on computer performance, SMP (symmetric multi-processor) systems are widely used and growing in scale. However, in traditional SMP systems, all processors share the system bus. Therefore, when the number of processors increases, the competition between the system bus becomes a bottleneck, therefore, the number of CPUs in the SMP system is generally dozens, and the scalability is greatly limited. NUMA effectively combines the programming capability of SMP system and the scalability of MPP (large-scale parallel processing) system to better solve the scalability problem of SMP system, it has become one of the mainstream architectures of high-performance servers. At present, famous foreign server manufacturers have successively launched high-performance Servers Based on NUMA architecture, such as HP Superdome, SGI's Altix 3000, IBM's x440, NEC's TX7, and AMD's Opteron. As Linux becomes more and more mature on the server platform, the support of the Linux kernel for the NUMA architecture is also improved, especially since 2.5, linux has carried out a lot of NUMA optimization work in terms of schedulers, storage management, and user-level APIs. At present, this work is still being improved, for example, the NUMA scheduler is added to 2.6.7-RC1 kernel. This article mainly discusses three aspects: storage management, scheduler and CpuMemSets.

Ii. NUMA Storage Management

The NUMA system is composed of multiple nodes connected through a high-speed Interconnection Network. It is two nodes in the SGI Altix 3000 ccNUMA system.

The node of the NUMA system is usually composed of a group of CPUs (for example, SGI Altix 3000 is two Itanium2 CPUs) and local memory. Some nodes may also have I/O subsystems. Because each node has its own local memory, the entire system is physically distributed, the latency between accessing the local memory at each node and accessing the remote memory at other nodes is different. To reduce the impact of non-consistent memory access on the system, the hardware design should minimize the latency of Memory Access (such as Cache consistency design), and the operating system must be able to perceive the hardware topology and optimize the system access.