NUMA Architecture Detailed

1. Several concepts of NUMA (Node,socket,core,thread)

For Socket,core and thread will have a lot of article introduction, here briefly, specifically see:

Summary: Socket is the CPU socket on the motherboard; Core is a separate set of program execution hardware units in the socket, such as registers, calculation units, etc. Thread: Is the concept of Hyper-threading Hyperthread, the execution unit of logic, the independent execution context, but the sharing of registers and compute units within the core.

The concept of node is more in NUMA architecture, which is actually used to solve the problem of core grouping, see Understanding (the OS CPU in the figure can understand thread, then the core is not drawn in the diagram), you can see that each socket has two node, A total of 4 sockets, each socket 2 node, each node has 8 thread, a total of 4 (socket) x2 (Node) x8 (4corex2 thread) = 64 thread.

In addition each node has its own internal CPU, bus and memory, but also can access other node memory, NUMA's biggest advantage is that it can easily increase the number of CPUs, because node has its own internal bus, so the increase in the number of CPUs can be achieved by increasing the number of node, If you simply increase the number of CPUs, the bus will cause great pressure, so the UMA structure can not support a lot of cores.

"This figure is from: NUMA best Practices for Dell PowerEdge 12th Generation Servers"

As mentioned above, because each node has its own CPU bus and memory, if the Vcpus of a virtual machine across different node, it will cause a CPU in node to access the memory in another node, which leads to increased memory access latency. In some special scenarios, such as in an NFV environment, where performance requirements are high, it is very important that the vcpus of the same virtual machine are allocated to the same node as PCPU, so that the features of the NUMA-aware virtual Machine Scheduler are added in the kilo version of OpenStack.

2. How to view the NUMA topology of a machine

The more commonly used command is LSCPU, the specific output is as follows:

2. dylan@hp3000:~$ LSCPU
4. Architecture:x86_64
6. CPU Op-mode (s): 32-bit, 64-bit
8. Byte Order:little Endian
10. CPU (s): 48//Total 48 logical CPUs (threads)
12. On-line CPU (s) list:0-47
14. Thread (s) per Core:2//2 x threads per core
16. Core (s) per socket:6//socket with 6 x cores
18. Socket (s): 4//A total of 4 sockets
20. NUMA node (s): 4//A total of 4 NUMA nodes
22. Vendor Id:genuineintel
24. CPU Family:6
26. Model:45
28. Stepping:7
30. CPU mhz:1200.000
32. bogomips:4790.83
34. Virtualization:vt-x
36. L1D cache:32k//l1 Data Cache 32k
38. l1i cache:32k//l1 Instruction Cache 32k (Cow x machine Performance, von Neumann + Harvard architecture)
40. L2 cache:256k
42. L3 cache:15360k
44. NUMA node0 CPU (s): 0-5,24-29
46. NUMA Node1 CPU (s): 6-11,30-35
48. NUMA Node2 CPU (s): 12-17,36-41
50. NUMA node3 CPU (s): 18-23,42-47

From the output, it can be seen that the current machine has 4 sockets, each sockets contains 1 NUMA node, each NUMA node has 6 cores, each cores contains 2 thread, so the total threads number =4 (sockets) x 1 (node) x6 (cores) x2 (threads) =48.

Alternatively, you can print out the number of socket,core and thread of the current machine using the script below.

2. #!/bin/bash
6. # Simple Print CPU topology
8. # Author:kodango
12. function Get_nr_processor ()
14. {
16. grep ' ^processor '/proc/cpuinfo | Wc-l
18. }
22. function Get_nr_socket ()
24. {
26. grep ' physical id '/proc/cpuinfo | Awk-f: ' {
28. Print $ | "Sort-un"} ' | Wc-l
30. }
34. function Get_nr_siblings ()
36. {
38. grep ' Siblings '/proc/cpuinfo | Awk-f: ' {
40. Print $ | "Sort-un"} '
42. }
46. function Get_nr_cores_of_socket ()
48. {
50. grep ' CPU cores '/proc/cpuinfo | Awk-f: ' {
52. Print $ | "Sort-un"} '
54. }
58. Echo ' ===== CPU topology Table ===== '
60. Echo
64. Echo ' +--------------+---------+-----------+ '
66. Echo ' | Processor ID | Core ID | Socket ID | '
68. Echo ' +--------------+---------+-----------+ '
72. while read line; Do
74. If [-Z "$line"]; Then
76. printf ' | %-12s | %-7s | %-9s |\n ' $p _id $c _id $s _id
78. Echo ' +--------------+---------+-----------+ '
80. Continue
82. Fi
86. If echo "$line" | Grep-q "^processor"; Then
88. P_id= ' echo ' $line | Awk-f: ' {print $} ' | Tr-d ""
90. Fi
94. If echo "$line" | Grep-q "^core id"; Then
96. C_id= ' echo ' $line | Awk-f: ' {print $} ' | Tr-d ""
98. Fi
102. If echo "$line" | Grep-q "^physical id"; Then
104. S_id= ' echo ' $line | Awk-f: ' {print $} ' | Tr-d ""
106. Fi
108. Done </proc/cpuinfo
112. Echo
116. Awk-f: ' {
118. if ($ ~/processor/) {
120. Gsub (//, "", $);
122. p_id=$2;
124. } else if ($ ~/physical ID/) {
126. Gsub (//, "", $);
128. s_id=$2;
130. ARR[S_ID]=ARR[S_ID] "" p_id
132. }
134. }
138. end{
140. For (i in ARR)
142. printf "Socket%s:%s\n", I, arr[i];
144. } '/proc/cpuinfo
148. Echo
150. Echo ' ===== CPU Info Summary ===== '
152. Echo
156. Nr_processor= ' Get_nr_processor '
158. echo "Logical processors: $nr _processor"
162. nr_socket= ' Get_nr_socket '
164. echo "Physical socket: $nr _socket"
168. nr_siblings= ' Get_nr_siblings '
170. echo "Siblings in one socket: $nr _siblings"
174. nr_cores= ' Get_nr_cores_of_socket '
176. echo "Cores in one socket: $nr _cores"
180. Let Nr_cores*=nr_socket
182. echo "Cores in total: $nr _cores"
186. If ["$nr _cores" = "$nr _processor"]; Then
188. echo "Hyper-threading:off"
190. Else
192. echo "Hyper-threading:on"
194. Fi
198. Echo
200. Echo ' ===== END ===== '

————————————————————

NUMA Architecture Detailed