Linux Performance Tuning Summary

System performance has always been a hot topic. Do operation of the past few years have been engaged in performance tuning, writing this article is also a summary of the work.

The first step in tuning is to say why tuning? That is, the system analysis, analysis also need to have indicators, good performance monitoring situation, see really need tuning to proceed. "Tuning" for tuning is not tuning, it is destruction.

Purpose of the performance analysis

1. Identify system performance bottlenecks
2. Provide a solution or reference for future optimization
3. To achieve good use of resources. Hardware resources and software configuration.

Factors that affect performance

To determine what the factors are, first determine what type of application you are using.
For example:

1. CPU-Intensive
   For example, a Web server like Nginx node. JS requires a CPU for batch processing and math to belong to this type
2. IO-intensive
   For example, MySQL, a common database, consumes a lot of memory and storage systems, and the CPU and network requirements are not high, the application uses the CPU to initiate IO requests, and then enters the sleep state.

Identify the type of application and start to analyze what can affect performance:

1. A large number of Web requests will fill the running queue, a large number of context switches, and interrupt
2. A large number of disk requests
3. Network card large amount of throughput
4. and memory exhaustion, etc...

That comes down to 4 things.

1. Cpu
2. Memory
3. / o
4. Network

Tools for System Detection

We know that these four blocks affect our performance, what tools do we have to test?

When a foreign great God summed up.

My personal work is often used to:
Htop Vmstat iotop SAR strace iftop SS Lsof Ethtool mtr etc.

In addition, Ali's Tsar and glances are recommended for system performance monitoring.

CPU Performance Monitoring and tuning

We can optimize by examining the CPU usage and by observing the context switches, interrupts, and code calls of the tool.

First clarify several terms:
Caching: The CPU provides hardware-level caching in order to provide memory I/O performance. Viewing the cache can be viewed using the lscpu-p command

# lscpu L1d cache:             32KL1i cache:             32KL2 cache:              256KL3 cache:              8192K

Level 1 caches are static caches and are divided into data caches and instruction caches.
2-and 3-level caches are dynamic caches, where the 2-level cache is a shared cache.

In order to increase the CPU cache hit rate, it is common practice to bind the CPU to a certain core, which is "CPU affinity".
Under Linux we can do this with the "taskset" command.

# taskset -pc 0 73890pid 73890‘s current affinity list: 0pid 73890‘s new affinity list: 0

But that's still a problem. For example, there is no guarantee of local memory allocation, so we need to use NUMA to solve the problem

NUMA: Non-conforming memory access mechanism. Every core in the house has its own memory, which is a local node, and has its own memory controller, which is called the neighbor node from the nearest memory node.

A simple topology for NUMA, originating from the Internet.

Numactl can bind a program to a specific NUMA node

Note: The database server does not use NUMA, and if you want to use it, please use Numactl-interleave=all in the database startup. As operations may have been a pit.

CPU Scheduling Policy

2. Real-Time scheduling strategy
   • Sched_fifo static scheduling policy, once the CPU is occupied, runs continuously until a higher priority task arrives or abandons itself.
   • SCHED_RR time polling policy, when the time slice of the process is exhausted, the system will redistribute the time slices and place them at the end of the ready queue. Placing at the end of the queue ensures that all RR tasks with the same priority are scheduled fairly.
     The real-time scheduling policy has a value of 1-99, and the higher the number the greater the priority.
4. General strategy
   • Sched_other The default scheduling policy determines the weights by the nice and counter values, the smaller the nice, the greater the counter, the greater the probability of being dispatched, i.e. the process that used the least CPU will be prioritized. The action value is 100-139, and the smaller the number the higher the priority.
   • Sched_batch
   • Sched_idle

Chrt modify real-time priority, in production generally do not modify, the default is RR scheduling

Sched_other is modified with nice and renice.
In addition, other support dynamic adjustment, if you manually directly nice modification.

Context Switches: Contextual switching

The Linux kernel treats each core as a separate processor. A kernel can run 50~50000 processes at the same time. Each thread will be allocated a time slice until the thread runs out of time slices or is preempted by a higher priority thread before it is re-placed back into the CPU queue. The process of switching threads is context switch. The higher the context switch, the greater the workload of the kernel scheduler.

Vmstat can see both high and low CS

Run queue

Each CPU has a running queue. Thread, either in the sleep state (blocking and waiting for IO), or in a running state. The longer the queue runs, the longer it will be waiting for the CPU to process the thread. The run queue is global and will be shared by all CPUs

Load is used to describe the running queue. Its value equals the thread that is currently being processed + the thread inside the run queue.

For example, the current system core number is 2, there are two threads executing rows, and 4 threads are running in the queue, then it's load=2+4.

Vmstat W uptime can be used to observe the load of the running queue.

CPU Performance Monitoring

With all that said, what values do you need to observe under normal circumstances?
First, NUMA and algorithms are optimized in special cases, which are generally not to be moved, and require binding adjustments based on your business scenario, such as virtualization, cloud computing, and so on, which may need to be adjusted.

The performance points that we need to observe on a daily basis are:

2. CPU utilization
   • US 60%-70%
   • Sy 30%-35%
   • ID 0%-5%
4. CS Context Switch
   • CS and CPU utilization related, if can maintain the above mentioned utilization large number of switching can accept
6. Run queue
   • Less than or equal to 4 best

Example:

# vmstat 1 5procs -----------memory---------- ---swap-- -----io---- --system-- -----cpu------ r  b   swpd   free   buff  cache   si   so    bi    bo   in   cs us sy id wa st 3  0 1150840 271628 260684 5530984    0    0     2     1    0    0 22  4 73  0  0 5  0 1150840 270264 260684 5531032    0    0     0     0 5873 6085 13 13 73  0  0 5  0 1150840 263940 260684 5531040    0    0     0     4 6721 7507 15 13 72  0  0 4  0 1150840 263320 260684 5531068    0    0     0     0 6111 7117 10 13 76  0  0 4  0 1150840 262328 260684 5531072    0    0     0     0 6854 7673 18 13 68  0  0

The CPU interrupt (in) and Context switch (CS) in the example are relatively high, indicating that the kernel has to switch processes back and forth, while in is also relatively high indicating that the CPU has been requesting resources.

Memory Memories

Terms
MMU:
The CPU is not able to deal with the hard disk, only the data is loaded into memory can be called by the CPU. When accessing memory, the CPU needs to be like a memory monitor request, a read-write request that monitors program control and allocates memory, which is called the MMU (memory management unit).

Linear address to physical address mapping, if you follow 1 bytes 1 byte mapping, you need a very large table, this conversion relationship is very complex. Therefore, the memory space is divided into another storage unit format, usually 4K.

The buffer TLB is required for each process to find the page table if it needs to access the memory, but it is not possible to find a TLB or a large number of lookups will cause slowness, so there is a hierarchical directory of page table. Page table can be divided into 1 levels of directories, 2 levels of directories and offsets.

There are two ways to manage large amounts of memory in the system:

1. increase the number of page tables in the Hardware memory snap-in
2. increase page size
   The first method is less realistic, and all of us consider the second method. That is: Large page.
   32-bit System 4m large page box 64-bit System 2m large page box, the thicker the page box is wasted more serious.
   View large pages of the system:
   cat/proc/meminfo

   anonhugepages:309248 KB
   hugepages_total:0
   hugepages_free:0
   hugepages_rsvd:0
   hugepages_surp:0
   hugepagesize:2048 KB
   directmap4k:6144 KB
   directmap2m:1042432 KB
   directmap1g:0 KB
   anonhugepages: transparent large page, THP is an abstraction layer that automatically creates, manages, and uses most aspects of oversized pages.
   Additionally HP must be set at boot time.
   Number of pages manually set for large pages:
   sysctl vm.nr_hugepages =

DMA: Direct Read memory
In the implementation of DMA transmission, the DMA controller is directly in charge of the bus, so there is a bus control transfer problem. That is, before the DMA transfer, the CPU to the bus control to the DMA controller, and after the end of the DMA transfer, the DMA controller should immediately return the bus control to the CPU. A full DMA transfer process must be 4 steps through DMA request, DMA response, DMA transfer, DMA termination.

Virtual Memory:
Every process on a 32-bit system accesses memory, each process as if its own 4 g of memory space is available, called Virtual memory (address), virtual memory into physical memory is done through the MMU. We try not to use virtual memory in production.

Several memory parameters that affect system performance:

2. Overcommit_memory excessive use of memory
   • 0 The default setting system determines whether it is excessive.
   • 1 Not excessive use
   • 2 excessive use but a certain proportion of the default percent of 50 is determined by overcommit_ratio (he is the default of 50), for example physical memory 8g,swap4g, can be used over 10g.
     Note: Try to avoid excessive use in production, such as Redis to turn off excessive use.
4. Spappines
   -Swap inactive processes into swap. Note: Try not to use swap.
   Production settings:
   ECHP >/proc/sys/vm/swappines
6. Reclaim Memory
   • This value is set to 1, 2, or 3 to let the kernel discard various combinations of page caches and slab caches.
     1 system is invalid and frees all paged memory as buffers
     2 The system frees all unused slab buffer memory. namely cached
     The 3 system frees all paged and slab buffered memory.
     Used in production:
     1. Run Sync
   1. Echo 3>/proc/sys/vm/drop_caches

/ o

The IO subsystem is generally the slowest part of a Linux system. One reason is its distance from the CPU, and the other reason is its physical structure. So try to reduce disk IO.

Disk Scheduling policy:

Where the CFQ policy is currently used.
CFQ: Completely fair dispatch. In its time fragment, the process can have a maximum of eight requests each time (the default
Recognition). The scheduler tries to estimate whether a program will emit more I/O in the near future based on historical data, and then CFQ will be idle, waiting for that
I/O, even if there are other processes waiting to emit I/O
Deadline: Each request must be serviced before the specified deadline.
NoOp: No Policy
Anticipatory: has been discarded, write more read less scenes use.

The Linux kernel accesses disk IO in page units, typically 4K.
View page:/usr/bin/time-v date

MPF
Linux maps the physical address space of the memory to virtual memory, and the kernel only maps the memory pages that are needed. When the app starts, the kernel searches for the CPU cache and physical memory in turn, finds out if there is a corresponding memory page, and if not, the kernel will initiate the MPF (Major page fault) to read the data from the disk and cache it in memory.

If the corresponding memory page is found in buffer cache, a MNPF (Minor page fault) will be generated.

/usr/bin/time-v HelloWorld
The first execution will find that most of the MPF
The second execution will find that most of the MNPF

The File Buffer Cache

The file buffer cache is used to reduce the MPF and increase MNPF, which will continue to grow until there is less available memory or the kernel needs to release some memory for other applications. Free memory is less, does not indicate the system memory is tight, can only indicate that the Linux system fully use memory to do the cache.

# cat /proc/meminfo MemTotal:        1004772 kBMemFree:           79104 kBBuffers:          105712 kB

Write a data page back to disk
You can use Fsync () or sync () to write back immediately, and if these functions are not called directly, Pdflush will periodically brush back to the disk.

Iotop can show the IO occupancy of all processes
Lsof can view all the calls and open files

Other commands:
Vmstat SAR iostat top htop, etc.

?

Linux Performance Tuning Summary