[Linux] process management, memory management, file system, system management, network operations Overview

Process/thread

Process concept: A process/thread is the smallest execution unit in a computer and the basis for implementing a time-sharing multi-user operating system. How can a process be implemented in Linux? If you are familiar with the kernel, you will know that the kernel is a pile of data structures. All the statuses of a process are included in the task_struck structure: storing the state of the Process, file/socket descriptor, the program address (stask/heap), time slice, DDL address, permission information, and other information. For example, the description of the process in the memory is displayed:

 

 

 

Processes in Linux are a linked list. Processes and sub-processes have a parent-child relationship. The process with process Number 1 is INIT. The sub-processes created by the process use the principle of copy at write time;

Thread concept: A process can contain many threads. Each thread of a process can easily share the memory. When a thread is created, only one stack space belonging to the thread is created;

Command: PS-Ef; PS-Elf; pstree; pidstat;

Process status: running; stopped, uninterruptible; interruptitble; sleeping; zombie;

Process destruction: All data structures (release file/socket handle, memory, etc.) will be released when the process is destroyed. After the process is released, a signal will be sent to the parent process, at this time, the sub-process is in the zombie state. The sub-process completely destroys the sub-process when the parent process responds to this signal. If the parent process does not respond to this signal, the sub-process state will change to the zombie state;

Process Scheduling: Linux is a preemptible scheduling policy, that is, processes running on the CPU can be preemptible by other processes at any time. How to schedule processes in Linux is very important. the Linux kernel allocates time slices of different lengths based on the priority of processes (Real-time priority) and whether processes are CPU-sensitive or I/O-sensitive; it can be said that the process scheduling policy is dynamic and intelligent; the time complexity of the process scheduling algorithm is O (1). By creating an array for different priorities of the process, that is, a priority corresponds to an array. the same priority is stored in the queue, and a binary bitmap is used to locate the priority queue to be scheduled from the array for constant time. Cleverly exchanges space for time to maximize the scheduling performance.

Context switching

Context switch refers to the context switch of processes/Threads. context switches are completed by the Linux kernel and the context information of the currently running processes is detached from the register to be saved in the process stack, load the new process to the register and execute it in the cache. Therefore, frequent context switching is one of the most influential factors in program performance. For example:

 
 

 

Reducing the context switching frequency is the key to improving the process performance. For our network servers, a large number of customer requests flood into the servers at any time, frequent CPU response to network interruptions will significantly affect the performance of our applications. How can we avoid frequent context switching? In addition to improving the priority of processes, the SMP Kernel provides the affinity technology to enable Network interruptions to a single CPU. In this way, frequent interruptions will not interfere with the execution of application processes on other CPUs;

View the context switching frequency: pidstat-W or vmstat; view affinity settings: CAT/proc/IRQ/$ IRQ/smp_affinity

Memory Management

In Linux, the memory is divided into two parts: kernel space and user space. For 32bit systems, the kernel space is 1 GB, and the user space is 3 GB. the Linux kernel code directly accesses the kernel space, applications are not allowed to access kernel-space;

In Linux design, "security" and "multi-process concurrency" are considered. User-space is virtualized, Which is isolated from the kernel space, each process has its own independent virtual process address space (which does not interfere with the addresses of other processes) when it is created; the virtual process space has 3 GB; in this way, each process looks like it has 3 GB of accessible memory space; it can be said that there is no memory virtualization, there is no concurrent running process, there is no multi-user support.

In Linux, how does one allocate memory? : The kernel allocates memory in units of pages. The page size is generally 4 kb and 8 KB. When a process is created, the memory (HEAP) applied by the application is all virtual memory addresses. when the process is running and the memory is actually accessed, if this memory has never been applied for physical memory or this memory is not in the physical memory (SWAp ). the process will be in the kernel state (called by the system), generate a page fault exception, apply for physical memory, and map it with the virtual memory.

How is the virtual memory mapped to the physical memory? Each process has a page table to maintain the ing between virtual memory and physical memory. When executing code, the CPU first converts the virtual memory to physical memory through MMU, and then accesses the memory address. The Linux kernel also uses the data structure TLB to cache the ing between virtual memory and physical memory. For example:

 

 

 

Command: view the Virtual Memory Distribution of processes: CAT/proc/$ PID/maps; view page fault: pidstat-R;

File

Linux's file sub-system blocks the implementation of various file systems; provides unified APIs for application file systems; and the file system maintains a cache in memory, the file content is cached according to the LRU algorithm. The kernel thread pdflush regularly or the percentage of memory occupied by the cache will be triggered to write the cache content to the disk. For example:

 

Common file systems: ext3, XFS, and JF (each file system is suitable for different application scenarios and delivers superior performance)

Command: fsck, fdisk, Mount-T ext3/dev/SDA/home/xxx; umount/home/xxx; iostat; SAR;

System (CPU/memory/Hard Disk/IO)

The Linux hierarchy is structured. For example, the CPU, memory, Io, and disk resources of the system are all centrally managed by the kernel. to use these resources, the application layer program must call the kernel; the kernel calling function at the application layer is completed through system call:

How is system calling implemented? At present, most operating systems are implemented through soft interruptions. When an application calls a kernel function, it sends an interrupt to the CPU at the same time. In the interrupt vector table, the CPU finds the handler corresponding to the interrupt number and runs the kernel code. At present, x86 and other CPUs have supported the system to call this function from the hardware layer, and the performance has improved a lot. Because the less a program, the better the system call performance. Run the strace yourexeapplication command to check that all your executable code calls those system calls;

Currently, the common machine architecture is the SMP structure (that is, all CPUs share L1/L2, memory, and peripherals), which is different from the NUMA structure; therefore, a process/thread will only run on one core at a certain moment; a process will frequently run on different CPUs and its performance will be reduced because each core does not share the high-speed cache, if the process is frequently switched between cores, the high-speed cache hit rate decreases, seriously affecting performance. The operating system kernel ensures that a process runs on one core as much as possible and ensures Load Balancing for each core. We know that the CPU needs to respond frequently to interruptions, and some of them are interrupted. The CPU must put down the processes being executed to respond to the interruptions. For example, for a server with high network traffic, the network is interrupted frequently; the operating system provides an affinity algorithm to bind a terminal number to a core. In this way, other cores will not be affected by interruptions, improving system performance; CAT
02>/proc/IRQ/XX/smp_affinity

Command: Top, uptime, vmstat; PS; pidstat;

Network (TCP/UDP)

The freedom of GNU/Linux comes from the network, and comes from the masterpiece completed by top programmers at various ends of the network. It can be said that without TCP/IP, Linux will not flourish. Linux is also unparalleled in the development of networks. The linux2.6 kernel supports synchronous and asynchronous network I/O. asynchronous support allows the current network server to support more requests. For example, Apache uses epoll to allow a small number of worder thread pools to respond to a large number of requests. For example:

 

Network connections are divided into listener connections and transmission connections. For example, the Apache linsten connection listener waits for customer requests on port 80. After customer requests come, the listener connection will establish three handshakes with client requests, the connection is thrown to the working thread to process the request. At this time, the client request and the worder thread establish the established State for data transmission.

Command: netstat, traceroute, ifconfig; Dig; hostname;