Linux Kernel Analysis Course summary

Linux Kernel Analysis Course summary

Name: Wang Zhaoxian

Study No.: 20135114

Note: Original works reproduced please specify the source + "Linux kernel analysis" MOOC course http://mooc.study.163.com/course/USTC-1000029000

First, how the computer works

    个人理解：计算机就是通过和用户进行交互，执行用户的指令，这些指令存放在内存中，通过寄存器存储，堆栈变化，来一步步顺序执行。

Second, the storage program computer working model

1.冯诺依曼体系结构—存储程序计算机    硬件角度（主板）：通过cpu中IP寄存器指向一个代码段运行某些指令；寄存区，指向内存的某一块区域（代码段）    程序员角度：将cpu抽象为一个for循环，只是执行下一条指令，从内存中取到下一条指令的内容。内存保存指令和数据，cpu负责解释和执行，通过总线连接。

Iii. Basics of X86 compilation

• 1.X86CPU Register
  32 bits: (Low 16 bits as 16-bit register AX,BX,CX,DX,BP,SI,DI,SP).
  General Register: EAX (accumulator), EBX (Base address register), ECX (count register), EDX (data register), EBP (stack base pointer), ESI (variable register), EDI (change register), ESP (stack top pointer).
  Segment Register: CS (code segment Register), DS (data segment Register), ES (additional segment register), SS (stack segment register), FS (additional segment register), GS (additional segment register).
  The CPU locates an instruction exactly according to the CS:EIP when the instruction is actually taken.
  A flag storage area that identifies some current states.
  64-bit registers: registers with R at the beginning
• 2. Assembly instruction B,w,l,q 8-bit, 16-bit, 32-bit, 64-bit, respectively
  Register mode, the register identifier that begins with%.
  The immediate number is the number that begins with $.
  Direct addressing is the data that accesses a specified memory address directly.
  Indirection is the memory that is accessed by the value of the Register as a memory address.
  A variable address is a value that alters a register at the time of an indirect addressing.
  MOV instruction: Register addressing MOVL%eax,%edx: puts the contents of the EAX register into the edx register.
• 3. Addressing method
  Immediate addressing MOVL $0x123,%edx: Put the 0x123 directly into the edx register;
  Direct addressing MOVL 0x123,%edx: Place the data pointed to by the memory address 0x123 in the edx register;
  Indirect addressing MOVL (%EBX),%edx: The value stored in the EBX register as the memory address, and the data is taken out into the edx register;
  Addressing MOVL 4 (%EBX),%edx: Add the value of the EBX register to 4 as the memory address, take out the data into the EDX register
• 4.Push instruction: Press stack PUSHL%eax
  Subtract 4 from the top pointer of the stack, and then place the value in the EAX register in the memory that the ESP points to.
• 5.POP instruction: Out of Stack popl%eax
  Place the data stored in the memory pointed to by the top pointer in the EAX register, and then add 4 to the top of the stack pointer.
• 6.Call command: Call 0x12345
  Put the current EIP on the stack and put the 0x12345 in the EIP register
• 7.RET directive: RET
  Restores the EIP value saved in the call instruction to Eip,ret after executing the EIP before calling, that is, the next instruction before
  * Indicates a pseudo-instruction, cannot be used directly by the programmer, the EIP register can not be directly modified, can only be indirectly modified by special instructions.

Iv. three magic weapons of computer work

• 1. Stored program computer working model, the most basic logical structure of computer system;
• 2. Function call stack, high-level language to run the foundation, only the machine language and assembly languages when the stack mechanism for the computer is not so important, but with high-level language and functions, the stack has become the basic function of the computer;
• 3. Interrupt, multi-channel program operating system base point, no interrupt mechanism program can only run from the beginning to the end of the possibility to start running other programs.

V. Stack-related registers: ESP (stack-top pointer) EBP (stack-bottom pointer)

    ebp在C语言中用作记录当前函数调用基址。    cs：eip：总是指向下一条的指令地址（顺序执行）。    跳转/分支：call，将当前cs：eip的值压入栈顶，cs：eip指向被调用函数的入口地址    Ret，将保存在栈顶的cs：eip的值弹出，放入cs：eip中。

Vi. Two Swords of computer work: interrupt context and process context switch Vii. key features of the Linux kernel:

• 1, support dynamic loading kernel module;
• 2, support symmetric multi-processing (SMP);
• 3, the kernel can be preempted (preemptive), allowing the kernel to run a task with the ability to prioritize execution;
• 4. Do not differentiate between threads and processes.

Viii. operating system and kernel

• Kernel: Interrupt Service Program in response to interrupt, manage multiple processes, share processor Time Scheduler, manage process address, space memory management program, network, interprocess communication and other functions.
• Kernel space: The system State and the protected memory space.
• System call: The application communicates with the kernel.

Nine, the system calls three layers of skin: Xyz,system-call and sys-xyz.

    什么是系统调用——系统调用就是用户程序和硬件设备之间的桥梁。用户程序在需要的时候，通过系统调用来使用硬件设备。

• 1, the kernel state: At high execution level, the code can execute the privileged instruction, access any physical address, the CPU execution level corresponds to the kernel state. In the corresponding low-level execution state, the control of the code will be limited. can only be active within the scope allowed by the corresponding level. Thus ensuring a more stable system.
• 2, the Intel x86 CPU has four different execution level 0-3,linux only uses 0 and three respectively to represent the kernel state and the user state.
• 3. The lowest two digits of the CS register represent the privilege level of the current code.
• 4, 0xc0000000 above the address space can only be accessed in the kernel state, the 0X00000000-0XBFFFFFFF address space can be accessed in both states. (Logical address)
• 5, interrupt processing is the main way to enter the kernel state from the user state.
• 6, System call is a special kind of interruption.
• 7. When switching from the user state to the kernel state, the register context of the user state must be saved. The Interrupt/int command saves some register values on the stack register. (User state stack top address, then state word, then CS:EIP value)
• 8, the first thing after the interruption occurs is to save the scene. To protect the scene is to enter the interrupt program, to save the value of the register required, the recovery site is to exit the interrupt program, reply to save the Register data. End of interrupt processing the last thing is to restore the scene.
• 9, the operating system for the user-state process and hardware devices to interact with a set of interfaces-system calls: The user from the bottom of the hardware programming to free up, greatly improve the security of the system, user programs are portable.
• 10, the application Programming interface API (a function), not every API corresponds to a system call.
• 11. When the user-state process calls a system call, the CPU switches to the kernel state and starts executing a kernel function
• 12. The system call number uses the EAX register. Each system call has at least one parameter.
• 13, the limit of the Register pass parameter: The length of each parameter cannot exceed the length of the register (32 bits), the number of parameters cannot exceed six (EBX,ECX,EDX,ESI,EDI,EBP) in addition to the system call number (EAX). If more than six, a single register parameter is used as a pointer to a piece of memory.

Ten, the existence of system calls, has the following important meanings:

    1.为用户空间提供一种硬件的抽象接口；    2.保证系统稳定和安全；    3.除异常和陷入，是内核唯一的合法入口。

XI. System Call Context

    内核在执行系统调用的时候处于进程上下文。current指针指向当前任务，即引发系统调用的那个进程。在进程上下文中，内核可以休眠并且可以被抢占。这表明即使是在内核空间中，当前进程也可以被其他进程抢占。因为新的进程可以执行相同的系统调用，所以必须保证系统调用是可重入的。当系统调用返回时，控制权仍然在system_call()中，它最终会负责切换到用户空间并让用户继续执行下去。

12. Process Control block PCB

    task_struct又称进程描述符，是操作系统用于管理控制进程的一个专门的数据结构，记录进程的各种属性，描述进程的动态变化过程，而PCB是系统感知进程存在的唯一标志。

13, the operating system three major functions: Process management (CORE), memory management, file system. 14. Process Type

• 1.I/O consumption process: Most of the time is used to submit I/O requests or wait for I/O requests, often in a running state, but running for a short time, waiting for the request process to be in a blocking state. such as interactive programs.
• 2. Processor-intensive processes: Most of the time is spent executing code, unless preempted or kept running.
• 3. Integrated type: I/O consumption type and processor consumption type.
• 4. The scheduling strategy seeks to strike a balance between rapid process responsiveness (short response times) and maximum system utilization (high throughput).

XV, process state transitions:

16. Process Management 1. Process descriptor and task structure

list）的双向循环链表中。链表中的每一项包含一个具体进程的所有信息，类型为task_struct，称为进程描述符（process descriptor），该结构定义在<linux/sched.h>文件中。Linux通过slab分配器分配task_struct结构，这样能达到对象复用和缓存着色（cache coloring）的目的。另一方面，为了避免使用额外的寄存器存储专门记录，让像x86这样寄存器较少的硬件体系结构只要通过栈指针就能计算出task_struct的位置，该结构为thread_info，在文件<asm/thread_info.h>中定义。

2. Process status

    task_struct中的state描述进程的当前状态。进程的状态一共有5种，而进程必然处于其中一种状态：

• 1) task_running (run)--the process is executable, it is either executing, or waiting to be executed in the run queue. This is the only possible state that the process executes in the user space, or it can be applied to the process being executed in the kernel space.
• 2) task_interruptible (interruptible)--The process is sleeping (i.e. it is blocked) waiting for certain conditions to be reached. Once these conditions are reached, the kernel will set the process state to run, and the process in this state will be woken up and put into operation because the signal is received.
• 3) Task_uninterruptible (non-disruptive)-this state is the same as the interruptible state, except that it will not be woken up for receiving a signal to be put into operation. This state usually occurs when the process must wait without interference or wait for the event to occur soon. Because tasks in this state do not respond to signals, they are less used than interruptible states.
• 4) Task_zombie (zombie)--the process has ended, but its parent process has not yet called the WAIT4 () system call. The process descriptor for the child process is still preserved for the parent process to be able to learn its message. Once the parent process calls WAIT4 (), the process descriptor is freed.

• 5) task_stopped (stop)--the process stops executing and the process is not operational or operational. Usually this state occurs when a signal such as Sigstop,sigtstp,sigttin,sigttou is received. In addition, any signal received during debugging will cause the process to enter this state.
  The status of the process needs to be adjusted, preferably using the Set_task_state (task, state) function, which, when necessary, sets the memory barrier to force the other processor to reorder (SMP).

  3. Process creation

  在Linux系统中，所有的进程都是PID为1的init进程的后代。内核在系统启动的最后阶段启动init进程。该进程读取系统的初始化脚本（initscript）并执行其他的相关程序，最终完成系统启动的整个进程。Linux提供两个函数去处理进程的创建和执行：fork()和exec()。首先，fork()通过拷贝当前进程创建一个子进程。子进程与父进程的区别仅仅在于PID（每个进程唯一），PPID（父进程的PID）和某些资源和统计量（例如挂起的信号）。exec()函数负责读取可执行文件并将其载入地址空间开始运行。fork()使用写时拷贝（copy-on-write）页实现。内核在fork进程时不复制整个进程地址空间，让父进程和子进程共享同一个拷贝，当需要写入时，数据才会被复制，使各进程拥有自己的拷贝。在页根本不会被写入的情况下（fork()后立即exec()），fork的实际开销只有复制父进程的页表以及给子进程创建唯一的task_struct。

  4. Threading implementation

  从Linux内核的角度来说，它并没有线程这个概念。Linux把所有的线程都当作进程来实现，内核并没有准备特别的调度算法或者定义特别的数据结构来表征线程。相反，每个线程都拥有唯一隶属于自己的task_struct，它看起来就像是一个普通的进程，只是该进程和其他一些进程共享某些资源，如地址空间。

  5. Process termination

  进程在运行结束，或接受到它既不能处理也不能忽略的信号，或异常时，都会被终结。此时，依靠do_exit()（在kernel/exit.c文件中）把与进程相关联的所有资源都被释放掉（假设进程是这些资源的唯一使用者）。进程不可运行（实际上也没有地址空间让它运行）并处于TASK_ZOMBIE状态。它占用的所有资源就是内核栈、thread_info和task_struct。在父进程获得已终结的子进程的信息后，或者通知内核它并不关注那些信息后，子进程的task_struct才被释放。如果父进程在子进程之前退出，必须有机制保证子进程能找到一个新的父类，否则的话这些成为孤儿的进程就会在退出时永远处于僵死状态，白白的耗费内存。解决方法是给子进程在当前线程组内找一个线程作为父亲，如果不行，就让init做它们的父进程。

17. Process scheduling

• 1. What is scheduling
  Now the operating system is multi-tasking, in order to enable more tasks to better run on the system at the same time, need a management program to manage the computer on the simultaneous running of the various tasks (that is, the process). This management program is the scheduler.
  Its features are simple to say: Decide which processes to run, which processes to wait for how long each process will run in addition, in order to achieve a better user experience, the running process can be immediately interrupted by other more urgent processes.
  In short, scheduling is a balanced process. On the one hand, it wants to ensure that each??? Process to maximize the use of the CPU (that is, as few switching processes, process switching too much, the CPU time will be wasted on switching), on the other hand, to ensure that the process can be fair to use the CPU (that is, to prevent a process for a long time exclusive CPU).
• 2. Scheduling implementation principle
  2.1. There are 2 ways to prioritize process priority processes
  One is the nice value, the range of nice values is -20~+19, and the higher the value the lower the priority, which means that the nice value is 20 with the highest process priority.
  One is the real-time priority, the real-time priority range is 0~99, contrary to the nice value definition, real-time priority is the higher the value the greater the priority. Real-time processes are some processes that require relatively high response times, so processes with real-time priority in the system are running queues, and they preempt the normal process run time.
  2.2. About Time slices
  With a priority, you can decide who will run first. However, for the scheduler, it is not the end of the run time, you must know how often the next schedule. Then there is the concept of time slices. A time slice is a numeric value that indicates how long a process can continue to run before it is preempted. It can also be thought of as the time that the process ran before the next scheduled occurrence (unless the process actively abandons the CPU, or there is a real-time process to preempt the CPU). The size of the time slice setting is not simple, set large, the system response is slow (long schedule), set small, process frequent switching brought about by the processor consumption. The default time slice is typically 10ms

18, executable program generation

    编译器预处理（负责把include的文件包含进来及宏替换等工作）；编译成汇编代码；编译器编译成目标代码；再链接成可执行文件；操作系统加载到内存中来执行。

19. Timing analysis of process scheduling and process scheduling

• 1. Different types of processes have different requirements for scheduling requirements:
  First Category:
  -i/o-bound: Frequent I/O, often spending a lot of time waiting for I/O operations to complete
  -cpu-bound: Computationally intensive, requiring a lot of CPU time to perform operations
  Second Category:
  -Batch process: Do not need to interact with the user, usually run in the background;
  -Real-time process: real-time demand, not blocked by low priority processes, short response time, stable;
  -Interactive process: requires frequent interaction with the user; fast response time
• 2. Scheduling policy: A set of rules that determines when and how to choose a new process to run.
• The 3.Linux process sorts by priority, calculates the process priority with a specific algorithm, and uses a value to indicate how the process is allocated appropriately to the CPU. The priority is dynamic, periodically adjusted according to the behavior of the process, which is not assigned to the CPU for a long time, and is already running on the CPU for a long time to decrease the priority level.
• The 4.Schedule function is used to implement scheduling, find a process in the queue, and assign the CPU to him.
• 5. Timing of Process scheduling
  Interrupt processing (including clock interrupts, I/O interrupts, system calls, and exceptions), call schedule () directly, or call schedule () based on the need_resched tag when returning to the user state;
  Kernel threads can directly call schedule () for process switching, or in the process of interrupt processing, which means that kernel threads as a special kind of process can be active scheduling, but also can be passively dispatched;
  The user-state process cannot implement the active scheduling, but can only be dispatched by a point in time after the kernel state, that is, scheduling during interrupt processing. The user-state process can only be dispatched passively;
  Kernel process is a special process that only the kernel state has no user state, and can be actively dispatched or passively dispatched.

Several special cases:

• By interrupting the timing of the processing process, the user-state process and kernel threads switch between each other and the kernel threads switch to each other, very similar to the most common situation, but the kernel thread is running in the process of interruption without process user state and kernel state conversion;
• Kernel thread actively calls schedule (), only the process context of the switch, there is no interrupt context switch, and the most general situation is slightly abbreviated;
• The system call that creates the child process starts at the execution point in the subprocess and returns the user state, such as fork (next-ip=ret-from-fork);
• A situation in which a new executable program is loaded and returned to the user state, such as EXECVE;

Self-summary

    通过几个月时间的学习，首先，我获得了很多知识，无论是网课还是书本都让我受益匪浅：动态的展示堆栈的变化，操作系统的三大法宝，扒开系统调用三层皮，API xyz，中断向量system_call，中断服务程序sys_xyz，再从操作入手，使用gdb跟踪；中断，进程切换、系统调用都都相当重要。但是经过这么长时间的学习，我感觉还是不能很好地系统化理解和掌握所学的Linux体系，不能将所学到的学以致用。知识点零散，上手难度不小，不便于深入理解和掌握，但我希望能通过接下来的进一步学习，努力打通各个知识点之间的连接，真正进入Linux知识的海洋。

Linux Kernel Analysis Course summary