Linux SMP Startup Process

1. SMP hardware architecture:

For SMP, it is easiest to understand that the system has multiple identical CPUs. All CPUs share the bus and have their own registers. Shared bus is used for memory and external device access. In the Linux operating system, multiple CPUs share the same ing in the system space, which is completely equivalent.

 

Because there are multiple CPUs in the system, this introduces a problem. When an external device is interrupted, which CPU is used for processing?

To this end, Intel proposed the architecture of Io APCI and local APCI.

Io APIC connects to various external devices, and can set the distribution type. According to the set distribution type, the local APIC of the CPU that sends the interrupt signal.

The local APIC is responsible for processing the interruption of the local CPU. The local APIC not only accepts the interruption of the io apic, but also needs to handle the exception of the local CPU. The local APIC also provides a timer.

 

How to determine which CPU is the boot CPU?

According to information from Intel, after the system is powered on, a CPU will be randomly selected as BSP Based on MP Initialization Protocol. Only BSP will run the BIOS program, and all other APs will enter the waiting status, BSP sends an IPI interrupt before it can run. For more information about the MP initialization protocol, see intel 64 and IA-32 ubuntures software developer's Manual Volume 3A: system programming.
Guide, Chapter 1.

 

How does the boot CPU control other CPUs to start running?

BSP can control the AP to run from the specified starting address through IPI message. The local APIC integrated in the CPU provides this function. You can send an IPI message to the specified CPU by writing the registers provided in the local APIC.

 

How do I obtain information about the system hardware CPU?

After the system initialization, the hardware will provide information about the CPU, bus, Io APIC, and so on at the specified location in the memory, that is, smp mp table. During Linux Initialization, this location is read to obtain system-related hardware information.

 

2. Introduction to the Linux SMP Startup Process

Setup_arch ()

Setup_memory ();

Reserve_bootmem (page_size, page_size );

Find_smp_config (); // locate the smp mp table

Smp_alloc_memory ();

Trampoline_base = (void *) alloc_bootmem_low_pages (page_size); // allocate the trampoline, which is used to start the AP boot code.

Get_smp_config (); // obtain the specific hardware information based on the smp mp table.

Trap_init ()

Init_apic_mappings ();

Mem_init ()

Zap_low_mappings (); If SMP is not defined, the address ing of the user space is clear.

Rest_init ();

Kernel_thread (init, null, clone_fs | clone_sighand );

Init ();

Set_cpus_allowed (current, cpu_mask_all );

Smp_prepare_cpus (max_cpus );

Smp_boot_cpus (max_cpus );

Connect_bsp_apic ();

Setup_local_apic (); // initialize the local APCI of BSP.

Map_cpu_to_logical_apicid ();

Do_boot_cpu (apicid, CPU) is called for each CPU)

Smp_init (); // scheduling starts for each CPU.

 

Trampoline. s ap Boot Code, hexadecimal code, enabling protection mode

Head. s creates Page Management for the AP

Initialize_secondary jump to start_secondary based on the information of the previous fork creation settings

Start_secondary checks whether the BSP is started. If the AP is started for task scheduling.

 

3. Code learning Summary

Find_smp_config (); to find the location of the MP table in the memory. For specific protocols, see Chapter 4th of the MP protocol.

This table describes the hardware information of the system, such as CPU, bus, and Io APIC.

Two related global variables: smp_found_config: whether to find the linear address of smp mp table and mpf_found smp mp table.

 

Smp_alloc_memory () allocates memory space for the Startup Program of the AP. Related global variable trampoline_base: Linear address of the allocated startup address.

 

Get_smp_config () obtains hardware information based on the content provided in the MP table.

 

Init_apic_mappings (); gets the ing addresses of Io APIC and local APIC.

 

Zap_low_mappings (); If SMP is not defined, the address ing of the user space is clear. Clear the table items in swapper_pg_dir.

 

Setup_local_apic (); initialize the local APCI of BSP.

 

Do_boot_cpu (apicid, CPU)

Idle = alloc_idle_task (CPU );

Task = copy_process (clone_vm, 0, idle_regs (& regs), 0, null, null, 0 );

Init_idle (task, CPU );

Copy the INIT process using copy_process and call the init_idle function to set the CPU that can run.

Idle-> thread. EIP = (unsigned long) start_secondary;

Modify thread. EIP in task_struct to run the start_secondary function after the AP Initialization is complete.

 

Start_eip = setup_trampoline ();

Call the setup_trampoline () function and copy the code from trampoline_data to trampoline_end to trampoline_base. trampoline_base is the memory applied for at setup_arch. The return value of start_eip is the physical address corresponding to trampoline_base.

 

Smpboot_setup_warm_reset_vector (start_eip); set the memory 40: 67 h to start_eip as the startup address.

 

Wakeup_secondary_cpu (apicid, start_eip); in this function, the BSP sends an IPI message to the target AP by operating the apic_rn register. This triggers the target AP to run from start_eip address to actual mode.

 

Trampoline. s

Entry (trampoline_data)

R_base =.

Wbinvd # needed for NUMA-Q shocould be harmless for others

MoV % CS, % ax # code and data in the same place

MoV % ax, % DS

 

CLI # We shoshould be safe anyway

 

Movl $0xa5a5a5a5, trampoline_data-r_base

This is to set the identity so that BSP knows that the AP is running here.

 

Lidtl boot_idt-r_base # Load IDT with 0, 0

Lgdtl boot_gdt-r_base # Load gdt with whatever is appropriate

Load LDT and gdt

 

XOR % ax, % ax

INC % ax # protected mode (PE) Bit

Lmsw % ax # Into Protected Mode

# Flush prefetch and jump to startup_32_smp in arch/i386/kernel/head. s

Ljmpl $ __boot_cs, $ (startup_32_smp-_ page_offset)

Start the protection mode and jump to startup_32_smp.

 

# These need to be in the same 64 K segment as the above;

# Hence we don't use the boot_gdt_descr defined in head. s

Boot_gdt:

. Word _ boot_ds + 7 # gdt limit

. Long boot_gdt_table-_ page_offset # gdt Base

 

Boot_idt:

. Word 0 # IDT Limit = 0

. Long 0 # IDT base = 0l

 

. Globl trampoline_end

Trampoline_end:

In this Code, set the identifier so that BSP can know that the AP has been running in this Code and load the base addresses of gdt and LDT tables.

Start the protection mode and jump to startup_32_smp.

 

Head. s code:

Entry (startup_32_smp)

ClD

Movl $ (_ boot_ds), % eax

Movl % eax, % DS

Movl % eax, % es

Movl % eax, % FS

Movl % eax, % GS

 

Xorl % EBX, % EBX

Incl % EBX

If it is an AP, set Bx to 1

 

Movl $ swapper_pg_dir-_ page_offset, % eax

Movl % eax, % H6/* set the page table pointer ..*/

Movl % Cr0, % eax

Orl $0x80000000, % eax

Movl % eax, % Cr0/* .. and set paging (PG) bit */

Ljmp $ __boot_cs, $ 1f/* clear prefetch and normalize % EIP */

Enable paging,

 

LSS stack_start, % ESP

Enable ESP to execute the process Kernel stack created by fork to jump to start_secondary

 

# Ifdef config_smp

Movb ready, % Cl

Movb $1, ready

Cmpb $0, % Cl

Je 1f # The first cpu cils start_kernel

# All other CPUs call initialize_secondary

Call initialize_secondary

JMP L6

1:

# Endif/* config_smp */

Call start_kernel

 

If the AP is started, call the initialize_secondary function.

Void _ devinit initialize_secondary (void)

{

/*

* We don't actually need to load the full TSS,

* Basically just the stack pointer and the EIP.

*/

 

ASM volatile (

"Movl % 0, % ESP/n/t"

"JMP * % 1"

:

: "R" (current-> thread. ESP), "R" (current-> thread. EIP ));

}

Set the stack to the stack when fork is created, and the IP address is the IP address when fork is created, so that start_secondary is redirected.

 

The start_secondary function handles the following:

While (! Cpu_isset (smp_processor_id (), smp_commenced_mask ))

Rep_nop ();

Smp_commenced_mask is used to determine whether to start AP operation. Smp_commenced_mask is set in smp_init.

Cpu_idle ();

If it is started, call cpu_idle for task scheduling.