Summary of kernel development, debugging and Optimization on godson

Author: Falcon
Copy Link
By Falcon <wuzhangjin@gmail.com>
2009-03-31

[I almost had to write a long article (I wrote two sections in E). I thought about it, but I copied it directly from weekreport and sorted it out.]

Recently, the Linux real-time extension patch rt_preempt has been transplanted to the loongson2f platform. Now it is basically finished. Here is a summary.

I. porting process

The porting process is as follows (serial numbers are generally promoted, but some may be cross-parallel ):

0. Search for loongson2f related information. The data collection is as follows:
Http://oss.lzu.edu.cn/blog/blog.php? Do_showone/tid_1986.html
More information: http://dev.lemote.com/drupal/download
Google + Baidu ==> entire Internet
1. Learn MIPS compilation. The learning notes are as follows:
Practical MIPS assembly language programming in Linux
Http://oss.lzu.edu.cn/blog/blog.php? Do_showone/tid_1991.html
There is no way to learn assembly. Read the CPU instruction manual, read assembly specifications, and use assembly tools to do exercises!
2. Understand MIPS/loongson2f Platform
Read the see mips run V2 and loongson2f user manual and use the box of Fu Yi 6003
It is useless to look at it. It is better to read it against the kernel source code!
3. Understand the rt_preempt-related section in the Linux kernel.
Read Linux kernel-related books such as lkd and ldd3, and carefully read kernel threads, interrupt processing, clock management, process synchronization, and so on.
While watching and practicing!
4. Use rt_preempt to read its source code.
Http://rt.wiki.kernel.org
Before porting, you need to understand what you want to transplant, but do not drill too deep at some points until you find the places related to the architecture and objectives )! Otherwise, the entire process will be difficult to control!
5. Port rt_preempt to qemu/MIPS (Malta ).
It mainly solves syntax errors and so on. At this time, it requires tools such as cssag, objdump, And nm.
Specify the cscope architecture: $ make cs1_arch = MIPS
6. Port rt_preempt to loongson2f based.
With the help of pmon bios and bootloader-level debugger, we found a bug. We can run rt_preempt on loongson2f.
For details about how to add configuration options and kernel compilation options, read: Documentation/kbuild /*
If it cannot run, do not fear it! The reason for running failure is only a very small bug. Analyze it carefully and use various debuggers to help solve it! For detailed debugging skills, refer to the following section!
7. Port rt_preempt's real-time kernel debugging tools ftrace and perf_counter
At this time, the main debugging tool is the omnipotent printk ~~ Of course, we also need to make the sub-routine and Other implementations of MIPS quite clear! Especially register protection!
Understand the principles of these tools, familiarize yourself with the relevant hardware, and then port it!
8. Sort out source code and documents
Sort out the source code. It is best to publish your stuff in the form of patches, otherwise the entire kernel may be dozens of MB! In this case, you must be familiar with diff & patch. Of course, there are also vimdiff and so on!
Write the document. Latex is my love!
9. Run the Migration Report.
Slides. Do not use M $ PPT or latex beamer. They are simple, beautiful, and time-saving!

Ii. debugging methods

The following lists the debugging methods used and not used in the porting process:

1. pmon

Pmon is the BIOS + bootloader for loongson2f serial machines. However, it is also a very good debugging tool for developers and supports breakpoint debugging and single-step debugging.
Test, memory, and register operations. The usage is very similar to that of GDB. For specific usage, enter the H command in the pmon command line to find the related information of debugger and memory.

Why is pmon used for debugging? Before the kernel initialization hardware trap processing (trap_init), the exception triggered during kernel execution should still be handled by pmon. In the porting process
A typical problem encountered in TLB
Miss, pmon tells us the approximate location of the problem, then uses the pmon breakpoint debugging and one-step debugging to locate the exact location of the problem, and analyzes the context to find the cause.

During debugging, you may need to disable the loongson2f cache for debugging. You can do this:

Mfc0 T0, cp0_config # Read the conifg register value of cp0 in the General Register t0 And T0, 0xfffffff8 # Low 3 position 0 Or T0, 0x2 # Low 3 Position 2, disable Cache Mtc0 T0, cp0_config # Write back cp0 conifg register

Reference: http://comcat.blog.openrays.org/blog-htm-do-showone-tid-148.html

2. Serial Port debugging

The advantage of serial port debugging is that you can view and analyze logs during kernel startup! Because the display information is limited, the previous data will be overwritten by the subsequent information, and the complete log cannot be viewed.
There is a serial port available on Fuxi. If you want to debug through serial port, follow these steps:

A. Use a serial line to connect it to another machine, start a serial debugging tool such as minicom on another machine, and configure the baud rate. The baud rate is 115200. For usage of minicom, see "Man minicom ".
B. run the command line from Fuxi to pmon (always press the Del key at startup), type "setvga 0", Press enter, and restart, after the pmon is restarted, the information will be entered to The minicom side.

C. To record more log information, run the script or screen command before starting minicom.
-L. After debugging, type exit to exit. In this way, you can view the operation log in the typescript or screenlog. * file. Of course, there is a simpler way to configure
The number of rows that can be rolled back by the terminal.

3. Puts, puthex, early_printk, etc.

Before the console initialization (lele_init), printk cannot print logs for us. What should I do? Write information directly to the serial port (Thx to Yanhua from lemote ):

// 32bit * (Char *) 0xbfd002f8 = 'A' // 64bit * (Char *) 0xffffffffbfd002f8 = 'A'

The serial port address used in FuXi 6003 is 0xbfd002f8, but 3f8 appears in 2E. For details, refer to the following link:

$ CAT/proc/ioports | grep serial 000002f8-000002ff: Serial

In fact, we can also use some encapsulated serial port operation functions, that is, puts,
Puthex or enable early_printk in the kernel directly. The method to enable early_printk is simple:
Kernel under the hacking menu
Enable the debugging option, and then select early_printk from the main menu, so that the kernel can use printk before lele_init.

4. Use the watch register (Thx to Mgr. Zhang from lemote)

Loongson2f provides us with a debug register, that is, the 18th register in cp0 (coprocessor 0) (see loongson2f Manual ). It can be used to track the read and write operations of a virtual memory address. The general usage is as follows:

63 3 2 1 0 Vaddr, 0, R, W

Set (monitor an address): Load T0, (vaddr | 0rw), mtc0 T0, 18
Clear: mtc0 $ zero, 18

This is quite useful. A typical example is: when you find that a piece of data is different from what you expected, you may want to find out which guy has modified the data! The search method is simple:

First through objdup-d
In vmlinux, find the virtual address corresponding to a data variable in the kernel, and set the watch register, once the data is tracked and modified (it can be seen that you have monitored the data set to read, write, or read/write ),
A watch exception will be triggered. In the exception handler, information about the registers will be printed and the culprit will be tracked!
If debugging is complete, clear it. Otherwise, no matter who is operating on the virtual address, report it to you! The clearing method is to write a 0 value to the watch register.

5. printk

After the console initialization (lele_init), printk can work. At this time, we can directly use it for debugging. In addition, after the kernel is started normally, you can use printk to debug kernel faults triggered by applications.

If you often use printk or program kernel modules, you should note that the first parameter of printk often contains words such as kerl_debug or <7>.
String, which indicates where the printk prints the log, but where it prints the log. Another lele_loglevel is used as a reference.

If the Log Level of the specified printk is smaller than that of lele_loglevel, the logs will be printed to the console. Otherwise, the logs may be sent to klogd or
/Proc/kmsg. We recommend that you set lele_loglevel to the maximum value during debugging. You can use/proc/sys/kernel/printk
.

6. koops

When the kernel fails, it will not "go away" and will always leave some "Last words" for us. These last words include backtrace information, register information, and so on. With this information, we can analyze the cause of kernel failure! This analysis process may require tools such as ksymoops and objdump.

7. kernel internal debugging tools & API

In fact, in addition to the above tools, the kernel also provides many other debugging methods, such as many debugging options under the kernel hacking menu, including Magic sysrq and tracer.

In addition to these debugging options, we also have some kernel APIs available, such as the printk mentioned above, and other such as bugs, warn_on,
Show_registers, show_regs, dump_stack/show_trace/show_stack,
Print_symbol.
Here we will introduce the usage of bugs:

Bug (! Irq_disabled ());

The above sentence is translated into natural language: If IRQ is not disabled now, it may cause serious kernel faults, so report it immediately and exit.

8. Other debugging methods

If there are no real machines, some emulator may be used for debugging. Loongson2f is highly compatible with MIPS.
There is no relationship between features (for the features of loongson2f, see its manual). You can directly use MIPS/Malta in qemu to simulate the development. This will accelerate the development process and protect
Hardware and file systems.
Here is an example of using qemu to start a MIPS kernel:

$ Qemu-system-mipsel-kernel vmlinux-2.6.26-1-4kc-malta-initrd initrd.gz -APPEND "root =/dev/ram0 init =/bin/sh console = ttys0 ramdisk_size = 3000" -Nographic-m malta-hda/dev/zero

If debugging is required, add-s and-S to the end of the preceding command, and then use GDB to connect to port 1234 enabled by qemu for debugging, the debugging process is the same as the debugging mode of gdb server + GDB commonly used in embedded development.

Similarly, to increase the interaction of the debugging process, we can use kgdb to debug the kernel. The basic process is similar to that of qemu + GDB, but kgdb must be enabled in the kernel.

9. Supplement: Use netconsole to remotely record system startup logs (THX taohl from lemote.com)

To track some inexplicable Kernel panic problems, another method similar to serial port debugging is netconsole. General Usage:
A. in the kernel, select the following options and directly compile them into the kernel instead of modules. After compilation, install the kernel and modules.

Device Drivers -->

Network Device
Support -->

Network Console logging support
B. Configuration and monitoring

Before configuration, specify two items: one is the system to be monitored (a) and the other is the system to be monitored (B ). To monitor the Startup Process of a system, we need to configure the kernel startup Selection in.
Item, that is, configured through/boot. cfg. In addition, if we start a monitoring program in B to record the startup log of.

In a, configure/boot. cfg and add the following after the kernel startup parameters:

No_auto_cmd debug netconsole = a-port @ A-IP/eth0, B-port @ B-IP/B-mach_addr

In B, start Netcat for monitoring.

$ Netcat-l-u-p B-Port

Now restart a and you can see the startup log of A in B.

Tip:
A. If you want the monitoring program to run in the background, you can use nohup to execute Netcat, that is

$ Nohup Netcat-l-u-p B-Port &

B.
If you want to constantly monitor the startup process of the system to capture Possible Kernel panic, you can add a reboot in/etc/rc. Local to record all the information,
You can set the console terminal level to the minimum through dmesg-N 8. In addition, if you want to record the startup process such as X, you can sleep for a period of time before reboot.

Iii. optimization tools

Okay: the debugging tool is introduced here. The following describes several auxiliary kernel optimization tools:

1. kft

The kernel function tracking tool can be used to track the execution process of kernel functions triggered by an application and the execution time of each kernel function, and then find the most worthy function for optimization.
 
Kft that has been transplanted to loongson2f:
Http://oss.lzu.edu.cn/blog/blog.php? Do_showone/tid_2027.html

2. kgcov

Kgcov is the profiling tool of the kernel. It can collect statistics on the code execution during kernel running and help developers find the hotspots conducive to optimization. This can be used by kft.

Kgcov that has been transplanted to loongson2f:
Http://oss.lzu.edu.cn/blog/blog.php? Do_showone/tid_2028.html

3. ftrace

Ftrace is a function.
Trace is similar to kft, but ftrace has developed into a kernel tracer framework, such as irqsoff, preemptoff,
Wakeup,
Tracer such as sched_switch appears, and ftrace has entered the kernel. kft and kgcov are only released as patches. Ftrace from
Rt_preempt.

Currently, ftrace that supports loongson2f has been transplanted and will not be released soon. In fact, some ftrace functions have been released with rt_preempt for loongson2f. You can download them here:
Http://lkml.indiana.edu/hypermail/linux/kernel/0903.1/00707.html

4. oprofile and perf_counter

Oprofile is a mature tool that uses the hardware performance counters provided by the processor for profile. Unlike kft, ftrace, and kgcov
It is achieved through hardware assistance, so it has little impact on the kernel, and because it can provide hardware event statistics corresponding to the code line level, therefore, it can also provide us with a more direct optimization "Build
".

Because loongson2f provides two performance counters and supports the oprofile kernel, The oprofile can be used directly on loongson2f, but remember to choose oprofile when compiling the kernel.

Perf_counter is a bit similar to oprofile. At present, it can perform basic performance counting, but it is not as powerful as oprofile, but it is likely to develop into a kernel
The profiling framework allows oprofile and other tools to work on it. Perf_counter is also from rt_preempt and has been transplanted
Loongson2f, but it will not be released soon.