Panic Fault Analysis of Solaris

1. after the Solaris crash, in addition to checking the system logs before the fault in/var/adm/messages, we also want to know the current system status and processes of the fault, what files are opened, what events are happening, and how can I understand the information?

2. Fortunately, Solaris provides us with such a mechanism. Solaris will generate UNIX under the/var/crash/'hostname' directory after the panic crash. X and vmcore. X file. These two files record the system status at the time of the crash. We can analyze them and obtain the information we need.

3. analysis method:

You can use MDB-k x or MDB-K Unix. x vmcore. X to analyze system dump files, for example, unix.3 and vmcore.3 in/var/crash/hostname,

Run the command MDB-K 3.

4. Common analysis commands

A. messages in the system message buffer during crash

The message buffer is a ring buffer, which has a lot of valuable information and can be used to know system messages in crash or a long time ago.

: Msgbuf

>:: Msgbufmessage sd94 at fjpfa1: Target 11 Lun bsd94 is/PCI @ 81,2000/fiber-channel @ 1/SD @ 11, bwarning: hme3: fault detected in device; Service degradedwarning: hme3: no response from Ethernet network: link down -- cable problem? Notice: hme3: fault detected in device; service still degradednotice: hme3: no response from Ethernet network: link down -- cable problem?
 

 

B. view the current system process

> :: Ps pid ppid pgid Sid uid flags ADDR namer 0 0 0 0 0x00000001 0000000001843840 schedr 3 0 0 0 0 0 0 0x00020001 00030006d0f818 fsflushr 2 0 0 0 0 0 0x00020001 0000030006d000020 pageoutr 1 0 0 0 0 0x42004000 0000030006d11028 initr 16365 1 16365 16365 100 0x42004000 0000030007d16028 ler 16338 1 16338 16338 100 0x42004000 000003012556eea8 ler 16336 1 16336 16336 100 0x42004000 00000300909ccdd8
 

C. Memory status

 

 

1. Call the backtrace of the stack

$ C
: Stack
: Stackregs

2. kernel symbol table

: Nm

3. Disassembly

<Kernel function>: dis

4. CPU registers

: Regs

5. Scheduling Queue (dispatch Queue)

: Cpuinfo-V

6. Physical memory and slab Subsystem

: Memstat
: Kmastat

7. All PROCESSES IN THE SYSTEM

: PS

8. All kernel threads

: Threadlist

9. thread status

<Kthread_t address>: Thread

10. A kernel thread call stack

<Kthread_t address>: findstack-V
<Proc_t address>: Walk thread |: findstack-V

11. synchronization object status

<Mutex address>: mutex
<Read/write lock address>: rwlock

12. Search for address reference

<Address>: kgrep
<Address>: whatthread