Notes on ORA-00471 Processing Methods

The company has launched a new service to the data center. The hardware is IBM X3850 X5. The hardware configuration is good, and the memory is 128 GB. SA has installed the system (sule 32bit) and notifies me to install the database. Busy for half a day. After Oracle 9204 is installed, prepare to create a tablespace to import data. Because the data volume is large, I plan to create a 10 GB Data File to store the written data. It was not long before the database went down after the command for creating the tablespace was entered. After that, I retried the operation again and the fault still persists. The following is the log information:

Alert. log:

Errors in file/u01/oracle/admin/orcl/bdump/orcl_pmon_13918.trc:
ORA-00471: DBWR process terminated with error.
Wed Aug 21 15:56:16 2013
PMON: terminating instance due to error 471
Instance terminated by PMON, pid = 13918

 

Trace file:

/U01/oracle/admin/orcl/bdump/orcl_pmon_13918.trc
Oracle9i Enterprise Edition Release 9.2.0.4.0-Production
With the Partitioning, OLAP and Oracle Data Mining options
JServer Release 9.2.0.4.0-Production
ORACLE_HOME =/u01/oracle/product/9.2.0
System name: Linux
Node name: dbprod
Release: 2.6.27.19-5-pae
Version: #1 SMP 04:40:21 + 0100
Machine: i686
Instance name: orcl
Redo thread mounted by this instance: 1
Oracle process number: 2
Unix process pid: 13918, image: oracle @ dbprod </a> (PMON)

* ** 15:56:16. 296
* ** Session id: (1.1) 15:56:16. 277
Error 471 detected in background process
ORA-00471: DBWR process terminated with error.

 

/Var/log/messages:

Aug 21 14:01:14 dzjg kernel: lowmem_reserve []: 0 872 62301
Aug 21 14:01:14 dzjg kernel: Node 0 Normal free: 3704kB min: 3744kB low: 4680kB high: 5616kB active: 3096kB inactive: 3092kB present: 893200kB

...

...

Aug 21 14:01:14 dzjg kernel: Out of memory: kill process 12867 (oracle) score 414016 or a child
Aug 21 14:01:14 dzjg kernel: Killed process 12867 (oracle)

Initially, it is determined that the system memory is insufficient, but the free-m shows that the system has 60 GB of memory. Check related documents online to analyze the fault as follows:

1. cause of failure:

The addressing range of a 32-bit CPU is limited. The Linux kernel defines the following three areas:

# DMA: 0x00000000-0x00999999 (0-16 MB)

 

# LowMem: 0x01000000-0x037999999 (16-896 MB)-size: 880 MB

 

# HighMem: 0x038000000-(determined by the server hardware)

 

In a 32-bit system, the memory is divided into the above three areas, DMA zone, Normal zone (also called LowMem) and HighMem zone. The DMA zone is mainly used for DMA data transmission, it is located in a low-address space, and its service object is various drivers. Its size is 16 MB. LowMem (also called Normal zone) is the address space that kernel can directly access, the size is 880 MB. Basically, all kernel data is in this region. If the kernel needs to access content in a higher address space, it must be mapped through Normal zone. If the address space is greater than 880 MB, HIghMem zone is used.

 

The intercepted system logs are as follows:

Aug 21 14:01:14 dzjg kernel: lowmem_reserve []: 0 872 62301
Aug 21 14:01:14 dzjg kernel: Node 0 Normal free: 3704kB min: 3744kB low: 4680kB high: 5616kB active: 3096kB inactive: 3092kB present: 893200kB

...

...

Aug 21 14:01:14 dzjg kernel: Out of memory: kill process 12867 (oracle) score 414016 or a child
Aug 21 14:01:14 dzjg kernel: Killed process 12867 (oracle)

We can see that the normal zone free 3704KB <min 3744KB, the kernel applies for the Minimum Memory 3744KB, at this time the system idle lowmem 3704KB, cannot meet the needs. This problem occurs because the kernel uses low memory to track all memory allocations. When low memory is exhausted, no matter how much memory is left in high memory, the out of memory killer mechanism will be triggered to kill the process to keep the system running normally. According to some documents, OOM-Killer has different performance in 2.4 and 2.6. In version 2.4, the new process (new memory application) is killed. On the other hand, the process that occupies the most memory is killed (this is very dangerous and can easily lead to system application paralysis ). Our current system kernel version is 2.6, which causes the system to kill the LGWR process of the database.

SPFILE error causing database startup failure (ORA-01565)

ORA-01172, ORA-01151 error handling

ORA-00600 [2662] troubleshooting

Troubleshooting for ORA-01078 and LRM-00109

1 2 Next Page