MAXAIO causes hang startup in Oracle

MAXAIO causes Oracle to start the hang problem Oracle database, 10.2.0.4 for linux x86. When it restarts normally, it freezes to the open stage. On the operating system, the CPU usage of some user processes started due to scheduled tasks is almost 100%, which is obviously waiting. The trc file is also quickly generated in the bdump directory of Oracle. The key points of these files are as follows: WARNING: io_submit failed due to kernel limitations MAXAIO for process = 0 pending aio = 0 WARNING: asynch I/O kernel limits is set at AIO-MAX-NR = 65536 AIO-NR = 65536 WARNING: Oracle process running out of OS kernel I/O resources (1) literally, MAXAIO of the operating system limits the process operations of Oracle users. Www.2cto.com to check the information, said it is a bug, but gives two solutions: one, increase the value of the operating system kernel parameter AIO-MAX-NR; two, disable the disk AIO mechanism. I used the method of modifying the system Kernel Parameter AIO-MAX-NR to solve this problem. 1. You can temporarily modify the Kernel Parameter aio-max-nr # echo>/proc/sys/fs/aio-max-nr 1048576. 2. permanently modify the Kernel Parameter aio-max-nr, in/etc/sysctl. add the following sentence to the conf file: fs. aio-max-nr = 1048576 use the following command to make the parameter take effect: www.2cto.com #/sbin/sysctl-p. top displays the result Tasks: 568 total, 6 running, 562 sleeping, 0 stopped, 0 zombieCpu (s): 20.4% us, 0.1% sy, 0.0% ni, 79.1% id, 0.4% wa, 0.0% hi, 0.0% si, 0.0% stMem: 132051284 k total, 117157820 k used, 14893464 k free, 197072 k buffersSwap: 5751260 k total, 2404292 k used, 3346968 k free, 114662552 k cached pid user pr ni virt res shr s % CPU % mem time + COMMAND 12975 oracle 25 0 1687 m 25 m 19 m R 99.8 0.0. 01 ora_p004_oncz 12981 oracle 25 0 1687 m 25 m 19 m R 99.8 0.0. 00 ora_p007_oncz 12983 oracle 25 0 1687 m 25 m 19 m R 99.8 0.0. 01 ora_p008_oncz 12985 oracle 25 0 1687 m 25 m 19 m R 99.8 0.0. 00 ora_p009_oncz 12002 oracle 25 0 1968 m 1.6g 1.3g R 90.5 1.3. 03 ora_j000_ofdb attachment, trc file information under the bdump directory www.2cto.com/u01/app/oracle/admin/oncz/bdump/login Database 10g Enterprise Edition Release 10.2.0.4.0-64bit ProductionWith the Partitioning, OLAP, Data Mining and Real Application Testing optionsORACLE_HOME =/u01/app/oracle/product/10.2.0/db_1System name: LinuxNode name: db-172-17-2-8Release: 2.6.18-348. el5Version: #1 SMP Tue Jan 8 17:53:53 EST 2013 Machine: x86_64Instance name: onczRedo thread mounted by this instance: 1 Oracle process number: 29 Unix process pid: 12983, image: oracle @ db-172-17-2-8 (P008) *** service name :() 15:55:08. 764 *** session id: (142.1) 15:55:08. 764ORA-27090: Message 27090 not found; product = RDBMS; facility = ORAAdditional information: 3 Additional information: 128 Additional information: 65536 WARNING: io_submit failed due to kernel limitations MAXAIO for process = 0 pending aio = 0 WARNING: asynch I/O kernel limits is set at AIO-MAX-NR = 65536 AIO-NR = 65536 WARNING: oracle process running out of OS kernel I/O resources (1) WARNING: Oracle process running out of OS kernel I/O resources (1) WARNING: oracle process running out of OS kernel I/O resources (1) WARNING: Oracle process running out of OS kernel I/O resources (1) attachment. See Bug 9949948 Linux: process spin under ksfdrwat0 if OS Async IO not configured high enough This note gives a brief overview of bug 9949948. the content was last updated on: 28-OCT-2011 Click here for details of each of the sections below. affects: Product (Component) Oracle Server (Rdbms) range of versions believed to be affected Versions> = 10.2.0.4 but BELOW 11.1 Versions confirmed as being affected 10.2.0.5Platforms affected Linux X86-64bitLinux 32bit It is believed to be a regression in default behaviour thus: regression introduced in 10.2.0.5Fixed: This issue is fixed in 11.1.0.6 (Base Release) 10.2.0.5.2 Patch Set Update10.2.0.5 Patch 5 on Windows PlatformsSymptoms: Related To: Hang (Process Spins) waits for "I/o slave wait" DISK_ASYNCH_IODescriptionThis problem is introduced in 10.2.0.5 It only affects platforms where Oracle has to reserve async IO slots, such as Linux platforms. if the OS async IO layer is underconfigured and an Oracle processcannot get sufficient AIO slots then rather than reverting to using non AIO call the process may go into an infinite spinunder ksfdrwat0.rediscovery notes: the spin will be preceded by messages in the trace file of the form: WARNING: io_submit failed due to kernel limitations MAXAIO for process = 0 pending aio = 0 WARNING: asynch I/O kernel limits is set at AIO-MAX-NR = 65536 AIO-NR = 65518 WARNING: 1 Oracle process running out of OS kernelI/O resources aiolimit = 0 Notice specifically that the value for aiolimit is reported as "0" for this bug. the process then spins in your typically with a stack showing skgfqio () ksfdgo () ksfdwtio () ksfdwat1 () equals () <Spin point ksfdblock () kcflwi () kcflci () kcblci () kcblcio () kcblgt () kcbldrget () It will show repeated waits for "I/o slave wait", which can be misleading as that is normally considered an idle wait event. workaround Raise the OS aio limits such that the number of concurrent slot requirements never exceeds the OS limit. ie: Increase AIO-MAX-NR OR Disable async IO (Set DISK_ASYNCH_IO = FALSE) See Note: 1313555.1 for additional notes on this issue. please note: The above is a summary description only. actual symptoms can vary. matching to any symptoms here does not confirm that you are encountering this problem. for questions about this bug please consult Oracle Support. referencesBug: 9949948 (This link will only work for PUBLISHED bugs) Note: 245840.1 Information on the sections in this article