Oracle Database DataFile Auto Offlile due to I/O error

Just arrived at the hotel, received a customer call said a database of data file IO error, through the VPN view found as follows:
Sun Oct 23:19:27 beist 2016
Trace dumping is performing id=[cdmp_20161030231927]
Sun Oct 23:19:27 beist 2016
Errors in FILE/ORACLE/APP/10.2/ADMIN/XXXX/BDUMP/XXXX2_SMON_11863216.TRC:
Ora-00376:file 595 cannot is read at this time
Ora-01110:data file 595: '/dev/rdata05vg_8g_48 '
Sun Oct 23:19:31 beist 2016
ORACLE Instance xxxx2 (pid =)-Error 376 encountered while recovering transaction (160, 1).
Sun Oct 23:19:31 beist 2016
Errors in FILE/ORACLE/APP/10.2/ADMIN/XXXX/BDUMP/XXXX2_SMON_11863216.TRC:
Ora-00376:file 595 cannot is read at this time
Ora-01110:data file 595: '/dev/rdata05vg_8g_48 '
Sun Oct 23:19:32 beist 2016
Errors in FILE/ORACLE/APP/10.2/ADMIN/XXXX/BDUMP/XXXX2_SMON_11863216.TRC:
Ora-00376:file 595 cannot is read at this time
Ora-01110:data file 595: '/dev/rdata05vg_8g_48 '
Sun Oct 23:19:33 beist 2016
It is not difficult to see that the error file can not be read; I actually logged on to 2 nodes ls-tr Check that the permissions are correct, and check the file through DBV to see that there is no bad block, so I decided to recover the file directly. In recover, I found an error in NFS mount problems:
sql> Recover datafile 595;
Ora-00279:change 15125505612642 generated at 10/30/2016 18:06:07 for thread 1
Ora-00289:suggestion:/arch2/1_108445_815539661.dbf
Ora-00280:change 15125505612642 for thread 1 was in sequence #108445

Specify log: {<ret>=suggested | AUTO | CANCEL}
/arch1/1_108445_815539661.dbf
Ora-00308:cannot Open archived log '/arch1/1_108445_815539661.dbf '
Ora-27054:nfs file system where the file is created or resides isn't mounted with correct options
Additional Information:6
This shows that there are a lot of environmental problems. Since the local node can not read, in order to return to normal for a short period of time, directly to the partial archive CP to the appropriate archive directory, again recover can successfully online the file, as follows:
Sun Oct 23:20:23 beist 2016
ALTER DATABASE datafile 595 online
Sun Oct 23:20:23 beist 2016
Completed:alter Database datafile 595 Online
Sun Oct 23:20:28 beist 2016
Smon:parallel Transaction Recovery tried
Sun Oct 23:23:02 beist 2016
Thread 2 advanced to log sequence 164113 (LGWR switch)
Current log# 7 seq# 164113 mem# 0:/dev/rora_redo2_01
We can see that the online file is really smooth, but it's okay to archive the database. So let's continue to analyze why this problem arises. A further search for alert log found that I had started to appear at 18:15 in the I-Euro error:
Sun Oct 18:15:04 beist 2016
Kcf:write/open Error block=0x29790 online=1
file=595/dev/rdata05vg_8g_48
error=27063 txt: ' IBM AIX RISC system/6000 error:5: I/O Error
Additional Information:-1
Additional information:131072 '
Automatic datafile offline due to write error on
File 595:/dev/rdata05vg_8g_48
Sun Oct 18:15:28 beist 2016
Thread 2 advanced to log sequence 164100 (LGWR switch)
Current log# seq# 164100 mem# 0:/dev/rora_redo2_06
Sun Oct 18:15:28 beist 2016
Errors in FILE/ORACLE/APP/10.2/ADMIN/XXXX/UDUMP/XXXX2_ORA_28705020.TRC:
Ora-00372:file 595 cannot is modified at this time
Ora-01110:data file 595: '/dev/rdata05vg_8g_48 '
Ora-00372:file 595 cannot is modified at this time
Ora-01110:data file 595: '/dev/rdata05vg_8g_48 '
We can see that because of the error, Oracle automatically offline the data file, which is actually the database protection mechanism (no relevant hidden parameters to control). Here I suspect most of the operating system where there is a problem, and sure enough errpt see found at 18:15 in the path error errors.
oracle:xxx$ (/oracle) errpt
IDENTIFIER TIMESTAMP T C resource_name DESCRIPTION
3d32b80d 1030181516 P S topsvcs NIM thread blocked
3d32b80d 1030181516 P S topsvcs NIM thread blocked
E86653C3 1030181516 P H lvdd I/O ERROR detected by LVM
B6267342 1030181516 P H hdisk46 DISK OPERATION ERROR
de3b8540 1030181516 P H hdisk46 PATH has FAILED
de3b8540 1030181416 P H hdisk46 PATH has FAILED

oracle:xxxx$ (/oracle/app/10.2/admin/xxxx/bdump) Errpt-aj de3b8540
---------------------------------------------------------------------------
Label:sc_disk_err7
identifier:de3b8540

Date/time:sun Oct 18:15:00 beist 2016
Sequence number:921
Machine id:00f7a4904c00
Node id:sti50l02
Class:h
Type:perm
Wpar:global
Resource name:hdisk46
Resource Class:disk
Resource Type:hitachi
......
......

Description
PATH has FAILED

Probable causes
ADAPTER Hardware OR CABLE
DASD DEVICE

Failure causes
Undetermined

Recommended Actions
Perform PROBLEM determination procedures
CHECK PATH

Detail Data
PATH ID
It is not difficult to see, due to the late afternoon related errors, resulting in the database IO exception, Oracle automatically offline the file. However, I just lspath check found all OK, actually should also be so, otherwise recover datafile will continue to report IO error. It's late at night, it's over here! Simply record!

Ps:
1 description of the relevant parameters of NFS mount

* NFS mount Option "Forcedirectio" is required on Solaris platforms when mounting the Ocr/crs files as using Oracle 10.1 .0.4 or 10.2.0.2 or later (Oracle unpublished bug 4466428)
* * AIX is-only supported with NAS on AIX 5.3 TL04 and higher with Oracle 10.2.0.1 and later
NAS devices are only supported with HPUX 11.23 or higher only
# These mount options are for Linux kernels 2.6 and above. For older kernels-check note 279393.1
# # The stated mount options for binaries are applicable only if the ORACLE home is shared.
Due to unpublished bug 5856342, it's necessary to-use the following Init.ora-when-using NAS with all parameter O F RAC on Linux (x86 & x86-64 platforms) until 10.2.0.4. This bug was fixed and included in 10.2.0.4 Patchset.
Filesystemio_options = Directio

2 if Oracle 11.2.0.2 starts, without installing patch 7691270, Oracle automatically crash the database after encountering an IO error, with an associated implied parameter: _datafile_write_ Errors_crash_instance
This parameter defaults to true after version 11.2.0.2 and contains the latest version of 11.2.0.4.
Enter value for Par:datafile_write
Old 3:where x.indx = y.indx and x.ksppinm like '%&par% '
New 3:where x.indx = Y.indx and x.ksppinm like '%datafile_write% '

name                                                 value                 Describ
--------------------------------------------- -------------------------------------------------------------------------------------
_datafile_write_ errors_crash_instance              TRUE                  datafile Write errors Crash instance