Record the Copyfailed problem encountered by a psu operation

I haven't installed or played PSU for a long time, and I still have a little hope in my mind. I thought it would take two hours to complete, after playing for more than four hours, I encountered some minor problems. The environment I played this time was 11.2.0.3.9. Here is a record. Let's take a look at the log: root @ crmdb1rac1: # orac

I haven't installed or played PSU for a long time, and I still have a little hope in my mind. I thought it would take two hours to complete, after playing for more than four hours, I encountered some minor problems. The environment I played this time was 11.2.0.3.9. Here is a record. If you don't talk nonsense, let's take a look at the log: root @ crmdb1rac1: // #/orac

I haven't installed or played PSU for a long time, and I still have a little hope in my mind. I thought it would take two hours to complete, after playing for more than four hours, I encountered some minor problems. The environment I played this time was 11.2.0.3.9. Here is a record. Let's take a look at the log:

root@crmdb1rac1:/#/oracle/app/11.2.0/grid/OPatch/opatch auto /oraclelog/opatch_soft -ocmrf /oracle/app/grid/ocm0709.rspExecuting /oracle/app/11.2.0/grid/perl/bin/perl /oracle/app/11.2.0/grid/OPatch/crs/patch11203.pl -patchdir /oraclelog -patchn opatch_soft -ocmrf /oracle/app/grid/ocm0709.rsp -paramfile /oracle/app/11.2.0/grid/crs/install/crsconfig_params/oracle/app/11.2.0/grid/crs/install/crsconfig_params/oracle/app/11.2.0/grid/crs/install/s_crsconfig_defsThis is the main log file: /oracle/app/11.2.0/grid/cfgtoollogs/opatchauto2014-07-10_00-18-14.logThis file will show your detected configuration and all the steps that opatchauto attempted to do on your system: /oracle/app/11.2.0/grid/cfgtoollogs/opatchauto2014-07-10_00-18-14.report.log2014-07-10 00:18:14: Starting Clusterware Patch SetupUsing configuration parameter file: /oracle/app/11.2.0/grid/crs/install/crsconfig_paramspatch /oraclelog/opatch_soft/17592127/custom/server/17592127  apply successful for home  /oracle/app/oracle/product/11.2.0/db  patch /oraclelog/opatch_soft/18031683  apply successful for home  /oracle/app/oracle/product/11.2.0/db CRS-2791: Starting shutdown of Oracle High Availability Services-managed resources on 'crmdb1rac1'CRS-2673: Attempting to stop 'ora.crsd' on 'crmdb1rac1'CRS-2790: Starting shutdown of Cluster Ready Services-managed resources on 'crmdb1rac1'CRS-2673: Attempting to stop 'ora.LISTENER_SCAN1.lsnr' on 'crmdb1rac1'CRS-2673: Attempting to stop 'ora.oc4j' on 'crmdb1rac1'CRS-2673: Attempting to stop 'ora.cvu' on 'crmdb1rac1'CRS-2673: Attempting to stop 'ora.crmdb1rac1.vip' on 'crmdb1rac1'CRS-2677: Stop of 'ora.cvu' on 'crmdb1rac1' succeededCRS-2672: Attempting to start 'ora.cvu' on 'crmdb1rac2'CRS-2677: Stop of 'ora.LISTENER_SCAN1.lsnr' on 'crmdb1rac1' succeededCRS-2673: Attempting to stop 'ora.scan1.vip' on 'crmdb1rac1'CRS-2676: Start of 'ora.cvu' on 'crmdb1rac2' succeededCRS-2677: Stop of 'ora.crmdb1rac1.vip' on 'crmdb1rac1' succeededCRS-2672: Attempting to start 'ora.crmdb1rac1.vip' on 'crmdb1rac2'CRS-2677: Stop of 'ora.scan1.vip' on 'crmdb1rac1' succeededCRS-2672: Attempting to start 'ora.scan1.vip' on 'crmdb1rac2'CRS-2676: Start of 'ora.crmdb1rac1.vip' on 'crmdb1rac2' succeededCRS-2676: Start of 'ora.scan1.vip' on 'crmdb1rac2' succeededCRS-2672: Attempting to start 'ora.LISTENER_SCAN1.lsnr' on 'crmdb1rac2'CRS-2676: Start of 'ora.LISTENER_SCAN1.lsnr' on 'crmdb1rac2' succeededCRS-2677: Stop of 'ora.oc4j' on 'crmdb1rac1' succeededCRS-2672: Attempting to start 'ora.oc4j' on 'crmdb1rac2'CRS-2676: Start of 'ora.oc4j' on 'crmdb1rac2' succeededCRS-2673: Attempting to stop 'ora.ons' on 'crmdb1rac1'CRS-2677: Stop of 'ora.ons' on 'crmdb1rac1' succeededCRS-2673: Attempting to stop 'ora.net1.network' on 'crmdb1rac1'CRS-2677: Stop of 'ora.net1.network' on 'crmdb1rac1' succeededCRS-2792: Shutdown of Cluster Ready Services-managed resources on 'crmdb1rac1' has completedCRS-2677: Stop of 'ora.crsd' on 'crmdb1rac1' succeededCRS-2673: Attempting to stop 'ora.mdnsd' on 'crmdb1rac1'CRS-2673: Attempting to stop 'ora.crf' on 'crmdb1rac1'CRS-2673: Attempting to stop 'ora.ctssd' on 'crmdb1rac1'CRS-2673: Attempting to stop 'ora.evmd' on 'crmdb1rac1'CRS-2673: Attempting to stop 'ora.cluster_interconnect.haip' on 'crmdb1rac1'CRS-2677: Stop of 'ora.mdnsd' on 'crmdb1rac1' succeededCRS-2677: Stop of 'ora.evmd' on 'crmdb1rac1' succeededCRS-2677: Stop of 'ora.crf' on 'crmdb1rac1' succeededCRS-2677: Stop of 'ora.cluster_interconnect.haip' on 'crmdb1rac1' succeededCRS-2677: Stop of 'ora.ctssd' on 'crmdb1rac1' succeededCRS-2673: Attempting to stop 'ora.cssd' on 'crmdb1rac1'CRS-2677: Stop of 'ora.cssd' on 'crmdb1rac1' succeededCRS-2673: Attempting to stop 'ora.gipcd' on 'crmdb1rac1'CRS-2677: Stop of 'ora.gipcd' on 'crmdb1rac1' succeededCRS-2673: Attempting to stop 'ora.gpnpd' on 'crmdb1rac1'CRS-2677: Stop of 'ora.gpnpd' on 'crmdb1rac1' succeededCRS-2793: Shutdown of Oracle High Availability Services-managed resources on 'crmdb1rac1' has completedCRS-4133: Oracle High Availability Services has been stopped.Successfully unlock /oracle/app/11.2.0/gridpatch /oraclelog/opatch_soft/17592127  apply failed  for home  /oracle/app/11.2.0/gridCRS-4123: Oracle High Availability Services has been started.

When executing the Successfully unlock/oracle/app/11.2.0/grid step in PSU. At this time, the screen output will be very slow. We can observe the log output in the background, and we find that there is a problem in the next copy operation. As follows:

 Patching component oracle.usm, 11.2.0.3.0... The following actions have failed: Copy failed from '/oracle/app/11.2.0/grid/.patch_storage/16619898_Jun_26_2013_21_46_53/files/lib/libeons.so' to '/oracle/app/11.2.0/grid/lib/libeons.so'...  Copy failed from '/oracle/app/11.2.0/grid/.patch_storage/16619898_Jun_26_2013_21_46_53/files/lib/libhasgen11.so' to '/oracle/app/11.2.0/grid/lib/libhasgen11.so'...  Copy failed from '/oracle/app/11.2.0/grid/.patch_storage/16619898_Jun_26_2013_21_46_53/files/lib/libocr11.so' to '/oracle/app/11.2.0/grid/lib/libocr11.so'...  Copy failed from '/oracle/app/11.2.0/grid/.patch_storage/16619898_Jun_26_2013_21_46_53/files/lib/libocrb11.so' to '/oracle/app/11.2.0/grid/lib/libocrb11.so'...  Copy failed from '/oracle/app/11.2.0/grid/.patch_storage/16619898_Jun_26_2013_21_46_53/files/lib/libocrutl11.so' to '/oracle/app/11.2.0/grid/lib/libocrutl11.so'... 

We can see that the copy operation fails when copying these files. This is because our Oracle so files are occupied in the memory. We need to execute slibclean to clear these caches. So I did the rollback operation, and then completely stopped the crs, and made several slibclean operations to start the crs normally. After these operations, I found that PSU still failed. So I checked some information. I backed up the. so files that failed the coyp and moved them to other directories. Then the crs cannot be started successfully. How can we beat PSU when crs cannot be started? So I copied the file back. I carefully observed that the crs will be automatically stopped when the PSU is hit, then we will run the Successfully unlock/oracle/app/11.2.0/grid we just mentioned. So can I execute the slibclean command to clear these caches in the stopped CRS phase?

This is the lib file that is still occupied by CRS at startup:

root@crmdb1rac1:/oracle/app/11.2.0/grid/.patch_storage#genkld |grep oracle 900000004d98000     b2c7 /oracle/app/oracle/product/11.2.0/db/jdk/jre/bin/libnio.a 900000004d77000    203b1 /oracle/app/oracle/product/11.2.0/db/jdk/jre/bin/libnet.a 900000004d5f000    17840 /oracle/app/oracle/product/11.2.0/db/jdk/jre/bin/libzip.a 900000004d31000    21700 /oracle/app/oracle/product/11.2.0/db/jdk/jre/bin/libj9ute23.so 900000004d1e000    12c7c /oracle/app/oracle/product/11.2.0/db/jdk/jre/bin/libiverel23.so 900000004cb6000    6792f /oracle/app/oracle/product/11.2.0/db/jdk/jre/bin/libjclscar_23.so 900000004c91000    2427d /oracle/app/oracle/product/11.2.0/db/jdk/jre/bin/libj9vrb23.so 900000004c5e000    32c3f /oracle/app/oracle/product/11.2.0/db/jdk/jre/bin/libj9jvmti23.so 900000004c2f000    2ed2c /oracle/app/oracle/product/11.2.0/db/jdk/jre/bin/libj9dyn23.so 900000004b97000    97a31 /oracle/app/oracle/product/11.2.0/db/jdk/jre/bin/libj9gc23.so 900000004b7f000    179e1 /oracle/app/oracle/product/11.2.0/db/jdk/jre/bin/libj9bcv23.so 9000000046cf000   4af269 /oracle/app/oracle/product/11.2.0/db/jdk/jre/bin/libj9jit23.so 9000000046be000    10eef /oracle/app/oracle/product/11.2.0/db/jdk/jre/bin/libj9trc23.so 900000004681000    2ac84 /oracle/app/oracle/product/11.2.0/db/jdk/jre/bin/libj9dmp23.so 9000000046ac000    11bab /oracle/app/oracle/product/11.2.0/db/jdk/jre/bin/libj9zlib23.so 900000004649000    37b9e /oracle/app/oracle/product/11.2.0/db/jdk/jre/bin/libjava.a 90000000091a000     1d41 /oracle/app/oracle/product/11.2.0/db/jdk/jre/bin/libwrappers.a 900000004588000    6f0fe /oracle/app/oracle/product/11.2.0/db/jdk/jre/bin/libj9vm23.so 9000000045f8000    50434 /oracle/app/oracle/product/11.2.0/db/jdk/jre/bin/libj9prt23.so 900000000918000     1d2f /oracle/app/oracle/product/11.2.0/db/jdk/jre/bin/libj9hookable23.so 90000000456c000    1b631 /oracle/app/oracle/product/11.2.0/db/jdk/jre/bin/j9vm/libjvm.so 900000004059000     bf6a /oracle/app/oracle/product/11.2.0/db/jdk/jre/bin/libj9thr23.so 900000000916000     1ff3 /oracle/app/oracle/product/11.2.0/db/jdk/jre/bin/libjsig.so 900000004049000     f6aa /oracle/app/oracle/product/11.2.0/db/jdk/jre/bin/classic/libjvm.so 90000000ff8e000  248bf1b /oracle/app/11.2.0/grid/lib/libclntsh.so 900000004513000    5852a /oracle/app/11.2.0/grid/lib/libcell11.so 90000000434a000   1b3f27 /oracle/app/11.2.0/grid/lib/libdb-4.6.so 90000000428a000    bf3a1 /oracle/app/11.2.0/grid/lib/libcrf11.so 9000000041d7000     ad7d /oracle/app/11.2.0/grid/lib/libeonsserver.so 9000000041b1000    15c4e /oracle/app/11.2.0/grid/lib/libonsx.so 900000004019000    2509a /oracle/app/11.2.0/grid/lib/libeons.so 90000000123b000  2dd6039 /oracle/app/11.2.0/grid/lib/libttsh11.so 900000004012000     6c16 /oracle/app/11.2.0/grid/lib/libskgxn2.so 90000000406c000   1449e1 /oracle/app/11.2.0/grid/lib/libgns11.so 90000000122e000     c5c5 /oracle/app/11.2.0/grid/lib/libocrutl11.so 90000000103e000   1ef18b /oracle/app/11.2.0/grid/lib/libocrb11.so 900000000f9f000    9e0a5 /oracle/app/11.2.0/grid/lib/libocr11.so 900000000921000   67dc0c /oracle/app/11.2.0/grid/lib/libhasgen11.so 90000000f7a9000   2adb17 /usr/openv/lib/libpioracle.so

This is the lib file that is occupied after CRS is disabled and I run slibclean several times.

 root@crmdb1rac1:/usr/sbin#genkld |grep oracle 900000001160000     b2c7 /oracle/app/11.2.0/grid/jdk/jre/bin/libnio.a 90000000113f000    203b1 /oracle/app/11.2.0/grid/jdk/jre/bin/libnet.a 900000001127000    17840 /oracle/app/11.2.0/grid/jdk/jre/bin/libzip.a 9000000010f9000    21700 /oracle/app/11.2.0/grid/jdk/jre/bin/libj9ute23.so 9000000010e6000    12c7c /oracle/app/11.2.0/grid/jdk/jre/bin/libiverel23.so 90000000107e000    6792f /oracle/app/11.2.0/grid/jdk/jre/bin/libjclscar_23.so 900000001059000    2427d /oracle/app/11.2.0/grid/jdk/jre/bin/libj9vrb23.so 900000001026000    32c3f /oracle/app/11.2.0/grid/jdk/jre/bin/libj9jvmti23.so 900000000ff7000    2ed2c /oracle/app/11.2.0/grid/jdk/jre/bin/libj9dyn23.so 900000000f5f000    97a31 /oracle/app/11.2.0/grid/jdk/jre/bin/libj9gc23.so 900000000a97000   4af269 /oracle/app/11.2.0/grid/jdk/jre/bin/libj9jit23.so 900000000a86000    10eef /oracle/app/11.2.0/grid/jdk/jre/bin/libj9trc23.so 900000000a49000    2ac84 /oracle/app/11.2.0/grid/jdk/jre/bin/libj9dmp23.so 900000000a74000    11bab /oracle/app/11.2.0/grid/jdk/jre/bin/libj9zlib23.so 900000000a0f000    37b9e /oracle/app/11.2.0/grid/jdk/jre/bin/libjava.a 900000000a47000     1d41 /oracle/app/11.2.0/grid/jdk/jre/bin/libwrappers.a 90000000094e000    6f0fe /oracle/app/11.2.0/grid/jdk/jre/bin/libj9vm23.so 9000000009be000    50434 /oracle/app/11.2.0/grid/jdk/jre/bin/libj9prt23.so 900000000432000     1d2f /oracle/app/11.2.0/grid/jdk/jre/bin/libj9hookable23.so 900000000926000    1b631 /oracle/app/11.2.0/grid/jdk/jre/bin/j9vm/libjvm.so 900000000942000     bf6a /oracle/app/11.2.0/grid/jdk/jre/bin/libj9thr23.so 900000000430000     1ff3 /oracle/app/11.2.0/grid/jdk/jre/bin/libjsig.so 900000000916000     f6aa /oracle/app/11.2.0/grid/jdk/jre/bin/classic/libjvm.so 90000000f7a9000   2adb17 /usr/openv/lib/libpioracle.so

We can see that there are basically some lib files related to JDK, and there is no need to copy these lib files. After completing these operations, the PSU patch is successfully installed. I don't know when to start. I have to wait a few times to finish the work.

Original article address: record the Copy failed problem encountered by a psu operation. Thank you for sharing it with the original author.