Monday, August 10, 2015

Applying of GI PSU Fails Due to -customLogDir Opton Used by OPatch

Use of opatch auto option has become the default for applying PSU for both GI and Oracle home in an automated fashion. Applying GI PSU (July 2015) for both GI and Oracle Home in a single instance setup failed with following message.
$ /opt/app/oracle/product/11.2.0/grid_4/OPatch/opatch version
OPatch Version:

# /opt/app/oracle/product/11.2.0/grid_4/OPatch/opatch auto ./20996923 -ocmrf ../ocm.rsp
Executing /opt/app/oracle/product/11.2.0/grid_4/perl/bin/perl /opt/app/oracle/product/11.2.0/grid_4/OPatch/crs/ -patchdir . -patchn 20996923 -ocmrf ../ocm.rsp -paramfile /opt/app/oracle/product/11.2.0/grid_4/crs/install/crsconfig_params
This is the main log file: /opt/app/oracle/product/11.2.0/grid_4/cfgtoollogs/opatchauto2015-08-10_14-06-35.log
This file will show your detected configuration and all the steps that opatchauto attempted to do on your system:
2015-08-10 14:06:35: Starting Oracle Restart Patch Setup
Using configuration parameter file: /opt/app/oracle/product/11.2.0/grid_4/crs/install/crsconfig_params
The opatch Component check failed. This patch is not applicable for /opt/app/oracle/product/11.2.0/dbhome_4
The opatch Component check failed. This patch is not applicable for /opt/app/oracle/product/11.2.0/dbhome_4
Conflict check failed for oracle home  /opt/app/oracle/product/11.2.0/dbhome_4
Conflict check failed
ERROR: Conflict-Check has failed . Please refer to /opt/app/oracle/product/11.2.0/grid_4/cfgtoollogs/opatchauto2015-08-10_14-06-35.log for details
Though the error message says conflict check failed, it is not the real reason in this case. Examining the log file revealed following
2015-08-10 14:07:13: Status of opatch version check  for /opt/app/oracle/product/11.2.0/grid_4 is 1
2015-08-10 14:07:13: Opatch version check passed for oracle home  /opt/app/oracle/product/11.2.0/grid_4
2015-08-10 14:07:13: Opatch version check passed  for all oracle homes
2015-08-10 14:07:13: Processing oracle home /opt/app/oracle/product/11.2.0/dbhome_4
2015-08-10 14:07:13: Opening file /etc/oracle/ocr.loc
2015-08-10 14:07:13: Value (TRUE) is set for key=local_only
2015-08-10 14:07:13: Home type of /opt/app/oracle/product/11.2.0/dbhome_4 is DB
2015-08-10 14:07:13: Oracle user for /opt/app/oracle/product/11.2.0/dbhome_4 is oracle
2015-08-10 14:07:13: Running as user oracle: /opt/app/oracle/product/11.2.0/dbhome_4/OPatch/opatch prereq CheckComponents -ph ./20996923/20760982 -oh /opt/app/oracle/product/11.2.0/dbhome_4 
-customLogDir /opt/app/oracle/product/11.2.0/grid_4/cfgtoollogs/opatchauto/core
2015-08-10 14:07:13: s_run_as_user2: Running /bin/su oracle -c ' /opt/app/oracle/product/11.2.0/dbhome_4/OPatch/opatch prereq CheckComponents -ph ./20996923/20760982 -oh /opt/app/oracle/product/11.2.0/dbhome_4 
-customLogDir /opt/app/oracle/product/11.2.0/grid_4/cfgtoollogs/opatchauto/core '
2015-08-10 14:07:14: Removing file /tmp/fileWJrEf9
2015-08-10 14:07:14: Successfully removed file: /tmp/fileWJrEf9
2015-08-10 14:07:14: /bin/su exited with rc=1

2015-08-10 14:07:14: The opatch Component check failed. This patch is not applicable for /opt/app/oracle/product/11.2.0/dbhome_4
2015-08-10 14:07:14: The component check failed with following error
2015-08-10 14:07:14: Input custom log directory "/opt/app/oracle/product/11.2.0/grid_4/cfgtoollogs/opatchauto/core" does not exists or not a valid directory.

 OPatch failed with error code 1
From the output it could be seen that when opatch is checking components for oracle home patching it is setting a custom log directory which is inside the grid home. This wouldn't be a problem if both GI home and Oracle home are owned by the same user. But in this case role separation is used and the directories cfgtoollogs,opatchauto,core all have permission set to either 700 or 755 and ownership grid:oinstall. This makes oracle user unable to write to these directories and the failure of the patch apply.
Metalink notes 2022797.1 and 1964569.1 (These MOS notes are related to 12c) suggest to use -customLogDir option to specify a directory but this is not applicable for "opatch auto" option.

Two ways to overcome this problem. One is to grant the oinstall group the write permission to cfgtoollogs/opatchauto/core before the patch apply and revoke once the the patch is successfully applied.
Second option is to use a lower version OPatch, in this case using OPatch version (minimum version required according to PSU's readme.html) enabled to successfully apply the patch.
As patch apply went ahead without a problem using a lower version of opatch mean that customLogDir option was introduced in higher version OPatch ( without the consideration to possible failures that could arise due to use of role separation.

Useful Metalink Notes
opatchauto wasn't able to run ... because 'bash' command could not be found [ID 2022797.1]
"opatch lsinventory" on CRS/GI home fails with error code 15 [ID 1964569.1]