Sunday, January 20, 2013

January 2013 PSU (11.2.0.3) Manual Steps for Apply/Rollback Patch vs OPatch Auto

It seems future of patch applying is to be "auto" if the January 2013 PSU for 11.2.0.3 is anything to go by. First is that manual steps has been taken out of the main readme.html that comes with the patch to a supplemental document (1494646.1). So more documents to go through than to have all the necessary information in one place.
Secondly from GI PSU 11.2.0.3.5 and up the GI PSU have a separate GI and DB portions, which means there's patch number GI component number and DB component number. These components must be substitued in the commands used for manually applying the patch and these commands now come in generic form such as
<GI_HOME>/OPatch/opatch napply -oh <GI_HOME> -local <UNZIPPED_PATCH_LOCATION>/<GI_components_number>
<GI_HOME>/OPatch/opatch apply -oh <GI_HOME> -local <UNZIPPED_PATCH_LOCATION>/<DB_PSU_number>
Manually applying patch gives the DBA the control and also to closely examine what goes on.
"opatch auto" has it's merits as well. One command to patch both GI and DB Homes. It brings down the clusterware stack and start it up once finished. One draw back is that even if one component fails rest of the patching will continue and one has to come back to it at the end of the patching. Manual patching is more interactive allowing any errors/failures to be corrected before continuing with the rest of the patching.
Auto patching is run as root and switch (or execute as) between the appropriate user (grid or oracle) to the relevant home as it apply the patch. Opatch auto also need an ocm response file. How To Create An OCM Response File For Opatch Silent Installation [ID 966023.1]
Output on the shell where opatch auto is run only shows the outcome of each patchs' apply process (pass or fail).
[root@rhel6m1 patches]# /opt/app/11.2.0/grid/OPatch/opatch auto /usr/local/patches -ocmrf ocm.rsp
Executing /usr/bin/perl /opt/app/11.2.0/grid/OPatch/crs/patch112.pl -patchdir /usr/local -patchn patches -ocmrf ocm.rsp -paramfile /opt/app/11.2.0/grid/crs/install/crsconfig_params
opatch auto log file location is /opt/app/11.2.0/grid/OPatch/crs/../../cfgtoollogs/opatchauto2013-01-17_11-33-35.log
Detected Oracle Clusterware install
Using configuration parameter file: /opt/app/11.2.0/grid/crs/install/crsconfig_params
patch /usr/local/patches/15876003/custom/server/15876003  apply successful for home  /opt/app/oracle/product/11.2.0/dbhome_1
patch /usr/local/patches/14727310  apply successful for home  /opt/app/oracle/product/11.2.0/dbhome_1
Successfully unlock /opt/app/11.2.0/grid
patch /usr/local/patches/15876003  apply successful for home  /opt/app/11.2.0/grid
patch /usr/local/patches/14727310  apply successful for home  /opt/app/11.2.0/grid
ACFS-9300: ADVM/ACFS distribution files found.
ACFS-9312: Existing ADVM/ACFS installation detected.
ACFS-9314: Removing previous ADVM/ACFS installation.
ACFS-9315: Previous ADVM/ACFS components successfully removed.
ACFS-9307: Installing requested ADVM/ACFS software.
ACFS-9308: Loading installed ADVM/ACFS drivers.
ACFS-9321: Creating udev for ADVM/ACFS.
ACFS-9323: Creating module dependencies - this may take some time.
ACFS-9154: Loading 'oracleoks.ko' driver.
ACFS-9154: Loading 'oracleadvm.ko' driver.
ACFS-9154: Loading 'oracleacfs.ko' driver.
ACFS-9327: Verifying ADVM/ACFS devices.
ACFS-9156: Detecting control device '/dev/asm/.asm_ctl_spec'.
ACFS-9156: Detecting control device '/dev/ofsctl'.
ACFS-9309: ADVM/ACFS installation correctness verified.
CRS-4123: Oracle High Availability Services has been started.
After apply query the lsinventory for the PSU
$GI_HOME/OPatch/opatch lsinventory -local -bugs_fixed | grep "GRID INFRASTRUCTURE PATCH SET UPDATE"
$ORACLE_HOME/OPatch/opatch lsinventory -local -bugs_fixed | grep "DATABASE PATCH SET UPDATE"
It all comes down to personal preference, manual or auto but it won't be a surprise if eventually "auto" becomes the "recommended" way while manual is only for "troubleshooting".




Below is a full log of applying GI PSU 11.2.0.3.5 on RAC system running on RHEL6 which shows the behind the scene work of opatch auto such as alternating between grid and oracle user.
2013-01-17 11:33:35: Using Oracle CRS home /opt/app/11.2.0/grid
2013-01-17 11:33:35: Checking parameters from paramfile /opt/app/11.2.0/grid/crs/install/crsconfig_params to validate installer variables
2013-01-17 11:33:35: The configuration parameter file /opt/app/11.2.0/grid/crs/install/crsconfig_params is valid
2013-01-17 11:33:35: ### Printing the configuration values from files:
2013-01-17 11:33:35:    /opt/app/11.2.0/grid/crs/install/crsconfig_params
2013-01-17 11:33:35:    /opt/app/11.2.0/grid/OPatch/crs/s_crsconfig_defs
2013-01-17 11:33:35: ASM_AU_SIZE=1
2013-01-17 11:33:35: ASM_DISCOVERY_STRING=/dev/sd*
2013-01-17 11:33:35: ASM_DISKS=/dev/sdb1,/dev/sdc1,/dev/sdd1
2013-01-17 11:33:35: ASM_DISK_GROUP=CLUSTER_DG
2013-01-17 11:33:35: ASM_REDUNDANCY=NORMAL
2013-01-17 11:33:35: ASM_SPFILE=
2013-01-17 11:33:35: ASM_UPGRADE=false
2013-01-17 11:33:35: CLSCFG_MISSCOUNT=
2013-01-17 11:33:35: CLUSTER_GUID=
2013-01-17 11:33:35: CLUSTER_NAME=rhel6m-cluster
2013-01-17 11:33:35: CRFHOME="/opt/app/11.2.0/grid"
2013-01-17 11:33:35: CRS_LIMIT_CORE=unlimited
2013-01-17 11:33:35: CRS_LIMIT_MEMLOCK=unlimited
2013-01-17 11:33:35: CRS_LIMIT_OPENFILE=65536
2013-01-17 11:33:35: CRS_LIMIT_STACK=2048
2013-01-17 11:33:35: CRS_NODEVIPS='rhel6m1-vip/255.255.255.0/eth0,rhel6m2-vip/255.255.255.0/eth0'
2013-01-17 11:33:35: CRS_STORAGE_OPTION=1
2013-01-17 11:33:35: CSS_LEASEDURATION=400
2013-01-17 11:33:35: DIRPREFIX=
2013-01-17 11:33:35: DISABLE_OPROCD=0
2013-01-17 11:33:35: EXTERNAL_ORACLE=/opt/oracle
2013-01-17 11:33:35: EXTERNAL_ORACLE_BIN=/opt/oracle/bin
2013-01-17 11:33:35: GNS_ADDR_LIST=
2013-01-17 11:33:35: GNS_ALLOW_NET_LIST=
2013-01-17 11:33:35: GNS_CONF=false
2013-01-17 11:33:35: GNS_DENY_ITF_LIST=
2013-01-17 11:33:35: GNS_DENY_NET_LIST=
2013-01-17 11:33:35: GNS_DOMAIN_LIST=
2013-01-17 11:33:35: GPNPCONFIGDIR=/opt/app/11.2.0/grid
2013-01-17 11:33:35: GPNPGCONFIGDIR=/opt/app/11.2.0/grid
2013-01-17 11:33:35: GPNP_PA=
2013-01-17 11:33:35: HOST_NAME_LIST=rhel6m1,rhel6m2
2013-01-17 11:33:35: ID=/etc/init.d
2013-01-17 11:33:35: INIT=/sbin/init
2013-01-17 11:33:35: ISROLLING=true
2013-01-17 11:33:35: IT=/etc/inittab
2013-01-17 11:33:35: JLIBDIR=/opt/app/11.2.0/grid/jlib
2013-01-17 11:33:35: JREDIR=/opt/app/11.2.0/grid/jdk/jre/
2013-01-17 11:33:35: LANGUAGE_ID=AMERICAN_AMERICA.AL32UTF8
2013-01-17 11:33:35: MSGFILE=/var/adm/messages
2013-01-17 11:33:35: NETWORKS="eth0"/192.168.0.0:public,"eth1"/192.168.1.0:cluster_interconnect
2013-01-17 11:33:35: NEW_HOST_NAME_LIST=
2013-01-17 11:33:35: NEW_NODEVIPS='rhel6m1-vip/255.255.255.0/eth0,rhel6m2-vip/255.255.255.0/eth0'
2013-01-17 11:33:35: NEW_NODE_NAME_LIST=
2013-01-17 11:33:35: NEW_PRIVATE_NAME_LIST=
2013-01-17 11:33:35: NODELIST=rhel6m1,rhel6m2
2013-01-17 11:33:35: NODE_NAME_LIST=rhel6m1,rhel6m2
2013-01-17 11:33:35: OCFS_CONFIG=
2013-01-17 11:33:35: OCRCONFIG=/etc/oracle/ocr.loc
2013-01-17 11:33:35: OCRCONFIGDIR=/etc/oracle
2013-01-17 11:33:35: OCRID=
2013-01-17 11:33:35: OCRLOC=ocr.loc
2013-01-17 11:33:35: OCR_LOCATIONS=NO_VAL
2013-01-17 11:33:35: OLASTGASPDIR=/etc/oracle/lastgasp
2013-01-17 11:33:35: OLD_CRS_HOME=
2013-01-17 11:33:35: OLRCONFIG=/etc/oracle/olr.loc
2013-01-17 11:33:35: OLRCONFIGDIR=/etc/oracle
2013-01-17 11:33:35: OLRLOC=olr.loc
2013-01-17 11:33:35: OPROCDCHECKDIR=/etc/oracle/oprocd/check
2013-01-17 11:33:35: OPROCDDIR=/etc/oracle/oprocd
2013-01-17 11:33:35: OPROCDFATALDIR=/etc/oracle/oprocd/fatal
2013-01-17 11:33:35: OPROCDSTOPDIR=/etc/oracle/oprocd/stop
2013-01-17 11:33:35: ORACLE_BASE=/opt/app/oracle
2013-01-17 11:33:35: ORACLE_HOME=/opt/app/11.2.0/grid
2013-01-17 11:33:35: ORACLE_OWNER=grid
2013-01-17 11:33:35: ORA_ASM_GROUP=asmadmin
2013-01-17 11:33:35: ORA_DBA_GROUP=oinstall
2013-01-17 11:33:35: PRIVATE_NAME_LIST=
2013-01-17 11:33:35: RCALLDIR=/etc/rc.d/rc0.d /etc/rc.d/rc1.d /etc/rc.d/rc2.d /etc/rc.d/rc3.d /etc/rc.d/rc4.d /etc/rc.d/rc5.d /etc/rc.d/rc6.d
2013-01-17 11:33:35: RCKDIR=/etc/rc.d/rc0.d /etc/rc.d/rc1.d /etc/rc.d/rc2.d /etc/rc.d/rc3.d /etc/rc.d/rc4.d /etc/rc.d/rc6.d
2013-01-17 11:33:35: RCSDIR=/etc/rc.d/rc3.d /etc/rc.d/rc5.d
2013-01-17 11:33:35: RC_KILL=K15
2013-01-17 11:33:35: RC_KILL_OLD=K96
2013-01-17 11:33:35: RC_KILL_OLD2=K19
2013-01-17 11:33:35: RC_START=S96
2013-01-17 11:33:35: REUSEDG=false
2013-01-17 11:33:35: SCAN_NAME=rhel6m-scan
2013-01-17 11:33:35: SCAN_PORT=1521
2013-01-17 11:33:35: SCRBASE=/etc/oracle/scls_scr
2013-01-17 11:33:35: SILENT=false
2013-01-17 11:33:35: SO_EXT=so
2013-01-17 11:33:35: SRVCFGLOC=srvConfig.loc
2013-01-17 11:33:35: SRVCONFIG=/var/opt/oracle/srvConfig.loc
2013-01-17 11:33:35: SRVCONFIGDIR=/var/opt/oracle
2013-01-17 11:33:35: TZ=Europe/London
2013-01-17 11:33:35: USER_IGNORED_PREREQ=false
2013-01-17 11:33:35: VNDR_CLUSTER=false
2013-01-17 11:33:35: VOTING_DISKS=NO_VAL
2013-01-17 11:33:35: ### Printing other configuration values ###
2013-01-17 11:33:35: CLSCFG_EXTRA_PARMS=
2013-01-17 11:33:35: HAS_GROUP=oinstall
2013-01-17 11:33:35: HAS_USER=root
2013-01-17 11:33:35: HOST=rhel6m1
2013-01-17 11:33:35: OLR_DIRECTORY=/opt/app/11.2.0/grid/cdata
2013-01-17 11:33:35: OLR_LOCATION=/opt/app/11.2.0/grid/cdata/rhel6m1.olr
2013-01-17 11:33:35: ORA_CRS_HOME=/opt/app/11.2.0/grid
2013-01-17 11:33:35: SUPERUSER=root
2013-01-17 11:33:35: UNLOCK=0
2013-01-17 11:33:35: VF_DISCOVERY_STRING=
2013-01-17 11:33:35: crscfg_trace=1
2013-01-17 11:33:35: crscfg_trace_file=/opt/app/11.2.0/grid/OPatch/crs/../../cfgtoollogs/opatchauto2013-01-17_11-33-35.log
2013-01-17 11:33:35: hosts=
2013-01-17 11:33:35: osdfile=/opt/app/11.2.0/grid/OPatch/crs/s_crsconfig_defs
2013-01-17 11:33:35: parameters_valid=1
2013-01-17 11:33:35: paramfile=/opt/app/11.2.0/grid/crs/install/crsconfig_params
2013-01-17 11:33:35: platform_family=unix
2013-01-17 11:33:35: srvctl_trc_suff=0
2013-01-17 11:33:35: user_is_superuser=1
2013-01-17 11:33:35: ### Printing of configuration values complete ###
2013-01-17 11:33:35: No -patchfile specified, assuming the patch is already uncompressed
2013-01-17 11:33:35: Oracle user for /opt/app/11.2.0/grid is grid
2013-01-17 11:33:35: silent mode option is -silent -ocmrf ocm.rsp
2013-01-17 11:33:35: Bundle.xml content is <bundle type = "GI_BUNDLE">
  <entities>
     <entity location="15876003">
       <target type="crs"/>
       <target type="siha"/>
     </entity>
     <entity location="15876003/custom/server/15876003">
       <target type="rac" />
       <target type="sidb"/>
     </entity>
     <entity location="14727310">
       <target type="crs"/>
       <target type="rac"/>
       <target type="sidb"/>
       <target type="siha"/>
     </entity>
   </entities>
 </bundle>

2013-01-17 11:33:35: The patch ids are 15876003 14727310
2013-01-17 11:33:35: The patch ids are 15876003 14727310
2013-01-17 11:33:35: Running as user grid: /opt/app/11.2.0/grid/OPatch/opatch query -get_patch_type /usr/local/patches/15876003 -oh /opt/app/11.2.0/grid
2013-01-17 11:33:35: s_run_as_user2: Running /bin/su grid -c ' /opt/app/11.2.0/grid/OPatch/opatch query -get_patch_type /usr/local/patches/15876003 -oh /opt/app/11.2.0/grid '
2013-01-17 11:33:39: Removing file /tmp/fileSKWEXl
2013-01-17 11:33:39: Successfully removed file: /tmp/fileSKWEXl
2013-01-17 11:33:39: /bin/su successfully executed

2013-01-17 11:33:39: output is  This patch is a "legacy_bundle_top" patch.

2013-01-17 11:33:39: Patch type is "legacy_bundle_top"
2013-01-17 11:33:39: Running as user grid: /opt/app/11.2.0/grid/OPatch/opatch query -get_patch_type /usr/local/patches/14727310 -oh /opt/app/11.2.0/grid
2013-01-17 11:33:39: s_run_as_user2: Running /bin/su grid -c ' /opt/app/11.2.0/grid/OPatch/opatch query -get_patch_type /usr/local/patches/14727310 -oh /opt/app/11.2.0/grid '
2013-01-17 11:33:40: Removing file /tmp/file1oyqPq
2013-01-17 11:33:40: Successfully removed file: /tmp/file1oyqPq
2013-01-17 11:33:40: /bin/su exited with rc=0
 75
2013-01-17 11:33:40: output is 
2013-01-17 11:33:40: Patch type is 
2013-01-17 11:33:40: GI patches are /usr/local/patches/15876003 /usr/local/patches/14727310
2013-01-17 11:33:40: DB patches are /usr/local/patches/15876003/custom/server/15876003 /usr/local/patches/14727310
2013-01-17 11:33:40: Running /opt/app/11.2.0/grid/bin/crsctl check cluster -n rhel6m1
2013-01-17 11:33:40: Executing cmd: /opt/app/11.2.0/grid/bin/crsctl check cluster -n rhel6m1
2013-01-17 11:33:40: Command output:
>  **************************************************************
>  rhel6m1:
>  CRS-4537: Cluster Ready Services is online
>  CRS-4529: Cluster Synchronization Services is online
>  CRS-4533: Event Manager is online
>  ************************************************************** 
>End Command output
2013-01-17 11:33:40: Looking for configured databases on node rhel6m1
2013-01-17 11:33:41: Databases configured on node rhel6m1 are: std11g2
2013-01-17 11:33:41: Determining ORACLE_HOME paths for configured databases
2013-01-17 11:33:41: Executing cmd: /opt/app/11.2.0/grid/bin/srvctl config database -d std11g2
2013-01-17 11:33:43: Command output:
>  Database unique name: std11g2
>  Database name: std11g2
>  Oracle home: /opt/app/oracle/product/11.2.0/dbhome_1
>  Oracle user: oracle
>  Spfile: +DATA/std11g2/spfilestd11g2.ora
>  Domain: 
>  Start options: open
>  Stop options: immediate
>  Database role: PRIMARY
>  Management policy: AUTOMATIC
>  Server pools: std11g2
>  Database instances: std11g21,std11g22
>  Disk Groups: DATA,FLASH
>  Mount point paths: 
>  Services: myservice
>  Type: RAC
>  Database is administrator managed 
>End Command output
2013-01-17 11:33:43: output is Oracle home: /opt/app/oracle/product/11.2.0/dbhome_1
2013-01-17 11:33:43: Oracle home for database std11g2 is /opt/app/oracle/product/11.2.0/dbhome_1
2013-01-17 11:33:43: Oracle Home /opt/app/oracle/product/11.2.0/dbhome_1 is configured with Database(s)-> std11g2
2013-01-17 11:33:43: Oracle user for /opt/app/oracle/product/11.2.0/dbhome_1 is oracle
2013-01-17 11:33:43: oracle home list is /opt/app/oracle/product/11.2.0/dbhome_1
2013-01-17 11:33:43: Processing oracle home /opt/app/oracle/product/11.2.0/dbhome_1
2013-01-17 11:33:43: Opening file /etc/oracle/ocr.loc
2013-01-17 11:33:43: Value (FALSE) is set for key=local_only
2013-01-17 11:33:43: Home type of /opt/app/oracle/product/11.2.0/dbhome_1 is DB
2013-01-17 11:33:43: Oracle user for /opt/app/oracle/product/11.2.0/dbhome_1 is oracle
2013-01-17 11:33:43: Oracle user for /opt/app/oracle/product/11.2.0/dbhome_1 is oracle
2013-01-17 11:33:43: Running as user oracle: /opt/app/oracle/product/11.2.0/dbhome_1/OPatch/opatch version -oh /opt/app/oracle/product/11.2.0/dbhome_1
2013-01-17 11:33:43: s_run_as_user2: Running /bin/su oracle -c ' /opt/app/oracle/product/11.2.0/dbhome_1/OPatch/opatch version -oh /opt/app/oracle/product/11.2.0/dbhome_1 '
2013-01-17 11:33:44: Removing file /tmp/fileTZwL3G
2013-01-17 11:33:44: Successfully removed file: /tmp/fileTZwL3G
2013-01-17 11:33:44: /bin/su successfully executed

2013-01-17 11:33:44: opatch version in oracle home /opt/app/oracle/product/11.2.0/dbhome_1  is 11.2.0.3.0

2013-01-17 11:33:44: Running as user oracle: /opt/app/oracle/product/11.2.0/dbhome_1/OPatch/opatch util checkMinimumOPatchVersion -ph /usr/local/patches/15876003/custom/server/15876003 -version 11.2.0.3.0 -oh /opt/app/oracle/product/11.2.0/dbhome_1
2013-01-17 11:33:44: s_run_as_user2: Running /bin/su oracle -c ' /opt/app/oracle/product/11.2.0/dbhome_1/OPatch/opatch util checkMinimumOPatchVersion -ph /usr/local/patches/15876003/custom/server/15876003 -version 11.2.0.3.0 -oh /opt/app/oracle/product/11.2.0/dbhome_1 '
2013-01-17 11:33:49: Removing file /tmp/filei5JgK0
2013-01-17 11:33:49: Successfully removed file: /tmp/filei5JgK0
2013-01-17 11:33:49: /bin/su successfully executed

2013-01-17 11:33:49: Running as user oracle: /opt/app/oracle/product/11.2.0/dbhome_1/OPatch/opatch util checkMinimumOPatchVersion -ph /usr/local/patches/14727310 -version 11.2.0.3.0 -oh /opt/app/oracle/product/11.2.0/dbhome_1
2013-01-17 11:33:49: s_run_as_user2: Running /bin/su oracle -c ' /opt/app/oracle/product/11.2.0/dbhome_1/OPatch/opatch util checkMinimumOPatchVersion -ph /usr/local/patches/14727310 -version 11.2.0.3.0 -oh /opt/app/oracle/product/11.2.0/dbhome_1 '
2013-01-17 11:33:52: Removing file /tmp/filekjM3wv
2013-01-17 11:33:52: Successfully removed file: /tmp/filekjM3wv
2013-01-17 11:33:52: /bin/su successfully executed

2013-01-17 11:33:52: Status of opatch version check  for /opt/app/oracle/product/11.2.0/dbhome_1 is 1
2013-01-17 11:33:52: Opatch version check passed for oracle home  /opt/app/oracle/product/11.2.0/dbhome_1
2013-01-17 11:33:52: Processing oracle home /opt/app/11.2.0/grid
2013-01-17 11:33:52: Opening file /etc/oracle/ocr.loc
2013-01-17 11:33:52: Value (FALSE) is set for key=local_only
2013-01-17 11:33:52: Home type of /opt/app/11.2.0/grid is CRS
2013-01-17 11:33:52: Oracle user for /opt/app/11.2.0/grid is grid
2013-01-17 11:33:52: Oracle user for /opt/app/11.2.0/grid is grid
2013-01-17 11:33:52: Running as user grid: /opt/app/11.2.0/grid/OPatch/opatch version -oh /opt/app/11.2.0/grid
2013-01-17 11:33:52: s_run_as_user2: Running /bin/su grid -c ' /opt/app/11.2.0/grid/OPatch/opatch version -oh /opt/app/11.2.0/grid '
2013-01-17 11:33:53: Removing file /tmp/fileQxFgZ8
2013-01-17 11:33:53: Successfully removed file: /tmp/fileQxFgZ8
2013-01-17 11:33:53: /bin/su successfully executed

2013-01-17 11:33:53: opatch version in oracle home /opt/app/11.2.0/grid  is 11.2.0.3.0

2013-01-17 11:33:53: Running as user grid: /opt/app/11.2.0/grid/OPatch/opatch util checkMinimumOPatchVersion -ph /usr/local/patches/15876003 -version 11.2.0.3.0 -oh /opt/app/11.2.0/grid
2013-01-17 11:33:53: s_run_as_user2: Running /bin/su grid -c ' /opt/app/11.2.0/grid/OPatch/opatch util checkMinimumOPatchVersion -ph /usr/local/patches/15876003 -version 11.2.0.3.0 -oh /opt/app/11.2.0/grid '
2013-01-17 11:33:57: Removing file /tmp/file1noRAO
2013-01-17 11:33:57: Successfully removed file: /tmp/file1noRAO
2013-01-17 11:33:57: /bin/su successfully executed

2013-01-17 11:33:57: Running as user grid: /opt/app/11.2.0/grid/OPatch/opatch util checkMinimumOPatchVersion -ph /usr/local/patches/14727310 -version 11.2.0.3.0 -oh /opt/app/11.2.0/grid
2013-01-17 11:33:57: s_run_as_user2: Running /bin/su grid -c ' /opt/app/11.2.0/grid/OPatch/opatch util checkMinimumOPatchVersion -ph /usr/local/patches/14727310 -version 11.2.0.3.0 -oh /opt/app/11.2.0/grid '
2013-01-17 11:34:00: Removing file /tmp/file3hDQCF
2013-01-17 11:34:00: Successfully removed file: /tmp/file3hDQCF
2013-01-17 11:34:00: /bin/su successfully executed

2013-01-17 11:34:00: Status of opatch version check  for /opt/app/11.2.0/grid is 1
2013-01-17 11:34:00: Opatch version check passed for oracle home  /opt/app/11.2.0/grid
2013-01-17 11:34:00: Opatch version check passed  for all oracle homes
2013-01-17 11:34:00: Opening file /etc/oracle/ocr.loc
2013-01-17 11:34:00: Value (FALSE) is set for key=local_only
2013-01-17 11:34:00: The cluster nodes are rhel6m1 rhel6m2
2013-01-17 11:34:00: checking if path /opt/app/oracle/product/11.2.0/dbhome_1 is shared
2013-01-17 11:34:00: Running as user grid: /opt/app/11.2.0/grid/bin/cluvfy comp ssa -t software -s /opt/app/oracle/product/11.2.0/dbhome_1 -n rhel6m1,rhel6m2 -display_status
2013-01-17 11:34:00: s_run_as_user2: Running /bin/su grid -c ' /opt/app/11.2.0/grid/bin/cluvfy comp ssa -t software -s /opt/app/oracle/product/11.2.0/dbhome_1 -n rhel6m1,rhel6m2 -display_status '
2013-01-17 11:35:02: Removing file /tmp/filejzzDOD
2013-01-17 11:35:02: Successfully removed file: /tmp/filejzzDOD
2013-01-17 11:35:02: /bin/su exited with rc=0
 1
2013-01-17 11:35:02: return code for shared check is 256
2013-01-17 11:35:02: output of sharedness check is 
 Verifying shared storage accessibility 
 
 Checking shared storage accessibility...
 
 "/opt/app/oracle/product/11.2.0/dbhome_1" is not shared
 
 
 Shared storage check failed on nodes "rhel6m2,rhel6m1"
 
 Verification of shared storage accessibility was unsuccessful on all the specified nodes. 
 NODE_STATUS::rhel6m2:VFAIL
 NODE_STATUS::rhel6m1:VFAIL
 OVERALL_STATUS::VFAIL

2013-01-17 11:35:02: the ishared value is 0
2013-01-17 11:35:02: The oracle home /opt/app/oracle/product/11.2.0/dbhome_1 is not shared
2013-01-17 11:35:02: Opening file /etc/oracle/ocr.loc
2013-01-17 11:35:02: Value (FALSE) is set for key=local_only
2013-01-17 11:35:02: The cluster nodes are rhel6m1 rhel6m2
2013-01-17 11:35:02: checking if path /opt/app/11.2.0/grid/crs/install is shared
2013-01-17 11:35:02: Running as user grid: /opt/app/11.2.0/grid/bin/cluvfy comp ssa -t software -s /opt/app/11.2.0/grid/crs/install -n rhel6m1,rhel6m2 -display_status
2013-01-17 11:35:02: s_run_as_user2: Running /bin/su grid -c ' /opt/app/11.2.0/grid/bin/cluvfy comp ssa -t software -s /opt/app/11.2.0/grid/crs/install -n rhel6m1,rhel6m2 -display_status '
2013-01-17 11:35:21: Removing file /tmp/file4Xaqdc
2013-01-17 11:35:21: Successfully removed file: /tmp/file4Xaqdc
2013-01-17 11:35:21: /bin/su exited with rc=0
 1
2013-01-17 11:35:21: return code for shared check is 256
2013-01-17 11:35:21: output of sharedness check is 
 Verifying shared storage accessibility 
 
 Checking shared storage accessibility...
 
 "/opt/app/11.2.0/grid/crs/install" is not shared
 
 
 Shared storage check failed on nodes "rhel6m2,rhel6m1"
 
 Verification of shared storage accessibility was unsuccessful on all the specified nodes. 
 NODE_STATUS::rhel6m2:VFAIL
 NODE_STATUS::rhel6m1:VFAIL
 OVERALL_STATUS::VFAIL

2013-01-17 11:35:21: the ishared value is 0
2013-01-17 11:35:21: The oracle home /opt/app/11.2.0/grid is not shared
2013-01-17 11:35:21: Processing oracle home /opt/app/oracle/product/11.2.0/dbhome_1
2013-01-17 11:35:21: Opening file /etc/oracle/ocr.loc
2013-01-17 11:35:21: Value (FALSE) is set for key=local_only
2013-01-17 11:35:21: Home type of /opt/app/oracle/product/11.2.0/dbhome_1 is DB
2013-01-17 11:35:21: Oracle user for /opt/app/oracle/product/11.2.0/dbhome_1 is oracle
2013-01-17 11:35:21: Running as user oracle: /opt/app/oracle/product/11.2.0/dbhome_1/OPatch/opatch prereq CheckComponents -ph /usr/local/patches/15876003/custom/server/15876003 -oh /opt/app/oracle/product/11.2.0/dbhome_1
2013-01-17 11:35:21: s_run_as_user2: Running /bin/su oracle -c ' /opt/app/oracle/product/11.2.0/dbhome_1/OPatch/opatch prereq CheckComponents -ph /usr/local/patches/15876003/custom/server/15876003 -oh /opt/app/oracle/product/11.2.0/dbhome_1 '
2013-01-17 11:35:24: Removing file /tmp/filedswEgx
2013-01-17 11:35:24: Successfully removed file: /tmp/filedswEgx
2013-01-17 11:35:24: /bin/su successfully executed

2013-01-17 11:35:24: Running as user oracle: /opt/app/oracle/product/11.2.0/dbhome_1/OPatch/opatch prereq CheckComponents -ph /usr/local/patches/14727310 -oh /opt/app/oracle/product/11.2.0/dbhome_1
2013-01-17 11:35:24: s_run_as_user2: Running /bin/su oracle -c ' /opt/app/oracle/product/11.2.0/dbhome_1/OPatch/opatch prereq CheckComponents -ph /usr/local/patches/14727310 -oh /opt/app/oracle/product/11.2.0/dbhome_1 '
2013-01-17 11:35:28: Removing file /tmp/filehlvsO1
2013-01-17 11:35:28: Successfully removed file: /tmp/filehlvsO1
2013-01-17 11:35:28: /bin/su successfully executed

2013-01-17 11:35:28: Oracle user for /opt/app/oracle/product/11.2.0/dbhome_1 is oracle
2013-01-17 11:35:28: Running as user oracle: /opt/app/oracle/product/11.2.0/dbhome_1/OPatch/opatch prereq CheckConflictAgainstOH -ph /usr/local/patches/15876003/custom/server/15876003 -oh /opt/app/oracle/product/11.2.0/dbhome_1
2013-01-17 11:35:28: s_run_as_user2: Running /bin/su oracle -c ' /opt/app/oracle/product/11.2.0/dbhome_1/OPatch/opatch prereq CheckConflictAgainstOH -ph /usr/local/patches/15876003/custom/server/15876003 -oh /opt/app/oracle/product/11.2.0/dbhome_1 '
2013-01-17 11:35:35: Removing file /tmp/fileBBS2qF
2013-01-17 11:35:35: Successfully removed file: /tmp/fileBBS2qF
2013-01-17 11:35:35: /bin/su successfully executed

2013-01-17 11:35:35: Running as user oracle: /opt/app/oracle/product/11.2.0/dbhome_1/OPatch/opatch prereq CheckConflictAgainstOH -ph /usr/local/patches/14727310 -oh /opt/app/oracle/product/11.2.0/dbhome_1
2013-01-17 11:35:35: s_run_as_user2: Running /bin/su oracle -c ' /opt/app/oracle/product/11.2.0/dbhome_1/OPatch/opatch prereq CheckConflictAgainstOH -ph /usr/local/patches/14727310 -oh /opt/app/oracle/product/11.2.0/dbhome_1 '
2013-01-17 11:35:42: Removing file /tmp/file1i2Z7B
2013-01-17 11:35:42: Successfully removed file: /tmp/file1i2Z7B
2013-01-17 11:35:42: /bin/su successfully executed

2013-01-17 11:35:42: Status of component/conflict check  for /opt/app/oracle/product/11.2.0/dbhome_1 is 1
2013-01-17 11:35:42:  Conflict check passes for oracle home  /opt/app/oracle/product/11.2.0/dbhome_1
2013-01-17 11:35:42: Processing oracle home /opt/app/11.2.0/grid
2013-01-17 11:35:42: Opening file /etc/oracle/ocr.loc
2013-01-17 11:35:42: Value (FALSE) is set for key=local_only
2013-01-17 11:35:42: Home type of /opt/app/11.2.0/grid is CRS
2013-01-17 11:35:42: Oracle user for /opt/app/11.2.0/grid is grid
2013-01-17 11:35:42: Running as user grid: /opt/app/11.2.0/grid/OPatch/opatch prereq CheckComponents -ph /usr/local/patches/15876003 -oh /opt/app/11.2.0/grid
2013-01-17 11:35:42: s_run_as_user2: Running /bin/su grid -c ' /opt/app/11.2.0/grid/OPatch/opatch prereq CheckComponents -ph /usr/local/patches/15876003 -oh /opt/app/11.2.0/grid '
2013-01-17 11:35:45: Removing file /tmp/filekTThuP
2013-01-17 11:35:45: Successfully removed file: /tmp/filekTThuP
2013-01-17 11:35:45: /bin/su successfully executed

2013-01-17 11:35:45: Running as user grid: /opt/app/11.2.0/grid/OPatch/opatch prereq CheckComponents -ph /usr/local/patches/14727310 -oh /opt/app/11.2.0/grid
2013-01-17 11:35:45: s_run_as_user2: Running /bin/su grid -c ' /opt/app/11.2.0/grid/OPatch/opatch prereq CheckComponents -ph /usr/local/patches/14727310 -oh /opt/app/11.2.0/grid '
2013-01-17 11:35:48: Removing file /tmp/filersHlyc
2013-01-17 11:35:48: Successfully removed file: /tmp/filersHlyc
2013-01-17 11:35:48: /bin/su successfully executed

2013-01-17 11:35:48: Oracle user for /opt/app/11.2.0/grid is grid
2013-01-17 11:35:48: Running as user grid: /opt/app/11.2.0/grid/OPatch/opatch prereq CheckConflictAgainstOH -ph /usr/local/patches/15876003 -oh /opt/app/11.2.0/grid
2013-01-17 11:35:48: s_run_as_user2: Running /bin/su grid -c ' /opt/app/11.2.0/grid/OPatch/opatch prereq CheckConflictAgainstOH -ph /usr/local/patches/15876003 -oh /opt/app/11.2.0/grid '
2013-01-17 11:35:56: Removing file /tmp/fileMd0bDH
2013-01-17 11:35:56: Successfully removed file: /tmp/fileMd0bDH
2013-01-17 11:35:56: /bin/su successfully executed

2013-01-17 11:35:56: Running as user grid: /opt/app/11.2.0/grid/OPatch/opatch prereq CheckConflictAgainstOH -ph /usr/local/patches/14727310 -oh /opt/app/11.2.0/grid
2013-01-17 11:35:56: s_run_as_user2: Running /bin/su grid -c ' /opt/app/11.2.0/grid/OPatch/opatch prereq CheckConflictAgainstOH -ph /usr/local/patches/14727310 -oh /opt/app/11.2.0/grid '
2013-01-17 11:36:03: Removing file /tmp/fileoLmZhv
2013-01-17 11:36:03: Successfully removed file: /tmp/fileoLmZhv
2013-01-17 11:36:03: /bin/su successfully executed

2013-01-17 11:36:03: Status of component/conflict check  for /opt/app/11.2.0/grid is 1
2013-01-17 11:36:03:  Conflict check passes for oracle home  /opt/app/11.2.0/grid
2013-01-17 11:36:03: Conflict check passed  for all oracle homes
2013-01-17 11:36:03: Processing oracle home /opt/app/oracle/product/11.2.0/dbhome_1
2013-01-17 11:36:03: Opening file /etc/oracle/ocr.loc
2013-01-17 11:36:03: Value (FALSE) is set for key=local_only
2013-01-17 11:36:03: Home type of /opt/app/oracle/product/11.2.0/dbhome_1 is DB
2013-01-17 11:36:03: Performing DB patch
2013-01-17 11:36:03: Oracle user for /opt/app/oracle/product/11.2.0/dbhome_1 is oracle
2013-01-17 11:36:03: Running as user oracle: /opt/app/oracle/product/11.2.0/dbhome_1/OPatch/opatch prereq CheckApplicable -ph /usr/local/patches/15876003/custom/server/15876003 -oh /opt/app/oracle/product/11.2.0/dbhome_1
2013-01-17 11:36:03: s_run_as_user2: Running /bin/su oracle -c ' /opt/app/oracle/product/11.2.0/dbhome_1/OPatch/opatch prereq CheckApplicable -ph /usr/local/patches/15876003/custom/server/15876003 -oh /opt/app/oracle/product/11.2.0/dbhome_1 '
2013-01-17 11:36:07: Removing file /tmp/file82jfJB
2013-01-17 11:36:07: Successfully removed file: /tmp/file82jfJB
2013-01-17 11:36:07: /bin/su successfully executed

2013-01-17 11:36:07: Running as user oracle: /opt/app/oracle/product/11.2.0/dbhome_1/OPatch/opatch prereq CheckApplicable -ph /usr/local/patches/14727310 -oh /opt/app/oracle/product/11.2.0/dbhome_1
2013-01-17 11:36:07: s_run_as_user2: Running /bin/su oracle -c ' /opt/app/oracle/product/11.2.0/dbhome_1/OPatch/opatch prereq CheckApplicable -ph /usr/local/patches/14727310 -oh /opt/app/oracle/product/11.2.0/dbhome_1 '
2013-01-17 11:36:11: Removing file /tmp/filegeYkOS
2013-01-17 11:36:11: Successfully removed file: /tmp/filegeYkOS
2013-01-17 11:36:11: /bin/su successfully executed

2013-01-17 11:36:11: Status of Applicable  check  for /opt/app/oracle/product/11.2.0/dbhome_1 is 1
2013-01-17 11:36:11: Executing cmd: /opt/app/11.2.0/grid/bin/crsctl query crs activeversion
2013-01-17 11:36:11: Command output:
>  Oracle Clusterware active version on the cluster is [11.2.0.3.0] 
>End Command output
2013-01-17 11:36:11: crs version is 11 2 0 3 0
2013-01-17 11:36:11: Running as user oracle: /opt/app/oracle/product/11.2.0/dbhome_1/bin/srvctl stop home -o /opt/app/oracle/product/11.2.0/dbhome_1 -s /opt/app/oracle/product/11.2.0/dbhome_1/srvm/admin/stophome.txt -n rhel6m1
2013-01-17 11:36:11: s_run_as_user2: Running /bin/su oracle -c ' /opt/app/oracle/product/11.2.0/dbhome_1/bin/srvctl stop home -o /opt/app/oracle/product/11.2.0/dbhome_1 -s /opt/app/oracle/product/11.2.0/dbhome_1/srvm/admin/stophome.txt -n rhel6m1 '
2013-01-17 11:36:29: Removing file /tmp/filegXdp4j
2013-01-17 11:36:29: Successfully removed file: /tmp/filegXdp4j
2013-01-17 11:36:29: /bin/su successfully executed

2013-01-17 11:36:29: /opt/app/oracle/product/11.2.0/dbhome_1/bin/srvctl stop home -o /opt/app/oracle/product/11.2.0/dbhome_1 -s /opt/app/oracle/product/11.2.0/dbhome_1/srvm/admin/stophome.txt -n rhel6m1 output is 
2013-01-17 11:36:29: Stopped resources from datbase home /opt/app/oracle/product/11.2.0/dbhome_1
2013-01-17 11:36:29: Running as user oracle: /usr/local/patches/15876003/custom/server/15876003/custom/scripts/prepatch.sh -dbhome /opt/app/oracle/product/11.2.0/dbhome_1
2013-01-17 11:36:29: s_run_as_user2: Running /bin/su oracle -c ' /usr/local/patches/15876003/custom/server/15876003/custom/scripts/prepatch.sh -dbhome /opt/app/oracle/product/11.2.0/dbhome_1 '
2013-01-17 11:36:29: Removing file /tmp/filePZl0Xw
2013-01-17 11:36:29: Successfully removed file: /tmp/filePZl0Xw
2013-01-17 11:36:29: /bin/su successfully executed

2013-01-17 11:36:29: Running as user oracle: true
2013-01-17 11:36:29: s_run_as_user2: Running /bin/su oracle -c ' true '
2013-01-17 11:36:29: Removing file /tmp/fileLQZBdK
2013-01-17 11:36:29: Successfully removed file: /tmp/fileLQZBdK
2013-01-17 11:36:29: /bin/su successfully executed

2013-01-17 11:36:29: prepatch execution for DB home ... success
2013-01-17 11:36:29: Oracle user for /opt/app/oracle/product/11.2.0/dbhome_1 is oracle
2013-01-17 11:36:29: Executing command /opt/app/oracle/product/11.2.0/dbhome_1/OPatch/opatch napply /usr/local/patches/15876003/custom/server/15876003 -local -silent -ocmrf ocm.rsp -oh /opt/app/oracle/product/11.2.0/dbhome_1 as oracle
2013-01-17 11:36:29: Running as user oracle: /opt/app/oracle/product/11.2.0/dbhome_1/OPatch/opatch napply /usr/local/patches/15876003/custom/server/15876003 -local -silent -ocmrf ocm.rsp -oh /opt/app/oracle/product/11.2.0/dbhome_1
2013-01-17 11:36:29: s_run_as_user2: Running /bin/su oracle -c ' /opt/app/oracle/product/11.2.0/dbhome_1/OPatch/opatch napply /usr/local/patches/15876003/custom/server/15876003 -local -silent -ocmrf ocm.rsp -oh /opt/app/oracle/product/11.2.0/dbhome_1 '
2013-01-17 11:37:17: Removing file /tmp/file2D3exX
2013-01-17 11:37:17: Successfully removed file: /tmp/file2D3exX
2013-01-17 11:37:17: /bin/su successfully executed

2013-01-17 11:37:17: status of apply patch is 0
2013-01-17 11:37:17: The apply patch output is Oracle Interim Patch Installer version 11.2.0.3.0
 Copyright (c) 2012, Oracle Corporation.  All rights reserved.
 
 
 Oracle Home       : /opt/app/oracle/product/11.2.0/dbhome_1
 Central Inventory : /opt/app/oraInventory
    from           : /opt/app/oracle/product/11.2.0/dbhome_1/oraInst.loc
 OPatch version    : 11.2.0.3.0
 OUI version       : 11.2.0.3.0
 Log file location : /opt/app/oracle/product/11.2.0/dbhome_1/cfgtoollogs/opatch/opatch2013-01-17_11-36-30AM_1.log
 
 Verifying environment and performing prerequisite checks...
 
 Conflicts/Supersets for each patch are:
 
 Patch : 15876003
 
  Bug Superset of 14275572
  Super set bugs are:
  14275572,  13919095,  13696251,  13348650,  12659561,  13039908,  13825231,  13036424,  12794268,  13011520,  13569812,  12758736,  13000491,  13498267,  13077654,  13001901,  13550689,  13430715,  13806545,  11675721,  14082976,  12771830,  12538907,  13947200,  13066371,  13483672,  12594616,  13540563,  12897651,  12897902,  13241779,  12896850,  12726222,  12829429,  12728585,  13079948,  12876314,  13090686,  12925041,  12995950,  13251796,  12650672,  12398492,  12848480,  13582411,  13652088,  12990582,  13857364,  12975811,  12917897,  13082238,  12947871,  13037709,  13371153,  12878750,  10114953,  11772838,  13058611,  13001955,  11836951,  12965049,  13440962,  12765467,  13727853,  13425727,  12885323,  13965075,  13339443,  12784559,  13332363,  13074261,  12971251,  13811209,  12709476,  13460353,  13523527,  12857064,  13719731,  13396284,  12899169,  13111013,  13323698,  12867511,  12639013,  12959140,  13085732,  12829917,  10317921,  13843080,  12934171,  12849377,  12349553,  13924431,  13869978,  12680491,  12914824,  13789135,  12730342,  13334158,  12950823,  10418841,  13355963,  13531373,  13776758,  12720728,  13620816,  13002015,  13023609,  13024624,  12791719
 
 Patches [   14275572 ] will be rolled back.
 
 
 Do you want to proceed? [y|n]
 Y (auto-answered by -silent)
 User Responded with: Y
 OPatch continues with these patches:   15876003  
 
 Do you want to proceed? [y|n]
 Y (auto-answered by -silent)
 User Responded with: Y
 All checks passed.
 
 Please shutdown Oracle instances running out of this ORACLE_HOME on the local system.
 (Oracle Home = '/opt/app/oracle/product/11.2.0/dbhome_1')
 
 
 Is the local system ready for patching? [y|n]
 Y (auto-answered by -silent)
 User Responded with: Y
 Backing up files...
 Applying interim patch '15876003' to OH '/opt/app/oracle/product/11.2.0/dbhome_1'
 Rolling back interim patch '14275572' from OH '/opt/app/oracle/product/11.2.0/dbhome_1'
 
 Patching component oracle.rdbms, 11.2.0.3.0...
 RollbackSession removing interim patch '14275572' from inventory
 
 
 OPatch back to application of the patch '15876003' after auto-rollback.
 
 
 Patching component oracle.rdbms, 11.2.0.3.0...
 
 Verifying the update...
 Patch 15876003 successfully applied.
 OPatch Session completed with warnings.
 Log file location: /opt/app/oracle/product/11.2.0/dbhome_1/cfgtoollogs/opatch/opatch2013-01-17_11-36-30AM_1.log
 
 OPatch completed with warnings.

2013-01-17 11:37:17: patch /usr/local/patches/15876003/custom/server/15876003  apply successful for home  /opt/app/oracle/product/11.2.0/dbhome_1
2013-01-17 11:37:17: Executing command /opt/app/oracle/product/11.2.0/dbhome_1/OPatch/opatch napply /usr/local/patches/14727310 -local -silent -ocmrf ocm.rsp -oh /opt/app/oracle/product/11.2.0/dbhome_1 as oracle
2013-01-17 11:37:17: Running as user oracle: /opt/app/oracle/product/11.2.0/dbhome_1/OPatch/opatch napply /usr/local/patches/14727310 -local -silent -ocmrf ocm.rsp -oh /opt/app/oracle/product/11.2.0/dbhome_1
2013-01-17 11:37:17: s_run_as_user2: Running /bin/su oracle -c ' /opt/app/oracle/product/11.2.0/dbhome_1/OPatch/opatch napply /usr/local/patches/14727310 -local -silent -ocmrf ocm.rsp -oh /opt/app/oracle/product/11.2.0/dbhome_1 '
2013-01-17 11:39:59: Removing file /tmp/file5tU7W9
2013-01-17 11:39:59: Successfully removed file: /tmp/file5tU7W9
2013-01-17 11:39:59: /bin/su successfully executed

2013-01-17 11:39:59: status of apply patch is 0
2013-01-17 11:39:59: The apply patch output is Oracle Interim Patch Installer version 11.2.0.3.0
 Copyright (c) 2012, Oracle Corporation.  All rights reserved.
 
 
 Oracle Home       : /opt/app/oracle/product/11.2.0/dbhome_1
 Central Inventory : /opt/app/oraInventory
    from           : /opt/app/oracle/product/11.2.0/dbhome_1/oraInst.loc
 OPatch version    : 11.2.0.3.0
 OUI version       : 11.2.0.3.0
 Log file location : /opt/app/oracle/product/11.2.0/dbhome_1/cfgtoollogs/opatch/opatch2013-01-17_11-37-18AM_1.log
 
 Verifying environment and performing prerequisite checks...
 OPatch continues with these patches:   14727310  
 
 Do you want to proceed? [y|n]
 Y (auto-answered by -silent)
 User Responded with: Y
 All checks passed.
 
 Please shutdown Oracle instances running out of this ORACLE_HOME on the local system.
 (Oracle Home = '/opt/app/oracle/product/11.2.0/dbhome_1')
 
 
 Is the local system ready for patching? [y|n]
 Y (auto-answered by -silent)
 User Responded with: Y
 Backing up files...
 Applying sub-patch '14727310' to OH '/opt/app/oracle/product/11.2.0/dbhome_1'
 
 Patching component oracle.rdbms, 11.2.0.3.0...
 
 Patching component oracle.rdbms.dbscripts, 11.2.0.3.0...
 
 Patching component oracle.rdbms.deconfig, 11.2.0.3.0...
 
 Patching component oracle.rdbms.rsf, 11.2.0.3.0...
 
 Patching component oracle.sdo.locator, 11.2.0.3.0...
 
 Patching component oracle.sysman.console.db, 11.2.0.3.0...
 
 Patching component oracle.sysman.oms.core, 10.2.0.4.4...
 
 Verifying the update...
 Composite patch 14727310 successfully applied.
 Log file location: /opt/app/oracle/product/11.2.0/dbhome_1/cfgtoollogs/opatch/opatch2013-01-17_11-37-18AM_1.log
 
 OPatch succeeded.

2013-01-17 11:39:59: patch /usr/local/patches/14727310  apply successful for home  /opt/app/oracle/product/11.2.0/dbhome_1
2013-01-17 11:39:59: Running as user oracle: /usr/local/patches/15876003/custom/server/15876003/custom/scripts/postpatch.sh -dbhome /opt/app/oracle/product/11.2.0/dbhome_1
2013-01-17 11:39:59: s_run_as_user2: Running /bin/su oracle -c ' /usr/local/patches/15876003/custom/server/15876003/custom/scripts/postpatch.sh -dbhome /opt/app/oracle/product/11.2.0/dbhome_1 '
2013-01-17 11:39:59: Removing file /tmp/fileeOwh17
2013-01-17 11:39:59: Successfully removed file: /tmp/fileeOwh17
2013-01-17 11:39:59: /bin/su successfully executed

2013-01-17 11:39:59: Running as user oracle: true
2013-01-17 11:39:59: s_run_as_user2: Running /bin/su oracle -c ' true '
2013-01-17 11:39:59: Removing file /tmp/file3qIa16
2013-01-17 11:39:59: Successfully removed file: /tmp/file3qIa16
2013-01-17 11:39:59: /bin/su successfully executed

2013-01-17 11:39:59: postpatch execution for DB home ... success
2013-01-17 11:39:59: Processing oracle home /opt/app/11.2.0/grid
2013-01-17 11:39:59: Opening file /etc/oracle/ocr.loc
2013-01-17 11:39:59: Value (FALSE) is set for key=local_only
2013-01-17 11:39:59: Home type of /opt/app/11.2.0/grid is CRS
2013-01-17 11:39:59: Unlock crshome...
2013-01-17 11:39:59: Exclude file used is /opt/app/11.2.0/grid/OPatch/crs/installPatch.excl
2013-01-17 11:39:59: Home location in olr.loc is 
2013-01-17 11:39:59: Executing cmd: /opt/app/11.2.0/grid/bin/crsctl stop crs -f
2013-01-17 11:41:59: Command output:
>  CRS-2791: Starting shutdown of Oracle High Availability Services-managed resources on 'rhel6m1'
>  CRS-2673: Attempting to stop 'ora.crsd' on 'rhel6m1'
>  CRS-2790: Starting shutdown of Cluster Ready Services-managed resources on 'rhel6m1'
>  CRS-2673: Attempting to stop 'ora.LISTENER.lsnr' on 'rhel6m1'
>  CRS-2673: Attempting to stop 'ora.CLUSTER_DG.dg' on 'rhel6m1'
>  CRS-2673: Attempting to stop 'ora.registry.acfs' on 'rhel6m1'
>  CRS-2673: Attempting to stop 'ora.DATA.dg' on 'rhel6m1'
>  CRS-2673: Attempting to stop 'ora.FLASH.dg' on 'rhel6m1'
>  CRS-2677: Stop of 'ora.LISTENER.lsnr' on 'rhel6m1' succeeded
>  CRS-2673: Attempting to stop 'ora.rhel6m1.vip' on 'rhel6m1'
>  CRS-2677: Stop of 'ora.rhel6m1.vip' on 'rhel6m1' succeeded
>  CRS-2672: Attempting to start 'ora.rhel6m1.vip' on 'rhel6m2'
>  CRS-2677: Stop of 'ora.DATA.dg' on 'rhel6m1' succeeded
>  CRS-2677: Stop of 'ora.registry.acfs' on 'rhel6m1' succeeded
>  CRS-2676: Start of 'ora.rhel6m1.vip' on 'rhel6m2' succeeded
>  CRS-2677: Stop of 'ora.FLASH.dg' on 'rhel6m1' succeeded
>  CRS-2677: Stop of 'ora.CLUSTER_DG.dg' on 'rhel6m1' succeeded
>  CRS-2673: Attempting to stop 'ora.asm' on 'rhel6m1'
>  CRS-2677: Stop of 'ora.asm' on 'rhel6m1' succeeded
>  CRS-2673: Attempting to stop 'ora.ons' on 'rhel6m1'
>  CRS-2677: Stop of 'ora.ons' on 'rhel6m1' succeeded
>  CRS-2673: Attempting to stop 'ora.net1.network' on 'rhel6m1'
>  CRS-2677: Stop of 'ora.net1.network' on 'rhel6m1' succeeded
>  CRS-2792: Shutdown of Cluster Ready Services-managed resources on 'rhel6m1' has completed
>  CRS-2677: Stop of 'ora.crsd' on 'rhel6m1' succeeded
>  CRS-2673: Attempting to stop 'ora.drivers.acfs' on 'rhel6m1'
>  CRS-2673: Attempting to stop 'ora.mdnsd' on 'rhel6m1'
>  CRS-2673: Attempting to stop 'ora.crf' on 'rhel6m1'
>  CRS-2673: Attempting to stop 'ora.ctssd' on 'rhel6m1'
>  CRS-2673: Attempting to stop 'ora.evmd' on 'rhel6m1'
>  CRS-2673: Attempting to stop 'ora.asm' on 'rhel6m1'
>  CRS-2677: Stop of 'ora.crf' on 'rhel6m1' succeeded
>  CRS-2677: Stop of 'ora.mdnsd' on 'rhel6m1' succeeded
>  CRS-2677: Stop of 'ora.evmd' on 'rhel6m1' succeeded
>  CRS-2677: Stop of 'ora.asm' on 'rhel6m1' succeeded
>  CRS-2673: Attempting to stop 'ora.cluster_interconnect.haip' on 'rhel6m1'
>  CRS-2677: Stop of 'ora.cluster_interconnect.haip' on 'rhel6m1' succeeded
>  CRS-2677: Stop of 'ora.ctssd' on 'rhel6m1' succeeded
>  CRS-2673: Attempting to stop 'ora.cssd' on 'rhel6m1'
>  CRS-2677: Stop of 'ora.drivers.acfs' on 'rhel6m1' succeeded
>  CRS-2677: Stop of 'ora.cssd' on 'rhel6m1' succeeded
>  CRS-2673: Attempting to stop 'ora.gipcd' on 'rhel6m1'
>  CRS-2677: Stop of 'ora.gipcd' on 'rhel6m1' succeeded
>  CRS-2673: Attempting to stop 'ora.gpnpd' on 'rhel6m1'
>  CRS-2677: Stop of 'ora.gpnpd' on 'rhel6m1' succeeded
>  CRS-2793: Shutdown of Oracle High Availability Services-managed resources on 'rhel6m1' has completed
>  CRS-4133: Oracle High Availability Services has been stopped. 
>End Command output
2013-01-17 11:41:59: /opt/app/11.2.0/grid/bin/crsctl stop crs -f
2013-01-17 11:41:59: Executing cmd: /opt/app/11.2.0/grid/bin/crsctl check cluster -n rhel6m1
2013-01-17 11:42:00: Command output:
>  CRS-4639: Could not contact Oracle High Availability Services
>  CRS-4000: Command Check failed, or completed with errors. 
>End Command output
2013-01-17 11:42:01: Executing cmd: /opt/app/11.2.0/grid/bin/crsctl check has
2013-01-17 11:42:02: Command output:
>  CRS-4639: Could not contact Oracle High Availability Services 
>End Command output
2013-01-17 11:44:10: Waiting for complete CRS stack to stop
2013-01-17 11:44:10: Invoking removeproc to clean oracle client procs
2013-01-17 11:44:10: Executing cmd: /sbin/fuser -k /opt/app/11.2.0/grid/bin/crsctl.bin
2013-01-17 11:44:10: fuser command output for /opt/app/11.2.0/grid/bin/crsctl.bin is 
2013-01-17 11:44:10: Oracle user for /opt/app/11.2.0/grid is grid
2013-01-17 11:44:10: Running as user grid: /opt/app/11.2.0/grid/OPatch/opatch prereq CheckApplicable -ph /usr/local/patches/15876003 -oh /opt/app/11.2.0/grid
2013-01-17 11:44:10: s_run_as_user2: Running /bin/su grid -c ' /opt/app/11.2.0/grid/OPatch/opatch prereq CheckApplicable -ph /usr/local/patches/15876003 -oh /opt/app/11.2.0/grid '
2013-01-17 11:44:14: Removing file /tmp/fileCAOkny
2013-01-17 11:44:14: Successfully removed file: /tmp/fileCAOkny
2013-01-17 11:44:14: /bin/su successfully executed

2013-01-17 11:44:14: Running as user grid: /opt/app/11.2.0/grid/OPatch/opatch prereq CheckApplicable -ph /usr/local/patches/14727310 -oh /opt/app/11.2.0/grid
2013-01-17 11:44:14: s_run_as_user2: Running /bin/su grid -c ' /opt/app/11.2.0/grid/OPatch/opatch prereq CheckApplicable -ph /usr/local/patches/14727310 -oh /opt/app/11.2.0/grid '
2013-01-17 11:44:17: Removing file /tmp/filepYiTfa
2013-01-17 11:44:17: Successfully removed file: /tmp/filepYiTfa
2013-01-17 11:44:17: /bin/su successfully executed

2013-01-17 11:44:17: Status of Applicable  check  for /opt/app/11.2.0/grid is 1
2013-01-17 11:44:17: Oracle user for /opt/app/11.2.0/grid is grid
2013-01-17 11:44:17: Executing command /opt/app/11.2.0/grid/OPatch/opatch napply /usr/local/patches/15876003 -local -silent -ocmrf ocm.rsp -oh /opt/app/11.2.0/grid as grid
2013-01-17 11:44:17: Running as user grid: /opt/app/11.2.0/grid/OPatch/opatch napply /usr/local/patches/15876003 -local -silent -ocmrf ocm.rsp -oh /opt/app/11.2.0/grid
2013-01-17 11:44:17: s_run_as_user2: Running /bin/su grid -c ' /opt/app/11.2.0/grid/OPatch/opatch napply /usr/local/patches/15876003 -local -silent -ocmrf ocm.rsp -oh /opt/app/11.2.0/grid '
2013-01-17 11:48:18: Removing file /tmp/fileftFTxU
2013-01-17 11:48:18: Successfully removed file: /tmp/fileftFTxU
2013-01-17 11:48:18: /bin/su successfully executed

2013-01-17 11:48:18: status of apply patch is 0
2013-01-17 11:48:18: The apply patch output is Oracle Interim Patch Installer version 11.2.0.3.0
 Copyright (c) 2012, Oracle Corporation.  All rights reserved.
 
 
 Oracle Home       : /opt/app/11.2.0/grid
 Central Inventory : /opt/app/oraInventory
    from           : /opt/app/11.2.0/grid/oraInst.loc
 OPatch version    : 11.2.0.3.0
 OUI version       : 11.2.0.3.0
 Log file location : /opt/app/11.2.0/grid/cfgtoollogs/opatch/opatch2013-01-17_11-44-18AM_1.log
 
 Verifying environment and performing prerequisite checks...
 
 Conflicts/Supersets for each patch are:
 
 Patch : 15876003
 
  Bug Superset of 14275572
  Super set bugs are:
  14275572,  13919095,  13696251,  13348650,  12659561,  13039908,  13825231,  13036424,  12794268,  13011520,  13569812,  12758736,  13000491,  13498267,  13077654,  13001901,  13550689,  13430715,  13806545,  11675721,  14082976,  12771830,  12538907,  13947200,  13066371,  13483672,  12594616,  13540563,  12897651,  12897902,  13241779,  12896850,  12726222,  12829429,  12728585,  13079948,  12876314,  13090686,  12925041,  12995950,  13251796,  12650672,  12398492,  12848480,  13582411,  13652088,  12990582,  13857364,  12975811,  12917897,  13082238,  12947871,  13037709,  13371153,  12878750,  10114953,  11772838,  13058611,  13001955,  11836951,  12965049,  13440962,  12765467,  13727853,  13425727,  12885323,  13965075,  13339443,  12784559,  13332363,  13074261,  12971251,  13811209,  12709476,  13460353,  13523527,  12857064,  13719731,  13396284,  12899169,  13111013,  13323698,  12867511,  12639013,  12959140,  13085732,  12829917,  10317921,  13843080,  12934171,  12849377,  12349553,  13924431,  13869978,  12680491,  12914824,  13789135,  12730342,  13334158,  12950823,  10418841,  13355963,  13531373,  13776758,  12720728,  13620816,  13002015,  13023609,  13024624,  12791719,  13886023,  13255295,  13821454,  12782756,  14625969,  14152875,  14186070,  12873909,  14214257,  12914722,  13243172,  12842804,  13045518,  12765868,  12772345,  12663376,  13345868,  14059576,  13683090,  12932852,  13889047,  12695029,  13146560,  13038806,  14251904,  14070200,  13820621,  14304758,  13396356,  13697828,  13258062,  12834777,  12996572,  13941934,  13657366,  13019958,  12810890,  13888719,  13502441,  13726162,  13880925,  14153867,  13506114,  12820045,  13604057,  12823838,  13877508,  12823042,  14494305,  13582706,  13617861,  12825835,  13263435,  13025879,  13853089,  14009845,  13410987,  13570879,  13637590,  12827493,  13247273,  13068077
 
 Patches [   14275572 ] will be rolled back.
 
 
 Do you want to proceed? [y|n]
 Y (auto-answered by -silent)
 User Responded with: Y
 OPatch continues with these patches:   15876003  
 
 Do you want to proceed? [y|n]
 Y (auto-answered by -silent)
 User Responded with: Y
 All checks passed.
 
 Please shutdown Oracle instances running out of this ORACLE_HOME on the local system.
 (Oracle Home = '/opt/app/11.2.0/grid')
 
 
 Is the local system ready for patching? [y|n]
 Y (auto-answered by -silent)
 User Responded with: Y
 Backing up files...
 Applying interim patch '15876003' to OH '/opt/app/11.2.0/grid'
 Rolling back interim patch '14275572' from OH '/opt/app/11.2.0/grid'
 
 Patching component oracle.crs, 11.2.0.3.0...
 
 Patching component oracle.usm, 11.2.0.3.0...
 RollbackSession removing interim patch '14275572' from inventory
 
 
 OPatch back to application of the patch '15876003' after auto-rollback.
 
 
 Patching component oracle.crs, 11.2.0.3.0...
 
 Patching component oracle.usm, 11.2.0.3.0...
 
 Verifying the update...
 Patch 15876003 successfully applied.
 OPatch Session completed with warnings.
 Log file location: /opt/app/11.2.0/grid/cfgtoollogs/opatch/opatch2013-01-17_11-44-18AM_1.log
 
 OPatch completed with warnings.

2013-01-17 11:48:18: patch /usr/local/patches/15876003  apply successful for home  /opt/app/11.2.0/grid
2013-01-17 11:48:18: Executing command /opt/app/11.2.0/grid/OPatch/opatch napply /usr/local/patches/14727310 -local -silent -ocmrf ocm.rsp -oh /opt/app/11.2.0/grid as grid
2013-01-17 11:48:18: Running as user grid: /opt/app/11.2.0/grid/OPatch/opatch napply /usr/local/patches/14727310 -local -silent -ocmrf ocm.rsp -oh /opt/app/11.2.0/grid
2013-01-17 11:48:18: s_run_as_user2: Running /bin/su grid -c ' /opt/app/11.2.0/grid/OPatch/opatch napply /usr/local/patches/14727310 -local -silent -ocmrf ocm.rsp -oh /opt/app/11.2.0/grid '
2013-01-17 11:49:39: Removing file /tmp/file5fgSjJ
2013-01-17 11:49:39: Successfully removed file: /tmp/file5fgSjJ
2013-01-17 11:49:39: /bin/su successfully executed

2013-01-17 11:49:39: status of apply patch is 0
2013-01-17 11:49:39: The apply patch output is Oracle Interim Patch Installer version 11.2.0.3.0
 Copyright (c) 2012, Oracle Corporation.  All rights reserved.
 
 
 Oracle Home       : /opt/app/11.2.0/grid
 Central Inventory : /opt/app/oraInventory
    from           : /opt/app/11.2.0/grid/oraInst.loc
 OPatch version    : 11.2.0.3.0
 OUI version       : 11.2.0.3.0
 Log file location : /opt/app/11.2.0/grid/cfgtoollogs/opatch/opatch2013-01-17_11-48-19AM_1.log
 
 Verifying environment and performing prerequisite checks...
 OPatch continues with these patches:   14727310  
 
 Do you want to proceed? [y|n]
 Y (auto-answered by -silent)
 User Responded with: Y
 All checks passed.
 
 Please shutdown Oracle instances running out of this ORACLE_HOME on the local system.
 (Oracle Home = '/opt/app/11.2.0/grid')
 
 
 Is the local system ready for patching? [y|n]
 Y (auto-answered by -silent)
 User Responded with: Y
 Backing up files...
 Applying sub-patch '14727310' to OH '/opt/app/11.2.0/grid'
 ApplySession: Optional component(s) [ oracle.sysman.console.db, 11.2.0.3.0 ] , [ oracle.sysman.oms.core, 10.2.0.4.4 ]  not present in the Oracle Home or a higher version is found.
 
 Patching component oracle.rdbms, 11.2.0.3.0...
 
 Patching component oracle.rdbms.dbscripts, 11.2.0.3.0...
 
 Patching component oracle.rdbms.deconfig, 11.2.0.3.0...
 
 Patching component oracle.rdbms.rsf, 11.2.0.3.0...
 
 Patching component oracle.sdo.locator, 11.2.0.3.0...
 
 Verifying the update...
 find: `./crf/admin/run/crfmond': Permission denied
 find: `./crf/admin/run/crflogd': Permission denied
 Composite patch 14727310 successfully applied.
 Log file location: /opt/app/11.2.0/grid/cfgtoollogs/opatch/opatch2013-01-17_11-48-19AM_1.log
 
 OPatch succeeded.

2013-01-17 11:49:39: patch /usr/local/patches/14727310  apply successful for home  /opt/app/11.2.0/grid
2013-01-17 11:49:39: Performing Post patch actions
2013-01-17 11:49:39: norestart flag is set to 
2013-01-17 11:49:39: Opening file /etc/oracle/ocr.loc
2013-01-17 11:49:39: Value (FALSE) is set for key=local_only
2013-01-17 11:49:39: Performing Post patch actions for Grid Home /opt/app/11.2.0/grid
2013-01-17 11:49:39: Executing cmd: /opt/app/11.2.0/grid/rdbms/install/rootadd_rdbms.sh
2013-01-17 11:49:39: setrdbmsfileperms succeeded
2013-01-17 11:49:39: Patching Oracle Clusterware
2013-01-17 11:49:39: norestart flag is set to 
2013-01-17 11:49:39: Executing cmd: /bin/rpm -q sles-release
2013-01-17 11:49:39: Command output:
>  package sles-release is not installed 
>End Command output
2013-01-17 11:49:39: init file = /opt/app/11.2.0/grid/crs/init/init.ohasd
2013-01-17 11:49:39: Copying file /opt/app/11.2.0/grid/crs/init/init.ohasd to /etc/init.d directory
2013-01-17 11:49:39: Setting init.ohasd permission in /etc/init.d directory
2013-01-17 11:49:39: init file = /opt/app/11.2.0/grid/crs/init/ohasd
2013-01-17 11:49:39: Copying file /opt/app/11.2.0/grid/crs/init/ohasd to /etc/init.d directory
2013-01-17 11:49:39: Setting ohasd permission in /etc/init.d directory
2013-01-17 11:49:39: Executing cmd: /bin/rpm -q sles-release
2013-01-17 11:49:39: Command output:
>  package sles-release is not installed 
>End Command output
2013-01-17 11:49:39: Removing "/etc/rc.d/rc3.d/S96ohasd"
2013-01-17 11:49:39: Removing file /etc/rc.d/rc3.d/S96ohasd
2013-01-17 11:49:39: Successfully removed file: /etc/rc.d/rc3.d/S96ohasd
2013-01-17 11:49:39: Creating a link "/etc/rc.d/rc3.d/S96ohasd" pointing to /etc/init.d/ohasd
2013-01-17 11:49:39: Removing "/etc/rc.d/rc5.d/S96ohasd"
2013-01-17 11:49:39: Removing file /etc/rc.d/rc5.d/S96ohasd
2013-01-17 11:49:39: Successfully removed file: /etc/rc.d/rc5.d/S96ohasd
2013-01-17 11:49:39: Creating a link "/etc/rc.d/rc5.d/S96ohasd" pointing to /etc/init.d/ohasd
2013-01-17 11:49:39: Removing "/etc/rc.d/rc0.d/K15ohasd"
2013-01-17 11:49:39: Removing file /etc/rc.d/rc0.d/K15ohasd
2013-01-17 11:49:39: Successfully removed file: /etc/rc.d/rc0.d/K15ohasd
2013-01-17 11:49:39: Creating a link "/etc/rc.d/rc0.d/K15ohasd" pointing to /etc/init.d/ohasd
2013-01-17 11:49:39: Removing "/etc/rc.d/rc1.d/K15ohasd"
2013-01-17 11:49:39: Removing file /etc/rc.d/rc1.d/K15ohasd
2013-01-17 11:49:39: Successfully removed file: /etc/rc.d/rc1.d/K15ohasd
2013-01-17 11:49:39: Creating a link "/etc/rc.d/rc1.d/K15ohasd" pointing to /etc/init.d/ohasd
2013-01-17 11:49:39: Removing "/etc/rc.d/rc2.d/K15ohasd"
2013-01-17 11:49:39: Removing file /etc/rc.d/rc2.d/K15ohasd
2013-01-17 11:49:39: Successfully removed file: /etc/rc.d/rc2.d/K15ohasd
2013-01-17 11:49:39: Creating a link "/etc/rc.d/rc2.d/K15ohasd" pointing to /etc/init.d/ohasd
2013-01-17 11:49:39: Removing "/etc/rc.d/rc3.d/K15ohasd"
2013-01-17 11:49:39: Removing file /etc/rc.d/rc3.d/K15ohasd
2013-01-17 11:49:39: Successfully removed file: /etc/rc.d/rc3.d/K15ohasd
2013-01-17 11:49:39: Creating a link "/etc/rc.d/rc3.d/K15ohasd" pointing to /etc/init.d/ohasd
2013-01-17 11:49:39: Removing "/etc/rc.d/rc4.d/K15ohasd"
2013-01-17 11:49:39: Removing file /etc/rc.d/rc4.d/K15ohasd
2013-01-17 11:49:39: Successfully removed file: /etc/rc.d/rc4.d/K15ohasd
2013-01-17 11:49:39: Creating a link "/etc/rc.d/rc4.d/K15ohasd" pointing to /etc/init.d/ohasd
2013-01-17 11:49:39: Removing "/etc/rc.d/rc6.d/K15ohasd"
2013-01-17 11:49:39: Removing file /etc/rc.d/rc6.d/K15ohasd
2013-01-17 11:49:39: Successfully removed file: /etc/rc.d/rc6.d/K15ohasd
2013-01-17 11:49:39: Creating a link "/etc/rc.d/rc6.d/K15ohasd" pointing to /etc/init.d/ohasd
2013-01-17 11:49:39: The file ohasd has been successfully linked to the RC directories
2013-01-17 11:49:39: Executing /opt/app/11.2.0/grid/bin/acfsroot install
2013-01-17 11:49:39: Executing cmd: /opt/app/11.2.0/grid/bin/acfsroot install
2013-01-17 11:51:21: Command output:
>  ACFS-9300: ADVM/ACFS distribution files found.
>  ACFS-9312: Existing ADVM/ACFS installation detected.
>  ACFS-9314: Removing previous ADVM/ACFS installation.
>  ACFS-9315: Previous ADVM/ACFS components successfully removed.
>  ACFS-9307: Installing requested ADVM/ACFS software.
>  ACFS-9308: Loading installed ADVM/ACFS drivers.
>  ACFS-9321: Creating udev for ADVM/ACFS.
>  ACFS-9323: Creating module dependencies - this may take some time.
>  ACFS-9154: Loading 'oracleoks.ko' driver.
>  ACFS-9154: Loading 'oracleadvm.ko' driver.
>  ACFS-9154: Loading 'oracleacfs.ko' driver.
>  ACFS-9327: Verifying ADVM/ACFS devices.
>  ACFS-9156: Detecting control device '/dev/asm/.asm_ctl_spec'.
>  ACFS-9156: Detecting control device '/dev/ofsctl'.
>  ACFS-9309: ADVM/ACFS installation correctness verified. 
>End Command output
2013-01-17 11:51:21: /opt/app/11.2.0/grid/bin/acfsroot install ... success
2013-01-17 11:51:21: USM driver install status is 1
2013-01-17 11:51:21: Validate crsctl command
2013-01-17 11:51:21: Validating /opt/app/11.2.0/grid/bin/crsctl
2013-01-17 11:51:21: Starting Oracle Clusterware
2013-01-17 11:51:21: Executing /opt/app/11.2.0/grid/bin/crsctl start crs
2013-01-17 11:51:21: Executing cmd: /opt/app/11.2.0/grid/bin/crsctl start crs
2013-01-17 11:51:28: Command output:
>  CRS-4123: Oracle High Availability Services has been started. 
>End Command output
2013-01-17 11:51:28: Executing cmd: /opt/app/11.2.0/grid/bin/crsctl stat resource
2013-01-17 11:51:30: Command output:
>  CRS-4535: Cannot communicate with Cluster Ready Services
>  CRS-4000: Command Status failed, or completed with errors. 
>End Command output
2013-01-17 11:51:30: Waiting for Oracle CRSD and EVMD to start
2013-01-17 11:51:35: Executing cmd: /opt/app/11.2.0/grid/bin/crsctl stat resource
2013-01-17 11:51:36: Command output:
>  CRS-4535: Cannot communicate with Cluster Ready Services
>  CRS-4000: Command Status failed, or completed with errors. 
>End Command output
2013-01-17 11:51:36: Waiting for Oracle CRSD and EVMD to start
2013-01-17 11:51:41: Executing cmd: /opt/app/11.2.0/grid/bin/crsctl stat resource
2013-01-17 11:51:43: Command output:
>  CRS-4535: Cannot communicate with Cluster Ready Services
>  CRS-4000: Command Status failed, or completed with errors. 
>End Command output
2013-01-17 11:51:43: Waiting for Oracle CRSD and EVMD to start
2013-01-17 11:51:48: Executing cmd: /opt/app/11.2.0/grid/bin/crsctl stat resource
2013-01-17 11:51:50: Command output:
>  CRS-4535: Cannot communicate with Cluster Ready Services
>  CRS-4000: Command Status failed, or completed with errors. 
>End Command output
2013-01-17 11:51:50: Waiting for Oracle CRSD and EVMD to start
2013-01-17 11:51:55: Executing cmd: /opt/app/11.2.0/grid/bin/crsctl stat resource
2013-01-17 11:51:57: Command output:
>  CRS-4535: Cannot communicate with Cluster Ready Services
>  CRS-4000: Command Status failed, or completed with errors. 
>End Command output
2013-01-17 11:51:57: Waiting for Oracle CRSD and EVMD to start
2013-01-17 11:52:02: Executing cmd: /opt/app/11.2.0/grid/bin/crsctl stat resource
2013-01-17 11:52:04: Command output:
>  CRS-4535: Cannot communicate with Cluster Ready Services
>  CRS-4000: Command Status failed, or completed with errors. 
>End Command output
2013-01-17 11:52:04: Waiting for Oracle CRSD and EVMD to start
2013-01-17 11:52:09: Executing cmd: /opt/app/11.2.0/grid/bin/crsctl stat resource
2013-01-17 11:52:10: Command output:
>  CRS-4535: Cannot communicate with Cluster Ready Services
>  CRS-4000: Command Status failed, or completed with errors. 
>End Command output
2013-01-17 11:52:10: Waiting for Oracle CRSD and EVMD to start
2013-01-17 11:52:15: Executing cmd: /opt/app/11.2.0/grid/bin/crsctl stat resource
2013-01-17 11:52:17: Command output:
>  CRS-4535: Cannot communicate with Cluster Ready Services
>  CRS-4000: Command Status failed, or completed with errors. 
>End Command output
2013-01-17 11:52:17: Waiting for Oracle CRSD and EVMD to start
2013-01-17 11:52:22: Executing cmd: /opt/app/11.2.0/grid/bin/crsctl stat resource
2013-01-17 11:52:24: Command output:
>  CRS-4535: Cannot communicate with Cluster Ready Services
>  CRS-4000: Command Status failed, or completed with errors. 
>End Command output
2013-01-17 11:52:24: Waiting for Oracle CRSD and EVMD to start
2013-01-17 11:52:29: Executing cmd: /opt/app/11.2.0/grid/bin/crsctl stat resource
2013-01-17 11:52:29: Command output:
>  CRS-4535: Cannot communicate with Cluster Ready Services
>  CRS-4000: Command Status failed, or completed with errors. 
>End Command output
2013-01-17 11:52:29: Waiting for Oracle CRSD and EVMD to start
2013-01-17 11:52:34: Executing cmd: /opt/app/11.2.0/grid/bin/crsctl stat resource
2013-01-17 11:52:34: Command output:
>  CRS-4535: Cannot communicate with Cluster Ready Services
>  CRS-4000: Command Status failed, or completed with errors. 
>End Command output
2013-01-17 11:52:34: Waiting for Oracle CRSD and EVMD to start
2013-01-17 11:52:39: Executing cmd: /opt/app/11.2.0/grid/bin/crsctl stat resource
2013-01-17 11:52:40: Command output:
>  NAME=ora.CLUSTER_DG.dg
>  TYPE=ora.diskgroup.type
>  TARGET=ONLINE           , ONLINE
>  STATE=ONLINE on rhel6m1, ONLINE on rhel6m2
>  
>  NAME=ora.DATA.dg
>  TYPE=ora.diskgroup.type
>  TARGET=ONLINE           , ONLINE
>  STATE=ONLINE on rhel6m1, ONLINE on rhel6m2
>  
>  NAME=ora.FLASH.dg
>  TYPE=ora.diskgroup.type
>  TARGET=ONLINE           , ONLINE
>  STATE=ONLINE on rhel6m1, ONLINE on rhel6m2
>  
>  NAME=ora.LISTENER.lsnr
>  TYPE=ora.listener.type
>  TARGET=ONLINE , ONLINE
>  STATE=OFFLINE, ONLINE on rhel6m2
>  
>  NAME=ora.LISTENER_SCAN1.lsnr
>  TYPE=ora.scan_listener.type
>  TARGET=ONLINE
>  STATE=ONLINE on rhel6m2
>  
>  NAME=ora.asm
>  TYPE=ora.asm.type
>  TARGET=ONLINE           , ONLINE
>  STATE=ONLINE on rhel6m1, ONLINE on rhel6m2
>  
>  NAME=ora.cvu
>  TYPE=ora.cvu.type
>  TARGET=ONLINE
>  STATE=ONLINE on rhel6m2
>  
>  NAME=ora.gsd
>  TYPE=ora.gsd.type
>  TARGET=OFFLINE, OFFLINE
>  STATE=OFFLINE, OFFLINE
>  
>  NAME=ora.net1.network
>  TYPE=ora.network.type
>  TARGET=ONLINE           , ONLINE
>  STATE=ONLINE on rhel6m1, ONLINE on rhel6m2
>  
>  NAME=ora.oc4j
>  TYPE=ora.oc4j.type
>  TARGET=ONLINE
>  STATE=ONLINE on rhel6m2
>  
>  NAME=ora.ons
>  TYPE=ora.ons.type
>  TARGET=ONLINE           , ONLINE
>  STATE=ONLINE on rhel6m1, ONLINE on rhel6m2
>  
>  NAME=ora.registry.acfs
>  TYPE=ora.registry.acfs.type
>  TARGET=ONLINE           , ONLINE
>  STATE=ONLINE on rhel6m1, ONLINE on rhel6m2
>  
>  NAME=ora.rhel6m1.vip
>  TYPE=ora.cluster_vip_net1.type
>  TARGET=ONLINE
>  STATE=ONLINE on rhel6m1
>  
>  NAME=ora.rhel6m2.vip
>  TYPE=ora.cluster_vip_net1.type
>  TARGET=ONLINE
>  STATE=ONLINE on rhel6m2
>  
>  NAME=ora.scan1.vip
>  TYPE=ora.scan_vip.type
>  TARGET=ONLINE
>  STATE=ONLINE on rhel6m2
>  
>  NAME=ora.std11g2.db
>  TYPE=ora.database.type
>  TARGET=OFFLINE, ONLINE
>  STATE=OFFLINE, ONLINE on rhel6m2
>  
>  NAME=ora.std11g2.myservice.svc
>  TYPE=ora.service.type
>  TARGET=OFFLINE, ONLINE
>  STATE=OFFLINE, ONLINE on rhel6m2
>   
>End Command output
2013-01-17 11:52:40: Running /opt/app/11.2.0/grid/bin/crsctl check cluster -n rhel6m1
2013-01-17 11:52:40: Executing cmd: /opt/app/11.2.0/grid/bin/crsctl check cluster -n rhel6m1
2013-01-17 11:52:40: Command output:
>  **************************************************************
>  rhel6m1:
>  CRS-4535: Cannot communicate with Cluster Ready Services
>  CRS-4529: Cluster Synchronization Services is online
>  CRS-4533: Event Manager is online
>  ************************************************************** 
>End Command output
2013-01-17 11:52:40: Checking the status of cluster
2013-01-17 11:52:45: Executing cmd: /opt/app/11.2.0/grid/bin/crsctl check cluster -n rhel6m1
2013-01-17 11:52:45: Command output:
>  **************************************************************
>  rhel6m1:
>  CRS-4535: Cannot communicate with Cluster Ready Services
>  CRS-4529: Cluster Synchronization Services is online
>  CRS-4533: Event Manager is online
>  ************************************************************** 
>End Command output
2013-01-17 11:52:45: Checking the status of cluster
2013-01-17 11:52:50: Executing cmd: /opt/app/11.2.0/grid/bin/crsctl check cluster -n rhel6m1
2013-01-17 11:52:50: Command output:
>  **************************************************************
>  rhel6m1:
>  CRS-4535: Cannot communicate with Cluster Ready Services
>  CRS-4529: Cluster Synchronization Services is online
>  CRS-4533: Event Manager is online
>  ************************************************************** 
>End Command output
2013-01-17 11:52:50: Checking the status of cluster
2013-01-17 11:52:55: Executing cmd: /opt/app/11.2.0/grid/bin/crsctl check cluster -n rhel6m1
2013-01-17 11:52:55: Command output:
>  **************************************************************
>  rhel6m1:
>  CRS-4537: Cluster Ready Services is online
>  CRS-4529: Cluster Synchronization Services is online
>  CRS-4533: Event Manager is online
>  ************************************************************** 
>End Command output
2013-01-17 11:52:55: Oracle CRS stack installed and running
2013-01-17 11:52:55: Performing Post patch actions
2013-01-17 11:52:55: norestart flag is set to 
2013-01-17 11:52:55: Opening file /etc/oracle/ocr.loc
2013-01-17 11:52:55: Value (FALSE) is set for key=local_only
2013-01-17 11:52:55: Performing Post Patch start  action for DB Home /opt/app/oracle/product/11.2.0/dbhome_1
2013-01-17 11:52:55: Executing cmd: /opt/app/11.2.0/grid/bin/crsctl stat resource -c rhel6m1
2013-01-17 11:52:56: Command output:
>  NAME=ora.CLUSTER_DG.dg
>  TYPE=ora.diskgroup.type
>  TARGET=ONLINE
>  STATE=ONLINE
>  
>  NAME=ora.DATA.dg
>  TYPE=ora.diskgroup.type
>  TARGET=ONLINE
>  STATE=ONLINE
>  
>  NAME=ora.FLASH.dg
>  TYPE=ora.diskgroup.type
>  TARGET=ONLINE
>  STATE=ONLINE
>  
>  NAME=ora.LISTENER.lsnr
>  TYPE=ora.listener.type
>  TARGET=ONLINE
>  STATE=ONLINE
>  
>  NAME=ora.asm
>  TYPE=ora.asm.type
>  TARGET=ONLINE
>  STATE=ONLINE
>  
>  NAME=ora.gsd
>  TYPE=ora.gsd.type
>  TARGET=OFFLINE
>  STATE=OFFLINE
>  
>  NAME=ora.net1.network
>  TYPE=ora.network.type
>  TARGET=ONLINE
>  STATE=ONLINE
>  
>  NAME=ora.ons
>  TYPE=ora.ons.type
>  TARGET=ONLINE
>  STATE=ONLINE
>  
>  NAME=ora.registry.acfs
>  TYPE=ora.registry.acfs.type
>  TARGET=ONLINE
>  STATE=ONLINE
>  
>  NAME=ora.rhel6m1.vip
>  TYPE=ora.cluster_vip_net1.type
>  CARDINALITY_ID=1
>  TARGET=ONLINE
>  STATE=ONLINE
>   
>End Command output
2013-01-17 11:52:56: Server assignments completed. Ready to start databases
2013-01-17 11:52:56: Running as user oracle: /opt/app/oracle/product/11.2.0/dbhome_1/bin/srvctl start home -o /opt/app/oracle/product/11.2.0/dbhome_1 -s /opt/app/oracle/product/11.2.0/dbhome_1/srvm/admin/stophome.txt -n rhel6m1
2013-01-17 11:52:56: s_run_as_user2: Running /bin/su oracle -c ' /opt/app/oracle/product/11.2.0/dbhome_1/bin/srvctl start home -o /opt/app/oracle/product/11.2.0/dbhome_1 -s /opt/app/oracle/product/11.2.0/dbhome_1/srvm/admin/stophome.txt -n rhel6m1 '
2013-01-17 11:53:16: Removing file /tmp/filePCaf67
2013-01-17 11:53:16: Successfully removed file: /tmp/filePCaf67
2013-01-17 11:53:16: /bin/su successfully executed

2013-01-17 11:53:16: /opt/app/oracle/product/11.2.0/dbhome_1/bin/srvctl start home -o /opt/app/oracle/product/11.2.0/dbhome_1 -s /opt/app/oracle/product/11.2.0/dbhome_1/srvm/admin/stophome.txt -n rhel6m1 output is 
2013-01-17 11:53:16: Started resources from datbase home /opt/app/oracle/product/11.2.0/dbhome_1