Tuesday, September 17, 2013

Downgrade Grid Infrastructure from 11.2.0.4 to 11.2.0.3

The post list the steps for downgrading grid infrastructure from 11.2.0.4 to 11.2.0.3. There's another post with showing downgrading GI from 11.2.0.4 to 11.1.0.7. The environment used for this downgrade had a successful upgrade to 11.2.0.4 from 11.2.0.3
[grid@rhel6m1 grid]$ crsctl query crs activeversion
Oracle Clusterware active version on the cluster is [11.2.0.4.0]
[grid@rhel6m1 grid]$ crsctl query crs softwareversion
Oracle Clusterware version on node [rhel6m1] is [11.2.0.4.0]
If the upgrade to 11.2.0.4 failed on one or on the last node or the downgrade is to a version prior to 11.2 then refer the metalink notes given at the end of the post on how to do a downgrade in such cases. This post also doesn't consider the downgrade of the database. Clusterware must be same or at a higher version than the database software. In this case the database software was still on 11.2.0.3 so downgrading from 11.2.0.4 to 11.2.0.3 wasn't an issue for the database. The old GI home (11.2.0.3 home) was not uninstalled nor removed after the upgrade to 11.2.0.4
<HOME NAME="Ora11g_gridinfrahome1" LOC="/opt/app/11.2.0/grid" TYPE="O" IDX="1"> <-- 11.2.0.3 GI HOME 
   <NODE_LIST>
      <NODE NAME="rhel6m1"/>
      <NODE NAME="rhel6m2"/>
   </NODE_LIST>
</HOME>
<HOME NAME="Ora11g_gridinfrahome2" LOC="/opt/app/11.2.0/grid4" TYPE="O" IDX="4" CRS="true"> <-- 11.2.0.4 GI HOME 
   <NODE_LIST>
      <NODE NAME="rhel6m1"/>
      <NODE NAME="rhel6m2"/>
   </NODE_LIST>
</HOME>
Before proceeding with the downgrade identify the "ocr-node". OCR node an Oracle term for the node which created the backup of the OCR when the GI was upgraded. This is usually the node on which the first rootupgrade.sh was run during the upgrade. Look in the cdata folder inside GI HOME and locate the backup ocr of the form ocrold_version.
[grid@rhel6m1 cdata]$ pwd
/opt/app/11.2.0/grid4/cdata
[grid@rhel6m1 cdata]$ ls -l
total 266508
drwxr-xr-x. 2 grid oinstall      4096 Sep  6 11:39 localhost
-rw-------. 1 root root        130663 Sep  6 11:56 ocr11.2.0.3.0
drwxr-xr-x. 2 grid oinstall      4096 Sep  6 11:57 rhel6m1
-rw-------. 1 root oinstall 272756736 Sep  6 12:39 rhel6m1.olr
drwxrwxr-x. 2 grid oinstall      4096 Sep  6 11:40 rhel6m-cluster
In this two node RAC the rootupgrade.sh was first run on rhel6m1 node and this become the OCR node. The downgrade should be run last on this node.
On all the nodes other than the OCR-node run the rootcrs.pl with the downgrade option from the new GI HOME (11.2.0.4 home).
[root@rhel6m2 grid]# $GI_HOME/crs/install/rootcrs.pl -downgrade -force -oldcrshome /opt/app/11.2.0/grid -version 11.2.0.3.0
Using configuration parameter file: /opt/app/11.2.0/grid4/crs/install/crsconfig_params
CRS-2791: Starting shutdown of Oracle High Availability Services-managed resources on 'rhel6m2'
CRS-2673: Attempting to stop 'ora.crsd' on 'rhel6m2'
CRS-2790: Starting shutdown of Cluster Ready Services-managed resources on 'rhel6m2'
CRS-2673: Attempting to stop 'ora.MYLISTENER.lsnr' on 'rhel6m2'
CRS-2673: Attempting to stop 'ora.std11g2.asa.domain.net.svc' on 'rhel6m2'
CRS-2673: Attempting to stop 'ora.oc4j' on 'rhel6m2'
CRS-2677: Stop of 'ora.std11g2.asa.domain.net.svc' on 'rhel6m2' succeeded
CRS-2673: Attempting to stop 'ora.CLUSTER_DG.dg' on 'rhel6m2'
CRS-2673: Attempting to stop 'ora.registry.acfs' on 'rhel6m2'
CRS-2673: Attempting to stop 'ora.std11g2.db' on 'rhel6m2'
CRS-2677: Stop of 'ora.MYLISTENER.lsnr' on 'rhel6m2' succeeded
CRS-2673: Attempting to stop 'ora.rhel6m2.vip' on 'rhel6m2'
CRS-2677: Stop of 'ora.rhel6m2.vip' on 'rhel6m2' succeeded
CRS-2672: Attempting to start 'ora.rhel6m2.vip' on 'rhel6m1'
CRS-2677: Stop of 'ora.std11g2.db' on 'rhel6m2' succeeded
CRS-2673: Attempting to stop 'ora.DATA.dg' on 'rhel6m2'
CRS-2673: Attempting to stop 'ora.FLASH.dg' on 'rhel6m2'
CRS-2677: Stop of 'ora.DATA.dg' on 'rhel6m2' succeeded
CRS-2677: Stop of 'ora.registry.acfs' on 'rhel6m2' succeeded
CRS-2677: Stop of 'ora.FLASH.dg' on 'rhel6m2' succeeded
CRS-2676: Start of 'ora.rhel6m2.vip' on 'rhel6m1' succeeded
CRS-2677: Stop of 'ora.CLUSTER_DG.dg' on 'rhel6m2' succeeded
CRS-2673: Attempting to stop 'ora.asm' on 'rhel6m2'
CRS-2677: Stop of 'ora.oc4j' on 'rhel6m2' succeeded
CRS-2672: Attempting to start 'ora.oc4j' on 'rhel6m1'
CRS-2677: Stop of 'ora.asm' on 'rhel6m2' succeeded
CRS-2676: Start of 'ora.oc4j' on 'rhel6m1' succeeded
CRS-2673: Attempting to stop 'ora.ons' on 'rhel6m2'
CRS-2677: Stop of 'ora.ons' on 'rhel6m2' succeeded
CRS-2673: Attempting to stop 'ora.net1.network' on 'rhel6m2'
CRS-2677: Stop of 'ora.net1.network' on 'rhel6m2' succeeded
CRS-2792: Shutdown of Cluster Ready Services-managed resources on 'rhel6m2' has completed
CRS-2677: Stop of 'ora.crsd' on 'rhel6m2' succeeded
CRS-2673: Attempting to stop 'ora.drivers.acfs' on 'rhel6m2'
CRS-2673: Attempting to stop 'ora.ctssd' on 'rhel6m2'
CRS-2673: Attempting to stop 'ora.evmd' on 'rhel6m2'
CRS-2673: Attempting to stop 'ora.asm' on 'rhel6m2'
CRS-2673: Attempting to stop 'ora.mdnsd' on 'rhel6m2'
CRS-2677: Stop of 'ora.evmd' on 'rhel6m2' succeeded
CRS-2677: Stop of 'ora.mdnsd' on 'rhel6m2' succeeded
CRS-2677: Stop of 'ora.asm' on 'rhel6m2' succeeded
CRS-2673: Attempting to stop 'ora.cluster_interconnect.haip' on 'rhel6m2'
CRS-2677: Stop of 'ora.cluster_interconnect.haip' on 'rhel6m2' succeeded
CRS-2677: Stop of 'ora.ctssd' on 'rhel6m2' succeeded
CRS-2673: Attempting to stop 'ora.cssd' on 'rhel6m2'
CRS-2677: Stop of 'ora.drivers.acfs' on 'rhel6m2' succeeded
CRS-2677: Stop of 'ora.cssd' on 'rhel6m2' succeeded
CRS-2673: Attempting to stop 'ora.crf' on 'rhel6m2'
CRS-2677: Stop of 'ora.crf' on 'rhel6m2' succeeded
CRS-2673: Attempting to stop 'ora.gipcd' on 'rhel6m2'
CRS-2677: Stop of 'ora.gipcd' on 'rhel6m2' succeeded
CRS-2673: Attempting to stop 'ora.gpnpd' on 'rhel6m2'
CRS-2677: Stop of 'ora.gpnpd' on 'rhel6m2' succeeded
CRS-2793: Shutdown of Oracle High Availability Services-managed resources on 'rhel6m2' has completed
CRS-4133: Oracle High Availability Services has been stopped.
Successfully downgraded Oracle Clusterware stack on this node



Once all the non-OCR Nodes are downgraded, run the downgrade on the OCR-Node with the lastnode option.
[root@rhel6m1 cdata]# $GI_HOME/crs/install/rootcrs.pl -downgrade -force -lastnode -oldcrshome /opt/app/11.2.0/grid -version 11.2.0.3.0
Using configuration parameter file: /opt/app/11.2.0/grid4/crs/install/crsconfig_params
CRS-2791: Starting shutdown of Oracle High Availability Services-managed resources on 'rhel6m1'
CRS-2673: Attempting to stop 'ora.crsd' on 'rhel6m1'
CRS-2790: Starting shutdown of Cluster Ready Services-managed resources on 'rhel6m1'
CRS-2673: Attempting to stop 'ora.MYLISTENER.lsnr' on 'rhel6m1'
CRS-2673: Attempting to stop 'ora.rhel6m2.vip' on 'rhel6m1'
CRS-2673: Attempting to stop 'ora.oc4j' on 'rhel6m1'
CRS-2673: Attempting to stop 'ora.cvu' on 'rhel6m1'
CRS-2673: Attempting to stop 'ora.MYSCANLISTENER_SCAN1.lsnr' on 'rhel6m1'
CRS-2677: Stop of 'ora.MYLISTENER.lsnr' on 'rhel6m1' succeeded
CRS-2677: Stop of 'ora.MYSCANLISTENER_SCAN1.lsnr' on 'rhel6m1' succeeded
CRS-2673: Attempting to stop 'ora.scan1.vip' on 'rhel6m1'
CRS-2677: Stop of 'ora.cvu' on 'rhel6m1' succeeded
CRS-2677: Stop of 'ora.rhel6m2.vip' on 'rhel6m1' succeeded
CRS-2673: Attempting to stop 'ora.std11g2.asa.domain.net.svc' on 'rhel6m1'
CRS-2677: Stop of 'ora.std11g2.asa.domain.net.svc' on 'rhel6m1' succeeded
CRS-2673: Attempting to stop 'ora.rhel6m1.vip' on 'rhel6m1'
CRS-2673: Attempting to stop 'ora.CLUSTER_DG.dg' on 'rhel6m1'
CRS-2673: Attempting to stop 'ora.registry.acfs' on 'rhel6m1'
CRS-2673: Attempting to stop 'ora.std11g2.db' on 'rhel6m1'
CRS-2677: Stop of 'ora.scan1.vip' on 'rhel6m1' succeeded
CRS-2677: Stop of 'ora.std11g2.db' on 'rhel6m1' succeeded
CRS-2673: Attempting to stop 'ora.DATA.dg' on 'rhel6m1'
CRS-2673: Attempting to stop 'ora.FLASH.dg' on 'rhel6m1'
CRS-2677: Stop of 'ora.rhel6m1.vip' on 'rhel6m1' succeeded
CRS-2677: Stop of 'ora.DATA.dg' on 'rhel6m1' succeeded
CRS-2677: Stop of 'ora.registry.acfs' on 'rhel6m1' succeeded
CRS-2677: Stop of 'ora.FLASH.dg' on 'rhel6m1' succeeded
CRS-2677: Stop of 'ora.CLUSTER_DG.dg' on 'rhel6m1' succeeded
CRS-2673: Attempting to stop 'ora.asm' on 'rhel6m1'
CRS-2677: Stop of 'ora.oc4j' on 'rhel6m1' succeeded
CRS-2677: Stop of 'ora.asm' on 'rhel6m1' succeeded
CRS-2673: Attempting to stop 'ora.ons' on 'rhel6m1'
CRS-2677: Stop of 'ora.ons' on 'rhel6m1' succeeded
CRS-2673: Attempting to stop 'ora.net1.network' on 'rhel6m1'
CRS-2677: Stop of 'ora.net1.network' on 'rhel6m1' succeeded
CRS-2792: Shutdown of Cluster Ready Services-managed resources on 'rhel6m1' has completed
CRS-2677: Stop of 'ora.crsd' on 'rhel6m1' succeeded
CRS-2673: Attempting to stop 'ora.drivers.acfs' on 'rhel6m1'
CRS-2673: Attempting to stop 'ora.crf' on 'rhel6m1'
CRS-2673: Attempting to stop 'ora.ctssd' on 'rhel6m1'
CRS-2673: Attempting to stop 'ora.evmd' on 'rhel6m1'
CRS-2673: Attempting to stop 'ora.asm' on 'rhel6m1'
CRS-2673: Attempting to stop 'ora.mdnsd' on 'rhel6m1'
CRS-2677: Stop of 'ora.crf' on 'rhel6m1' succeeded
CRS-2677: Stop of 'ora.evmd' on 'rhel6m1' succeeded
CRS-2677: Stop of 'ora.mdnsd' on 'rhel6m1' succeeded
CRS-2677: Stop of 'ora.asm' on 'rhel6m1' succeeded
CRS-2673: Attempting to stop 'ora.cluster_interconnect.haip' on 'rhel6m1'
CRS-2677: Stop of 'ora.cluster_interconnect.haip' on 'rhel6m1' succeeded
CRS-2677: Stop of 'ora.ctssd' on 'rhel6m1' succeeded
CRS-2673: Attempting to stop 'ora.cssd' on 'rhel6m1'
CRS-2677: Stop of 'ora.cssd' on 'rhel6m1' succeeded
CRS-2673: Attempting to stop 'ora.gipcd' on 'rhel6m1'
CRS-2677: Stop of 'ora.drivers.acfs' on 'rhel6m1' succeeded
CRS-2677: Stop of 'ora.gipcd' on 'rhel6m1' succeeded
CRS-2673: Attempting to stop 'ora.gpnpd' on 'rhel6m1'
CRS-2677: Stop of 'ora.gpnpd' on 'rhel6m1' succeeded
CRS-2793: Shutdown of Oracle High Availability Services-managed resources on 'rhel6m1' has completed
CRS-4133: Oracle High Availability Services has been stopped.
CRS-4123: Oracle High Availability Services has been started.
CRS-2672: Attempting to start 'ora.mdnsd' on 'rhel6m1'
CRS-2676: Start of 'ora.mdnsd' on 'rhel6m1' succeeded
CRS-2672: Attempting to start 'ora.gpnpd' on 'rhel6m1'
CRS-2676: Start of 'ora.gpnpd' on 'rhel6m1' succeeded
CRS-2672: Attempting to start 'ora.cssdmonitor' on 'rhel6m1'
CRS-2672: Attempting to start 'ora.gipcd' on 'rhel6m1'
CRS-2676: Start of 'ora.cssdmonitor' on 'rhel6m1' succeeded
CRS-2676: Start of 'ora.gipcd' on 'rhel6m1' succeeded
CRS-2672: Attempting to start 'ora.cssd' on 'rhel6m1'
CRS-2672: Attempting to start 'ora.diskmon' on 'rhel6m1'
CRS-2676: Start of 'ora.diskmon' on 'rhel6m1' succeeded
CRS-2676: Start of 'ora.cssd' on 'rhel6m1' succeeded
CRS-2672: Attempting to start 'ora.drivers.acfs' on 'rhel6m1'
CRS-2679: Attempting to clean 'ora.cluster_interconnect.haip' on 'rhel6m1'
CRS-2672: Attempting to start 'ora.ctssd' on 'rhel6m1'
CRS-2681: Clean of 'ora.cluster_interconnect.haip' on 'rhel6m1' succeeded
CRS-2672: Attempting to start 'ora.cluster_interconnect.haip' on 'rhel6m1'
CRS-2676: Start of 'ora.ctssd' on 'rhel6m1' succeeded
CRS-2676: Start of 'ora.drivers.acfs' on 'rhel6m1' succeeded
CRS-2676: Start of 'ora.cluster_interconnect.haip' on 'rhel6m1' succeeded
CRS-2672: Attempting to start 'ora.asm' on 'rhel6m1'
CRS-2676: Start of 'ora.asm' on 'rhel6m1' succeeded
CRS-2791: Starting shutdown of Oracle High Availability Services-managed resources on 'rhel6m1'
CRS-2673: Attempting to stop 'ora.mdnsd' on 'rhel6m1'
CRS-2673: Attempting to stop 'ora.drivers.acfs' on 'rhel6m1'
CRS-2673: Attempting to stop 'ora.ctssd' on 'rhel6m1'
CRS-2673: Attempting to stop 'ora.asm' on 'rhel6m1'
CRS-2677: Stop of 'ora.mdnsd' on 'rhel6m1' succeeded
CRS-2677: Stop of 'ora.asm' on 'rhel6m1' succeeded
CRS-2673: Attempting to stop 'ora.cluster_interconnect.haip' on 'rhel6m1'
CRS-2677: Stop of 'ora.cluster_interconnect.haip' on 'rhel6m1' succeeded
CRS-2677: Stop of 'ora.ctssd' on 'rhel6m1' succeeded
CRS-2673: Attempting to stop 'ora.cssd' on 'rhel6m1'
CRS-2677: Stop of 'ora.cssd' on 'rhel6m1' succeeded
CRS-2673: Attempting to stop 'ora.gipcd' on 'rhel6m1'
CRS-2677: Stop of 'ora.drivers.acfs' on 'rhel6m1' succeeded
CRS-2677: Stop of 'ora.gipcd' on 'rhel6m1' succeeded
CRS-2673: Attempting to stop 'ora.gpnpd' on 'rhel6m1'
CRS-2677: Stop of 'ora.gpnpd' on 'rhel6m1' succeeded
CRS-2793: Shutdown of Oracle High Availability Services-managed resources on 'rhel6m1' has completed
CRS-4133: Oracle High Availability Services has been stopped.
CRS-4123: Oracle High Availability Services has been started.
CRS-2672: Attempting to start 'ora.mdnsd' on 'rhel6m1'
CRS-2676: Start of 'ora.mdnsd' on 'rhel6m1' succeeded
CRS-2672: Attempting to start 'ora.gpnpd' on 'rhel6m1'
CRS-2676: Start of 'ora.gpnpd' on 'rhel6m1' succeeded
CRS-2672: Attempting to start 'ora.cssdmonitor' on 'rhel6m1'
CRS-2672: Attempting to start 'ora.gipcd' on 'rhel6m1'
CRS-2676: Start of 'ora.cssdmonitor' on 'rhel6m1' succeeded
CRS-2676: Start of 'ora.gipcd' on 'rhel6m1' succeeded
CRS-2672: Attempting to start 'ora.cssd' on 'rhel6m1'
CRS-2672: Attempting to start 'ora.diskmon' on 'rhel6m1'
CRS-2676: Start of 'ora.diskmon' on 'rhel6m1' succeeded
CRS-2676: Start of 'ora.cssd' on 'rhel6m1' succeeded
CRS-2672: Attempting to start 'ora.drivers.acfs' on 'rhel6m1'
CRS-2679: Attempting to clean 'ora.cluster_interconnect.haip' on 'rhel6m1'
CRS-2672: Attempting to start 'ora.ctssd' on 'rhel6m1'
CRS-2681: Clean of 'ora.cluster_interconnect.haip' on 'rhel6m1' succeeded
CRS-2672: Attempting to start 'ora.cluster_interconnect.haip' on 'rhel6m1'
CRS-2676: Start of 'ora.drivers.acfs' on 'rhel6m1' succeeded
CRS-2676: Start of 'ora.ctssd' on 'rhel6m1' succeeded
CRS-2676: Start of 'ora.cluster_interconnect.haip' on 'rhel6m1' succeeded
CRS-2672: Attempting to start 'ora.asm' on 'rhel6m1'
CRS-2676: Start of 'ora.asm' on 'rhel6m1' succeeded
Successfully downgraded OCR to 11.2.0.3.0
CRS-2672: Attempting to start 'ora.crsd' on 'rhel6m1'
CRS-2676: Start of 'ora.crsd' on 'rhel6m1' succeeded
CRS-2791: Starting shutdown of Oracle High Availability Services-managed resources on 'rhel6m1'
CRS-2673: Attempting to stop 'ora.crsd' on 'rhel6m1'
CRS-2677: Stop of 'ora.crsd' on 'rhel6m1' succeeded
CRS-2673: Attempting to stop 'ora.mdnsd' on 'rhel6m1'
CRS-2673: Attempting to stop 'ora.ctssd' on 'rhel6m1'
CRS-2673: Attempting to stop 'ora.asm' on 'rhel6m1'
CRS-2673: Attempting to stop 'ora.drivers.acfs' on 'rhel6m1'
CRS-2677: Stop of 'ora.mdnsd' on 'rhel6m1' succeeded
CRS-2677: Stop of 'ora.asm' on 'rhel6m1' succeeded
CRS-2673: Attempting to stop 'ora.cluster_interconnect.haip' on 'rhel6m1'
CRS-2677: Stop of 'ora.ctssd' on 'rhel6m1' succeeded
CRS-2677: Stop of 'ora.cluster_interconnect.haip' on 'rhel6m1' succeeded
CRS-2673: Attempting to stop 'ora.cssd' on 'rhel6m1'
CRS-2677: Stop of 'ora.drivers.acfs' on 'rhel6m1' succeeded
CRS-2677: Stop of 'ora.cssd' on 'rhel6m1' succeeded
CRS-2673: Attempting to stop 'ora.gipcd' on 'rhel6m1'
CRS-2677: Stop of 'ora.gipcd' on 'rhel6m1' succeeded
CRS-2673: Attempting to stop 'ora.gpnpd' on 'rhel6m1'
CRS-2677: Stop of 'ora.gpnpd' on 'rhel6m1' succeeded
CRS-2793: Shutdown of Oracle High Availability Services-managed resources on 'rhel6m1' has completed
CRS-4133: Oracle High Availability Services has been stopped.
Successfully downgraded Oracle Clusterware stack on this node
Run '/opt/app/11.2.0/grid/bin/crsctl start crs' on all nodes
Before running the crs start there are few things to be done. As part of the upgrade to 11.2.0.4 trace file analyzer (TFA) is installed. Remove the init script for TFA
#cd /etc/init
[root@rhel6m1 init]# rm oracle-tfa.conf
rm: remove regular file `oracle-tfa.conf'? y
[root@rhel6m1 init]# rm ../init.d/init.tfa
rm: remove regular file `../init.d/init.tfa'? y
Verify the ASM home in the /etc/oratb is pointing to the old GI home
cat /etc/oratab
+ASM2:/opt/app/11.2.0/grid:N            # line added by Agent
Remove gpnp profiles if the rootupgrade.sh was run concurrently on multiple nodes. Also remove the clusterware checkpoint files.
rm -rf /opt/app/11.2.0/grid4/gpnp/*
rm -rf $ORACLE_BASE/Clusterware/ckptGridHA_rhel6m1.xml
Start clusterware stack on all the nodes with crsctl start crs. Verify the active version is the downgraded version.
[grid@rhel6m2 ~]$ crsctl query crs activeversion
Oracle Clusterware active version on the cluster is [11.2.0.3.0]
[grid@rhel6m2 ~]$ crsctl query crs softwareversion
Oracle Clusterware version on node [rhel6m2] is [11.2.0.3.0]
Run a ocrcheck to verify the OCR intergrity
[root@rhel6m2 gpnp]# ocrcheck
Status of Oracle Cluster Registry is as follows :
         Version                  :          3
         Total space (kbytes)     :     262120
         Used space (kbytes)      :       3388
         Available space (kbytes) :     258732
         ID                       : 2072206343
         Device/File Name         : +CLUSTER_DG
                                    Device/File integrity check succeeded
                                    Device/File not configured
         Cluster registry integrity check succeeded
         Logical corruption check succeeded
The inventory will still have the CRS=true on the 11.2.0.4 home. Run the runInstaller to set the CRS=true to the 11.2.0.3 home.
[grid@rhel6m1 bin]$ ./runInstaller -updateNodeList ORACLE_HOME=/opt/app/11.2.0/grid CRS=true
[grid@rhel6m1 bin]$ ./runInstaller -updateNodeList ORACLE_HOME=/opt/app/11.2.0/grid4 CRS=false

<HOME NAME="Ora11g_gridinfrahome1" LOC="/opt/app/11.2.0/grid" TYPE="O" IDX="1" CRS="true">
   <NODE_LIST>
      <NODE NAME="rhel6m1"/>
      <NODE NAME="rhel6m2"/>
   </NODE_LIST>
</HOME>
<HOME NAME="Ora11g_gridinfrahome2" LOC="/opt/app/11.2.0/grid4" TYPE="O" IDX="4" >
   <NODE_LIST>
      <NODE NAME="rhel6m1"/>
      <NODE NAME="rhel6m2"/>
   </NODE_LIST>
</HOME>
As the last step detach the 11.2.0.4 GI Home from the inventory and remove it
 ./runInstaller -detachHome ORACLE_HOME=/opt/app/11.2.0/grid4 -silent
rm -rf /opt/app/11.2.0/grid4 # run on all nodes

Related Post
Downgrade Grid Infrastructure from 12.1.0.2 to 11.2.0.4
Downgrade Grid Infrastructure from 11.2.0.4 to 11.1.0.7

Useful metalink notes
How to Downgrade 11.2.0.2 Grid Infrastructure Cluster to 11.2.0.1 [ 1364230.1]
How to Downgrade 11.2.0.3 Grid Infrastructure Cluster to Lower 11.2 GI or Pre-11.2 CRS [ 1364946.1]
How to Update Inventory to Set/Unset "CRS=true" Flag for Oracle Clusterware Home [ 1053393.1]
Oracle Clusterware (GI or CRS) Related Abbreviations, Acronyms and Procedures [ 1374275.1]

Monday, September 9, 2013

Upgrading RAC from 11.2.0.3 to 11.2.0.4 - Database

After upgrading GI from 11.2.0.3 to 11.2.0.4 next process in upgrading the RAC is to upgrade the database. This include upgrading the database software and database itself. In this post an out-of-place upgrade is done for the database software. It is possible to do an in-place upgrade of the database software but is not recommended by Oracle.
Refer metalink note 1562142.1 for list of bug fixes for 11.2.0.4 patch set and 11.2.0.4 section of Readme document for additional information.
Though at the time of this post not updated to reflect any information related to 11.2.0.4, contents of 837570.1 and 1276368.1 are still valid for 11.2.0.3 to 11.2.0.4 upgrade.
Out-of-place upgrade is similar to software only installation. Since the current Oracle home is not affected, this doesn't require any database down time.
Before executing the runInstaller edit the cvu_config file to fix the pdksh issue. Once done start the database software installation for the upgrade by executing runInstaller.
The new home (11.2.0.4) is dbhome_1 and previous home (11.2.0.3) location is dbhome_2 (this is the same environment used for standard to enterprise conversion).
Summary page.

Once the software installation is complete (after root.sh is run) check the ownership of the oralce binaries. Since this is an environment that uses role separation the group ownership of the oracle binaries must be asmamdin. However these correct permissions were not set during the software installation. 11.2.0.3 home's oracle binaries
[oracle@rhel6m1 11.2.0]$ ls -l dbhome_2/bin/oracle*
-rwsr-s--x. 1 oracle asmadmin 232617534 Aug 15 10:35 dbhome_2/bin/oracle
-rwsr-s--x. 1 oracle asmadmin 232399041 Aug 14 15:45 dbhome_2/bin/oracleO
11.2.0.4 home's oracle binaries
[oracle@rhel6m1 11.2.0]$ ls -l dbhome_1/bin/oracle*
-rwsr-s--x. 1 oracle oinstall 239626641 Sep  9 15:08 dbhome_1/bin/oracle
-rwxr-x---. 1 oracle oinstall         0 Aug 24 11:43 dbhome_1/bin/oracleO
Therefore manual intervention is required to set the correct permissions. Run setasmgidwrap as grid user (on all the nodes)
[grid@rhel6m1 ~]$ $GI_HOME/bin/setasmgidwrap o=/opt/app/oracle/product/11.2.0/dbhome_1/bin/oracle
[oracle@rhel6m1 11.2.0]$ ls -l dbhome_1/bin/oracle*
-rwsr-s--x. 1 oracle asmadmin 239626641 Sep  9 15:08 dbhome_1/bin/oracle
If COST is setup copy the sqlnet.ora file from the 11.2.0.3 home to 11.2.0.4 home ($ORACLE_HOME/network/admin).
This conclude the database software upgrade. The next step is to upgrade the database and DBUA was used for this process.




Metalink note 1562142.1 list timezone related upgrade requirements. In this case the timezone was 14
SQL> SELECT version FROM v$timezone_file;

VERSION
-------
14
As such it doesn't require any steps to be carried out before or after the upgrade related to timezone. If the timezone version value is other than 14 then refer above metalink note.
Output generated by running utlu112i.sql is given below.
SQL> @utlu112i.sql
Oracle Database 11.2 Pre-Upgrade Information Tool 09-09-2013 15:31:51
Script Version: 11.2.0.4.0 Build: 001
.
**********************************************************************
Database:
**********************************************************************
--> name:          STD11G2
--> version:       11.2.0.3.0
--> compatible:    11.2.0.0.0
--> blocksize:     8192
--> platform:      Linux x86 64-bit
--> timezone file: V14
.
**********************************************************************
Tablespaces: [make adjustments in the current environment]
**********************************************************************
--> SYSTEM tablespace is adequate for the upgrade.
.... minimum required size: 716 MB
--> SYSAUX tablespace is adequate for the upgrade.
.... minimum required size: 673 MB
--> UNDOTBS1 tablespace is adequate for the upgrade.
.... minimum required size: 400 MB
--> TEMP tablespace is adequate for the upgrade.
.... minimum required size: 60 MB
.
**********************************************************************
Flashback: OFF
**********************************************************************
**********************************************************************
Update Parameters: [Update Oracle Database 11.2 init.ora or spfile]
Note: Pre-upgrade tool was run on a lower version 64-bit database.
**********************************************************************
--> If Target Oracle is 32-Bit, refer here for Update Parameters:
-- No update parameter changes are required.
.

--> If Target Oracle is 64-Bit, refer here for Update Parameters:
-- No update parameter changes are required.
.
**********************************************************************
Renamed Parameters: [Update Oracle Database 11.2 init.ora or spfile]
**********************************************************************
-- No renamed parameters found. No changes are required.
.
**********************************************************************
Obsolete/Deprecated Parameters: [Update Oracle Database 11.2 init.ora or spfile]
**********************************************************************
-- No obsolete parameters found. No changes are required
.

**********************************************************************
Components: [The following database components will be upgraded or installed]
**********************************************************************
--> Oracle Catalog Views         [upgrade]  VALID
--> Oracle Packages and Types    [upgrade]  VALID
--> JServer JAVA Virtual Machine [upgrade]  VALID
--> Oracle XDK for Java          [upgrade]  VALID
--> Real Application Clusters    [upgrade]  VALID
--> Oracle Workspace Manager     [upgrade]  VALID
--> Oracle XML Database          [upgrade]  VALID
--> Oracle Java Packages         [upgrade]  VALID
--> Oracle Ultra Search          [upgrade]
--> Expression Filter            [upgrade]  VALID
--> Rule Manager                 [upgrade]  VALID
.
**********************************************************************
Miscellaneous Warnings
**********************************************************************
WARNING: --> The "cluster_database" parameter is currently "TRUE"
.... and must be set to "FALSE" prior to running a manual upgrade.
WARNING: --> Database contains INVALID objects prior to upgrade.
.... The list of invalid SYS/SYSTEM objects was written to
.... registry$sys_inv_objs.
.... The list of non-SYS/SYSTEM objects was written to
.... registry$nonsys_inv_objs.
.... Use utluiobj.sql after the upgrade to identify any new invalid
.... objects due to the upgrade.
.... USER ASANGA has 4 INVALID objects.
WARNING: --> Your recycle bin is turned on and currently contains no objects.
.... Because it is REQUIRED that the recycle bin be empty prior to upgrading
.... and your recycle bin is turned on, you may need to execute the command:
.... prior to executing your upgrade to confirm the recycle bin is empty.
.
**********************************************************************
Recommendations
**********************************************************************
Oracle recommends gathering dictionary statistics prior to
upgrading the database.
To gather dictionary statistics execute the following command
while connected as SYSDBA:

    EXECUTE dbms_stats.gather_dictionary_stats;

**********************************************************************
Oracle recommends reviewing any defined events prior to upgrading.

To view existing non-default events execute the following commands
while connected AS SYSDBA:
  Events:
    SELECT (translate(value,chr(13)||chr(10),' ')) FROM sys.v$parameter2
      WHERE  UPPER(name) ='EVENT' AND  isdefault='FALSE'

  Trace Events:
    SELECT (translate(value,chr(13)||chr(10),' ')) from sys.v$parameter2
      WHERE UPPER(name) = '_TRACE_EVENTS' AND isdefault='FALSE'

Changes will need to be made in the init.ora or spfile.

**********************************************************************
There are no invalid components before the upgrade. Noteably this environment doesn't have a enterprise manager console setup. If this is setup then backup emconsole using emdwgrd.
Take a full backup of the database before running DBUA. Once backup is complete start DBUA from the new home. Verify that DBUA invoked is from the new home (PATH variable may contain old home's dbua).
The following warning was shown during the upgrade. This is the same warning given when a similar system was upgraded to 12c. As the database started as a standard edition database and then got converted to enterprise, it doesn't have the text component installed. Same as the earlier incident with 12c this was ignored and upgrade completed successfully.
Upgrade summary.

After the upgrade the remote_listener parameter values goes back to scan-ip:port, even though before the upgrade this was changed to include individual scan IP values while setting up COST. Modify the remote_listener again as per 1340831.1 if COST is used. If the sole reason for implementing cost is preventing remote registration then use Valid Node Checking for Registration (VNCR) instead of setting up COST. VNCR is available from 11.2.0.4 onwards.
Remove the default listener created during the GI upgrade.
Deinstall the 11.2.0.3 Oracle home or detach it from inventory and delete
./runInstaller -silent -detachHome ORACLE_HOME=/opt/app/oracle/product/11.2.0/dbhome_2
rm -rf dbhome_2
11.2.0.4 has introduced a new value 11 for the OPTIMIZER_DYNAMIC_SAMPLING parameter which automatically lets the optimizer decide whether dynamic statistics (use to be called sampling) are useful and which level to use. This feature is enabled by setting OPTIMIZER_DYNAMIC_SAMPLING to 11. See performance tuning guide for more.

This conclude the upgrade of RAC from 11.2.0.3 to 11.2.0.4.

Useful metalink notes
Complete Checklist for Manual Upgrades to 11gR2 [837570.1]
11.2.0.4 Patch Set - List of Bug Fixes by Problem Type [1562142.1]
Actions For DST Updates When Upgrading To Or Applying The 11.2.0.4 Patchset [1579838.1]
Complete checklist for out-of-place manual upgrade from previous 11.2.0.N version to the latest 11.2.0.N patchset. [1276368.1]
Things to Consider Before Upgrading to 11.2.0.4 to Avoid Poor Performance or Wrong Results [ID 1645862.1]
Oracle 11gR2 Upgrade Companion [ID 785351.1]

Related Posts
Upgrading from 10.2.0.4 to 10.2.0.5 (Clusterware, RAC, ASM)
Upgrade from 10.2.0.5 to 11.2.0.3 (Clusterware, RAC, ASM)
Upgrade from 11.1.0.7 to 11.2.0.3 (Clusterware, ASM & RAC)
Upgrade from 11.1.0.7 to 11.2.0.4 (Clusterware, ASM & RAC)
Upgrading from 11.1.0.7 to 11.2.0.3 with Transient Logical Standby
Upgrading from 11.2.0.1 to 11.2.0.3 with in-place upgrade for RAC
In-place upgrade from 11.2.0.2 to 11.2.0.3
Upgrading from 11.2.0.2 to 11.2.0.3 with Physical Standby - 1
Upgrading from 11.2.0.2 to 11.2.0.3 with Physical Standby - 2
Upgrading from 11gR2 (11.2.0.3) to 12c (12.1.0.1) Grid Infrastructure

Friday, September 6, 2013

Upgrading RAC from 11.2.0.3 to 11.2.0.4 - Grid Infrastructure

11.2.0.4 patchset is available under patch number 13390677. This post list the steps related to upgrading from 11.2.0.3 to 11.2.0.4. It's not a comprehensive post of "how to upgrade" and list only major steps and things to look out for during the upgrade. At the time of this post, there's not much metalink notes related to 11.2.0.4 upgrade. However step is almost same as previous posts that showed upgrading to 11.2.0.3 and some of the information in metalink notes applicable to 11.2.0.3 are still useful.
There are few new features introduced with 11.2.0.4 grid infrastructure. Some of these "features" were already available as separate software packages and now being bundled with Grid and Database software. Trace File Analyzer (1513912.1) comes as part of GI and automatically installed. During the upgrade this is installed when rootupgrade.sh is run. It will be added to upstart (or inittab) to start at boot time. With this on a 11.2.0.4 installation there will be two Oracle related processes added to init.
14907 ?        Ss     0:00 /bin/sh /etc/init.d/init.tfa run
16456 ?        Ss     0:00 /bin/sh /etc/init.d/init.ohasd run
RACcheck (1268927.1, 1457357.1) is another software that is bundled in 11.2.0.4 database software. For upgrades RACcheck is useful to check the upgrade readiness of the RAC (1457357.1) which was available from 11.2.0.3 onwards.
Oracle documentation says that installing clusterware files on block or raw devices is not supported. This was also the behavior from 11.2.0.1 as well, that block or raw devices are not supported when a fresh installation is done. But upgrading from a system that uses block or raw devices was not an issue. It's not clear why this is specifically mentioned in de-supported section of 11.2.0.4 and if this is now applicable for upgrades as well. (cannot upgrade to 12c if cluster files are in block or raw devices). However under the 11.2 section of the latest documentation it still says "Installing Oracle Clusterware files on raw or block devices is no longer supported, unless an existing system is being upgraded." Upgrading to 11.2.0.4 when cluster files are on block or raw devices will be verified later is verified here. System used for this post was the one earlier converted from standard edition to enterprise edition and had cluster files in ASM.
Refer new feature guide for complete list of features available with 11.2.0.4.
The system used is 11.2.0.3 RAC system on RHEL 6 and has listener names (scan listener and listener) and ports changed to non-default values. During that upgrade this caused issues at the end of the execution of the rootupgrade.sh script on the last node. The rootupgrade.sh seem to be looking for default listener name and without it some of the steps seem to fail. However output also shows that it is able to detect non-default listener names. It's not sure if any step actually failed or not as ultimately cluster version was upgraded to 11.2.0.4. Below is an few lines of output during rootupgrade.sh execution on the last node.
...
[main] [ 2013-09-06 12:07:20.698 BST ] [OracleHome.getVersion:1003]  ORACLE_HOME=/opt/app/11.2.0/grid4
[main] [ 2013-09-06 12:07:20.807 BST ] [OracleHome.getVersion:1008]  /opt/app/11.2.0/grid4/bin/sqlplus Banner:

SQL*Plus: Release 11.2.0.4.0 Production

[main] [ 2013-09-06 12:07:20.808 BST ] [OracleHome.getVersion:1026]  Current version from sqlplus: 11.2.0.4.0
[main] [ 2013-09-06 12:07:21.050 BST ] [NetworkUtils.checkListenerStatus:495]  Checking default listener status
[main] [ 2013-09-06 12:07:21.050 BST ] [NetworkUtils.checkListenerStatus:500]  Checking listener status: LISTENER
[main] [ 2013-09-06 12:07:21.070 BST ] [NetworkUtils.checkListenerStatus:506]  lsnrct status output

LSNRCTL for Linux: Version 11.2.0.4.0 - Production on 06-SEP-2013 12:07:21

Copyright (c) 1991, 2013, Oracle.  All rights reserved.

Connecting to (ADDRESS=(PROTOCOL=tcp)(HOST=)(PORT=1521))
TNS-12541: TNS:no listener
 TNS-12560: TNS:protocol adapter error
  TNS-00511: No listener
   Linux Error: 111: Connection refused
[main] [ 2013-09-06 12:07:21.082 BST ] [NetworkUtils.getOneLocalListenerProtocolAddress:3085]  lsnr MYSCANLISTENER_SCAN1
[main] [ 2013-09-06 12:07:21.139 BST ] [InventoryUtil.getOUIInvSession:347]  setting OUI READ level to ACCESSLEVEL_READ_LOCKLESS
...
[main] [ 2013-09-06 12:07:21.722 BST ] [OCR.isCluster:968]  Bolean result = true
[main] [ 2013-09-06 12:07:21.790 BST ] [CRSNative.isEntityRegistered:737]  entity: ora.MYSCANLISTENER_SCAN1.lsnr, type: 1, registered: true
[main] [ 2013-09-06 12:07:21.817 BST ] [CRSNative.isEntityRegistered:737]  entity: ora.MYSCANLISTENER_SCAN1.lsnr, type: 1, registered: true
[main] [ 2013-09-06 12:07:21.818 BST ] [CRSNative.getStat:1447]  About to call crs stat: name: ora.MYSCANLISTENER_SCAN1.lsnr, type: 1, node: null
[main] [ 2013-09-06 12:07:21.818 BST ] [CRSNative.getStat:1449]  looking for following attributes:
[main] [ 2013-09-06 12:07:21.819 BST ] [CRSNative.getStat:1451]                 'TYPE'
[main] [ 2013-09-06 12:07:21.848 BST ] [CRSNative.getStat:1464]  crs found 1 attributes
[main] [ 2013-09-06 12:07:21.849 BST ] [CRSNative.getStat:1474]         Name: 'TYPE'; Value: 'ora.scan_listener.type'
[main] [ 2013-09-06 12:07:21.850 BST ] [NodeAppsFactoryImpl.getListener:1140]  Resource type is ora.scan_listener.type
[main] [ 2013-09-06 12:07:21.876 BST ] [NetworkUtils.getOneLocalListenerProtocolAddress:3099]  Error: Error while getting listener address list
[main] [ 2013-09-06 12:07:21.878 BST ] [NetworkUtils.getOneLocalListenerProtocolAddress:3100]  PRCN-2066 : Failed to retrieve cluster listener for MYSCANLISTENER_SCAN1 because it has the type of Single Client Access Name Listener
...
[ 2013-09-06 12:07:22.379 BST ] [RuntimeExec.runCommand:192]  Returning from RunTimeExec.runCommand
[main] [ 2013-09-06 12:07:22.416 BST ] [CRSNative.isEntityRegistered:737]  entity: ora.MYLISTENER.lsnr, type: 1, registered: true
[main] [ 2013-09-06 12:07:22.448 BST ] [CRSNative.isEntityRegistered:737]  entity: ora.MYLISTENER.lsnr, type: 1, registered: true
[main] [ 2013-09-06 12:07:22.448 BST ] [CRSNative.getStat:1447]  About to call crs stat: name: ora.MYLISTENER.lsnr, type: 1, node: null
[main] [ 2013-09-06 12:07:22.449 BST ] [CRSNative.getStat:1449]  looking for following attributes:
[main] [ 2013-09-06 12:07:22.449 BST ] [CRSNative.getStat:1451]                 'TYPE'
[main] [ 2013-09-06 12:07:22.481 BST ] [CRSNative.getStat:1464]  crs found 1 attributes
[main] [ 2013-09-06 12:07:22.482 BST ] [CRSNative.getStat:1474]         Name: 'TYPE'; Value: 'ora.listener.type'
[main] [ 2013-09-06 12:07:22.483 BST ] [NodeAppsFactoryImpl.getListener:1140]  Resource type is ora.listener.type
[main] [ 2013-09-06 12:07:22.483 BST ] [CRSNative.getStat:1447]  About to call crs stat: name: ora.MYLISTENER.lsnr, type: 1, node: null
[main] [ 2013-09-06 12:07:22.485 BST ] [CRSNative.getStat:1449]  looking for following attributes:
[main] [ 2013-09-06 12:07:22.486 BST ] [CRSNative.getStat:1451]                 'ENDPOINTS'
[main] [ 2013-09-06 12:07:22.515 BST ] [CRSNative.getStat:1464]  crs found 1 attributes
[main] [ 2013-09-06 12:07:22.515 BST ] [CRSNative.getStat:1474]         Name: 'ENDPOINTS'; Value: 'TCP:9120'
[main] [ 2013-09-06 12:07:22.518 BST ] [EndpointsImpl.:276]  Get listener endpoints from TCP:9120
[main] [ 2013-09-06 12:07:22.518 BST ] [EndpointsImpl.:302]  Token count is 1
[main] [ 2013-09-06 12:07:22.519 BST ] [EndpointsImpl.:324]  LsnrProtocol type = TCP, values = 9120
[main] [ 2013-09-06 12:07:22.521 BST ] [EndpointsImpl$LsnrProtocol.addPortValue:86]  Calling parseInt to check 9120
[main] [ 2013-09-06 12:07:22.522 BST ] [EndpointsImpl$LsnrProtocol.addPortValue:97]  Get port number 9120
...
[main] [ 2013-09-06 12:07:22.532 BST ] [NetworkUtils.getOneLocalListenerProtocolAddress:3116]  addr (ADDRESS=(PROTOCOL=TCP)(HOST=rhel6m2.domain.net)(PORT=9120))
[main] [ 2013-09-06 12:07:22.534 BST ] [NetworkUtils.getLocalListenerAddresses:1310]  listener[1]=MYLISTENER address=(ADDRESS=(PROTOCOL=TCP)(HOST=rhel6m2.domain.net)(PORT=9120))
...
[main] [ 2013-09-06 12:07:30.536 BST ] [CRSNativeResult.addComp:162]  add comp: name ora.LISTENER.lsnr, rc 210, msg CRS-0210: Could not find resource 'ora.LISTENER.lsnr'.
[main] [ 2013-09-06 12:07:30.539 BST ] [CRSNative.isEntityRegistered:737]  entity: ora.LISTENER.lsnr, type: 1, registered: false
ASM upgrade has finished on last node.
null
Configuring ASM instance in EM failed. The following error occured:
PRCR-1001 : Resource ora.LISTENER.lsnr does not exist
Configure Oracle Grid Infrastructure for a Cluster ... succeeded
To be on the safe side if non-default listener name/port is used it's best to create a default listener and start it before the upgrade and remove once the upgrade is completed. Same problem was there upgrading to 12c using DBUA
[grid@rhel6m1 admin]$ srvctl add listener -listener listener -p 1521
[grid@rhel6m1 admin]$ srvctl start listener -l listener
[grid@rhel6m1 admin]$ lsnrctl status
Running RACcheck post upgrade check (run as oracle user)
./raccheck -u -o post
on the existing 11.2.0.3 system gives the following warning on the resulting html file."The default setting for memory_target in 11.2.0.4 and 12.1 versions has been increased to 1024M from the previous default of 256M. Oracle also advises increasing memory_target to 1024M as a minimum in version 11.2 as a proactive measure to match the new defaults. No patch is required in order to increase the value of memory_target." This fact is also noted in Bug 14292825 - ORA-4031 in ASM as default memory parameters values for 11.2 ASM instances low (Doc ID 14292825.8). So it's best to increase the memory target for the ASM instances before the upgrade. (update 2015/12/03. Seems there's further changes to minimum amount, doc 1363369.1 suggest minimum value of 1536M.)
SQL> show parameter memory

NAME                                 TYPE        VALUE
------------------------------------ ----------- ------------------------------
memory_max_target                    big integer 272M
memory_target                        big integer 272M
SQL> alter system set memory_max_target=1024M scope=spfile sid='*';
System altered.

SQL> alter system set memory_target=1024M scope=spfile sid='*';
System altered.
Restart cluster stack on each node
crsctl stop cluster -n rhel6m1
crsctl start cluster -n rhel6m1

SQL> show parameter memory
NAME                                 TYPE        VALUE
------------------------------------ ----------- ------------------------------
memory_max_target                    big integer 1G
memory_target                        big integer 1G
Use both RACCheck pre upgrade check
./raccheck -u -o pre
and cluvfy to check the upgrade readiness to 11.2.0.4. Cluvfy output is shown below. Since the system is RHEL 6 the pdksh issue is there and cluvfy fails. Manually verify that all other checks have passed except for pdksh package check.
[grid@rhel6m1 grid]$ ./runcluvfy.sh stage -pre crsinst -upgrade -n rhel6m1,rhel6m2 -rolling -src_crshome /opt/app/11.2.0/grid -dest_crshome /opt/app/11.2.0/grid4 -dest_version 11.2.0.4.0 -fixup -fixupdir /home/grid/fixupscript -verbose

Performing pre-checks for cluster services setup

Checking node reachability...

Check: Node reachability from node "rhel6m1"
  Destination Node                      Reachable?
  ------------------------------------  ------------------------
  rhel6m1                               yes
  rhel6m2                               yes
Result: Node reachability check passed from node "rhel6m1"


Checking user equivalence...

Check: User equivalence for user "grid"
  Node Name                             Status
  ------------------------------------  ------------------------
  rhel6m2                               passed
  rhel6m1                               passed
Result: User equivalence check passed for user "grid"

Checking CRS user consistency
Result: CRS user consistency check successful

Checking node connectivity...

Checking hosts config file...
  Node Name                             Status
  ------------------------------------  ------------------------
  rhel6m2                               passed
  rhel6m1                               passed

Verification of the hosts config file successful


Interface information for node "rhel6m2"
 Name   IP Address      Subnet          Gateway         Def. Gateway    HW Address        MTU
 ------ --------------- --------------- --------------- --------------- ----------------- ------
 eth0   192.168.0.86    192.168.0.0     0.0.0.0         192.168.0.100   08:00:27:6A:D4:18 1500
 eth0   192.168.0.90    192.168.0.0     0.0.0.0         192.168.0.100   08:00:27:6A:D4:18 1500
 eth0   192.168.0.91    192.168.0.0     0.0.0.0         192.168.0.100   08:00:27:6A:D4:18 1500
 eth1   192.168.1.88    192.168.1.0     0.0.0.0         192.168.0.100   08:00:27:D4:AC:BE 1500
 eth1   169.254.160.199 169.254.0.0     0.0.0.0         192.168.0.100   08:00:27:D4:AC:BE 1500


Interface information for node "rhel6m1"
 Name   IP Address      Subnet          Gateway         Def. Gateway    HW Address        MTU
 ------ --------------- --------------- --------------- --------------- ----------------- ------
 eth0   192.168.0.85    192.168.0.0     0.0.0.0         192.168.0.100   08:00:27:44:1B:94 1500
 eth0   192.168.0.89    192.168.0.0     0.0.0.0         192.168.0.100   08:00:27:44:1B:94 1500
 eth1   192.168.1.87    192.168.1.0     0.0.0.0         192.168.0.100   08:00:27:E6:AB:27 1500
 eth1   169.254.31.63   169.254.0.0     0.0.0.0         192.168.0.100   08:00:27:E6:AB:27 1500


Check: Node connectivity for interface "eth0"
  Source                          Destination                     Connected?
  ------------------------------  ------------------------------  ----------------
  rhel6m2[192.168.0.86]           rhel6m2[192.168.0.90]           yes
  rhel6m2[192.168.0.86]           rhel6m2[192.168.0.91]           yes
  rhel6m2[192.168.0.86]           rhel6m1[192.168.0.85]           yes
  rhel6m2[192.168.0.86]           rhel6m1[192.168.0.89]           yes
  rhel6m2[192.168.0.90]           rhel6m2[192.168.0.91]           yes
  rhel6m2[192.168.0.90]           rhel6m1[192.168.0.85]           yes
  rhel6m2[192.168.0.90]           rhel6m1[192.168.0.89]           yes
  rhel6m2[192.168.0.91]           rhel6m1[192.168.0.85]           yes
  rhel6m2[192.168.0.91]           rhel6m1[192.168.0.89]           yes
  rhel6m1[192.168.0.85]           rhel6m1[192.168.0.89]           yes
Result: Node connectivity passed for interface "eth0"


Check: TCP connectivity of subnet "192.168.0.0"
  Source                          Destination                     Connected?
  ------------------------------  ------------------------------  ----------------
  rhel6m1:192.168.0.85            rhel6m2:192.168.0.86            passed
  rhel6m1:192.168.0.85            rhel6m2:192.168.0.90            passed
  rhel6m1:192.168.0.85            rhel6m2:192.168.0.91            passed
  rhel6m1:192.168.0.85            rhel6m1:192.168.0.89            passed
Result: TCP connectivity check passed for subnet "192.168.0.0"


Check: Node connectivity for interface "eth1"
  Source                          Destination                     Connected?
  ------------------------------  ------------------------------  ----------------
  rhel6m2[192.168.1.88]           rhel6m1[192.168.1.87]           yes
Result: Node connectivity passed for interface "eth1"


Check: TCP connectivity of subnet "192.168.1.0"
  Source                          Destination                     Connected?
  ------------------------------  ------------------------------  ----------------
  rhel6m1:192.168.1.87            rhel6m2:192.168.1.88            passed
Result: TCP connectivity check passed for subnet "192.168.1.0"

Checking subnet mask consistency...
Subnet mask consistency check passed for subnet "192.168.0.0".
Subnet mask consistency check passed for subnet "192.168.1.0".
Subnet mask consistency check passed.

Result: Node connectivity check passed

Checking multicast communication...

Checking subnet "192.168.0.0" for multicast communication with multicast group "230.0.1.0"...
Check of subnet "192.168.0.0" for multicast communication with multicast group "230.0.1.0" passed.

Checking subnet "192.168.1.0" for multicast communication with multicast group "230.0.1.0"...
Check of subnet "192.168.1.0" for multicast communication with multicast group "230.0.1.0" passed.

Check of multicast communication passed.

Checking OCR integrity...

OCR integrity check passed

Checking ASMLib configuration.
  Node Name                             Status
  ------------------------------------  ------------------------
  rhel6m2                               passed
  rhel6m1                               passed
Result: Check for ASMLib configuration passed.

Check: Total memory
  Node Name     Available                 Required                  Status
  ------------  ------------------------  ------------------------  ----------
  rhel6m2       3.7414GB (3923160.0KB)    1.5GB (1572864.0KB)       passed
  rhel6m1       3.7414GB (3923160.0KB)    1.5GB (1572864.0KB)       passed
Result: Total memory check passed

Check: Available memory
  Node Name     Available                 Required                  Status
  ------------  ------------------------  ------------------------  ----------
  rhel6m2       2.2198GB (2327612.0KB)    50MB (51200.0KB)          passed
  rhel6m1       2.2859GB (2396920.0KB)    50MB (51200.0KB)          passed
Result: Available memory check passed

Check: Swap space
  Node Name     Available                 Required                  Status
  ------------  ------------------------  ------------------------  ----------
  rhel6m2       4GB (4194296.0KB)         3.7414GB (3923160.0KB)    passed
  rhel6m1       4GB (4194296.0KB)         3.7414GB (3923160.0KB)    passed
Result: Swap space check passed

Check: Free disk space for "rhel6m2:/opt/app/11.2.0/grid4,rhel6m2:/tmp"
  Path              Node Name     Mount point   Available     Required      Status
  ----------------  ------------  ------------  ------------  ------------  ------------
  /opt/app/11.2.0/grid4  rhel6m2       /             13.3555GB     7.5GB         passed
  /tmp              rhel6m2       /             13.3555GB     7.5GB         passed
Result: Free disk space check passed for "rhel6m2:/opt/app/11.2.0/grid4,rhel6m2:/tmp"

Check: Free disk space for "rhel6m1:/opt/app/11.2.0/grid4,rhel6m1:/tmp"
  Path              Node Name     Mount point   Available     Required      Status
  ----------------  ------------  ------------  ------------  ------------  ------------
  /opt/app/11.2.0/grid4  rhel6m1       /             11.6013GB     7.5GB         passed
  /tmp              rhel6m1       /             11.6013GB     7.5GB         passed
Result: Free disk space check passed for "rhel6m1:/opt/app/11.2.0/grid4,rhel6m1:/tmp"

Check: User existence for "grid"
  Node Name     Status                    Comment
  ------------  ------------------------  ------------------------
  rhel6m2       passed                    exists(502)
  rhel6m1       passed                    exists(502)

Checking for multiple users with UID value 502
Result: Check for multiple users with UID value 502 passed
Result: User existence check passed for "grid"

Check: Group existence for "oinstall"
  Node Name     Status                    Comment
  ------------  ------------------------  ------------------------
  rhel6m2       passed                    exists
  rhel6m1       passed                    exists
Result: Group existence check passed for "oinstall"

Check: Membership of user "grid" in group "oinstall" [as Primary]
  Node Name         User Exists   Group Exists  User in Group  Primary       Status
  ----------------  ------------  ------------  ------------  ------------  ------------
  rhel6m2           yes           yes           yes           yes           passed
  rhel6m1           yes           yes           yes           yes           passed
Result: Membership check for user "grid" in group "oinstall" [as Primary] passed

Check: Run level
  Node Name     run level                 Required                  Status
  ------------  ------------------------  ------------------------  ----------
  rhel6m2       3                         3,5                       passed
  rhel6m1       3                         3,5                       passed
Result: Run level check passed

Check: Hard limits for "maximum open file descriptors"
  Node Name         Type          Available     Required      Status
  ----------------  ------------  ------------  ------------  ----------------
  rhel6m2           hard          65536         65536         passed
  rhel6m1           hard          65536         65536         passed
Result: Hard limits check passed for "maximum open file descriptors"

Check: Soft limits for "maximum open file descriptors"
  Node Name         Type          Available     Required      Status
  ----------------  ------------  ------------  ------------  ----------------
  rhel6m2           soft          1024          1024          passed
  rhel6m1           soft          1024          1024          passed
Result: Soft limits check passed for "maximum open file descriptors"

Check: Hard limits for "maximum user processes"
  Node Name         Type          Available     Required      Status
  ----------------  ------------  ------------  ------------  ----------------
  rhel6m2           hard          16384         16384         passed
  rhel6m1           hard          16384         16384         passed
Result: Hard limits check passed for "maximum user processes"

Check: Soft limits for "maximum user processes"
  Node Name         Type          Available     Required      Status
  ----------------  ------------  ------------  ------------  ----------------
  rhel6m2           soft          2047          2047          passed
  rhel6m1           soft          2047          2047          passed
Result: Soft limits check passed for "maximum user processes"

There are no oracle patches required for home "/opt/app/11.2.0/grid".

There are no oracle patches required for home "/opt/app/11.2.0/grid4".

Check: System architecture
  Node Name     Available                 Required                  Status
  ------------  ------------------------  ------------------------  ----------
  rhel6m2       x86_64                    x86_64                    passed
  rhel6m1       x86_64                    x86_64                    passed
Result: System architecture check passed

Check: Kernel version
  Node Name     Available                 Required                  Status
  ------------  ------------------------  ------------------------  ----------
  rhel6m2       2.6.32-220.el6.x86_64     2.6.9                     passed
  rhel6m1       2.6.32-220.el6.x86_64     2.6.9                     passed
Result: Kernel version check passed

Check: Kernel parameter for "semmsl"
  Node Name         Current       Configured    Required      Status        Comment
  ----------------  ------------  ------------  ------------  ------------  ------------
  rhel6m2           3010          3010          250           passed
  rhel6m1           3010          3010          250           passed
Result: Kernel parameter check passed for "semmsl"

Check: Kernel parameter for "semmns"
  Node Name         Current       Configured    Required      Status        Comment
  ----------------  ------------  ------------  ------------  ------------  ------------
  rhel6m2           385280        385280        32000         passed
  rhel6m1           385280        385280        32000         passed
Result: Kernel parameter check passed for "semmns"

Check: Kernel parameter for "semopm"
  Node Name         Current       Configured    Required      Status        Comment
  ----------------  ------------  ------------  ------------  ------------  ------------
  rhel6m2           3010          3010          100           passed
  rhel6m1           3010          3010          100           passed
Result: Kernel parameter check passed for "semopm"

Check: Kernel parameter for "semmni"
  Node Name         Current       Configured    Required      Status        Comment
  ----------------  ------------  ------------  ------------  ------------  ------------
  rhel6m2           128           128           128           passed
  rhel6m1           128           128           128           passed
Result: Kernel parameter check passed for "semmni"

Check: Kernel parameter for "shmmax"
  Node Name         Current       Configured    Required      Status        Comment
  ----------------  ------------  ------------  ------------  ------------  ------------
  rhel6m2           68719476736   68719476736   2008657920    passed
  rhel6m1           68719476736   68719476736   2008657920    passed
Result: Kernel parameter check passed for "shmmax"

Check: Kernel parameter for "shmmni"
  Node Name         Current       Configured    Required      Status        Comment
  ----------------  ------------  ------------  ------------  ------------  ------------
  rhel6m2           4096          4096          4096          passed
  rhel6m1           4096          4096          4096          passed
Result: Kernel parameter check passed for "shmmni"

Check: Kernel parameter for "shmall"
  Node Name         Current       Configured    Required      Status        Comment
  ----------------  ------------  ------------  ------------  ------------  ------------
  rhel6m2           4294967296    4294967296    2097152       passed
  rhel6m1           4294967296    4294967296    2097152       passed
Result: Kernel parameter check passed for "shmall"

Check: Kernel parameter for "file-max"
  Node Name         Current       Configured    Required      Status        Comment
  ----------------  ------------  ------------  ------------  ------------  ------------
  rhel6m2           6815744       6815744       6815744       passed
  rhel6m1           6815744       6815744       6815744       passed
Result: Kernel parameter check passed for "file-max"

Check: Kernel parameter for "ip_local_port_range"
  Node Name         Current       Configured    Required      Status        Comment
  ----------------  ------------  ------------  ------------  ------------  ------------
  rhel6m2           between 9000.0 & 65500.0  between 9000.0 & 65500.0  between 9000.0 & 65500.0  passed
  rhel6m1           between 9000.0 & 65500.0  between 9000.0 & 65500.0  between 9000.0 & 65500.0  passed
Result: Kernel parameter check passed for "ip_local_port_range"

Check: Kernel parameter for "rmem_default"
  Node Name         Current       Configured    Required      Status        Comment
  ----------------  ------------  ------------  ------------  ------------  ------------
  rhel6m2           4194304       4194304       262144        passed
  rhel6m1           4194304       4194304       262144        passed
Result: Kernel parameter check passed for "rmem_default"

Check: Kernel parameter for "rmem_max"
  Node Name         Current       Configured    Required      Status        Comment
  ----------------  ------------  ------------  ------------  ------------  ------------
  rhel6m2           4194304       4194304       4194304       passed
  rhel6m1           4194304       4194304       4194304       passed
Result: Kernel parameter check passed for "rmem_max"

Check: Kernel parameter for "wmem_default"
  Node Name         Current       Configured    Required      Status        Comment
  ----------------  ------------  ------------  ------------  ------------  ------------
  rhel6m2           1048576       1048576       262144        passed
  rhel6m1           1048576       1048576       262144        passed
Result: Kernel parameter check passed for "wmem_default"

Check: Kernel parameter for "wmem_max"
  Node Name         Current       Configured    Required      Status        Comment
  ----------------  ------------  ------------  ------------  ------------  ------------
  rhel6m2           2097152       2097152       1048576       passed
  rhel6m1           2097152       2097152       1048576       passed
Result: Kernel parameter check passed for "wmem_max"

Check: Kernel parameter for "aio-max-nr"
  Node Name         Current       Configured    Required      Status        Comment
  ----------------  ------------  ------------  ------------  ------------  ------------
  rhel6m2           3145728       3145728       1048576       passed
  rhel6m1           3145728       3145728       1048576       passed
Result: Kernel parameter check passed for "aio-max-nr"

Check: Package existence for "make"
  Node Name     Available                 Required                  Status
  ------------  ------------------------  ------------------------  ----------
  rhel6m2       make-3.81-19.el6          make-3.80                 passed
  rhel6m1       make-3.81-19.el6          make-3.80                 passed
Result: Package existence check passed for "make"

Check: Package existence for "binutils"
  Node Name     Available                 Required                  Status
  ------------  ------------------------  ------------------------  ----------
  rhel6m2       binutils-2.20.51.0.2-5.28.el6  binutils-2.15.92.0.2      passed
  rhel6m1       binutils-2.20.51.0.2-5.28.el6  binutils-2.15.92.0.2      passed
Result: Package existence check passed for "binutils"

Check: Package existence for "gcc(x86_64)"
  Node Name     Available                 Required                  Status
  ------------  ------------------------  ------------------------  ----------
  rhel6m2       gcc(x86_64)-4.4.6-3.el6   gcc(x86_64)-3.4.6         passed
  rhel6m1       gcc(x86_64)-4.4.6-3.el6   gcc(x86_64)-3.4.6         passed
Result: Package existence check passed for "gcc(x86_64)"

Check: Package existence for "libaio(x86_64)"
  Node Name     Available                 Required                  Status
  ------------  ------------------------  ------------------------  ----------
  rhel6m2       libaio(x86_64)-0.3.107-10.el6  libaio(x86_64)-0.3.105    passed
  rhel6m1       libaio(x86_64)-0.3.107-10.el6  libaio(x86_64)-0.3.105    passed
Result: Package existence check passed for "libaio(x86_64)"

Check: Package existence for "glibc(x86_64)"
  Node Name     Available                 Required                  Status
  ------------  ------------------------  ------------------------  ----------
  rhel6m2       glibc(x86_64)-2.12-1.47.el6  glibc(x86_64)-2.3.4-2.41  passed
  rhel6m1       glibc(x86_64)-2.12-1.47.el6  glibc(x86_64)-2.3.4-2.41  passed
Result: Package existence check passed for "glibc(x86_64)"

Check: Package existence for "compat-libstdc++-33(x86_64)"
  Node Name     Available                 Required                  Status
  ------------  ------------------------  ------------------------  ----------
  rhel6m2       compat-libstdc++-33(x86_64)-3.2.3-69.el6  compat-libstdc++-33(x86_64)-3.2.3  passed
  rhel6m1       compat-libstdc++-33(x86_64)-3.2.3-69.el6  compat-libstdc++-33(x86_64)-3.2.3  passed
Result: Package existence check passed for "compat-libstdc++-33(x86_64)"

Check: Package existence for "elfutils-libelf(x86_64)"
  Node Name     Available                 Required                  Status
  ------------  ------------------------  ------------------------  ----------
  rhel6m2       elfutils-libelf(x86_64)-0.152-1.el6  elfutils-libelf(x86_64)-0.97  passed
  rhel6m1       elfutils-libelf(x86_64)-0.152-1.el6  elfutils-libelf(x86_64)-0.97  passed
Result: Package existence check passed for "elfutils-libelf(x86_64)"

Check: Package existence for "elfutils-libelf-devel"
  Node Name     Available                 Required                  Status
  ------------  ------------------------  ------------------------  ----------
  rhel6m2       elfutils-libelf-devel-0.152-1.el6  elfutils-libelf-devel-0.97  passed
  rhel6m1       elfutils-libelf-devel-0.152-1.el6  elfutils-libelf-devel-0.97  passed
Result: Package existence check passed for "elfutils-libelf-devel"

Check: Package existence for "glibc-common"
  Node Name     Available                 Required                  Status
  ------------  ------------------------  ------------------------  ----------
  rhel6m2       glibc-common-2.12-1.47.el6  glibc-common-2.3.4        passed
  rhel6m1       glibc-common-2.12-1.47.el6  glibc-common-2.3.4        passed
Result: Package existence check passed for "glibc-common"

Check: Package existence for "glibc-devel(x86_64)"
  Node Name     Available                 Required                  Status
  ------------  ------------------------  ------------------------  ----------
  rhel6m2       glibc-devel(x86_64)-2.12-1.47.el6  glibc-devel(x86_64)-2.3.4  passed
  rhel6m1       glibc-devel(x86_64)-2.12-1.47.el6  glibc-devel(x86_64)-2.3.4  passed
Result: Package existence check passed for "glibc-devel(x86_64)"

Check: Package existence for "glibc-headers"
  Node Name     Available                 Required                  Status
  ------------  ------------------------  ------------------------  ----------
  rhel6m2       glibc-headers-2.12-1.47.el6  glibc-headers-2.3.4       passed
  rhel6m1       glibc-headers-2.12-1.47.el6  glibc-headers-2.3.4       passed
Result: Package existence check passed for "glibc-headers"

Check: Package existence for "gcc-c++(x86_64)"
  Node Name     Available                 Required                  Status
  ------------  ------------------------  ------------------------  ----------
  rhel6m2       gcc-c++(x86_64)-4.4.6-3.el6  gcc-c++(x86_64)-3.4.6     passed
  rhel6m1       gcc-c++(x86_64)-4.4.6-3.el6  gcc-c++(x86_64)-3.4.6     passed
Result: Package existence check passed for "gcc-c++(x86_64)"

Check: Package existence for "libaio-devel(x86_64)"
  Node Name     Available                 Required                  Status
  ------------  ------------------------  ------------------------  ----------
  rhel6m2       libaio-devel(x86_64)-0.3.107-10.el6  libaio-devel(x86_64)-0.3.105  passed
  rhel6m1       libaio-devel(x86_64)-0.3.107-10.el6  libaio-devel(x86_64)-0.3.105  passed
Result: Package existence check passed for "libaio-devel(x86_64)"

Check: Package existence for "libgcc(x86_64)"
  Node Name     Available                 Required                  Status
  ------------  ------------------------  ------------------------  ----------
  rhel6m2       libgcc(x86_64)-4.4.6-3.el6  libgcc(x86_64)-3.4.6      passed
  rhel6m1       libgcc(x86_64)-4.4.6-3.el6  libgcc(x86_64)-3.4.6      passed
Result: Package existence check passed for "libgcc(x86_64)"

Check: Package existence for "libstdc++(x86_64)"
  Node Name     Available                 Required                  Status
  ------------  ------------------------  ------------------------  ----------
  rhel6m2       libstdc++(x86_64)-4.4.6-3.el6  libstdc++(x86_64)-3.4.6   passed
  rhel6m1       libstdc++(x86_64)-4.4.6-3.el6  libstdc++(x86_64)-3.4.6   passed
Result: Package existence check passed for "libstdc++(x86_64)"

Check: Package existence for "libstdc++-devel(x86_64)"
  Node Name     Available                 Required                  Status
  ------------  ------------------------  ------------------------  ----------
  rhel6m2       libstdc++-devel(x86_64)-4.4.6-3.el6  libstdc++-devel(x86_64)-3.4.6  passed
  rhel6m1       libstdc++-devel(x86_64)-4.4.6-3.el6  libstdc++-devel(x86_64)-3.4.6  passed
Result: Package existence check passed for "libstdc++-devel(x86_64)"

Check: Package existence for "sysstat"
  Node Name     Available                 Required                  Status
  ------------  ------------------------  ------------------------  ----------
  rhel6m2       sysstat-9.0.4-18.el6      sysstat-5.0.5             passed
  rhel6m1       sysstat-9.0.4-18.el6      sysstat-5.0.5             passed
Result: Package existence check passed for "sysstat"

Check: Package existence for "pdksh"
  Node Name     Available                 Required                  Status
  ------------  ------------------------  ------------------------  ----------
  rhel6m2       missing                   pdksh-5.2.14              failed
  rhel6m1       missing                   pdksh-5.2.14              failed
Result: Package existence check failed for "pdksh"

Check: Package existence for "expat(x86_64)"
  Node Name     Available                 Required                  Status
  ------------  ------------------------  ------------------------  ----------
  rhel6m2       expat(x86_64)-2.0.1-9.1.el6  expat(x86_64)-1.95.7      passed
  rhel6m1       expat(x86_64)-2.0.1-9.1.el6  expat(x86_64)-1.95.7      passed
Result: Package existence check passed for "expat(x86_64)"

Checking for multiple users with UID value 0
Result: Check for multiple users with UID value 0 passed

Check: Current group ID
Result: Current group ID check passed

Starting check for consistency of primary group of root user
  Node Name                             Status
  ------------------------------------  ------------------------
  rhel6m2                               passed
  rhel6m1                               passed

Check for consistency of root user's primary group passed

Check: Package existence for "cvuqdisk"
  Node Name     Available                 Required                  Status
  ------------  ------------------------  ------------------------  ----------
  rhel6m2       cvuqdisk-1.0.9-1          cvuqdisk-1.0.9-1          passed
  rhel6m1       cvuqdisk-1.0.9-1          cvuqdisk-1.0.9-1          passed
Result: Package existence check passed for "cvuqdisk"

Starting Clock synchronization checks using Network Time Protocol(NTP)...

NTP Configuration file check started...
Network Time Protocol(NTP) configuration file not found on any of the nodes. Oracle Cluster Time Synchronization Service(CTSS) can be used instead of NTP for time synchronization on the cluster nodes
No NTP Daemons or Services were found to be running

Result: Clock synchronization check using Network Time Protocol(NTP) passed

Checking Core file name pattern consistency...
Core file name pattern consistency check passed.

Checking to make sure user "grid" is not in "root" group
  Node Name     Status                    Comment
  ------------  ------------------------  ------------------------
  rhel6m2       passed                    does not exist
  rhel6m1       passed                    does not exist
Result: User "grid" is not part of "root" group. Check passed

Check default user file creation mask
  Node Name     Available                 Required                  Comment
  ------------  ------------------------  ------------------------  ----------
  rhel6m2       0022                      0022                      passed
  rhel6m1       0022                      0022                      passed
Result: Default user file creation mask check passed
Checking consistency of file "/etc/resolv.conf" across nodes

Checking the file "/etc/resolv.conf" to make sure only one of domain and search entries is defined
File "/etc/resolv.conf" does not have both domain and search entries defined
Checking if domain entry in file "/etc/resolv.conf" is consistent across the nodes...
domain entry in file "/etc/resolv.conf" is consistent across nodes
Checking if search entry in file "/etc/resolv.conf" is consistent across the nodes...
search entry in file "/etc/resolv.conf" is consistent across nodes
Checking file "/etc/resolv.conf" to make sure that only one search entry is defined
All nodes have one search entry defined in file "/etc/resolv.conf"
Checking all nodes to make sure that search entry is "domain.net" as found on node "rhel6m2"
All nodes of the cluster have same value for 'search'
Checking DNS response time for an unreachable node
  Node Name                             Status
  ------------------------------------  ------------------------
  rhel6m2                               passed
  rhel6m1                               passed
The DNS response time for an unreachable node is within acceptable limit on all nodes

File "/etc/resolv.conf" is consistent across nodes


UDev attributes check for OCR locations started...
Result: UDev attributes check passed for OCR locations


UDev attributes check for Voting Disk locations started...
Result: UDev attributes check passed for Voting Disk locations

Check: Time zone consistency
Result: Time zone consistency check passed
Checking VIP configuration.
Checking VIP Subnet configuration.
Check for VIP Subnet configuration passed.
Checking VIP reachability
Check for VIP reachability passed.

Checking Oracle Cluster Voting Disk configuration...

ASM Running check passed. ASM is running on all specified nodes

Oracle Cluster Voting Disk configuration check passed

Clusterware version consistency passed

Pre-check for cluster services setup was unsuccessful on all the nodes.
Manually create the new out of place directory to be used for the 11.2.0.4 grid installation. Since the parent directories are owned by root the grid user installation process will not be able to create the new directory.
Create the directory on all the nodes of the cluster
# cd /opt/app/11.2.0
# mkdir grid4
# chown grid:oinstall grid4
# chmod 775 grid4



Start the GI upgrade by executing runInstaller.

When prompted execute the rootupgrade.sh script on each node.
It was also observed that COST related files were copied to the new GI HOME during the upgrade and doesn't require any manual intervention. However once the upgrade is completed and before the old GI home is removed change the wallet location in the sqlnet.ora file inside $ORACLE_HOME/network/admin. Upgrade is a rolling upgrade and script will bring down cluster stack on the node and once upgrade is finish bring it back up. Other nodes are open for use during this time.
[root@rhel6m1 grid]# /opt/app/11.2.0/grid4/rootupgrade.sh
Performing root user operation for Oracle 11g

The following environment variables are set as:
    ORACLE_OWNER= grid
    ORACLE_HOME=  /opt/app/11.2.0/grid4

Enter the full pathname of the local bin directory: [/usr/local/bin]:
The contents of "dbhome" have not changed. No need to overwrite.
The contents of "oraenv" have not changed. No need to overwrite.
The contents of "coraenv" have not changed. No need to overwrite.

Entries will be added to the /etc/oratab file as needed by
Database Configuration Assistant when a database is created
Finished running generic part of root script.
Now product-specific root actions will be performed.
Using configuration parameter file: /opt/app/11.2.0/grid4/crs/install/crsconfig_params
Creating trace directory
Installing Trace File Analyzer

ASM upgrade has started on first node.

CRS-2791: Starting shutdown of Oracle High Availability Services-managed resources on 'rhel6m1'
CRS-2673: Attempting to stop 'ora.crsd' on 'rhel6m1'
CRS-2790: Starting shutdown of Cluster Ready Services-managed resources on 'rhel6m1'
CRS-2673: Attempting to stop 'ora.CLUSTER_DG.dg' on 'rhel6m1'
CRS-2673: Attempting to stop 'ora.registry.acfs' on 'rhel6m1'
CRS-2673: Attempting to stop 'ora.std11g2.asa.domain.net.svc' on 'rhel6m1'
CRS-2677: Stop of 'ora.std11g2.asa.domain.net.svc' on 'rhel6m1' succeeded
CRS-2673: Attempting to stop 'ora.std11g2.db' on 'rhel6m1'
CRS-2673: Attempting to stop 'ora.LISTENER.lsnr' on 'rhel6m1'
CRS-2673: Attempting to stop 'ora.MYLISTENER.lsnr' on 'rhel6m1'
CRS-2677: Stop of 'ora.LISTENER.lsnr' on 'rhel6m1' succeeded
CRS-2677: Stop of 'ora.MYLISTENER.lsnr' on 'rhel6m1' succeeded
CRS-2673: Attempting to stop 'ora.rhel6m1.vip' on 'rhel6m1'
CRS-2677: Stop of 'ora.registry.acfs' on 'rhel6m1' succeeded
CRS-2677: Stop of 'ora.std11g2.db' on 'rhel6m1' succeeded
CRS-2673: Attempting to stop 'ora.DATA.dg' on 'rhel6m1'
CRS-2673: Attempting to stop 'ora.FLASH.dg' on 'rhel6m1'
CRS-2677: Stop of 'ora.rhel6m1.vip' on 'rhel6m1' succeeded
CRS-2672: Attempting to start 'ora.rhel6m1.vip' on 'rhel6m2'
CRS-2677: Stop of 'ora.DATA.dg' on 'rhel6m1' succeeded
CRS-2677: Stop of 'ora.FLASH.dg' on 'rhel6m1' succeeded
CRS-2676: Start of 'ora.rhel6m1.vip' on 'rhel6m2' succeeded
CRS-2677: Stop of 'ora.CLUSTER_DG.dg' on 'rhel6m1' succeeded
CRS-2673: Attempting to stop 'ora.asm' on 'rhel6m1'
CRS-2677: Stop of 'ora.asm' on 'rhel6m1' succeeded
CRS-2673: Attempting to stop 'ora.ons' on 'rhel6m1'
CRS-2677: Stop of 'ora.ons' on 'rhel6m1' succeeded
CRS-2673: Attempting to stop 'ora.net1.network' on 'rhel6m1'
CRS-2677: Stop of 'ora.net1.network' on 'rhel6m1' succeeded
CRS-2792: Shutdown of Cluster Ready Services-managed resources on 'rhel6m1' has completed
CRS-2677: Stop of 'ora.crsd' on 'rhel6m1' succeeded
CRS-2673: Attempting to stop 'ora.crf' on 'rhel6m1'
CRS-2673: Attempting to stop 'ora.ctssd' on 'rhel6m1'
CRS-2673: Attempting to stop 'ora.evmd' on 'rhel6m1'
CRS-2673: Attempting to stop 'ora.asm' on 'rhel6m1'
CRS-2673: Attempting to stop 'ora.mdnsd' on 'rhel6m1'
CRS-2673: Attempting to stop 'ora.drivers.acfs' on 'rhel6m1'
CRS-2677: Stop of 'ora.crf' on 'rhel6m1' succeeded
CRS-2677: Stop of 'ora.evmd' on 'rhel6m1' succeeded
CRS-2677: Stop of 'ora.mdnsd' on 'rhel6m1' succeeded
CRS-2677: Stop of 'ora.ctssd' on 'rhel6m1' succeeded
CRS-2677: Stop of 'ora.asm' on 'rhel6m1' succeeded
CRS-2673: Attempting to stop 'ora.cluster_interconnect.haip' on 'rhel6m1'
CRS-2677: Stop of 'ora.cluster_interconnect.haip' on 'rhel6m1' succeeded
CRS-2673: Attempting to stop 'ora.cssd' on 'rhel6m1'
CRS-2677: Stop of 'ora.drivers.acfs' on 'rhel6m1' succeeded
CRS-2677: Stop of 'ora.cssd' on 'rhel6m1' succeeded
CRS-2673: Attempting to stop 'ora.gipcd' on 'rhel6m1'
CRS-2677: Stop of 'ora.gipcd' on 'rhel6m1' succeeded
CRS-2673: Attempting to stop 'ora.gpnpd' on 'rhel6m1'
CRS-2677: Stop of 'ora.gpnpd' on 'rhel6m1' succeeded
CRS-2793: Shutdown of Oracle High Availability Services-managed resources on 'rhel6m1' has completed
CRS-4133: Oracle High Availability Services has been stopped.
OLR initialization - successful
Replacing Clusterware entries in upstart
clscfg: EXISTING configuration version 5 detected.
clscfg: version 5 is 11g Release 2.
Successfully accumulated necessary OCR keys.
Creating OCR keys for user 'root', privgrp 'root'..
Operation successful.
Configure Oracle Grid Infrastructure for a Cluster ... succeeded
Active version still remain the lower version until all the nodes are upgraded.
[grid@rhel6m1 ~]$ crsctl query crs releaseversion
Oracle High Availability Services release version on the local node is [11.2.0.4.0]
[grid@rhel6m1 ~]$ crsctl query crs softwareversion
Oracle Clusterware version on node [rhel6m1] is [11.2.0.4.0]
[grid@rhel6m1 ~]$ crsctl query crs activeversion
Oracle Clusterware active version on the cluster is [11.2.0.3.0]
At this time /etc/oratab will also be upgraded to reflect the new GI_HOME
[grid@rhel6m1 ~]$ cat /etc/oratab
#Backup file is  /opt/app/oracle/product/11.2.0/dbhome_2/srvm/admin/oratab.bak.rhel6m1 line added by Agent
#
+ASM1:/opt/app/11.2.0/grid4:N           # line added by Agent
Executing on the last node will result in unusual amount of logging (compared to previous upgrades) after the Oracle Clusterware operating version was successfully set to 11.2.0.4.0 line. Not all the lines of this out put is shown below.
[root@rhel6m2 grid]#  /opt/app/11.2.0/grid4/rootupgrade.sh
Performing root user operation for Oracle 11g

The following environment variables are set as:
    ORACLE_OWNER= grid
    ORACLE_HOME=  /opt/app/11.2.0/grid4

Enter the full pathname of the local bin directory: [/usr/local/bin]:
The contents of "dbhome" have not changed. No need to overwrite.
The contents of "oraenv" have not changed. No need to overwrite.
The contents of "coraenv" have not changed. No need to overwrite.

Entries will be added to the /etc/oratab file as needed by
Database Configuration Assistant when a database is created
Finished running generic part of root script.
Now product-specific root actions will be performed.
Using configuration parameter file: /opt/app/11.2.0/grid4/crs/install/crsconfig_params
Creating trace directory
Installing Trace File Analyzer
CRS-2791: Starting shutdown of Oracle High Availability Services-managed resources on 'rhel6m2'
CRS-2673: Attempting to stop 'ora.crsd' on 'rhel6m2'
CRS-2790: Starting shutdown of Cluster Ready Services-managed resources on 'rhel6m2'
CRS-2673: Attempting to stop 'ora.CLUSTER_DG.dg' on 'rhel6m2'
CRS-2673: Attempting to stop 'ora.registry.acfs' on 'rhel6m2'
CRS-2673: Attempting to stop 'ora.std11g2.asa.domain.net.svc' on 'rhel6m2'
CRS-2677: Stop of 'ora.std11g2.asa.domain.net.svc' on 'rhel6m2' succeeded
CRS-2673: Attempting to stop 'ora.std11g2.db' on 'rhel6m2'
CRS-2673: Attempting to stop 'ora.LISTENER.lsnr' on 'rhel6m2'
CRS-2673: Attempting to stop 'ora.MYLISTENER.lsnr' on 'rhel6m2'
CRS-2677: Stop of 'ora.LISTENER.lsnr' on 'rhel6m2' succeeded
CRS-2677: Stop of 'ora.MYLISTENER.lsnr' on 'rhel6m2' succeeded
CRS-2673: Attempting to stop 'ora.rhel6m2.vip' on 'rhel6m2'
CRS-2677: Stop of 'ora.registry.acfs' on 'rhel6m2' succeeded
CRS-2677: Stop of 'ora.std11g2.db' on 'rhel6m2' succeeded
CRS-2673: Attempting to stop 'ora.DATA.dg' on 'rhel6m2'
CRS-2673: Attempting to stop 'ora.FLASH.dg' on 'rhel6m2'
CRS-2677: Stop of 'ora.rhel6m2.vip' on 'rhel6m2' succeeded
CRS-2672: Attempting to start 'ora.rhel6m2.vip' on 'rhel6m2'
CRS-2677: Stop of 'ora.DATA.dg' on 'rhel6m2' succeeded
CRS-2677: Stop of 'ora.FLASH.dg' on 'rhel6m2' succeeded
CRS-2676: Start of 'ora.rhel6m2.vip' on 'rhel6m2' succeeded
CRS-2677: Stop of 'ora.CLUSTER_DG.dg' on 'rhel6m2' succeeded
CRS-2673: Attempting to stop 'ora.asm' on 'rhel6m2'
CRS-2677: Stop of 'ora.asm' on 'rhel6m2' succeeded
CRS-2673: Attempting to stop 'ora.ons' on 'rhel6m2'
CRS-2677: Stop of 'ora.ons' on 'rhel6m2' succeeded
CRS-2673: Attempting to stop 'ora.net1.network' on 'rhel6m2'
CRS-2677: Stop of 'ora.net1.network' on 'rhel6m2' succeeded
CRS-2792: Shutdown of Cluster Ready Services-managed resources on 'rhel6m2' has completed
CRS-2677: Stop of 'ora.crsd' on 'rhel6m2' succeeded
CRS-2673: Attempting to stop 'ora.crf' on 'rhel6m2'
CRS-2673: Attempting to stop 'ora.ctssd' on 'rhel6m2'
CRS-2673: Attempting to stop 'ora.evmd' on 'rhel6m2'
CRS-2673: Attempting to stop 'ora.asm' on 'rhel6m2'
CRS-2673: Attempting to stop 'ora.mdnsd' on 'rhel6m2'
CRS-2673: Attempting to stop 'ora.drivers.acfs' on 'rhel6m2'
CRS-2677: Stop of 'ora.crf' on 'rhel6m2' succeeded
CRS-2677: Stop of 'ora.evmd' on 'rhel6m2' succeeded
CRS-2677: Stop of 'ora.mdnsd' on 'rhel6m2' succeeded
CRS-2677: Stop of 'ora.ctssd' on 'rhel6m2' succeeded
CRS-2677: Stop of 'ora.asm' on 'rhel6m2' succeeded
CRS-2673: Attempting to stop 'ora.cluster_interconnect.haip' on 'rhel6m2'
CRS-2677: Stop of 'ora.cluster_interconnect.haip' on 'rhel6m2' succeeded
CRS-2673: Attempting to stop 'ora.cssd' on 'rhel6m2'
CRS-2677: Stop of 'ora.drivers.acfs' on 'rhel6m2' succeeded
CRS-2677: Stop of 'ora.cssd' on 'rhel6m2' succeeded
CRS-2673: Attempting to stop 'ora.gipcd' on 'rhel6m2'
CRS-2677: Stop of 'ora.gipcd' on 'rhel6m2' succeeded
CRS-2673: Attempting to stop 'ora.gpnpd' on 'rhel6m2'
CRS-2677: Stop of 'ora.gpnpd' on 'rhel6m2' succeeded
CRS-2793: Shutdown of Oracle High Availability Services-managed resources on 'rhel6m2' has completed
CRS-4133: Oracle High Availability Services has been stopped.
OLR initialization - successful
Replacing Clusterware entries in upstart
clscfg: EXISTING configuration version 5 detected.
clscfg: version 5 is 11g Release 2.
Successfully accumulated necessary OCR keys.
Creating OCR keys for user 'root', privgrp 'root'..
Operation successful.
Started to upgrade the Oracle Clusterware. This operation may take a few minutes.
Started to upgrade the CSS.
Started to upgrade the CRS.
The CRS was successfully upgraded.
Successfully upgraded the Oracle Clusterware.
Oracle Clusterware operating version was successfully set to 11.2.0.4.0
...
...
[main] [ 2013-09-06 15:03:50.561 BST ] [RuntimeExec.runCommand:170]  RunTimeExec: error>
[main] [ 2013-09-06 15:03:50.562 BST ] [RuntimeExec.runCommand:192]  Returning from RunTimeExec.runCommand
[main] [ 2013-09-06 15:03:50.572 BST ] [Utils.getLocalHost:481]  Hostname retrieved: rhel6m2.domain.net, returned: rhel6m2
[main] [ 2013-09-06 15:03:50.574 BST ] [Utils.getLocalHost:481]  Hostname retrieved: rhel6m2.domain.net, returned: rhel6m2
[main] [ 2013-09-06 15:03:51.585 BST ] [SQLEngine.done:2189]  Done called

ASM upgrade has finished on last node.

Configure Oracle Grid Infrastructure for a Cluster ... succeeded
Once the last node is also upgraded the active version is updated
[grid@rhel6m2 ~]$ crsctl query crs activeversion
Oracle Clusterware active version on the cluster is [11.2.0.4.0]
Click OK on the configuration dialog to continue with rest of the upgrade steps.

If the 11.2.0.3 GI HOME is no longer needed deatch it from the inventory and remove
./runInstaller -silent -detachHome ORACLE_HOME=/opt/app/11.2.0/grid 
rm -rf /opt/app/11.2.0/grid
To check the post upgrade health of the cluster use
cluvfy stage -post crsinst -n rhel6m1,rhel6m2 -verbose
or/and
./raccheck -u -o post
.
This conclude the grid infrastructure part of the RAC upgrade. Next step is to upgrade the database software and finally the databsae.

Related Posts
Upgrading from 10.2.0.4 to 10.2.0.5 (Clusterware, RAC, ASM)
Upgrade from 10.2.0.5 to 11.2.0.3 (Clusterware, RAC, ASM)
Upgrade from 11.1.0.7 to 11.2.0.3 (Clusterware, ASM & RAC)
Upgrade from 11.1.0.7 to 11.2.0.4 (Clusterware, ASM & RAC)
Upgrading from 11.1.0.7 to 11.2.0.3 with Transient Logical Standby
Upgrading from 11.2.0.1 to 11.2.0.3 with in-place upgrade for RAC
In-place upgrade from 11.2.0.2 to 11.2.0.3
Upgrading from 11.2.0.2 to 11.2.0.3 with Physical Standby - 1
Upgrading from 11.2.0.2 to 11.2.0.3 with Physical Standby - 2
Upgrading from 11gR2 (11.2.0.3) to 12c (12.1.0.1) Grid Infrastructure
Upgrading RAC from 11.2.0.4 to 12.1.0.2 - Grid Infrastructure