[grid@rhel6m1 grid]$ crsctl query crs activeversion Oracle Clusterware active version on the cluster is [11.2.0.4.0] [grid@rhel6m1 grid]$ crsctl query crs softwareversion Oracle Clusterware version on node [rhel6m1] is [11.2.0.4.0]If the upgrade to 11.2.0.4 failed on one or on the last node or the downgrade is to a version prior to 11.2 then refer the metalink notes given at the end of the post on how to do a downgrade in such cases. This post also doesn't consider the downgrade of the database. Clusterware must be same or at a higher version than the database software. In this case the database software was still on 11.2.0.3 so downgrading from 11.2.0.4 to 11.2.0.3 wasn't an issue for the database. The old GI home (11.2.0.3 home) was not uninstalled nor removed after the upgrade to 11.2.0.4
<HOME NAME="Ora11g_gridinfrahome1" LOC="/opt/app/11.2.0/grid" TYPE="O" IDX="1"> <-- 11.2.0.3 GI HOME
<NODE_LIST>
<NODE NAME="rhel6m1"/>
<NODE NAME="rhel6m2"/>
</NODE_LIST>
</HOME>
<HOME NAME="Ora11g_gridinfrahome2" LOC="/opt/app/11.2.0/grid4" TYPE="O" IDX="4" CRS="true"> <-- 11.2.0.4 GI HOME
<NODE_LIST>
<NODE NAME="rhel6m1"/>
<NODE NAME="rhel6m2"/>
</NODE_LIST>
</HOME>Before proceeding with the downgrade identify the "ocr-node". OCR node an Oracle term for the node which created the backup of the OCR when the GI was upgraded. This is usually the node on which the first rootupgrade.sh was run during the upgrade. Look in the cdata folder inside GI HOME and locate the backup ocr of the form ocrold_version.[grid@rhel6m1 cdata]$ pwd /opt/app/11.2.0/grid4/cdata [grid@rhel6m1 cdata]$ ls -l total 266508 drwxr-xr-x. 2 grid oinstall 4096 Sep 6 11:39 localhost -rw-------. 1 root root 130663 Sep 6 11:56 ocr11.2.0.3.0 drwxr-xr-x. 2 grid oinstall 4096 Sep 6 11:57 rhel6m1 -rw-------. 1 root oinstall 272756736 Sep 6 12:39 rhel6m1.olr drwxrwxr-x. 2 grid oinstall 4096 Sep 6 11:40 rhel6m-clusterIn this two node RAC the rootupgrade.sh was first run on rhel6m1 node and this become the OCR node. The downgrade should be run last on this node.
On all the nodes other than the OCR-node run the rootcrs.pl with the downgrade option from the new GI HOME (11.2.0.4 home).
[root@rhel6m2 grid]# $GI_HOME/crs/install/rootcrs.pl -downgrade -force -oldcrshome /opt/app/11.2.0/grid -version 11.2.0.3.0 Using configuration parameter file: /opt/app/11.2.0/grid4/crs/install/crsconfig_params CRS-2791: Starting shutdown of Oracle High Availability Services-managed resources on 'rhel6m2' CRS-2673: Attempting to stop 'ora.crsd' on 'rhel6m2' CRS-2790: Starting shutdown of Cluster Ready Services-managed resources on 'rhel6m2' CRS-2673: Attempting to stop 'ora.MYLISTENER.lsnr' on 'rhel6m2' CRS-2673: Attempting to stop 'ora.std11g2.asa.domain.net.svc' on 'rhel6m2' CRS-2673: Attempting to stop 'ora.oc4j' on 'rhel6m2' CRS-2677: Stop of 'ora.std11g2.asa.domain.net.svc' on 'rhel6m2' succeeded CRS-2673: Attempting to stop 'ora.CLUSTER_DG.dg' on 'rhel6m2' CRS-2673: Attempting to stop 'ora.registry.acfs' on 'rhel6m2' CRS-2673: Attempting to stop 'ora.std11g2.db' on 'rhel6m2' CRS-2677: Stop of 'ora.MYLISTENER.lsnr' on 'rhel6m2' succeeded CRS-2673: Attempting to stop 'ora.rhel6m2.vip' on 'rhel6m2' CRS-2677: Stop of 'ora.rhel6m2.vip' on 'rhel6m2' succeeded CRS-2672: Attempting to start 'ora.rhel6m2.vip' on 'rhel6m1' CRS-2677: Stop of 'ora.std11g2.db' on 'rhel6m2' succeeded CRS-2673: Attempting to stop 'ora.DATA.dg' on 'rhel6m2' CRS-2673: Attempting to stop 'ora.FLASH.dg' on 'rhel6m2' CRS-2677: Stop of 'ora.DATA.dg' on 'rhel6m2' succeeded CRS-2677: Stop of 'ora.registry.acfs' on 'rhel6m2' succeeded CRS-2677: Stop of 'ora.FLASH.dg' on 'rhel6m2' succeeded CRS-2676: Start of 'ora.rhel6m2.vip' on 'rhel6m1' succeeded CRS-2677: Stop of 'ora.CLUSTER_DG.dg' on 'rhel6m2' succeeded CRS-2673: Attempting to stop 'ora.asm' on 'rhel6m2' CRS-2677: Stop of 'ora.oc4j' on 'rhel6m2' succeeded CRS-2672: Attempting to start 'ora.oc4j' on 'rhel6m1' CRS-2677: Stop of 'ora.asm' on 'rhel6m2' succeeded CRS-2676: Start of 'ora.oc4j' on 'rhel6m1' succeeded CRS-2673: Attempting to stop 'ora.ons' on 'rhel6m2' CRS-2677: Stop of 'ora.ons' on 'rhel6m2' succeeded CRS-2673: Attempting to stop 'ora.net1.network' on 'rhel6m2' CRS-2677: Stop of 'ora.net1.network' on 'rhel6m2' succeeded CRS-2792: Shutdown of Cluster Ready Services-managed resources on 'rhel6m2' has completed CRS-2677: Stop of 'ora.crsd' on 'rhel6m2' succeeded CRS-2673: Attempting to stop 'ora.drivers.acfs' on 'rhel6m2' CRS-2673: Attempting to stop 'ora.ctssd' on 'rhel6m2' CRS-2673: Attempting to stop 'ora.evmd' on 'rhel6m2' CRS-2673: Attempting to stop 'ora.asm' on 'rhel6m2' CRS-2673: Attempting to stop 'ora.mdnsd' on 'rhel6m2' CRS-2677: Stop of 'ora.evmd' on 'rhel6m2' succeeded CRS-2677: Stop of 'ora.mdnsd' on 'rhel6m2' succeeded CRS-2677: Stop of 'ora.asm' on 'rhel6m2' succeeded CRS-2673: Attempting to stop 'ora.cluster_interconnect.haip' on 'rhel6m2' CRS-2677: Stop of 'ora.cluster_interconnect.haip' on 'rhel6m2' succeeded CRS-2677: Stop of 'ora.ctssd' on 'rhel6m2' succeeded CRS-2673: Attempting to stop 'ora.cssd' on 'rhel6m2' CRS-2677: Stop of 'ora.drivers.acfs' on 'rhel6m2' succeeded CRS-2677: Stop of 'ora.cssd' on 'rhel6m2' succeeded CRS-2673: Attempting to stop 'ora.crf' on 'rhel6m2' CRS-2677: Stop of 'ora.crf' on 'rhel6m2' succeeded CRS-2673: Attempting to stop 'ora.gipcd' on 'rhel6m2' CRS-2677: Stop of 'ora.gipcd' on 'rhel6m2' succeeded CRS-2673: Attempting to stop 'ora.gpnpd' on 'rhel6m2' CRS-2677: Stop of 'ora.gpnpd' on 'rhel6m2' succeeded CRS-2793: Shutdown of Oracle High Availability Services-managed resources on 'rhel6m2' has completed CRS-4133: Oracle High Availability Services has been stopped. Successfully downgraded Oracle Clusterware stack on this node
Once all the non-OCR Nodes are downgraded, run the downgrade on the OCR-Node with the lastnode option.
[root@rhel6m1 cdata]# $GI_HOME/crs/install/rootcrs.pl -downgrade -force -lastnode -oldcrshome /opt/app/11.2.0/grid -version 11.2.0.3.0 Using configuration parameter file: /opt/app/11.2.0/grid4/crs/install/crsconfig_params CRS-2791: Starting shutdown of Oracle High Availability Services-managed resources on 'rhel6m1' CRS-2673: Attempting to stop 'ora.crsd' on 'rhel6m1' CRS-2790: Starting shutdown of Cluster Ready Services-managed resources on 'rhel6m1' CRS-2673: Attempting to stop 'ora.MYLISTENER.lsnr' on 'rhel6m1' CRS-2673: Attempting to stop 'ora.rhel6m2.vip' on 'rhel6m1' CRS-2673: Attempting to stop 'ora.oc4j' on 'rhel6m1' CRS-2673: Attempting to stop 'ora.cvu' on 'rhel6m1' CRS-2673: Attempting to stop 'ora.MYSCANLISTENER_SCAN1.lsnr' on 'rhel6m1' CRS-2677: Stop of 'ora.MYLISTENER.lsnr' on 'rhel6m1' succeeded CRS-2677: Stop of 'ora.MYSCANLISTENER_SCAN1.lsnr' on 'rhel6m1' succeeded CRS-2673: Attempting to stop 'ora.scan1.vip' on 'rhel6m1' CRS-2677: Stop of 'ora.cvu' on 'rhel6m1' succeeded CRS-2677: Stop of 'ora.rhel6m2.vip' on 'rhel6m1' succeeded CRS-2673: Attempting to stop 'ora.std11g2.asa.domain.net.svc' on 'rhel6m1' CRS-2677: Stop of 'ora.std11g2.asa.domain.net.svc' on 'rhel6m1' succeeded CRS-2673: Attempting to stop 'ora.rhel6m1.vip' on 'rhel6m1' CRS-2673: Attempting to stop 'ora.CLUSTER_DG.dg' on 'rhel6m1' CRS-2673: Attempting to stop 'ora.registry.acfs' on 'rhel6m1' CRS-2673: Attempting to stop 'ora.std11g2.db' on 'rhel6m1' CRS-2677: Stop of 'ora.scan1.vip' on 'rhel6m1' succeeded CRS-2677: Stop of 'ora.std11g2.db' on 'rhel6m1' succeeded CRS-2673: Attempting to stop 'ora.DATA.dg' on 'rhel6m1' CRS-2673: Attempting to stop 'ora.FLASH.dg' on 'rhel6m1' CRS-2677: Stop of 'ora.rhel6m1.vip' on 'rhel6m1' succeeded CRS-2677: Stop of 'ora.DATA.dg' on 'rhel6m1' succeeded CRS-2677: Stop of 'ora.registry.acfs' on 'rhel6m1' succeeded CRS-2677: Stop of 'ora.FLASH.dg' on 'rhel6m1' succeeded CRS-2677: Stop of 'ora.CLUSTER_DG.dg' on 'rhel6m1' succeeded CRS-2673: Attempting to stop 'ora.asm' on 'rhel6m1' CRS-2677: Stop of 'ora.oc4j' on 'rhel6m1' succeeded CRS-2677: Stop of 'ora.asm' on 'rhel6m1' succeeded CRS-2673: Attempting to stop 'ora.ons' on 'rhel6m1' CRS-2677: Stop of 'ora.ons' on 'rhel6m1' succeeded CRS-2673: Attempting to stop 'ora.net1.network' on 'rhel6m1' CRS-2677: Stop of 'ora.net1.network' on 'rhel6m1' succeeded CRS-2792: Shutdown of Cluster Ready Services-managed resources on 'rhel6m1' has completed CRS-2677: Stop of 'ora.crsd' on 'rhel6m1' succeeded CRS-2673: Attempting to stop 'ora.drivers.acfs' on 'rhel6m1' CRS-2673: Attempting to stop 'ora.crf' on 'rhel6m1' CRS-2673: Attempting to stop 'ora.ctssd' on 'rhel6m1' CRS-2673: Attempting to stop 'ora.evmd' on 'rhel6m1' CRS-2673: Attempting to stop 'ora.asm' on 'rhel6m1' CRS-2673: Attempting to stop 'ora.mdnsd' on 'rhel6m1' CRS-2677: Stop of 'ora.crf' on 'rhel6m1' succeeded CRS-2677: Stop of 'ora.evmd' on 'rhel6m1' succeeded CRS-2677: Stop of 'ora.mdnsd' on 'rhel6m1' succeeded CRS-2677: Stop of 'ora.asm' on 'rhel6m1' succeeded CRS-2673: Attempting to stop 'ora.cluster_interconnect.haip' on 'rhel6m1' CRS-2677: Stop of 'ora.cluster_interconnect.haip' on 'rhel6m1' succeeded CRS-2677: Stop of 'ora.ctssd' on 'rhel6m1' succeeded CRS-2673: Attempting to stop 'ora.cssd' on 'rhel6m1' CRS-2677: Stop of 'ora.cssd' on 'rhel6m1' succeeded CRS-2673: Attempting to stop 'ora.gipcd' on 'rhel6m1' CRS-2677: Stop of 'ora.drivers.acfs' on 'rhel6m1' succeeded CRS-2677: Stop of 'ora.gipcd' on 'rhel6m1' succeeded CRS-2673: Attempting to stop 'ora.gpnpd' on 'rhel6m1' CRS-2677: Stop of 'ora.gpnpd' on 'rhel6m1' succeeded CRS-2793: Shutdown of Oracle High Availability Services-managed resources on 'rhel6m1' has completed CRS-4133: Oracle High Availability Services has been stopped. CRS-4123: Oracle High Availability Services has been started. CRS-2672: Attempting to start 'ora.mdnsd' on 'rhel6m1' CRS-2676: Start of 'ora.mdnsd' on 'rhel6m1' succeeded CRS-2672: Attempting to start 'ora.gpnpd' on 'rhel6m1' CRS-2676: Start of 'ora.gpnpd' on 'rhel6m1' succeeded CRS-2672: Attempting to start 'ora.cssdmonitor' on 'rhel6m1' CRS-2672: Attempting to start 'ora.gipcd' on 'rhel6m1' CRS-2676: Start of 'ora.cssdmonitor' on 'rhel6m1' succeeded CRS-2676: Start of 'ora.gipcd' on 'rhel6m1' succeeded CRS-2672: Attempting to start 'ora.cssd' on 'rhel6m1' CRS-2672: Attempting to start 'ora.diskmon' on 'rhel6m1' CRS-2676: Start of 'ora.diskmon' on 'rhel6m1' succeeded CRS-2676: Start of 'ora.cssd' on 'rhel6m1' succeeded CRS-2672: Attempting to start 'ora.drivers.acfs' on 'rhel6m1' CRS-2679: Attempting to clean 'ora.cluster_interconnect.haip' on 'rhel6m1' CRS-2672: Attempting to start 'ora.ctssd' on 'rhel6m1' CRS-2681: Clean of 'ora.cluster_interconnect.haip' on 'rhel6m1' succeeded CRS-2672: Attempting to start 'ora.cluster_interconnect.haip' on 'rhel6m1' CRS-2676: Start of 'ora.ctssd' on 'rhel6m1' succeeded CRS-2676: Start of 'ora.drivers.acfs' on 'rhel6m1' succeeded CRS-2676: Start of 'ora.cluster_interconnect.haip' on 'rhel6m1' succeeded CRS-2672: Attempting to start 'ora.asm' on 'rhel6m1' CRS-2676: Start of 'ora.asm' on 'rhel6m1' succeeded CRS-2791: Starting shutdown of Oracle High Availability Services-managed resources on 'rhel6m1' CRS-2673: Attempting to stop 'ora.mdnsd' on 'rhel6m1' CRS-2673: Attempting to stop 'ora.drivers.acfs' on 'rhel6m1' CRS-2673: Attempting to stop 'ora.ctssd' on 'rhel6m1' CRS-2673: Attempting to stop 'ora.asm' on 'rhel6m1' CRS-2677: Stop of 'ora.mdnsd' on 'rhel6m1' succeeded CRS-2677: Stop of 'ora.asm' on 'rhel6m1' succeeded CRS-2673: Attempting to stop 'ora.cluster_interconnect.haip' on 'rhel6m1' CRS-2677: Stop of 'ora.cluster_interconnect.haip' on 'rhel6m1' succeeded CRS-2677: Stop of 'ora.ctssd' on 'rhel6m1' succeeded CRS-2673: Attempting to stop 'ora.cssd' on 'rhel6m1' CRS-2677: Stop of 'ora.cssd' on 'rhel6m1' succeeded CRS-2673: Attempting to stop 'ora.gipcd' on 'rhel6m1' CRS-2677: Stop of 'ora.drivers.acfs' on 'rhel6m1' succeeded CRS-2677: Stop of 'ora.gipcd' on 'rhel6m1' succeeded CRS-2673: Attempting to stop 'ora.gpnpd' on 'rhel6m1' CRS-2677: Stop of 'ora.gpnpd' on 'rhel6m1' succeeded CRS-2793: Shutdown of Oracle High Availability Services-managed resources on 'rhel6m1' has completed CRS-4133: Oracle High Availability Services has been stopped. CRS-4123: Oracle High Availability Services has been started. CRS-2672: Attempting to start 'ora.mdnsd' on 'rhel6m1' CRS-2676: Start of 'ora.mdnsd' on 'rhel6m1' succeeded CRS-2672: Attempting to start 'ora.gpnpd' on 'rhel6m1' CRS-2676: Start of 'ora.gpnpd' on 'rhel6m1' succeeded CRS-2672: Attempting to start 'ora.cssdmonitor' on 'rhel6m1' CRS-2672: Attempting to start 'ora.gipcd' on 'rhel6m1' CRS-2676: Start of 'ora.cssdmonitor' on 'rhel6m1' succeeded CRS-2676: Start of 'ora.gipcd' on 'rhel6m1' succeeded CRS-2672: Attempting to start 'ora.cssd' on 'rhel6m1' CRS-2672: Attempting to start 'ora.diskmon' on 'rhel6m1' CRS-2676: Start of 'ora.diskmon' on 'rhel6m1' succeeded CRS-2676: Start of 'ora.cssd' on 'rhel6m1' succeeded CRS-2672: Attempting to start 'ora.drivers.acfs' on 'rhel6m1' CRS-2679: Attempting to clean 'ora.cluster_interconnect.haip' on 'rhel6m1' CRS-2672: Attempting to start 'ora.ctssd' on 'rhel6m1' CRS-2681: Clean of 'ora.cluster_interconnect.haip' on 'rhel6m1' succeeded CRS-2672: Attempting to start 'ora.cluster_interconnect.haip' on 'rhel6m1' CRS-2676: Start of 'ora.drivers.acfs' on 'rhel6m1' succeeded CRS-2676: Start of 'ora.ctssd' on 'rhel6m1' succeeded CRS-2676: Start of 'ora.cluster_interconnect.haip' on 'rhel6m1' succeeded CRS-2672: Attempting to start 'ora.asm' on 'rhel6m1' CRS-2676: Start of 'ora.asm' on 'rhel6m1' succeeded Successfully downgraded OCR to 11.2.0.3.0 CRS-2672: Attempting to start 'ora.crsd' on 'rhel6m1' CRS-2676: Start of 'ora.crsd' on 'rhel6m1' succeeded CRS-2791: Starting shutdown of Oracle High Availability Services-managed resources on 'rhel6m1' CRS-2673: Attempting to stop 'ora.crsd' on 'rhel6m1' CRS-2677: Stop of 'ora.crsd' on 'rhel6m1' succeeded CRS-2673: Attempting to stop 'ora.mdnsd' on 'rhel6m1' CRS-2673: Attempting to stop 'ora.ctssd' on 'rhel6m1' CRS-2673: Attempting to stop 'ora.asm' on 'rhel6m1' CRS-2673: Attempting to stop 'ora.drivers.acfs' on 'rhel6m1' CRS-2677: Stop of 'ora.mdnsd' on 'rhel6m1' succeeded CRS-2677: Stop of 'ora.asm' on 'rhel6m1' succeeded CRS-2673: Attempting to stop 'ora.cluster_interconnect.haip' on 'rhel6m1' CRS-2677: Stop of 'ora.ctssd' on 'rhel6m1' succeeded CRS-2677: Stop of 'ora.cluster_interconnect.haip' on 'rhel6m1' succeeded CRS-2673: Attempting to stop 'ora.cssd' on 'rhel6m1' CRS-2677: Stop of 'ora.drivers.acfs' on 'rhel6m1' succeeded CRS-2677: Stop of 'ora.cssd' on 'rhel6m1' succeeded CRS-2673: Attempting to stop 'ora.gipcd' on 'rhel6m1' CRS-2677: Stop of 'ora.gipcd' on 'rhel6m1' succeeded CRS-2673: Attempting to stop 'ora.gpnpd' on 'rhel6m1' CRS-2677: Stop of 'ora.gpnpd' on 'rhel6m1' succeeded CRS-2793: Shutdown of Oracle High Availability Services-managed resources on 'rhel6m1' has completed CRS-4133: Oracle High Availability Services has been stopped. Successfully downgraded Oracle Clusterware stack on this node Run '/opt/app/11.2.0/grid/bin/crsctl start crs' on all nodesBefore running the crs start there are few things to be done. As part of the upgrade to 11.2.0.4 trace file analyzer (TFA) is installed. Remove the init script for TFA
#cd /etc/init [root@rhel6m1 init]# rm oracle-tfa.conf rm: remove regular file `oracle-tfa.conf'? y [root@rhel6m1 init]# rm ../init.d/init.tfa rm: remove regular file `../init.d/init.tfa'? yVerify the ASM home in the /etc/oratb is pointing to the old GI home
cat /etc/oratab +ASM2:/opt/app/11.2.0/grid:N # line added by AgentRemove gpnp profiles if the rootupgrade.sh was run concurrently on multiple nodes. Also remove the clusterware checkpoint files.
rm -rf /opt/app/11.2.0/grid4/gpnp/* rm -rf $ORACLE_BASE/Clusterware/ckptGridHA_rhel6m1.xmlStart clusterware stack on all the nodes with crsctl start crs. Verify the active version is the downgraded version.
[grid@rhel6m2 ~]$ crsctl query crs activeversion Oracle Clusterware active version on the cluster is [11.2.0.3.0] [grid@rhel6m2 ~]$ crsctl query crs softwareversion Oracle Clusterware version on node [rhel6m2] is [11.2.0.3.0]Run a ocrcheck to verify the OCR intergrity
[root@rhel6m2 gpnp]# ocrcheck
Status of Oracle Cluster Registry is as follows :
Version : 3
Total space (kbytes) : 262120
Used space (kbytes) : 3388
Available space (kbytes) : 258732
ID : 2072206343
Device/File Name : +CLUSTER_DG
Device/File integrity check succeeded
Device/File not configured
Cluster registry integrity check succeeded
Logical corruption check succeededThe inventory will still have the CRS=true on the 11.2.0.4 home. Run the runInstaller to set the CRS=true to the 11.2.0.3 home.[grid@rhel6m1 bin]$ ./runInstaller -updateNodeList ORACLE_HOME=/opt/app/11.2.0/grid CRS=true
[grid@rhel6m1 bin]$ ./runInstaller -updateNodeList ORACLE_HOME=/opt/app/11.2.0/grid4 CRS=false
<HOME NAME="Ora11g_gridinfrahome1" LOC="/opt/app/11.2.0/grid" TYPE="O" IDX="1" CRS="true">
<NODE_LIST>
<NODE NAME="rhel6m1"/>
<NODE NAME="rhel6m2"/>
</NODE_LIST>
</HOME>
<HOME NAME="Ora11g_gridinfrahome2" LOC="/opt/app/11.2.0/grid4" TYPE="O" IDX="4" >
<NODE_LIST>
<NODE NAME="rhel6m1"/>
<NODE NAME="rhel6m2"/>
</NODE_LIST>
</HOME>As the last step detach the 11.2.0.4 GI Home from the inventory and remove it./runInstaller -detachHome ORACLE_HOME=/opt/app/11.2.0/grid4 -silent rm -rf /opt/app/11.2.0/grid4 # run on all nodes
Related Post
Downgrade Grid Infrastructure from 12.1.0.2 to 11.2.0.4
Downgrade Grid Infrastructure from 11.2.0.4 to 11.1.0.7
Useful metalink notes
How to Downgrade 11.2.0.2 Grid Infrastructure Cluster to 11.2.0.1 [ 1364230.1]
How to Downgrade 11.2.0.3 Grid Infrastructure Cluster to Lower 11.2 GI or Pre-11.2 CRS [ 1364946.1]
How to Update Inventory to Set/Unset "CRS=true" Flag for Oracle Clusterware Home [ 1053393.1]
Oracle Clusterware (GI or CRS) Related Abbreviations, Acronyms and Procedures [ 1374275.1]
