[grid@rhel6m1 ~]$ crsctl query crs activeversion
Oracle Clusterware active version on the cluster is [12.1.0.2.0]
[grid@rhel6m1 ~]$ crsctl query crs softwareversion
Oracle Clusterware version on node [rhel6m1] is [12.1.0.2.0]
[root@rhel6m1 grid]# ocrcheck
Status of Oracle Cluster Registry is as follows :
Version : 4
Total space (kbytes) : 409568
Used space (kbytes) : 9068
Available space (kbytes) : 400500
ID : 2072206343
Device/File Name : +CLUSTER_DG The environment used here is the same environment used to upgrade from 11.2.0.3 to 11.2.0.4. As such the OCR-node (node containing the backup of the OCR created during the upgrade to 12.1.0.2) contains both 11.2.0.3 and 11.2.0.4 OCR backups but with different permissions.[grid@rhel6m1 cdata]$ ls -lrt /opt/app/12.1.0/grid2/cdata/ total 266644 drwxr-xr-x. 2 grid oinstall 4096 Mar 26 17:22 localhost -rw-r--r--. 1 grid oinstall 132453 Mar 26 17:34 ocr11.2.0.3.0 -rw-------. 1 root root 130412 Mar 26 17:44 ocr11.2.0.4.0 drwxr-xr-x. 2 grid oinstall 4096 Mar 26 17:46 rhel6m1 -rw-------. 1 root oinstall 503484416 Mar 26 17:54 rhel6m1.olr drwxrwxr-x. 2 grid oinstall 4096 Mar 26 17:58 rhel6m-clusterDuring the downgrade the the higher version is detected and restored (no harm even if the ocr11.2.0.3.0 is removed before the downgrade).
Also before the downgrade commands are run make sure only the 12.1 related binaries are in the PATH variable (crsctl, srvctl) and ORACLE_HOME points to 12.1. At times the downgrade failed with
Died at /opt/app/12.1.0/grid2/crs/install/crsdeconfig.pm line 2336. The command '/opt/app/12.1.0/grid2/perl/bin/perl -I/opt/app/12.1.0/grid2/perl/lib -I/opt/app/12.1.0/grid2/crs/install /opt/app/12.1.0/grid2/crs/install/rootcrs.pl -downgrade -lastnode' execution failedfor which solution according to 1917922.1 is to run the downgrade command from a location where grid user has write access to. In this case it was run from grid user home so MOS note solution may not be applicable.
Another time the downgrade on the OCR-node failed with
Using configuration parameter file: /opt/app/12.1.0/grid2/crs/install/crsconfig_params 2015-04-01 10:43:00.047 CLSD: An error occurred while attempting to generate a full name. Logging may not be active for this process Additional diagnostics: CLSU-00100: operating system function: sclsdgcwd failed with error data: -1 CLSU-00103: error location: sclsdgcwd5 (:CLSD00183:)MOS notes 1961344.1 and 1911258.1 gives causes and solutions for this but neither case was valid in this case as the permission of directories mentioned on these MOS were already set to correct values. These issues went away after removing from PATH any references to 11.2 binaries.
To begin the downgrade run the rootcrs.sh with downgrade option on all but the OCR-node. In this case the rhel6m1 is the OCR-node and downgrade first run on rhel6m2.
[root@rhel6m2 grid]# /opt/app/12.1.0/grid2/crs/install/rootcrs.sh -downgrade Using configuration parameter file: /opt/app/12.1.0/grid2/crs/install/crsconfig_params CRS-2791: Starting shutdown of Oracle High Availability Services-managed resources on 'rhel6m2' CRS-2673: Attempting to stop 'ora.crsd' on 'rhel6m2' CRS-2790: Starting shutdown of Cluster Ready Services-managed resources on 'rhel6m2' CRS-2673: Attempting to stop 'ora.CLUSTER_DG.dg' on 'rhel6m2' CRS-2673: Attempting to stop 'ora.MYSCANLISTENER_SCAN1.lsnr' on 'rhel6m2' CRS-2673: Attempting to stop 'ora.std11g2.asa.domain.net.svc' on 'rhel6m2' CRS-2673: Attempting to stop 'ora.oc4j' on 'rhel6m2' CRS-2677: Stop of 'ora.CLUSTER_DG.dg' on 'rhel6m2' succeeded CRS-2677: Stop of 'ora.std11g2.asa.domain.net.svc' on 'rhel6m2' succeeded CRS-2673: Attempting to stop 'ora.std11g2.db' on 'rhel6m2' CRS-2673: Attempting to stop 'ora.MYLISTENER.lsnr' on 'rhel6m2' CRS-2677: Stop of 'ora.MYSCANLISTENER_SCAN1.lsnr' on 'rhel6m2' succeeded CRS-2673: Attempting to stop 'ora.scan1.vip' on 'rhel6m2' CRS-2677: Stop of 'ora.MYLISTENER.lsnr' on 'rhel6m2' succeeded CRS-2673: Attempting to stop 'ora.rhel6m2.vip' on 'rhel6m2' CRS-2677: Stop of 'ora.std11g2.db' on 'rhel6m2' succeeded CRS-2673: Attempting to stop 'ora.FLASH.dg' on 'rhel6m2' CRS-2673: Attempting to stop 'ora.DATA.dg' on 'rhel6m2' CRS-2677: Stop of 'ora.scan1.vip' on 'rhel6m2' succeeded CRS-2672: Attempting to start 'ora.scan1.vip' on 'rhel6m1' CRS-2677: Stop of 'ora.DATA.dg' on 'rhel6m2' succeeded CRS-2677: Stop of 'ora.rhel6m2.vip' on 'rhel6m2' succeeded CRS-2672: Attempting to start 'ora.rhel6m2.vip' on 'rhel6m1' CRS-2676: Start of 'ora.scan1.vip' on 'rhel6m1' succeeded CRS-2672: Attempting to start 'ora.MYSCANLISTENER_SCAN1.lsnr' on 'rhel6m1' CRS-2677: Stop of 'ora.FLASH.dg' on 'rhel6m2' succeeded CRS-2673: Attempting to stop 'ora.asm' on 'rhel6m2' CRS-2677: Stop of 'ora.asm' on 'rhel6m2' succeeded CRS-2676: Start of 'ora.rhel6m2.vip' on 'rhel6m1' succeeded CRS-2676: Start of 'ora.MYSCANLISTENER_SCAN1.lsnr' on 'rhel6m1' succeeded CRS-2677: Stop of 'ora.oc4j' on 'rhel6m2' succeeded CRS-2672: Attempting to start 'ora.oc4j' on 'rhel6m1' CRS-2676: Start of 'ora.oc4j' on 'rhel6m1' succeeded CRS-2673: Attempting to stop 'ora.ons' on 'rhel6m2' CRS-2677: Stop of 'ora.ons' on 'rhel6m2' succeeded CRS-2673: Attempting to stop 'ora.net1.network' on 'rhel6m2' CRS-2677: Stop of 'ora.net1.network' on 'rhel6m2' succeeded CRS-2792: Shutdown of Cluster Ready Services-managed resources on 'rhel6m2' has completed CRS-2677: Stop of 'ora.crsd' on 'rhel6m2' succeeded CRS-2673: Attempting to stop 'ora.ctssd' on 'rhel6m2' CRS-2673: Attempting to stop 'ora.evmd' on 'rhel6m2' CRS-2673: Attempting to stop 'ora.storage' on 'rhel6m2' CRS-2673: Attempting to stop 'ora.mdnsd' on 'rhel6m2' CRS-2673: Attempting to stop 'ora.gpnpd' on 'rhel6m2' CRS-2673: Attempting to stop 'ora.drivers.acfs' on 'rhel6m2' CRS-2677: Stop of 'ora.storage' on 'rhel6m2' succeeded CRS-2673: Attempting to stop 'ora.asm' on 'rhel6m2' CRS-2677: Stop of 'ora.drivers.acfs' on 'rhel6m2' succeeded CRS-2677: Stop of 'ora.ctssd' on 'rhel6m2' succeeded CRS-2677: Stop of 'ora.mdnsd' on 'rhel6m2' succeeded CRS-2677: Stop of 'ora.evmd' on 'rhel6m2' succeeded CRS-2677: Stop of 'ora.gpnpd' on 'rhel6m2' succeeded CRS-2677: Stop of 'ora.asm' on 'rhel6m2' succeeded CRS-2673: Attempting to stop 'ora.cluster_interconnect.haip' on 'rhel6m2' CRS-2677: Stop of 'ora.cluster_interconnect.haip' on 'rhel6m2' succeeded CRS-2673: Attempting to stop 'ora.cssd' on 'rhel6m2' CRS-2677: Stop of 'ora.cssd' on 'rhel6m2' succeeded CRS-2673: Attempting to stop 'ora.crf' on 'rhel6m2' CRS-2677: Stop of 'ora.crf' on 'rhel6m2' succeeded CRS-2673: Attempting to stop 'ora.gipcd' on 'rhel6m2' CRS-2677: Stop of 'ora.gipcd' on 'rhel6m2' succeeded CRS-2793: Shutdown of Oracle High Availability Services-managed resources on 'rhel6m2' has completed CRS-4133: Oracle High Availability Services has been stopped. 2015/04/01 12:07:02 CLSRSC-4001: Installing Oracle Trace File Analyzer (TFA) Collector. 2015/04/01 12:07:02 CLSRSC-4002: Successfully installed Oracle Trace File Analyzer (TFA) Collector. Successfully downgraded Oracle Clusterware stack on this node
Since this is a two node RAC the downgrade command is run with lastnode option on rhel6m1 (OCR-node). This will remove the GI management repository, downgrade the OCR
[root@rhel6m1 grid]# /opt/app/12.1.0/grid2/crs/install/rootcrs.sh -downgrade -lastnode Using configuration parameter file: /opt/app/12.1.0/grid2/crs/install/crsconfig_params CRS-2791: Starting shutdown of Oracle High Availability Services-managed resources on 'rhel6m1' CRS-2673: Attempting to stop 'ora.crsd' on 'rhel6m1' CRS-2790: Starting shutdown of Cluster Ready Services-managed resources on 'rhel6m1' CRS-2673: Attempting to stop 'ora.MYLISTENER.lsnr' on 'rhel6m1' CRS-2673: Attempting to stop 'ora.std11g2.asa.domain.net.svc' on 'rhel6m1' CRS-2673: Attempting to stop 'ora.MYSCANLISTENER_SCAN1.lsnr' on 'rhel6m1' CRS-2673: Attempting to stop 'ora.rhel6m2.vip' on 'rhel6m1' CRS-2673: Attempting to stop 'ora.oc4j' on 'rhel6m1' CRS-2673: Attempting to stop 'ora.mgmtdb' on 'rhel6m1' CRS-2673: Attempting to stop 'ora.cvu' on 'rhel6m1' CRS-2677: Stop of 'ora.cvu' on 'rhel6m1' succeeded CRS-2677: Stop of 'ora.MYSCANLISTENER_SCAN1.lsnr' on 'rhel6m1' succeeded CRS-2673: Attempting to stop 'ora.scan1.vip' on 'rhel6m1' CRS-2677: Stop of 'ora.MYLISTENER.lsnr' on 'rhel6m1' succeeded CRS-2677: Stop of 'ora.std11g2.asa.domain.net.svc' on 'rhel6m1' succeeded CRS-2673: Attempting to stop 'ora.std11g2.db' on 'rhel6m1' CRS-2677: Stop of 'ora.std11g2.db' on 'rhel6m1' succeeded CRS-2673: Attempting to stop 'ora.DATA.dg' on 'rhel6m1' CRS-2673: Attempting to stop 'ora.FLASH.dg' on 'rhel6m1' CRS-2677: Stop of 'ora.rhel6m2.vip' on 'rhel6m1' succeeded CRS-2673: Attempting to stop 'ora.rhel6m1.vip' on 'rhel6m1' CRS-2677: Stop of 'ora.DATA.dg' on 'rhel6m1' succeeded CRS-2677: Stop of 'ora.scan1.vip' on 'rhel6m1' succeeded CRS-2677: Stop of 'ora.rhel6m1.vip' on 'rhel6m1' succeeded CRS-2677: Stop of 'ora.mgmtdb' on 'rhel6m1' succeeded CRS-2673: Attempting to stop 'ora.CLUSTER_DG.dg' on 'rhel6m1' CRS-2673: Attempting to stop 'ora.MGMTLSNR' on 'rhel6m1' CRS-2677: Stop of 'ora.FLASH.dg' on 'rhel6m1' succeeded CRS-2677: Stop of 'ora.MGMTLSNR' on 'rhel6m1' succeeded CRS-2677: Stop of 'ora.CLUSTER_DG.dg' on 'rhel6m1' succeeded CRS-2673: Attempting to stop 'ora.asm' on 'rhel6m1' CRS-2677: Stop of 'ora.asm' on 'rhel6m1' succeeded CRS-2677: Stop of 'ora.oc4j' on 'rhel6m1' succeeded CRS-2673: Attempting to stop 'ora.ons' on 'rhel6m1' CRS-2677: Stop of 'ora.ons' on 'rhel6m1' succeeded CRS-2673: Attempting to stop 'ora.net1.network' on 'rhel6m1' CRS-2677: Stop of 'ora.net1.network' on 'rhel6m1' succeeded CRS-2792: Shutdown of Cluster Ready Services-managed resources on 'rhel6m1' has completed CRS-2677: Stop of 'ora.crsd' on 'rhel6m1' succeeded CRS-2673: Attempting to stop 'ora.storage' on 'rhel6m1' CRS-2673: Attempting to stop 'ora.mdnsd' on 'rhel6m1' CRS-2673: Attempting to stop 'ora.gpnpd' on 'rhel6m1' CRS-2673: Attempting to stop 'ora.drivers.acfs' on 'rhel6m1' CRS-2677: Stop of 'ora.storage' on 'rhel6m1' succeeded CRS-2677: Stop of 'ora.drivers.acfs' on 'rhel6m1' succeeded CRS-2673: Attempting to stop 'ora.crf' on 'rhel6m1' CRS-2673: Attempting to stop 'ora.ctssd' on 'rhel6m1' CRS-2673: Attempting to stop 'ora.evmd' on 'rhel6m1' CRS-2673: Attempting to stop 'ora.asm' on 'rhel6m1' CRS-2677: Stop of 'ora.mdnsd' on 'rhel6m1' succeeded CRS-2677: Stop of 'ora.gpnpd' on 'rhel6m1' succeeded CRS-2677: Stop of 'ora.crf' on 'rhel6m1' succeeded CRS-2677: Stop of 'ora.ctssd' on 'rhel6m1' succeeded CRS-2677: Stop of 'ora.evmd' on 'rhel6m1' succeeded CRS-2677: Stop of 'ora.asm' on 'rhel6m1' succeeded CRS-2673: Attempting to stop 'ora.cluster_interconnect.haip' on 'rhel6m1' CRS-2677: Stop of 'ora.cluster_interconnect.haip' on 'rhel6m1' succeeded CRS-2673: Attempting to stop 'ora.cssd' on 'rhel6m1' CRS-2677: Stop of 'ora.cssd' on 'rhel6m1' succeeded CRS-2673: Attempting to stop 'ora.gipcd' on 'rhel6m1' CRS-2677: Stop of 'ora.gipcd' on 'rhel6m1' succeeded CRS-2793: Shutdown of Oracle High Availability Services-managed resources on 'rhel6m1' has completed CRS-4133: Oracle High Availability Services has been stopped. ASM downgrade operation succeeded CRS-2791: Starting shutdown of Oracle High Availability Services-managed resources on 'rhel6m1' CRS-2673: Attempting to stop 'ora.crsd' on 'rhel6m1' CRS-2677: Stop of 'ora.crsd' on 'rhel6m1' succeeded CRS-2673: Attempting to stop 'ora.gpnpd' on 'rhel6m1' CRS-2673: Attempting to stop 'ora.ctssd' on 'rhel6m1' CRS-2673: Attempting to stop 'ora.evmd' on 'rhel6m1' CRS-2673: Attempting to stop 'ora.storage' on 'rhel6m1' CRS-2673: Attempting to stop 'ora.crf' on 'rhel6m1' CRS-2673: Attempting to stop 'ora.drivers.acfs' on 'rhel6m1' CRS-2673: Attempting to stop 'ora.mdnsd' on 'rhel6m1' CRS-2677: Stop of 'ora.storage' on 'rhel6m1' succeeded CRS-2673: Attempting to stop 'ora.asm' on 'rhel6m1' CRS-2677: Stop of 'ora.drivers.acfs' on 'rhel6m1' succeeded CRS-2677: Stop of 'ora.ctssd' on 'rhel6m1' succeeded CRS-2677: Stop of 'ora.evmd' on 'rhel6m1' succeeded CRS-2677: Stop of 'ora.gpnpd' on 'rhel6m1' succeeded CRS-2677: Stop of 'ora.crf' on 'rhel6m1' succeeded CRS-2677: Stop of 'ora.mdnsd' on 'rhel6m1' succeeded CRS-2677: Stop of 'ora.asm' on 'rhel6m1' succeeded CRS-2673: Attempting to stop 'ora.cluster_interconnect.haip' on 'rhel6m1' CRS-2677: Stop of 'ora.cluster_interconnect.haip' on 'rhel6m1' succeeded CRS-2673: Attempting to stop 'ora.cssd' on 'rhel6m1' CRS-2677: Stop of 'ora.cssd' on 'rhel6m1' succeeded CRS-2673: Attempting to stop 'ora.gipcd' on 'rhel6m1' CRS-2677: Stop of 'ora.gipcd' on 'rhel6m1' succeeded CRS-2793: Shutdown of Oracle High Availability Services-managed resources on 'rhel6m1' has completed CRS-4133: Oracle High Availability Services has been stopped. CRS-4123: Oracle High Availability Services has been started. CRS-2791: Starting shutdown of Oracle High Availability Services-managed resources on 'rhel6m1' CRS-2673: Attempting to stop 'ora.drivers.acfs' on 'rhel6m1' CRS-2677: Stop of 'ora.drivers.acfs' on 'rhel6m1' succeeded CRS-2793: Shutdown of Oracle High Availability Services-managed resources on 'rhel6m1' has completed CRS-4133: Oracle High Availability Services has been stopped. CRS-4123: Starting Oracle High Availability Services-managed resources CRS-2672: Attempting to start 'ora.mdnsd' on 'rhel6m1' CRS-2672: Attempting to start 'ora.evmd' on 'rhel6m1' CRS-2676: Start of 'ora.mdnsd' on 'rhel6m1' succeeded CRS-2676: Start of 'ora.evmd' on 'rhel6m1' succeeded CRS-2672: Attempting to start 'ora.gpnpd' on 'rhel6m1' CRS-2676: Start of 'ora.gpnpd' on 'rhel6m1' succeeded CRS-2672: Attempting to start 'ora.gipcd' on 'rhel6m1' CRS-2676: Start of 'ora.gipcd' on 'rhel6m1' succeeded CRS-2672: Attempting to start 'ora.cssdmonitor' on 'rhel6m1' CRS-2676: Start of 'ora.cssdmonitor' on 'rhel6m1' succeeded CRS-2672: Attempting to start 'ora.cssd' on 'rhel6m1' CRS-2672: Attempting to start 'ora.diskmon' on 'rhel6m1' CRS-2676: Start of 'ora.diskmon' on 'rhel6m1' succeeded CRS-2676: Start of 'ora.cssd' on 'rhel6m1' succeeded CRS-2672: Attempting to start 'ora.cluster_interconnect.haip' on 'rhel6m1' CRS-2672: Attempting to start 'ora.ctssd' on 'rhel6m1' CRS-2676: Start of 'ora.ctssd' on 'rhel6m1' succeeded CRS-2676: Start of 'ora.cluster_interconnect.haip' on 'rhel6m1' succeeded CRS-2672: Attempting to start 'ora.asm' on 'rhel6m1' CRS-2676: Start of 'ora.asm' on 'rhel6m1' succeeded CRS-2672: Attempting to start 'ora.storage' on 'rhel6m1' CRS-2676: Start of 'ora.storage' on 'rhel6m1' succeeded CRS-2672: Attempting to start 'ora.crf' on 'rhel6m1' CRS-2676: Start of 'ora.crf' on 'rhel6m1' succeeded CRS-2672: Attempting to start 'ora.crsd' on 'rhel6m1' CRS-2676: Start of 'ora.crsd' on 'rhel6m1' succeeded CRS-6023: Starting Oracle Cluster Ready Services-managed resources CRS-6017: Processing resource auto-start for servers: rhel6m1 CRS-6016: Resource auto-start has completed for server rhel6m1 CRS-6024: Completed start of Oracle Cluster Ready Services-managed resources CRS-4123: Oracle High Availability Services has been started. Starting the Grid Infrastructure Management Repository database Using PFILE='/opt/app/12.1.0/grid2/dbs/init-dropmgmtdb.ora' Command output: SQL*Plus: Release 12.1.0.2.0 Production on Wed Apr 1 12:12:12 2015 Copyright (c) 1982, 2014, Oracle. All rights reserved. Connected to an idle instance. SQL> ORACLE instance started. Total System Global Area 788529152 bytes Fixed Size 2929352 bytes Variable Size 314576184 bytes Database Buffers 465567744 bytes Redo Buffers 5455872 bytes Database mounted. SQL> Disconnected from Oracle Database 12c Enterprise Edition Release 12.1.0.2.0 - 64bit Production With the Partitioning, Automatic Storage Management and Advanced Analytics options Starting the Grid Infrastructure Management Repository database succeeded Creating new pfile for the Management Repository database SQL*Plus: Release 12.1.0.2.0 Production on Wed Apr 1 12:12:25 2015 Copyright (c) 1982, 2014, Oracle. All rights reserved. Connected to: Oracle Database 12c Enterprise Edition Release 12.1.0.2.0 - 64bit Production With the Partitioning, Automatic Storage Management and Advanced Analytics options SQL> File created. SQL> Disconnected from Oracle Database 12c Enterprise Edition Release 12.1.0.2.0 - 64bit Production With the Partitioning, Automatic Storage Management and Advanced Analytics options Succeeded in creating PFILE '/opt/app/12.1.0/grid2/dbs/init-dropmgmtdbSAVED.ora' Dropping the Grid Infrastructure Management Repository database Dropping the Grid Infrastructure Management Repository database succeeded CRS-2791: Starting shutdown of Oracle High Availability Services-managed resources on 'rhel6m1' CRS-2673: Attempting to stop 'ora.crsd' on 'rhel6m1' CRS-2790: Starting shutdown of Cluster Ready Services-managed resources on 'rhel6m1' CRS-2673: Attempting to stop 'ora.CLUSTER_DG.dg' on 'rhel6m1' CRS-2677: Stop of 'ora.CLUSTER_DG.dg' on 'rhel6m1' succeeded CRS-2673: Attempting to stop 'ora.DATA.dg' on 'rhel6m1' CRS-2673: Attempting to stop 'ora.FLASH.dg' on 'rhel6m1' CRS-2677: Stop of 'ora.DATA.dg' on 'rhel6m1' succeeded CRS-2677: Stop of 'ora.FLASH.dg' on 'rhel6m1' succeeded CRS-2673: Attempting to stop 'ora.asm' on 'rhel6m1' CRS-2677: Stop of 'ora.asm' on 'rhel6m1' succeeded CRS-2792: Shutdown of Cluster Ready Services-managed resources on 'rhel6m1' has completed CRS-2677: Stop of 'ora.crsd' on 'rhel6m1' succeeded CRS-2673: Attempting to stop 'ora.evmd' on 'rhel6m1' CRS-2673: Attempting to stop 'ora.drivers.acfs' on 'rhel6m1' CRS-2673: Attempting to stop 'ora.ctssd' on 'rhel6m1' CRS-2673: Attempting to stop 'ora.storage' on 'rhel6m1' CRS-2673: Attempting to stop 'ora.mdnsd' on 'rhel6m1' CRS-2673: Attempting to stop 'ora.gpnpd' on 'rhel6m1' CRS-2677: Stop of 'ora.drivers.acfs' on 'rhel6m1' succeeded CRS-2677: Stop of 'ora.storage' on 'rhel6m1' succeeded CRS-2673: Attempting to stop 'ora.asm' on 'rhel6m1' CRS-2677: Stop of 'ora.evmd' on 'rhel6m1' succeeded CRS-2677: Stop of 'ora.ctssd' on 'rhel6m1' succeeded CRS-2677: Stop of 'ora.mdnsd' on 'rhel6m1' succeeded CRS-2677: Stop of 'ora.gpnpd' on 'rhel6m1' succeeded CRS-2677: Stop of 'ora.asm' on 'rhel6m1' succeeded CRS-2673: Attempting to stop 'ora.cluster_interconnect.haip' on 'rhel6m1' CRS-2677: Stop of 'ora.cluster_interconnect.haip' on 'rhel6m1' succeeded CRS-2673: Attempting to stop 'ora.cssd' on 'rhel6m1' CRS-2677: Stop of 'ora.cssd' on 'rhel6m1' succeeded CRS-2673: Attempting to stop 'ora.crf' on 'rhel6m1' CRS-2677: Stop of 'ora.crf' on 'rhel6m1' succeeded CRS-2673: Attempting to stop 'ora.gipcd' on 'rhel6m1' CRS-2677: Stop of 'ora.gipcd' on 'rhel6m1' succeeded CRS-2793: Shutdown of Oracle High Availability Services-managed resources on 'rhel6m1' has completed CRS-4133: Oracle High Availability Services has been stopped. CRS-4123: Oracle High Availability Services has been started. CRS-2672: Attempting to start 'ora.evmd' on 'rhel6m1' CRS-2672: Attempting to start 'ora.mdnsd' on 'rhel6m1' CRS-2676: Start of 'ora.mdnsd' on 'rhel6m1' succeeded CRS-2676: Start of 'ora.evmd' on 'rhel6m1' succeeded CRS-2672: Attempting to start 'ora.gpnpd' on 'rhel6m1' CRS-2676: Start of 'ora.gpnpd' on 'rhel6m1' succeeded CRS-2672: Attempting to start 'ora.cssdmonitor' on 'rhel6m1' CRS-2672: Attempting to start 'ora.gipcd' on 'rhel6m1' CRS-2676: Start of 'ora.cssdmonitor' on 'rhel6m1' succeeded CRS-2676: Start of 'ora.gipcd' on 'rhel6m1' succeeded CRS-2672: Attempting to start 'ora.cssd' on 'rhel6m1' CRS-2672: Attempting to start 'ora.diskmon' on 'rhel6m1' CRS-2676: Start of 'ora.diskmon' on 'rhel6m1' succeeded CRS-2676: Start of 'ora.cssd' on 'rhel6m1' succeeded CRS-2672: Attempting to start 'ora.crf' on 'rhel6m1' CRS-2672: Attempting to start 'ora.ctssd' on 'rhel6m1' CRS-2672: Attempting to start 'ora.cluster_interconnect.haip' on 'rhel6m1' CRS-2676: Start of 'ora.crf' on 'rhel6m1' succeeded CRS-2676: Start of 'ora.ctssd' on 'rhel6m1' succeeded CRS-2676: Start of 'ora.cluster_interconnect.haip' on 'rhel6m1' succeeded CRS-2672: Attempting to start 'ora.asm' on 'rhel6m1' CRS-2676: Start of 'ora.asm' on 'rhel6m1' succeeded CRS-2672: Attempting to start 'ora.storage' on 'rhel6m1' CRS-2676: Start of 'ora.storage' on 'rhel6m1' succeeded CRS-2672: Attempting to start 'ora.crsd' on 'rhel6m1' CRS-2676: Start of 'ora.crsd' on 'rhel6m1' succeeded CRS-2673: Attempting to stop 'ora.crsd' on 'rhel6m1' CRS-2677: Stop of 'ora.crsd' on 'rhel6m1' succeeded CRS-2791: Starting shutdown of Oracle High Availability Services-managed resources on 'rhel6m1' CRS-2673: Attempting to stop 'ora.ctssd' on 'rhel6m1' CRS-2673: Attempting to stop 'ora.evmd' on 'rhel6m1' CRS-2673: Attempting to stop 'ora.storage' on 'rhel6m1' CRS-2673: Attempting to stop 'ora.drivers.acfs' on 'rhel6m1' CRS-2673: Attempting to stop 'ora.gpnpd' on 'rhel6m1' CRS-2673: Attempting to stop 'ora.mdnsd' on 'rhel6m1' CRS-2677: Stop of 'ora.storage' on 'rhel6m1' succeeded CRS-2673: Attempting to stop 'ora.asm' on 'rhel6m1' CRS-2677: Stop of 'ora.drivers.acfs' on 'rhel6m1' succeeded CRS-2677: Stop of 'ora.ctssd' on 'rhel6m1' succeeded CRS-2677: Stop of 'ora.evmd' on 'rhel6m1' succeeded CRS-2677: Stop of 'ora.gpnpd' on 'rhel6m1' succeeded CRS-2677: Stop of 'ora.mdnsd' on 'rhel6m1' succeeded CRS-2677: Stop of 'ora.asm' on 'rhel6m1' succeeded CRS-2673: Attempting to stop 'ora.cluster_interconnect.haip' on 'rhel6m1' CRS-2677: Stop of 'ora.cluster_interconnect.haip' on 'rhel6m1' succeeded CRS-2673: Attempting to stop 'ora.cssd' on 'rhel6m1' CRS-2677: Stop of 'ora.cssd' on 'rhel6m1' succeeded CRS-2673: Attempting to stop 'ora.crf' on 'rhel6m1' CRS-2677: Stop of 'ora.crf' on 'rhel6m1' succeeded CRS-2673: Attempting to stop 'ora.gipcd' on 'rhel6m1' CRS-2677: Stop of 'ora.gipcd' on 'rhel6m1' succeeded CRS-2793: Shutdown of Oracle High Availability Services-managed resources on 'rhel6m1' has completed CRS-4133: Oracle High Availability Services has been stopped. CRS-4123: Oracle High Availability Services has been started. CRS-2672: Attempting to start 'ora.mdnsd' on 'rhel6m1' CRS-2676: Start of 'ora.mdnsd' on 'rhel6m1' succeeded CRS-2672: Attempting to start 'ora.gpnpd' on 'rhel6m1' CRS-2676: Start of 'ora.gpnpd' on 'rhel6m1' succeeded CRS-2672: Attempting to start 'ora.cssdmonitor' on 'rhel6m1' CRS-2672: Attempting to start 'ora.gipcd' on 'rhel6m1' CRS-2676: Start of 'ora.cssdmonitor' on 'rhel6m1' succeeded CRS-2676: Start of 'ora.gipcd' on 'rhel6m1' succeeded CRS-2672: Attempting to start 'ora.cssd' on 'rhel6m1' CRS-2672: Attempting to start 'ora.diskmon' on 'rhel6m1' CRS-2676: Start of 'ora.diskmon' on 'rhel6m1' succeeded CRS-2676: Start of 'ora.cssd' on 'rhel6m1' succeeded CRS-2672: Attempting to start 'ora.drivers.acfs' on 'rhel6m1' CRS-2679: Attempting to clean 'ora.cluster_interconnect.haip' on 'rhel6m1' CRS-2672: Attempting to start 'ora.ctssd' on 'rhel6m1' CRS-2681: Clean of 'ora.cluster_interconnect.haip' on 'rhel6m1' succeeded CRS-2672: Attempting to start 'ora.cluster_interconnect.haip' on 'rhel6m1' CRS-2674: Start of 'ora.drivers.acfs' on 'rhel6m1' failed CRS-2676: Start of 'ora.ctssd' on 'rhel6m1' succeeded CRS-2676: Start of 'ora.cluster_interconnect.haip' on 'rhel6m1' succeeded CRS-2672: Attempting to start 'ora.asm' on 'rhel6m1' CRS-2676: Start of 'ora.asm' on 'rhel6m1' succeeded 2015/04/01 12:15:49 CLSRSC-338: Successfully downgraded OCR to version 11.2.0.4.0 CRS-2672: Attempting to start 'ora.crsd' on 'rhel6m1' CRS-2676: Start of 'ora.crsd' on 'rhel6m1' succeeded 2015/04/01 12:20:01 CLSRSC-4001: Installing Oracle Trace File Analyzer (TFA) Collector. 2015/04/01 12:20:01 CLSRSC-4002: Successfully installed Oracle Trace File Analyzer (TFA) Collector. Successfully downgraded Oracle Clusterware stack on this node Run '/opt/app/11.2.0/grid4/bin/crsctl start crs' on all nodes to complete downgradeBefore starting the cluster with 11.2 update the inventory by setting crs=true for 11.2 GI home. At this point 12.1 GI home will have crs=true.
<HOME NAME="Ora11g_gridinfrahome2" LOC="/opt/app/11.2.0/grid4" TYPE="O" IDX="4">
<NODE_LIST>
<NODE NAME="rhel6m1"/>
<NODE NAME="rhel6m2"/>
</NODE_LIST>
</HOME>
<HOME NAME="OraGI12Home1" LOC="/opt/app/12.1.0/grid2" TYPE="O" IDX="5" CRS="true">
<NODE_LIST>
<NODE NAME="rhel6m1"/>
<NODE NAME="rhel6m2"/>
</NODE_LIST>
</HOME>Run the following to update the inventorycd /opt/app/12.1.0/grid2/oui/bin/ [grid@rhel6m1 bin]$ ./runInstaller -nowait -waitforcompletion -ignoreSysPrereqs -updateNodeList -silent CRS=false ORACLE_HOME=/opt/app/12.1.0/grid2 Starting Oracle Universal Installer... Checking swap space: must be greater than 500 MB. Actual 4095 MB Passed The inventory pointer is located at /etc/oraInst.loc 'UpdateNodeList' was successful. [grid@rhel6m1 bin]$ ./runInstaller -nowait -waitforcompletion -ignoreSysPrereqs -updateNodeList -silent CRS=true ORACLE_HOME=/opt/app/11.2.0/grid4 Starting Oracle Universal Installer... Checking swap space: must be greater than 500 MB. Actual 4095 MB Passed The inventory pointer is located at /etc/oraInst.loc 'UpdateNodeList' was successful.Verify the inventory update
<HOME NAME="Ora11g_gridinfrahome2" LOC="/opt/app/11.2.0/grid4" TYPE="O" IDX="4" CRS="true">
<NODE_LIST>
<NODE NAME="rhel6m1"/>
<NODE NAME="rhel6m2"/>
</NODE_LIST>
</HOME>
<HOME NAME="OraGI12Home1" LOC="/opt/app/12.1.0/grid2" TYPE="O" IDX="5">
<NODE_LIST>
<NODE NAME="rhel6m1"/>
<NODE NAME="rhel6m2"/>
</NODE_LIST>
</HOME>Also make sure the contents in /etc/init.d/ohasd and /etc/init.d/init.ohasd refers the 11.2 as the ORA_CRS_HOME and there are no references to 12.1. Few times the last downgrade node (rhel6m1) did contain references to 12.1 even though downgrade command had successfully completed. However these files on the other node had the correct references to 11.2.[root@rhel6m1 grid]# cat /etc/init.d/ohasd | grep ORA_CRS_HOME
ORA_CRS_HOME=/opt/app/11.2.0/grid4
export ORA_CRS_HOME
[root@rhel6m1 grid]# cat /etc/init.d/init.ohasd | grep ORA_CRS_HOME
ORA_CRS_HOME=/opt/app/11.2.0/grid4
export ORA_CRS_HOME
PERL="/opt/app/11.2.0/grid4/perl/bin/perl -I${ORA_CRS_HOME}/perl/lib"Also as part of the downgrade /etc/oratab is modified. All references to MGMTDB are moved and ASM instances homes are set to 11.2. But this only happens on last node (rhel6m1). For other nodes /etc/oratab did not have any ASM related entry. Manually add this ASM related entry to /etc/oratab+ASM2:/opt/app/11.2.0/grid4:NBefore starting the crs unset any references to 12.1 binaries from the environment (PATH, ORACLE_HOME variables). Make sure crsctl use to start the crs comes from 11.2 GI home.
[grid@rhel6m1 bin]$ . oraenv ORACLE_SID = [+ASM1] ? +ASM1 The Oracle base remains unchanged with value /opt/app/oracle [grid@rhel6m1 bin]$ which crsctl /opt/app/11.2.0/grid4/bin/crsctlStart the crs on all nodes.
# crsctl start crs
crsctl query crs activeversion -f
Oracle Clusterware active version on the cluster is [11.2.0.4.0]. The cluster upgrade state is [NORMAL].
crsctl query crs softwareversion
Oracle Clusterware version on node [rhel6m1] is [11.2.0.4.0]
[root@rhel6m1 grid]# ocrcheck
Status of Oracle Cluster Registry is as follows :
Version : 3
Total space (kbytes) : 262120
Used space (kbytes) : 3356
Available space (kbytes) : 258764
ID : 2072206343
Device/File Name : +CLUSTER_DGThis concludes the steps to successfully downgrading GI from 12.1.0.2 to 11.2.0.4.There could be occasions the downgrade command successfully completes but start of crs fails. Symptoms in this case include unable to discover any voting disks crsctl query css votedisk doesn't return any vote disk infomration and ocssd.log will have entries similar to
2015-03-31 14:00:09.618: [ CSSD][898090752]clssnmvDiskVerify: Successful discovery of 0 disks 2015-03-31 14:00:09.618: [ CSSD][898090752]clssnmCompleteInitVFDiscovery: Completing initial voting file discovery 2015-03-31 14:00:09.618: [ CSSD][898090752]clssnmvFindInitialConfigs: No voting files foundOther times includes corrupted OCR with ocssd.log having entries similar to
2015-03-26 11:33:05.633: [ CRSMAIN][3817551648] Initialing cluclu context... [ OCRMAS][3776734976]th_calc_av:8': Failed in vsnupr. Incorrect SV stored in OCR. Key [SYSTEM.version.hostnames.] Value [] 2015-03-26 11:33:06.618: [ OCRSRV][3776734976]th_upgrade:9 Shutdown CacheMaster. prev AV [186647552] new calc av [186647552] my sv [186647552]No root cause was found of these cases and could only assume this may be due to some of the earlier mentioned reasons such as having backups of OCR from previous upgrades i.e. ocr11.2.0.3.0 (But it must be said a successful downgrade was archived while ocr11.2.0.3.0 was in the cdata directory), wrong binaries referred during downgrade due to environment variable settings. The only option to recover from such a situation is to restore the OCR taken while cluster was 11.2.
crsctl stop crs -f # run on all nodes crsctl start crs -excl -nocrs # run only on one node ocrconfig -restore /opt/app/11.2.0/grid4/cdata/rhel6m-cluster/backup_20150331_170634.ocr crsctl replace votedisk +cluster_dgUseful metalink notes
rootcrs.pl -downgrade -lastnode" Fails if Current Directory Can not be Accessed by Grid User [ID 1917922.1]
SRVCTL commands fails with error CLSU-00100: Operating System function: sclsdgcwd failed with error data: -1 [ID 1961344.1]
Srvctl Status Commands Report CLSU-00100, CLSU-00101 and CLSU-00103 Errors [ID 1911258.1]
Related Posts
Downgrade Grid Infrastructure from 11.2.0.4 to 11.2.0.3
Downgrade Grid Infrastructure from 11.2.0.4 to 11.1.0.7
