Downgrade Grid Infrastructure from 12.2.0.1 to 12.1.0.2

This post list steps for downgrading grid infrastructure from 12.2.0.1 to 12.1.0.2. The grid infrastructure upgrade has been successful on all nodes. If upgrade to 12.2 had failed on any of the nodes then follow oracle documentation for additional steps/options required for such cases. Only the GI was upgraded to 12.2 and the database remained on 12.1.0.2. The current version of the cluster and nodes is as below

$ crsctl query crs activeversion
Oracle Clusterware active version on the cluster is [12.2.0.1.0]

$ crsctl query crs softwareversion -all
Oracle Clusterware version on node [rhel12c1] is [12.2.0.1.0]
Oracle Clusterware version on node [rhel12c2] is [12.2.0.1.0]

1. As the first step for downgrade, remove the MGMTDB.

[grid@rhel12c1 ~]$ dbca -silent -deleteDatabase -sourceDB -MGMTDB
Connecting to database
4% complete
9% complete
14% complete
19% complete
23% complete
28% complete
47% complete
Updating network configuration files
52% complete
Deleting instance and datafiles
76% complete
100% complete
Look at the log file "/opt/app/oracle/cfgtoollogs/dbca/_mgmtdb.log" for further details.

2. Run rootcrs.sh with downgrade option from any of the nodes

[root@rhel12c1 grid]# ./crs/install/rootcrs.sh -downgrade
Using configuration parameter file: /opt/app/12.2.0/grid/crs/install/crsconfig_params
The log of current session can be found at:
  /opt/app/oracle/crsdata/rhel12c1/crsconfig/crsdowngrade_rhel12c1_2017-09-25_04-26-42PM.log
CRS-2791: Starting shutdown of Oracle High Availability Services-managed resources on 'rhel12c1'
CRS-2673: Attempting to stop 'ora.crsd' on 'rhel12c1'
CRS-2790: Starting shutdown of Cluster Ready Services-managed resources on server 'rhel12c1'
CRS-2673: Attempting to stop 'ora.gns' on 'rhel12c1'
CRS-2673: Attempting to stop 'ora.MGMTLSNR' on 'rhel12c1'
CRS-2673: Attempting to stop 'ora.std12c1.tbx.svc' on 'rhel12c1'
CRS-2673: Attempting to stop 'ora.qosmserver' on 'rhel12c1'
CRS-2677: Stop of 'ora.std12c1.tbx.svc' on 'rhel12c1' succeeded
CRS-2673: Attempting to stop 'ora.std12c1.db' on 'rhel12c1'
CRS-2677: Stop of 'ora.gns' on 'rhel12c1' succeeded
CRS-2677: Stop of 'ora.std12c1.db' on 'rhel12c1' succeeded
CRS-2673: Attempting to stop 'ora.CLUSFS.dg' on 'rhel12c1'
CRS-2673: Attempting to stop 'ora.GIMR.dg' on 'rhel12c1'
CRS-2673: Attempting to stop 'ora.DATA.dg' on 'rhel12c1'
CRS-2673: Attempting to stop 'ora.FRA.dg' on 'rhel12c1'
CRS-2673: Attempting to stop 'ora.LISTENER.lsnr' on 'rhel12c1'
CRS-2673: Attempting to stop 'ora.LISTENER_SCAN2.lsnr' on 'rhel12c1'
CRS-2673: Attempting to stop 'ora.LISTENER_SCAN3.lsnr' on 'rhel12c1'
CRS-2673: Attempting to stop 'ora.cvu' on 'rhel12c1'
CRS-2673: Attempting to stop 'ora.gns.vip' on 'rhel12c1'
CRS-2677: Stop of 'ora.FRA.dg' on 'rhel12c1' succeeded
CRS-2677: Stop of 'ora.DATA.dg' on 'rhel12c1' succeeded
CRS-2677: Stop of 'ora.GIMR.dg' on 'rhel12c1' succeeded
CRS-2677: Stop of 'ora.CLUSFS.dg' on 'rhel12c1' succeeded
CRS-2673: Attempting to stop 'ora.asm' on 'rhel12c1'
CRS-2677: Stop of 'ora.MGMTLSNR' on 'rhel12c1' succeeded
CRS-2677: Stop of 'ora.LISTENER_SCAN3.lsnr' on 'rhel12c1' succeeded
CRS-2673: Attempting to stop 'ora.scan3.vip' on 'rhel12c1'
CRS-2677: Stop of 'ora.LISTENER_SCAN2.lsnr' on 'rhel12c1' succeeded
CRS-2673: Attempting to stop 'ora.scan2.vip' on 'rhel12c1'
CRS-2677: Stop of 'ora.LISTENER.lsnr' on 'rhel12c1' succeeded
CRS-2673: Attempting to stop 'ora.rhel12c1.vip' on 'rhel12c1'
CRS-2677: Stop of 'ora.cvu' on 'rhel12c1' succeeded
CRS-2677: Stop of 'ora.asm' on 'rhel12c1' succeeded
CRS-2673: Attempting to stop 'ora.ASMNET1LSNR_ASM.lsnr' on 'rhel12c1'
CRS-2677: Stop of 'ora.gns.vip' on 'rhel12c1' succeeded
CRS-2677: Stop of 'ora.qosmserver' on 'rhel12c1' succeeded
CRS-2677: Stop of 'ora.scan2.vip' on 'rhel12c1' succeeded
CRS-2677: Stop of 'ora.scan3.vip' on 'rhel12c1' succeeded
CRS-2677: Stop of 'ora.rhel12c1.vip' on 'rhel12c1' succeeded
CRS-2677: Stop of 'ora.ASMNET1LSNR_ASM.lsnr' on 'rhel12c1' succeeded
CRS-2672: Attempting to start 'ora.MGMTLSNR' on 'rhel12c2'
CRS-2672: Attempting to start 'ora.qosmserver' on 'rhel12c2'
CRS-2672: Attempting to start 'ora.gns.vip' on 'rhel12c2'
CRS-2672: Attempting to start 'ora.cvu' on 'rhel12c2'
CRS-2676: Start of 'ora.cvu' on 'rhel12c2' succeeded
CRS-2676: Start of 'ora.gns.vip' on 'rhel12c2' succeeded
CRS-2672: Attempting to start 'ora.gns' on 'rhel12c2'
CRS-2676: Start of 'ora.MGMTLSNR' on 'rhel12c2' succeeded
CRS-2676: Start of 'ora.gns' on 'rhel12c2' succeeded
CRS-2672: Attempting to start 'ora.scan2.vip' on 'rhel12c2'
CRS-2672: Attempting to start 'ora.scan3.vip' on 'rhel12c2'
CRS-2672: Attempting to start 'ora.rhel12c1.vip' on 'rhel12c2'
CRS-2676: Start of 'ora.qosmserver' on 'rhel12c2' succeeded
CRS-2676: Start of 'ora.scan2.vip' on 'rhel12c2' succeeded
CRS-2672: Attempting to start 'ora.LISTENER_SCAN2.lsnr' on 'rhel12c2'
CRS-2676: Start of 'ora.scan3.vip' on 'rhel12c2' succeeded
CRS-2672: Attempting to start 'ora.LISTENER_SCAN3.lsnr' on 'rhel12c2'
CRS-2676: Start of 'ora.rhel12c1.vip' on 'rhel12c2' succeeded
CRS-2676: Start of 'ora.LISTENER_SCAN2.lsnr' on 'rhel12c2' succeeded
CRS-2676: Start of 'ora.LISTENER_SCAN3.lsnr' on 'rhel12c2' succeeded
CRS-2673: Attempting to stop 'ora.ons' on 'rhel12c1'
CRS-2677: Stop of 'ora.ons' on 'rhel12c1' succeeded
CRS-2673: Attempting to stop 'ora.net1.network' on 'rhel12c1'
CRS-2677: Stop of 'ora.net1.network' on 'rhel12c1' succeeded
CRS-2792: Shutdown of Cluster Ready Services-managed resources on 'rhel12c1' has completed
CRS-2677: Stop of 'ora.crsd' on 'rhel12c1' succeeded
CRS-2673: Attempting to stop 'ora.asm' on 'rhel12c1'
CRS-2673: Attempting to stop 'ora.crf' on 'rhel12c1'
CRS-2673: Attempting to stop 'ora.drivers.acfs' on 'rhel12c1'
CRS-2673: Attempting to stop 'ora.gpnpd' on 'rhel12c1'
CRS-2673: Attempting to stop 'ora.mdnsd' on 'rhel12c1'
CRS-2677: Stop of 'ora.drivers.acfs' on 'rhel12c1' succeeded
CRS-2677: Stop of 'ora.gpnpd' on 'rhel12c1' succeeded
CRS-2677: Stop of 'ora.crf' on 'rhel12c1' succeeded
CRS-2677: Stop of 'ora.mdnsd' on 'rhel12c1' succeeded
CRS-2677: Stop of 'ora.asm' on 'rhel12c1' succeeded
CRS-2673: Attempting to stop 'ora.cluster_interconnect.haip' on 'rhel12c1'
CRS-2677: Stop of 'ora.cluster_interconnect.haip' on 'rhel12c1' succeeded
CRS-2673: Attempting to stop 'ora.ctssd' on 'rhel12c1'
CRS-2673: Attempting to stop 'ora.evmd' on 'rhel12c1'
CRS-2677: Stop of 'ora.ctssd' on 'rhel12c1' succeeded
CRS-2677: Stop of 'ora.evmd' on 'rhel12c1' succeeded
CRS-2673: Attempting to stop 'ora.cssd' on 'rhel12c1'
CRS-2677: Stop of 'ora.cssd' on 'rhel12c1' succeeded
CRS-2673: Attempting to stop 'ora.gipcd' on 'rhel12c1'
CRS-2677: Stop of 'ora.gipcd' on 'rhel12c1' succeeded
CRS-2793: Shutdown of Oracle High Availability Services-managed resources on 'rhel12c1' has completed
CRS-4133: Oracle High Availability Services has been stopped.
2017/09/25 16:28:17 CLSRSC-4006: Removing Oracle Trace File Analyzer (TFA) Collector.
2017/09/25 16:28:43 CLSRSC-4007: Successfully removed Oracle Trace File Analyzer (TFA) Collector.
2017/09/25 16:28:43 CLSRSC-4001: Installing Oracle Trace File Analyzer (TFA) Collector.
2017/09/25 16:29:21 CLSRSC-4002: Successfully installed Oracle Trace File Analyzer (TFA) Collector.
2017/09/25 16:29:21 CLSRSC-591: successfully downgraded Oracle Clusterware stack on this node

3. Run the downgrade from the last node.

[root@rhel12c2 grid]# ./crs/install/rootcrs.sh -downgrade
Using configuration parameter file: /opt/app/12.2.0/grid/crs/install/crsconfig_params
The log of current session can be found at:
  /opt/app/oracle/crsdata/rhel12c2/crsconfig/crsdowngrade_rhel12c2_2017-09-25_04-30-45PM.log
CRS-2791: Starting shutdown of Oracle High Availability Services-managed resources on 'rhel12c2'
CRS-2673: Attempting to stop 'ora.crsd' on 'rhel12c2'
CRS-2790: Starting shutdown of Cluster Ready Services-managed resources on server 'rhel12c2'
CRS-2673: Attempting to stop 'ora.std12c1.tbx.svc' on 'rhel12c2'
CRS-2673: Attempting to stop 'ora.qosmserver' on 'rhel12c2'
CRS-2673: Attempting to stop 'ora.gns' on 'rhel12c2'
CRS-2673: Attempting to stop 'ora.MGMTLSNR' on 'rhel12c2'
CRS-2677: Stop of 'ora.std12c1.tbx.svc' on 'rhel12c2' succeeded
CRS-2673: Attempting to stop 'ora.std12c1.db' on 'rhel12c2'
CRS-2677: Stop of 'ora.gns' on 'rhel12c2' succeeded
CRS-2677: Stop of 'ora.std12c1.db' on 'rhel12c2' succeeded
CRS-2673: Attempting to stop 'ora.CLUSFS.dg' on 'rhel12c2'
CRS-2673: Attempting to stop 'ora.GIMR.dg' on 'rhel12c2'
CRS-2673: Attempting to stop 'ora.DATA.dg' on 'rhel12c2'
CRS-2673: Attempting to stop 'ora.FRA.dg' on 'rhel12c2'
CRS-2673: Attempting to stop 'ora.LISTENER.lsnr' on 'rhel12c2'
CRS-2673: Attempting to stop 'ora.LISTENER_SCAN1.lsnr' on 'rhel12c2'
CRS-2673: Attempting to stop 'ora.LISTENER_SCAN2.lsnr' on 'rhel12c2'
CRS-2673: Attempting to stop 'ora.LISTENER_SCAN3.lsnr' on 'rhel12c2'
CRS-2673: Attempting to stop 'ora.cvu' on 'rhel12c2'
CRS-2673: Attempting to stop 'ora.gns.vip' on 'rhel12c2'
CRS-2673: Attempting to stop 'ora.rhel12c1.vip' on 'rhel12c2'
CRS-2677: Stop of 'ora.GIMR.dg' on 'rhel12c2' succeeded
CRS-2677: Stop of 'ora.DATA.dg' on 'rhel12c2' succeeded
CRS-2677: Stop of 'ora.CLUSFS.dg' on 'rhel12c2' succeeded
CRS-2677: Stop of 'ora.LISTENER.lsnr' on 'rhel12c2' succeeded
CRS-2677: Stop of 'ora.LISTENER_SCAN2.lsnr' on 'rhel12c2' succeeded
CRS-2673: Attempting to stop 'ora.scan2.vip' on 'rhel12c2'
CRS-2677: Stop of 'ora.LISTENER_SCAN3.lsnr' on 'rhel12c2' succeeded
CRS-2677: Stop of 'ora.LISTENER_SCAN1.lsnr' on 'rhel12c2' succeeded
CRS-2673: Attempting to stop 'ora.scan1.vip' on 'rhel12c2'
CRS-2673: Attempting to stop 'ora.scan3.vip' on 'rhel12c2'
CRS-2677: Stop of 'ora.FRA.dg' on 'rhel12c2' succeeded
CRS-2673: Attempting to stop 'ora.asm' on 'rhel12c2'
CRS-2677: Stop of 'ora.qosmserver' on 'rhel12c2' succeeded
CRS-2677: Stop of 'ora.cvu' on 'rhel12c2' succeeded
CRS-2677: Stop of 'ora.asm' on 'rhel12c2' succeeded
CRS-2673: Attempting to stop 'ora.ASMNET1LSNR_ASM.lsnr' on 'rhel12c2'
CRS-2677: Stop of 'ora.rhel12c1.vip' on 'rhel12c2' succeeded
CRS-2673: Attempting to stop 'ora.rhel12c2.vip' on 'rhel12c2'
CRS-2677: Stop of 'ora.gns.vip' on 'rhel12c2' succeeded
CRS-2677: Stop of 'ora.MGMTLSNR' on 'rhel12c2' succeeded
CRS-2677: Stop of 'ora.scan1.vip' on 'rhel12c2' succeeded
CRS-2677: Stop of 'ora.scan3.vip' on 'rhel12c2' succeeded
CRS-2677: Stop of 'ora.scan2.vip' on 'rhel12c2' succeeded
CRS-2677: Stop of 'ora.rhel12c2.vip' on 'rhel12c2' succeeded
CRS-2677: Stop of 'ora.ASMNET1LSNR_ASM.lsnr' on 'rhel12c2' succeeded
CRS-2673: Attempting to stop 'ora.ons' on 'rhel12c2'
CRS-2677: Stop of 'ora.ons' on 'rhel12c2' succeeded
CRS-2673: Attempting to stop 'ora.net1.network' on 'rhel12c2'
CRS-2677: Stop of 'ora.net1.network' on 'rhel12c2' succeeded
CRS-2792: Shutdown of Cluster Ready Services-managed resources on 'rhel12c2' has completed
CRS-2677: Stop of 'ora.crsd' on 'rhel12c2' succeeded
CRS-2673: Attempting to stop 'ora.storage' on 'rhel12c2'
CRS-2673: Attempting to stop 'ora.crf' on 'rhel12c2'
CRS-2673: Attempting to stop 'ora.drivers.acfs' on 'rhel12c2'
CRS-2673: Attempting to stop 'ora.gpnpd' on 'rhel12c2'
CRS-2673: Attempting to stop 'ora.mdnsd' on 'rhel12c2'
CRS-2677: Stop of 'ora.drivers.acfs' on 'rhel12c2' succeeded
CRS-2677: Stop of 'ora.crf' on 'rhel12c2' succeeded
CRS-2677: Stop of 'ora.gpnpd' on 'rhel12c2' succeeded
CRS-2677: Stop of 'ora.storage' on 'rhel12c2' succeeded
CRS-2673: Attempting to stop 'ora.asm' on 'rhel12c2'
CRS-2677: Stop of 'ora.mdnsd' on 'rhel12c2' succeeded
CRS-2677: Stop of 'ora.asm' on 'rhel12c2' succeeded
CRS-2673: Attempting to stop 'ora.cluster_interconnect.haip' on 'rhel12c2'
CRS-2677: Stop of 'ora.cluster_interconnect.haip' on 'rhel12c2' succeeded
CRS-2673: Attempting to stop 'ora.ctssd' on 'rhel12c2'
CRS-2673: Attempting to stop 'ora.evmd' on 'rhel12c2'
CRS-2677: Stop of 'ora.ctssd' on 'rhel12c2' succeeded
CRS-2677: Stop of 'ora.evmd' on 'rhel12c2' succeeded
CRS-2673: Attempting to stop 'ora.cssd' on 'rhel12c2'
CRS-2677: Stop of 'ora.cssd' on 'rhel12c2' succeeded
CRS-2673: Attempting to stop 'ora.gipcd' on 'rhel12c2'
CRS-2677: Stop of 'ora.gipcd' on 'rhel12c2' succeeded
CRS-2793: Shutdown of Oracle High Availability Services-managed resources on 'rhel12c2' has completed
CRS-4133: Oracle High Availability Services has been stopped.
CRS-2791: Starting shutdown of Oracle High Availability Services-managed resources on 'rhel12c2'
CRS-2673: Attempting to stop 'ora.crsd' on 'rhel12c2'
CRS-2677: Stop of 'ora.crsd' on 'rhel12c2' succeeded
CRS-2673: Attempting to stop 'ora.storage' on 'rhel12c2'
CRS-2673: Attempting to stop 'ora.crf' on 'rhel12c2'
CRS-2673: Attempting to stop 'ora.drivers.acfs' on 'rhel12c2'
CRS-2673: Attempting to stop 'ora.gpnpd' on 'rhel12c2'
CRS-2673: Attempting to stop 'ora.mdnsd' on 'rhel12c2'
CRS-2677: Stop of 'ora.drivers.acfs' on 'rhel12c2' succeeded
CRS-2677: Stop of 'ora.crf' on 'rhel12c2' succeeded
CRS-2677: Stop of 'ora.gpnpd' on 'rhel12c2' succeeded
CRS-2677: Stop of 'ora.storage' on 'rhel12c2' succeeded
CRS-2673: Attempting to stop 'ora.asm' on 'rhel12c2'
CRS-2677: Stop of 'ora.mdnsd' on 'rhel12c2' succeeded
CRS-2677: Stop of 'ora.asm' on 'rhel12c2' succeeded
CRS-2673: Attempting to stop 'ora.cluster_interconnect.haip' on 'rhel12c2'
CRS-2677: Stop of 'ora.cluster_interconnect.haip' on 'rhel12c2' succeeded
CRS-2673: Attempting to stop 'ora.ctssd' on 'rhel12c2'
CRS-2673: Attempting to stop 'ora.evmd' on 'rhel12c2'
CRS-2677: Stop of 'ora.ctssd' on 'rhel12c2' succeeded
CRS-2677: Stop of 'ora.evmd' on 'rhel12c2' succeeded
CRS-2673: Attempting to stop 'ora.cssd' on 'rhel12c2'
CRS-2677: Stop of 'ora.cssd' on 'rhel12c2' succeeded
CRS-2673: Attempting to stop 'ora.gipcd' on 'rhel12c2'
CRS-2677: Stop of 'ora.gipcd' on 'rhel12c2' succeeded
CRS-2793: Shutdown of Oracle High Availability Services-managed resources on 'rhel12c2' has completed
CRS-4133: Oracle High Availability Services has been stopped.
CRS-4123: Starting Oracle High Availability Services-managed resources
CRS-2672: Attempting to start 'ora.mdnsd' on 'rhel12c2'
CRS-2672: Attempting to start 'ora.evmd' on 'rhel12c2'
CRS-2676: Start of 'ora.mdnsd' on 'rhel12c2' succeeded
CRS-2676: Start of 'ora.evmd' on 'rhel12c2' succeeded
CRS-2672: Attempting to start 'ora.gpnpd' on 'rhel12c2'
CRS-2676: Start of 'ora.gpnpd' on 'rhel12c2' succeeded
CRS-2672: Attempting to start 'ora.gipcd' on 'rhel12c2'
CRS-2676: Start of 'ora.gipcd' on 'rhel12c2' succeeded
CRS-2672: Attempting to start 'ora.cssdmonitor' on 'rhel12c2'
CRS-2676: Start of 'ora.cssdmonitor' on 'rhel12c2' succeeded
CRS-2672: Attempting to start 'ora.cssd' on 'rhel12c2'
CRS-2672: Attempting to start 'ora.diskmon' on 'rhel12c2'
CRS-2676: Start of 'ora.diskmon' on 'rhel12c2' succeeded
CRS-2676: Start of 'ora.cssd' on 'rhel12c2' succeeded
CRS-2672: Attempting to start 'ora.cluster_interconnect.haip' on 'rhel12c2'
CRS-2672: Attempting to start 'ora.ctssd' on 'rhel12c2'
CRS-2676: Start of 'ora.ctssd' on 'rhel12c2' succeeded
CRS-2676: Start of 'ora.cluster_interconnect.haip' on 'rhel12c2' succeeded
CRS-2672: Attempting to start 'ora.asm' on 'rhel12c2'
CRS-2676: Start of 'ora.asm' on 'rhel12c2' succeeded
CRS-2672: Attempting to start 'ora.storage' on 'rhel12c2'
CRS-2676: Start of 'ora.storage' on 'rhel12c2' succeeded
CRS-2672: Attempting to start 'ora.crf' on 'rhel12c2'
CRS-2676: Start of 'ora.crf' on 'rhel12c2' succeeded
CRS-2672: Attempting to start 'ora.crsd' on 'rhel12c2'
CRS-2676: Start of 'ora.crsd' on 'rhel12c2' succeeded
CRS-6023: Starting Oracle Cluster Ready Services-managed resources
CRS-2664: Resource 'ora.CLUSFS.dg' is already running on 'rhel12c2'
CRS-2664: Resource 'ora.DATA.dg' is already running on 'rhel12c2'
CRS-2664: Resource 'ora.FRA.dg' is already running on 'rhel12c2'
CRS-2664: Resource 'ora.GIMR.dg' is already running on 'rhel12c2'
CRS-6017: Processing resource auto-start for servers: rhel12c2
CRS-2672: Attempting to start 'ora.ons' on 'rhel12c2'
CRS-2672: Attempting to start 'ora.cvu' on 'rhel12c2'
CRS-2672: Attempting to start 'ora.qosmserver' on 'rhel12c2'
CRS-2672: Attempting to start 'ora.MGMTLSNR' on 'rhel12c2'
CRS-2676: Start of 'ora.cvu' on 'rhel12c2' succeeded
CRS-2676: Start of 'ora.MGMTLSNR' on 'rhel12c2' succeeded
CRS-2676: Start of 'ora.ons' on 'rhel12c2' succeeded
CRS-2672: Attempting to start 'ora.scan2.vip' on 'rhel12c2'
CRS-2672: Attempting to start 'ora.rhel12c2.vip' on 'rhel12c2'
CRS-2672: Attempting to start 'ora.scan1.vip' on 'rhel12c2'
CRS-2672: Attempting to start 'ora.scan3.vip' on 'rhel12c2'
CRS-2672: Attempting to start 'ora.rhel12c1.vip' on 'rhel12c2'
CRS-2676: Start of 'ora.scan2.vip' on 'rhel12c2' succeeded
CRS-2672: Attempting to start 'ora.LISTENER_SCAN2.lsnr' on 'rhel12c2'
CRS-2676: Start of 'ora.qosmserver' on 'rhel12c2' succeeded
CRS-2676: Start of 'ora.rhel12c2.vip' on 'rhel12c2' succeeded
CRS-2672: Attempting to start 'ora.LISTENER.lsnr' on 'rhel12c2'
CRS-2676: Start of 'ora.scan1.vip' on 'rhel12c2' succeeded
CRS-2672: Attempting to start 'ora.LISTENER_SCAN1.lsnr' on 'rhel12c2'
CRS-2676: Start of 'ora.scan3.vip' on 'rhel12c2' succeeded
CRS-2672: Attempting to start 'ora.LISTENER_SCAN3.lsnr' on 'rhel12c2'
CRS-2676: Start of 'ora.rhel12c1.vip' on 'rhel12c2' succeeded
CRS-2676: Start of 'ora.LISTENER_SCAN2.lsnr' on 'rhel12c2' succeeded
CRS-2676: Start of 'ora.LISTENER.lsnr' on 'rhel12c2' succeeded
CRS-2676: Start of 'ora.LISTENER_SCAN1.lsnr' on 'rhel12c2' succeeded
CRS-2672: Attempting to start 'ora.std12c1.db' on 'rhel12c2'
CRS-2676: Start of 'ora.LISTENER_SCAN3.lsnr' on 'rhel12c2' succeeded
CRS-2676: Start of 'ora.std12c1.db' on 'rhel12c2' succeeded
CRS-2672: Attempting to start 'ora.std12c1.tbx.svc' on 'rhel12c2'
CRS-2676: Start of 'ora.std12c1.tbx.svc' on 'rhel12c2' succeeded
CRS-6016: Resource auto-start has completed for server rhel12c2
CRS-6024: Completed start of Oracle Cluster Ready Services-managed resources
CRS-4123: Oracle High Availability Services has been started.
CRS-2791: Starting shutdown of Oracle High Availability Services-managed resources on 'rhel12c2'
CRS-2673: Attempting to stop 'ora.crsd' on 'rhel12c2'
CRS-2790: Starting shutdown of Cluster Ready Services-managed resources on server 'rhel12c2'
CRS-2673: Attempting to stop 'ora.qosmserver' on 'rhel12c2'
CRS-2673: Attempting to stop 'ora.gns' on 'rhel12c2'
CRS-2673: Attempting to stop 'ora.MGMTLSNR' on 'rhel12c2'
CRS-2673: Attempting to stop 'ora.std12c1.tbx.svc' on 'rhel12c2'
CRS-2677: Stop of 'ora.std12c1.tbx.svc' on 'rhel12c2' succeeded
CRS-2673: Attempting to stop 'ora.std12c1.db' on 'rhel12c2'
CRS-2677: Stop of 'ora.gns' on 'rhel12c2' succeeded
CRS-2677: Stop of 'ora.std12c1.db' on 'rhel12c2' succeeded
CRS-2673: Attempting to stop 'ora.CLUSFS.dg' on 'rhel12c2'
CRS-2673: Attempting to stop 'ora.GIMR.dg' on 'rhel12c2'
CRS-2673: Attempting to stop 'ora.DATA.dg' on 'rhel12c2'
CRS-2673: Attempting to stop 'ora.FRA.dg' on 'rhel12c2'
CRS-2673: Attempting to stop 'ora.LISTENER.lsnr' on 'rhel12c2'
CRS-2673: Attempting to stop 'ora.LISTENER_SCAN1.lsnr' on 'rhel12c2'
CRS-2673: Attempting to stop 'ora.LISTENER_SCAN2.lsnr' on 'rhel12c2'
CRS-2673: Attempting to stop 'ora.LISTENER_SCAN3.lsnr' on 'rhel12c2'
CRS-2673: Attempting to stop 'ora.cvu' on 'rhel12c2'
CRS-2673: Attempting to stop 'ora.gns.vip' on 'rhel12c2'
CRS-2673: Attempting to stop 'ora.rhel12c1.vip' on 'rhel12c2'
CRS-2677: Stop of 'ora.CLUSFS.dg' on 'rhel12c2' succeeded
CRS-2677: Stop of 'ora.DATA.dg' on 'rhel12c2' succeeded
CRS-2677: Stop of 'ora.FRA.dg' on 'rhel12c2' succeeded
CRS-2677: Stop of 'ora.GIMR.dg' on 'rhel12c2' succeeded
CRS-2673: Attempting to stop 'ora.asm' on 'rhel12c2'
CRS-2677: Stop of 'ora.LISTENER.lsnr' on 'rhel12c2' succeeded
CRS-2677: Stop of 'ora.LISTENER_SCAN1.lsnr' on 'rhel12c2' succeeded
CRS-2673: Attempting to stop 'ora.scan1.vip' on 'rhel12c2'
CRS-2677: Stop of 'ora.LISTENER_SCAN2.lsnr' on 'rhel12c2' succeeded
CRS-2677: Stop of 'ora.LISTENER_SCAN3.lsnr' on 'rhel12c2' succeeded
CRS-2673: Attempting to stop 'ora.scan2.vip' on 'rhel12c2'
CRS-2673: Attempting to stop 'ora.scan3.vip' on 'rhel12c2'
CRS-2677: Stop of 'ora.asm' on 'rhel12c2' succeeded
CRS-2673: Attempting to stop 'ora.ASMNET1LSNR_ASM.lsnr' on 'rhel12c2'
CRS-2677: Stop of 'ora.qosmserver' on 'rhel12c2' succeeded
CRS-2677: Stop of 'ora.gns.vip' on 'rhel12c2' succeeded
CRS-2677: Stop of 'ora.cvu' on 'rhel12c2' succeeded
CRS-2677: Stop of 'ora.MGMTLSNR' on 'rhel12c2' succeeded
CRS-2677: Stop of 'ora.ASMNET1LSNR_ASM.lsnr' on 'rhel12c2' succeeded
CRS-2677: Stop of 'ora.rhel12c1.vip' on 'rhel12c2' succeeded
CRS-2673: Attempting to stop 'ora.rhel12c2.vip' on 'rhel12c2'
CRS-2677: Stop of 'ora.scan3.vip' on 'rhel12c2' succeeded
CRS-2677: Stop of 'ora.scan1.vip' on 'rhel12c2' succeeded
CRS-2677: Stop of 'ora.scan2.vip' on 'rhel12c2' succeeded
CRS-2677: Stop of 'ora.rhel12c2.vip' on 'rhel12c2' succeeded
CRS-2673: Attempting to stop 'ora.ons' on 'rhel12c2'
CRS-2677: Stop of 'ora.ons' on 'rhel12c2' succeeded
CRS-2673: Attempting to stop 'ora.net1.network' on 'rhel12c2'
CRS-2677: Stop of 'ora.net1.network' on 'rhel12c2' succeeded
CRS-2792: Shutdown of Cluster Ready Services-managed resources on 'rhel12c2' has completed
CRS-2677: Stop of 'ora.crsd' on 'rhel12c2' succeeded
CRS-2673: Attempting to stop 'ora.asm' on 'rhel12c2'
CRS-2673: Attempting to stop 'ora.crf' on 'rhel12c2'
CRS-2673: Attempting to stop 'ora.drivers.acfs' on 'rhel12c2'
CRS-2673: Attempting to stop 'ora.gpnpd' on 'rhel12c2'
CRS-2673: Attempting to stop 'ora.mdnsd' on 'rhel12c2'
CRS-2677: Stop of 'ora.drivers.acfs' on 'rhel12c2' succeeded
CRS-2677: Stop of 'ora.gpnpd' on 'rhel12c2' succeeded
CRS-2677: Stop of 'ora.crf' on 'rhel12c2' succeeded
CRS-2677: Stop of 'ora.mdnsd' on 'rhel12c2' succeeded
CRS-2677: Stop of 'ora.asm' on 'rhel12c2' succeeded
CRS-2673: Attempting to stop 'ora.cluster_interconnect.haip' on 'rhel12c2'
CRS-2677: Stop of 'ora.cluster_interconnect.haip' on 'rhel12c2' succeeded
CRS-2673: Attempting to stop 'ora.ctssd' on 'rhel12c2'
CRS-2673: Attempting to stop 'ora.evmd' on 'rhel12c2'
CRS-2677: Stop of 'ora.ctssd' on 'rhel12c2' succeeded
CRS-2677: Stop of 'ora.evmd' on 'rhel12c2' succeeded
CRS-2673: Attempting to stop 'ora.cssd' on 'rhel12c2'
CRS-2677: Stop of 'ora.cssd' on 'rhel12c2' succeeded
CRS-2673: Attempting to stop 'ora.gipcd' on 'rhel12c2'
CRS-2677: Stop of 'ora.gipcd' on 'rhel12c2' succeeded
CRS-2793: Shutdown of Oracle High Availability Services-managed resources on 'rhel12c2' has completed
CRS-4133: Oracle High Availability Services has been stopped.
CRS-4123: Oracle High Availability Services has been started.
CRS-2672: Attempting to start 'ora.evmd' on 'rhel12c2'
CRS-2672: Attempting to start 'ora.mdnsd' on 'rhel12c2'
CRS-2676: Start of 'ora.mdnsd' on 'rhel12c2' succeeded
CRS-2676: Start of 'ora.evmd' on 'rhel12c2' succeeded
CRS-2672: Attempting to start 'ora.gpnpd' on 'rhel12c2'
CRS-2676: Start of 'ora.gpnpd' on 'rhel12c2' succeeded
CRS-2672: Attempting to start 'ora.cssdmonitor' on 'rhel12c2'
CRS-2672: Attempting to start 'ora.gipcd' on 'rhel12c2'
CRS-2676: Start of 'ora.cssdmonitor' on 'rhel12c2' succeeded
CRS-2676: Start of 'ora.gipcd' on 'rhel12c2' succeeded
CRS-2672: Attempting to start 'ora.cssd' on 'rhel12c2'
CRS-2672: Attempting to start 'ora.diskmon' on 'rhel12c2'
CRS-2676: Start of 'ora.diskmon' on 'rhel12c2' succeeded
CRS-2676: Start of 'ora.cssd' on 'rhel12c2' succeeded
CRS-2672: Attempting to start 'ora.crf' on 'rhel12c2'
CRS-2672: Attempting to start 'ora.ctssd' on 'rhel12c2'
CRS-2672: Attempting to start 'ora.cluster_interconnect.haip' on 'rhel12c2'
CRS-2676: Start of 'ora.crf' on 'rhel12c2' succeeded
CRS-2676: Start of 'ora.ctssd' on 'rhel12c2' succeeded
CRS-2676: Start of 'ora.cluster_interconnect.haip' on 'rhel12c2' succeeded
CRS-2672: Attempting to start 'ora.asm' on 'rhel12c2'
CRS-2676: Start of 'ora.asm' on 'rhel12c2' succeeded
CRS-2672: Attempting to start 'ora.storage' on 'rhel12c2'
CRS-2676: Start of 'ora.storage' on 'rhel12c2' succeeded
CRS-2672: Attempting to start 'ora.crsd' on 'rhel12c2'
CRS-2676: Start of 'ora.crsd' on 'rhel12c2' succeeded
CRS-2791: Starting shutdown of Oracle High Availability Services-managed resources on 'rhel12c2'
CRS-2673: Attempting to stop 'ora.crsd' on 'rhel12c2'
CRS-2677: Stop of 'ora.crsd' on 'rhel12c2' succeeded
CRS-2673: Attempting to stop 'ora.storage' on 'rhel12c2'
CRS-2673: Attempting to stop 'ora.crf' on 'rhel12c2'
CRS-2673: Attempting to stop 'ora.drivers.acfs' on 'rhel12c2'
CRS-2673: Attempting to stop 'ora.gpnpd' on 'rhel12c2'
CRS-2673: Attempting to stop 'ora.mdnsd' on 'rhel12c2'
CRS-2677: Stop of 'ora.drivers.acfs' on 'rhel12c2' succeeded
CRS-2677: Stop of 'ora.crf' on 'rhel12c2' succeeded
CRS-2677: Stop of 'ora.gpnpd' on 'rhel12c2' succeeded
CRS-2677: Stop of 'ora.storage' on 'rhel12c2' succeeded
CRS-2673: Attempting to stop 'ora.asm' on 'rhel12c2'
CRS-2677: Stop of 'ora.mdnsd' on 'rhel12c2' succeeded
CRS-2677: Stop of 'ora.asm' on 'rhel12c2' succeeded
CRS-2673: Attempting to stop 'ora.cluster_interconnect.haip' on 'rhel12c2'
CRS-2677: Stop of 'ora.cluster_interconnect.haip' on 'rhel12c2' succeeded
CRS-2673: Attempting to stop 'ora.ctssd' on 'rhel12c2'
CRS-2673: Attempting to stop 'ora.evmd' on 'rhel12c2'
CRS-2677: Stop of 'ora.evmd' on 'rhel12c2' succeeded
CRS-2677: Stop of 'ora.ctssd' on 'rhel12c2' succeeded
CRS-2673: Attempting to stop 'ora.cssd' on 'rhel12c2'
CRS-2677: Stop of 'ora.cssd' on 'rhel12c2' succeeded
CRS-2673: Attempting to stop 'ora.gipcd' on 'rhel12c2'
CRS-2677: Stop of 'ora.gipcd' on 'rhel12c2' succeeded
CRS-2793: Shutdown of Oracle High Availability Services-managed resources on 'rhel12c2' has completed
CRS-4133: Oracle High Availability Services has been stopped.
CRS-4123: Oracle High Availability Services has been started.
CRS-2672: Attempting to start 'ora.evmd' on 'rhel12c2'
CRS-2672: Attempting to start 'ora.mdnsd' on 'rhel12c2'
CRS-2676: Start of 'ora.evmd' on 'rhel12c2' succeeded
CRS-2676: Start of 'ora.mdnsd' on 'rhel12c2' succeeded
CRS-2672: Attempting to start 'ora.gpnpd' on 'rhel12c2'
CRS-2676: Start of 'ora.gpnpd' on 'rhel12c2' succeeded
CRS-2672: Attempting to start 'ora.cssdmonitor' on 'rhel12c2'
CRS-2672: Attempting to start 'ora.gipcd' on 'rhel12c2'
CRS-2676: Start of 'ora.cssdmonitor' on 'rhel12c2' succeeded
CRS-2676: Start of 'ora.gipcd' on 'rhel12c2' succeeded
CRS-2672: Attempting to start 'ora.cssd' on 'rhel12c2'
CRS-2672: Attempting to start 'ora.diskmon' on 'rhel12c2'
CRS-2676: Start of 'ora.diskmon' on 'rhel12c2' succeeded
CRS-2676: Start of 'ora.cssd' on 'rhel12c2' succeeded
CRS-2672: Attempting to start 'ora.drivers.acfs' on 'rhel12c2'
CRS-2672: Attempting to start 'ora.cluster_interconnect.haip' on 'rhel12c2'
CRS-2672: Attempting to start 'ora.ctssd' on 'rhel12c2'
CRS-2674: Start of 'ora.drivers.acfs' on 'rhel12c2' failed
CRS-2676: Start of 'ora.ctssd' on 'rhel12c2' succeeded
CRS-2676: Start of 'ora.cluster_interconnect.haip' on 'rhel12c2' succeeded
CRS-2672: Attempting to start 'ora.asm' on 'rhel12c2'
CRS-2676: Start of 'ora.asm' on 'rhel12c2' succeeded
2017/09/25 16:39:54 CLSRSC-338: Successfully downgraded OCR to version 12.1.0.2.0
CRS-2672: Attempting to start 'ora.crf' on 'rhel12c2'
CRS-2672: Attempting to start 'ora.storage' on 'rhel12c2'
CRS-2676: Start of 'ora.storage' on 'rhel12c2' succeeded
CRS-2676: Start of 'ora.crf' on 'rhel12c2' succeeded
CRS-2672: Attempting to start 'ora.crsd' on 'rhel12c2'
CRS-2676: Start of 'ora.crsd' on 'rhel12c2' succeeded
2017/09/25 16:40:24 CLSRSC-4006: Removing Oracle Trace File Analyzer (TFA) Collector.
2017/09/25 16:40:50 CLSRSC-4007: Successfully removed Oracle Trace File Analyzer (TFA) Collector.
2017/09/25 16:40:50 CLSRSC-4001: Installing Oracle Trace File Analyzer (TFA) Collector.
2017/09/25 16:41:25 CLSRSC-4002: Successfully installed Oracle Trace File Analyzer (TFA) Collector.
2017/09/25 16:41:25 CLSRSC-591: successfully downgraded Oracle Clusterware stack on this node
2017/09/25 16:41:25 CLSRSC-640: To complete the downgrade operation, ensure that the node inventory on all nodes points to the configured Grid Infrastructure home '/opt/app/12.1.0/grid2'.
2017/09/25 16:41:27 CLSRSC-592: Run 'crsctl start crs' from home /opt/app/12.1.0/grid2 on each node to complete downgrade.

4. Upgrade the oracle inventory by removing the CRS association from 12.2 GI home and adding it to 12.1 home.

[grid@rhel12c1 bin]$ pwd
/opt/app/12.2.0/grid/oui/bin

./runInstaller -nowait -waitforcompletion -ignoreSysPrereqs -updateNodeList -silent CRS=false ORACLE_HOME=/opt/app/12.2.0/grid "CLUSTER_NODES=rhel12c1,rhel12c2" -doNotUpdateNodeList 
Starting Oracle Universal Installer...

Checking swap space: must be greater than 500 MB.   Actual 5119 MB    Passed
The inventory pointer is located at /etc/oraInst.loc
'UpdateNodeList' was successful.

[grid@rhel12c1 bin]$ pwd
/opt/app/12.1.0/grid2/oui/bin

./runInstaller -nowait -waitforcompletion -ignoreSysPrereqs -updateNodeList -silent CRS=true ORACLE_HOME=/opt/app/12.1.0/grid2 "CLUSTER_NODES=rhel12c1,rhel12c2"

Starting Oracle Universal Installer...

Checking swap space: must be greater than 500 MB.   Actual 5119 MB    Passed
The inventory pointer is located at /etc/oraInst.loc
'UpdateNodeList' was successful.

Verify the update on inventory.xml

<HOME NAME="OraGI12Home1" LOC="/opt/app/12.1.0/grid2" TYPE="O" IDX="1" CRS="true">
   <NODE_LIST>
      <NODE NAME="rhel12c1"/>
      <NODE NAME="rhel12c2"/>
   </NODE_LIST>
</HOME>

<HOME NAME="OraGI12Home2" LOC="/opt/app/12.2.0/grid" TYPE="O" IDX="3"/>

5. If the /etc/oratab is missing any ASM references with downgraded 12.1 GI home, add the entries manually.

6. Before re-creatign the 12.1 MGMTDB check if there's any references to mgmtdb (using crsctl stat res -t). If there are any references then recreating the MGMTDB will fail with

PRCD-1000 : Database _mgmtdb already exists
PRCR-1086 : resource ora.mgmtdb is already registered

mgmtdb could be removed with

srvctl remove mgmtdb
Remove the database _mgmtdb? (y/[n]) y

7. Once all residual information on mgmtdb is removed create the MGMTDB root container.

[grid@rhel12c1 ~]$ $GI_HOME/bin/dbca -silent -createDatabase -createAsContainerDatabase true -templateName MGMTSeed_Database.dbc -sid -MGMTDB -gdbName _mgmtdb -storageType ASM -diskGroupName +GIMR -datafileJarLocation $GI_HOME/assistants/dbca/templates -characterset AL32UTF8 -autoGeneratePasswords -skipUserTemplateCheck
 Registering database with Oracle Grid Infrastructure
5% complete
Copying database files
7% complete
9% complete
16% complete
23% complete
30% complete
37% complete
41% complete
Creating and starting Oracle instance
43% complete
48% complete
49% complete
50% complete
55% complete
60% complete
61% complete
64% complete
Completing Database Creation
68% complete
79% complete
89% complete
100% complete
Look at the log file "/opt/app/oracle/cfgtoollogs/dbca/_mgmtdb/_mgmtdb3.log" for further details.

8. Create the PDB inside the MGMTDB root. The PDB should be named same as the cluster name. Cluster name could be found out with

$ cemutlo -n
prod-cluster

If cluster name has a dash (-) then when specify the PDB name replace the dash (-) with an underscore (_). Without this change the creation of the PDB will fail. More on this was mentioned on an earlier post.

[grid@rhel12c1 ~]$ $GI_HOME/bin/dbca -silent -createPluggableDatabase -sourceDB -MGMTDB -pdbName prod_cluster -createPDBFrom RMANBACKUP -PDBBackUpfile  $GI_HOME/assistants/dbca/templates/mgmtseed_pdb.dfb -PDBMetadataFile  $GI_HOME/assistants/dbca/templates/mgmtseed_pdb.xml -createAsClone true -internalSkipGIHomeCheck

4% complete
12% complete
21% complete
38% complete
55% complete
85% complete
Completing Pluggable Database Creation
100% complete
Look at the log file "/opt/app/oracle/cfgtoollogs/dbca/_mgmtdb/prod_cluster/_mgmtdb0.log" for further details.

9. Finally run mgmtca and verify mgmtdb resoruce is available

run mgmtca

ora.mgmtdb
      1        ONLINE  ONLINE       rhel12c1                 Open,STABLE

10. As mentioned in upgarde to 12.2 the SCAN name changes if GNS is used. This cluster uses GNS. Check if the SCAN name has been reverted back to 12.1 format.

srvctl config scan
SCAN name: prod-cluster-scan.rac.domain.net, Network: 1
Subnet IPv4: 192.168.0.0/255.255.255.0/eth0, dhcp
Subnet IPv6:
SCAN 0 IPv4 VIP: -/scan1-vip/192.168.0.92
SCAN VIP is enabled.
SCAN VIP is individually enabled on nodes:
SCAN VIP is individually disabled on nodes:
SCAN 1 IPv4 VIP: -/scan2-vip/192.168.0.89
SCAN VIP is enabled.
SCAN VIP is individually enabled on nodes:
SCAN VIP is individually disabled on nodes:
SCAN 2 IPv4 VIP: -/scan3-vip/192.168.0.90
SCAN VIP is enabled.
SCAN VIP is individually enabled on nodes:
SCAN VIP is individually disabled on nodes:

This concludes the downgrade to 12.1.0.2 from 12.2.0.1

Related posts
Downgrade Grid Infrastructure from 12.2.0.1 to 11.2.0.4
Downgrade Grid Infrastructure from 12.1.0.2 to 11.2.0.4
Downgrade Grid Infrastructure from 11.2.0.4 to 11.2.0.3
Downgrade Grid Infrastructure from 11.2.0.4 to 11.1.0.7

Useful metalink notes
How to Upgrade to/Downgrade from Grid Infrastructure 12.2 and Known Issues [ID 2240959.1]

Downgrade Grid Infrastructure from 12.2.0.1 to 11.2.0.4

This post list steps for downgrading grid infrastructure from 12.2.0.1 to 11.2.0.4. The upgrade to 12.2 GI was completed successfully on all nodes. No irreversible changes such as compatible.[asm|rdbms] updates have been done as part of the upgrade. Steps to downgrade would be different if the upgrade had failed on some nodes. For more refer Oracle documentation.
1. The current active version of the cluster is 12.2.0.1 and all nodes have the same version.

crsctl query crs activeversion
Oracle Clusterware active version on the cluster is [12.2.0.1.0]

crsctl query crs softwareversion -all
Oracle Clusterware version on node [rhel6m1] is [12.2.0.1.0]
Oracle Clusterware version on node [rhel6m2] is [12.2.0.1.0]

2. As part of the upgrade the OCR and vote disk were moved to ASM disk group with external redundancy called +GIMR to meet the space requirement for 12.2 upgrade. (Refer previous post on 11.2.0.4 to 12.2 GI upgrade for more) After the upgrade the OCR and Vote disk were moved back to a disk group of normal redundancy called +CLUSTER_DG. The +GIMR is used as the OCR backup location by the 12.2 GI.

ocrconfig -showbackuploc
The Oracle Cluster Registry backup location is [+GIMR]

ocrcheck
Status of Oracle Cluster Registry is as follows :
         Version                  :          4
         Total space (kbytes)     :     409568
         Used space (kbytes)      :      12056
         Available space (kbytes) :     397512
         ID                       : 1487892601
         Device/File Name         : +CLUSTER_DG

3. As shown above 12.2 the OCR and OCR backups reside in ASM and are accessible from all nodes. This is different from previous version where OCR backups reside on the local file system by default. This required to find out the "OCR-Node" before the downgrade. In 12.2 backups are in ASM any node can carry out the downgrade, no need to make a particular node the last to down grade. Previous version OCR node had to be the last one to downgrade. The OCR backup taken at the time of the upgrade could be listed with asmcmd.

ASMCMD> pwd
+gimr/rhel6m-cluster/OCRBACKUP

ASMCMD> ls -l *11.2.0.4*
Type       Redund  Striped  Time             Sys  Name
OCRBACKUP  UNPROT  COARSE   SEP 15 14:00:00  Y    rhel6m-cluster_backup11.2.0.4.0.ocr.256.954763871

4. First step in the downgrade to delete the management repository database (MGMTDB). Use DBCA silent option to drop the MGMTDB.

dbca -silent -deleteDatabase -sourceDB -MGMTDB
Connecting to database
4% complete
9% complete
14% complete
19% complete
23% complete
28% complete
47% complete
Updating network configuration files
52% complete
Deleting instance and datafiles
76% complete
100% complete
Look at the log file "/opt/app/oracle/cfgtoollogs/dbca/_mgmtdb.log" for further details.

After the delete verify that there's no ora.mgmtdb resource.

crsctl stat res -t
--------------------------------------------------------------------------------
Name           Target  State        Server                   State details
--------------------------------------------------------------------------------
Local Resources
--------------------------------------------------------------------------------
ora.ASMNET1LSNR_ASM.lsnr
               ONLINE  ONLINE       rhel6m1                  STABLE
               ONLINE  ONLINE       rhel6m2                  STABLE
ora.CLUSTER_DG.dg
               ONLINE  ONLINE       rhel6m1                  STABLE
               ONLINE  ONLINE       rhel6m2                  STABLE
ora.DATA.dg
               ONLINE  ONLINE       rhel6m1                  STABLE
               ONLINE  ONLINE       rhel6m2                  STABLE
ora.FLASH.dg
               ONLINE  ONLINE       rhel6m1                  STABLE
               ONLINE  ONLINE       rhel6m2                  STABLE
ora.GIMR.dg
               ONLINE  ONLINE       rhel6m1                  STABLE
               ONLINE  ONLINE       rhel6m2                  STABLE
ora.MYLISTENER.lsnr
               ONLINE  ONLINE       rhel6m1                  STABLE
               ONLINE  ONLINE       rhel6m2                  STABLE
ora.chad
               ONLINE  OFFLINE      rhel6m1                  STABLE
               ONLINE  OFFLINE      rhel6m2                  STABLE
ora.net1.network
               ONLINE  ONLINE       rhel6m1                  STABLE
               ONLINE  ONLINE       rhel6m2                  STABLE
ora.ons
               ONLINE  ONLINE       rhel6m1                  STABLE
               ONLINE  ONLINE       rhel6m2                  STABLE
ora.proxy_advm
               OFFLINE OFFLINE      rhel6m1                  STABLE
               OFFLINE OFFLINE      rhel6m2                  STABLE
--------------------------------------------------------------------------------
Cluster Resources
--------------------------------------------------------------------------------
ora.MGMTLSNR
      1        ONLINE  ONLINE       rhel6m1                  169.254.31.63 192.16
                                                             8.1.87,STABLE
ora.MYLISTENER_SCAN1.lsnr
      1        ONLINE  ONLINE       rhel6m1                  STABLE
ora.MYLISTENER_SCAN2.lsnr
      1        ONLINE  ONLINE       rhel6m2                  STABLE
ora.MYLISTENER_SCAN3.lsnr
      1        ONLINE  ONLINE       rhel6m2                  STABLE
ora.asm
      1        ONLINE  ONLINE       rhel6m1                  Started,STABLE
      2        ONLINE  ONLINE       rhel6m2                  Started,STABLE
ora.cvu
      1        ONLINE  ONLINE       rhel6m2                  STABLE
ora.qosmserver
      1        ONLINE  ONLINE       rhel6m2                  STABLE
ora.rhel6m1.vip
      1        ONLINE  ONLINE       rhel6m1                  STABLE
ora.rhel6m2.vip
      1        ONLINE  ONLINE       rhel6m2                  STABLE
ora.scan1.vip
      1        ONLINE  ONLINE       rhel6m1                  STABLE
ora.scan2.vip
      1        ONLINE  ONLINE       rhel6m2                  STABLE
ora.scan3.vip
      1        ONLINE  ONLINE       rhel6m2                  STABLE
ora.std11g2.db
      1        ONLINE  ONLINE       rhel6m1                  Open,HOME=/opt/app/o
                                                             racle/product/11.2.0
                                                             /dbhome_4,STABLE
      2        ONLINE  ONLINE       rhel6m2                  Open,HOME=/opt/app/o
                                                             racle/product/11.2.0
                                                             /dbhome_4,STABLE
ora.std11g2.myservice.svc
      1        ONLINE  ONLINE       rhel6m1                  STABLE
      2        ONLINE  ONLINE       rhel6m2                  STABLE
ora.std11g2.bx.domain.net.svc
      1        ONLINE  ONLINE       rhel6m2                  STABLE
      2        ONLINE  ONLINE       rhel6m1                  STABLE

If there's any residual details about mgmtdb or if it is not clearly removed then downgrade could fail.

2017/09/12 14:16:38 CLSRSC-503: Deleting the Grid Infrastructure Management Repository database failed.
Died at /opt/app/12.2.0/grid/crs/install/crsdowngrade.pm line 1143.
The command '/opt/app/12.2.0/grid/perl/bin/perl -I/opt/app/12.2.0/grid/perl/lib -I/opt/app/12.2.0/grid/crs/install /opt/app/12.2.0/grid/crs/install/rootcrs.pl -downgrade' execution failed

5. Run the downgrade from all but one of the nodes. Since this is a 2 node cluster downgrade has to be done one node at a time.

[root@rhel6m1 grid]# /opt/app/12.2.0/grid/crs/install/rootcrs.sh -downgrade
Using configuration parameter file: /opt/app/12.2.0/grid/crs/install/crsconfig_params
The log of current session can be found at:
  /opt/app/oracle/crsdata/rhel6m1/crsconfig/crsdowngrade_rhel6m1_2017-09-15_01-58-55PM.log
CRS-2791: Starting shutdown of Oracle High Availability Services-managed resources on 'rhel6m1'
CRS-2673: Attempting to stop 'ora.crsd' on 'rhel6m1'
CRS-2790: Starting shutdown of Cluster Ready Services-managed resources on server 'rhel6m1'
CRS-2673: Attempting to stop 'ora.std11g2.myservice.svc' on 'rhel6m1'
CRS-2673: Attempting to stop 'ora.std11g2.bx.domain.net.svc' on 'rhel6m1'
CRS-2673: Attempting to stop 'ora.MGMTLSNR' on 'rhel6m1'
CRS-2677: Stop of 'ora.std11g2.myservice.svc' on 'rhel6m1' succeeded
CRS-2677: Stop of 'ora.std11g2.bx.domain.net.svc' on 'rhel6m1' succeeded
CRS-2673: Attempting to stop 'ora.std11g2.db' on 'rhel6m1'
CRS-2677: Stop of 'ora.MGMTLSNR' on 'rhel6m1' succeeded
CRS-2677: Stop of 'ora.std11g2.db' on 'rhel6m1' succeeded
CRS-2673: Attempting to stop 'ora.CLUSTER_DG.dg' on 'rhel6m1'
CRS-2673: Attempting to stop 'ora.GIMR.dg' on 'rhel6m1'
CRS-2673: Attempting to stop 'ora.DATA.dg' on 'rhel6m1'
CRS-2673: Attempting to stop 'ora.FLASH.dg' on 'rhel6m1'
CRS-2673: Attempting to stop 'ora.MYLISTENER.lsnr' on 'rhel6m1'
CRS-2673: Attempting to stop 'ora.MYLISTENER_SCAN1.lsnr' on 'rhel6m1'
CRS-2677: Stop of 'ora.DATA.dg' on 'rhel6m1' succeeded
CRS-2677: Stop of 'ora.FLASH.dg' on 'rhel6m1' succeeded
CRS-2677: Stop of 'ora.CLUSTER_DG.dg' on 'rhel6m1' succeeded
CRS-2677: Stop of 'ora.GIMR.dg' on 'rhel6m1' succeeded
CRS-2673: Attempting to stop 'ora.asm' on 'rhel6m1'
CRS-2677: Stop of 'ora.MYLISTENER.lsnr' on 'rhel6m1' succeeded
CRS-2673: Attempting to stop 'ora.rhel6m1.vip' on 'rhel6m1'
CRS-2677: Stop of 'ora.asm' on 'rhel6m1' succeeded
CRS-2673: Attempting to stop 'ora.ASMNET1LSNR_ASM.lsnr' on 'rhel6m1'
CRS-2677: Stop of 'ora.MYLISTENER_SCAN1.lsnr' on 'rhel6m1' succeeded
CRS-2673: Attempting to stop 'ora.scan1.vip' on 'rhel6m1'
CRS-2677: Stop of 'ora.rhel6m1.vip' on 'rhel6m1' succeeded
CRS-2677: Stop of 'ora.scan1.vip' on 'rhel6m1' succeeded
CRS-2677: Stop of 'ora.ASMNET1LSNR_ASM.lsnr' on 'rhel6m1' succeeded
CRS-2672: Attempting to start 'ora.MGMTLSNR' on 'rhel6m2'
CRS-2672: Attempting to start 'ora.scan1.vip' on 'rhel6m2'
CRS-2672: Attempting to start 'ora.rhel6m1.vip' on 'rhel6m2'
CRS-2676: Start of 'ora.scan1.vip' on 'rhel6m2' succeeded
CRS-2672: Attempting to start 'ora.MYLISTENER_SCAN1.lsnr' on 'rhel6m2'
CRS-2676: Start of 'ora.rhel6m1.vip' on 'rhel6m2' succeeded
CRS-2676: Start of 'ora.MGMTLSNR' on 'rhel6m2' succeeded
CRS-2676: Start of 'ora.MYLISTENER_SCAN1.lsnr' on 'rhel6m2' succeeded
CRS-2673: Attempting to stop 'ora.ons' on 'rhel6m1'
CRS-2677: Stop of 'ora.ons' on 'rhel6m1' succeeded
CRS-2673: Attempting to stop 'ora.net1.network' on 'rhel6m1'
CRS-2677: Stop of 'ora.net1.network' on 'rhel6m1' succeeded
CRS-2792: Shutdown of Cluster Ready Services-managed resources on 'rhel6m1' has completed
CRS-2677: Stop of 'ora.crsd' on 'rhel6m1' succeeded
CRS-2673: Attempting to stop 'ora.storage' on 'rhel6m1'
CRS-2673: Attempting to stop 'ora.crf' on 'rhel6m1'
CRS-2673: Attempting to stop 'ora.drivers.acfs' on 'rhel6m1'
CRS-2673: Attempting to stop 'ora.gpnpd' on 'rhel6m1'
CRS-2673: Attempting to stop 'ora.mdnsd' on 'rhel6m1'
CRS-2677: Stop of 'ora.drivers.acfs' on 'rhel6m1' succeeded
CRS-2677: Stop of 'ora.storage' on 'rhel6m1' succeeded
CRS-2673: Attempting to stop 'ora.asm' on 'rhel6m1'
CRS-2677: Stop of 'ora.crf' on 'rhel6m1' succeeded
CRS-2677: Stop of 'ora.gpnpd' on 'rhel6m1' succeeded
CRS-2677: Stop of 'ora.mdnsd' on 'rhel6m1' succeeded
CRS-2677: Stop of 'ora.asm' on 'rhel6m1' succeeded
CRS-2673: Attempting to stop 'ora.cluster_interconnect.haip' on 'rhel6m1'
CRS-2677: Stop of 'ora.cluster_interconnect.haip' on 'rhel6m1' succeeded
CRS-2673: Attempting to stop 'ora.ctssd' on 'rhel6m1'
CRS-2673: Attempting to stop 'ora.evmd' on 'rhel6m1'
CRS-2677: Stop of 'ora.ctssd' on 'rhel6m1' succeeded
CRS-2677: Stop of 'ora.evmd' on 'rhel6m1' succeeded
CRS-2673: Attempting to stop 'ora.cssd' on 'rhel6m1'
CRS-2677: Stop of 'ora.cssd' on 'rhel6m1' succeeded
CRS-2673: Attempting to stop 'ora.gipcd' on 'rhel6m1'
CRS-2677: Stop of 'ora.gipcd' on 'rhel6m1' succeeded
CRS-2793: Shutdown of Oracle High Availability Services-managed resources on 'rhel6m1' has completed
CRS-4133: Oracle High Availability Services has been stopped.
2017/09/15 14:01:49 CLSRSC-4006: Removing Oracle Trace File Analyzer (TFA) Collector.
2017/09/15 14:02:16 CLSRSC-4007: Successfully removed Oracle Trace File Analyzer (TFA) Collector.
2017/09/15 14:02:16 CLSRSC-4001: Installing Oracle Trace File Analyzer (TFA) Collector.
2017/09/15 14:02:55 CLSRSC-4002: Successfully installed Oracle Trace File Analyzer (TFA) Collector.
2017/09/15 14:02:55 CLSRSC-591: successfully downgraded Oracle Clusterware stack on this node

6. Run the downgrade from the last node. There's no need to specify the "-lastnode" option anymore as with previous downgrades. Specifying this would result in command failing. It is automatically detected as the last node. This could be verified by referring the generated log (output given at the end).

[root@rhel6m2 ~]# /opt/app/12.2.0/grid/crs/install/rootcrs.sh -downgrade
Using configuration parameter file: /opt/app/12.2.0/grid/crs/install/crsconfig_params
The log of current session can be found at:
  /opt/app/oracle/crsdata/rhel6m2/crsconfig/crsdowngrade_rhel6m2_2017-09-15_02-03-24PM.log
CRS-2791: Starting shutdown of Oracle High Availability Services-managed resources on 'rhel6m2'
CRS-2673: Attempting to stop 'ora.crsd' on 'rhel6m2'
CRS-2790: Starting shutdown of Cluster Ready Services-managed resources on server 'rhel6m2'
CRS-2673: Attempting to stop 'ora.qosmserver' on 'rhel6m2'
CRS-2673: Attempting to stop 'ora.std11g2.bx.domain.net.svc' on 'rhel6m2'
CRS-2673: Attempting to stop 'ora.std11g2.myservice.svc' on 'rhel6m2'
CRS-2673: Attempting to stop 'ora.MGMTLSNR' on 'rhel6m2'
CRS-2677: Stop of 'ora.std11g2.bx.domain.net.svc' on 'rhel6m2' succeeded
CRS-2677: Stop of 'ora.std11g2.myservice.svc' on 'rhel6m2' succeeded
CRS-2673: Attempting to stop 'ora.std11g2.db' on 'rhel6m2'
CRS-2677: Stop of 'ora.std11g2.db' on 'rhel6m2' succeeded
CRS-2673: Attempting to stop 'ora.CLUSTER_DG.dg' on 'rhel6m2'
CRS-2673: Attempting to stop 'ora.GIMR.dg' on 'rhel6m2'
CRS-2673: Attempting to stop 'ora.DATA.dg' on 'rhel6m2'
CRS-2673: Attempting to stop 'ora.FLASH.dg' on 'rhel6m2'
CRS-2673: Attempting to stop 'ora.MYLISTENER.lsnr' on 'rhel6m2'
CRS-2673: Attempting to stop 'ora.MYLISTENER_SCAN1.lsnr' on 'rhel6m2'
CRS-2673: Attempting to stop 'ora.MYLISTENER_SCAN2.lsnr' on 'rhel6m2'
CRS-2673: Attempting to stop 'ora.MYLISTENER_SCAN3.lsnr' on 'rhel6m2'
CRS-2673: Attempting to stop 'ora.cvu' on 'rhel6m2'
CRS-2673: Attempting to stop 'ora.rhel6m1.vip' on 'rhel6m2'
CRS-2677: Stop of 'ora.FLASH.dg' on 'rhel6m2' succeeded
CRS-2677: Stop of 'ora.CLUSTER_DG.dg' on 'rhel6m2' succeeded
CRS-2677: Stop of 'ora.DATA.dg' on 'rhel6m2' succeeded
CRS-2677: Stop of 'ora.cvu' on 'rhel6m2' succeeded
CRS-2677: Stop of 'ora.MYLISTENER.lsnr' on 'rhel6m2' succeeded
CRS-2677: Stop of 'ora.MYLISTENER_SCAN1.lsnr' on 'rhel6m2' succeeded
CRS-2673: Attempting to stop 'ora.scan1.vip' on 'rhel6m2'
CRS-2677: Stop of 'ora.MYLISTENER_SCAN3.lsnr' on 'rhel6m2' succeeded
CRS-2673: Attempting to stop 'ora.scan3.vip' on 'rhel6m2'
CRS-2677: Stop of 'ora.MYLISTENER_SCAN2.lsnr' on 'rhel6m2' succeeded
CRS-2673: Attempting to stop 'ora.scan2.vip' on 'rhel6m2'
CRS-2677: Stop of 'ora.GIMR.dg' on 'rhel6m2' succeeded
CRS-2673: Attempting to stop 'ora.asm' on 'rhel6m2'
CRS-2677: Stop of 'ora.rhel6m1.vip' on 'rhel6m2' succeeded
CRS-2677: Stop of 'ora.asm' on 'rhel6m2' succeeded
CRS-2673: Attempting to stop 'ora.ASMNET1LSNR_ASM.lsnr' on 'rhel6m2'
CRS-2677: Stop of 'ora.scan1.vip' on 'rhel6m2' succeeded
CRS-2677: Stop of 'ora.MGMTLSNR' on 'rhel6m2' succeeded
CRS-2673: Attempting to stop 'ora.rhel6m2.vip' on 'rhel6m2'
CRS-2677: Stop of 'ora.qosmserver' on 'rhel6m2' succeeded
CRS-2677: Stop of 'ora.ASMNET1LSNR_ASM.lsnr' on 'rhel6m2' succeeded
CRS-2677: Stop of 'ora.scan3.vip' on 'rhel6m2' succeeded
CRS-2677: Stop of 'ora.scan2.vip' on 'rhel6m2' succeeded
CRS-2677: Stop of 'ora.rhel6m2.vip' on 'rhel6m2' succeeded
CRS-2673: Attempting to stop 'ora.ons' on 'rhel6m2'
CRS-2677: Stop of 'ora.ons' on 'rhel6m2' succeeded
CRS-2673: Attempting to stop 'ora.net1.network' on 'rhel6m2'
CRS-2677: Stop of 'ora.net1.network' on 'rhel6m2' succeeded
CRS-2792: Shutdown of Cluster Ready Services-managed resources on 'rhel6m2' has completed
CRS-2677: Stop of 'ora.crsd' on 'rhel6m2' succeeded
CRS-2673: Attempting to stop 'ora.asm' on 'rhel6m2'
CRS-2673: Attempting to stop 'ora.crf' on 'rhel6m2'
CRS-2673: Attempting to stop 'ora.drivers.acfs' on 'rhel6m2'
CRS-2673: Attempting to stop 'ora.gpnpd' on 'rhel6m2'
CRS-2673: Attempting to stop 'ora.mdnsd' on 'rhel6m2'
CRS-2677: Stop of 'ora.drivers.acfs' on 'rhel6m2' succeeded
CRS-2677: Stop of 'ora.crf' on 'rhel6m2' succeeded
CRS-2677: Stop of 'ora.gpnpd' on 'rhel6m2' succeeded
CRS-2677: Stop of 'ora.mdnsd' on 'rhel6m2' succeeded
CRS-2677: Stop of 'ora.asm' on 'rhel6m2' succeeded
CRS-2673: Attempting to stop 'ora.cluster_interconnect.haip' on 'rhel6m2'
CRS-2677: Stop of 'ora.cluster_interconnect.haip' on 'rhel6m2' succeeded
CRS-2673: Attempting to stop 'ora.ctssd' on 'rhel6m2'
CRS-2673: Attempting to stop 'ora.evmd' on 'rhel6m2'
CRS-2677: Stop of 'ora.ctssd' on 'rhel6m2' succeeded
CRS-2677: Stop of 'ora.evmd' on 'rhel6m2' succeeded
CRS-2673: Attempting to stop 'ora.cssd' on 'rhel6m2'
CRS-2677: Stop of 'ora.cssd' on 'rhel6m2' succeeded
CRS-2673: Attempting to stop 'ora.gipcd' on 'rhel6m2'
CRS-2677: Stop of 'ora.gipcd' on 'rhel6m2' succeeded
CRS-2793: Shutdown of Oracle High Availability Services-managed resources on 'rhel6m2' has completed
CRS-4133: Oracle High Availability Services has been stopped.

ASM downgrade operation succeeded

CRS-2791: Starting shutdown of Oracle High Availability Services-managed resources on 'rhel6m2'
CRS-2673: Attempting to stop 'ora.crsd' on 'rhel6m2'
CRS-2677: Stop of 'ora.crsd' on 'rhel6m2' succeeded
CRS-2673: Attempting to stop 'ora.storage' on 'rhel6m2'
CRS-2673: Attempting to stop 'ora.crf' on 'rhel6m2'
CRS-2673: Attempting to stop 'ora.drivers.acfs' on 'rhel6m2'
CRS-2673: Attempting to stop 'ora.gpnpd' on 'rhel6m2'
CRS-2673: Attempting to stop 'ora.mdnsd' on 'rhel6m2'
CRS-2677: Stop of 'ora.drivers.acfs' on 'rhel6m2' succeeded
CRS-2677: Stop of 'ora.crf' on 'rhel6m2' succeeded
CRS-2677: Stop of 'ora.gpnpd' on 'rhel6m2' succeeded
CRS-2677: Stop of 'ora.storage' on 'rhel6m2' succeeded
CRS-2673: Attempting to stop 'ora.asm' on 'rhel6m2'
CRS-2677: Stop of 'ora.mdnsd' on 'rhel6m2' succeeded
CRS-2677: Stop of 'ora.asm' on 'rhel6m2' succeeded
CRS-2673: Attempting to stop 'ora.cluster_interconnect.haip' on 'rhel6m2'
CRS-2677: Stop of 'ora.cluster_interconnect.haip' on 'rhel6m2' succeeded
CRS-2673: Attempting to stop 'ora.ctssd' on 'rhel6m2'
CRS-2673: Attempting to stop 'ora.evmd' on 'rhel6m2'
CRS-2677: Stop of 'ora.ctssd' on 'rhel6m2' succeeded
CRS-2677: Stop of 'ora.evmd' on 'rhel6m2' succeeded
CRS-2673: Attempting to stop 'ora.cssd' on 'rhel6m2'
CRS-2677: Stop of 'ora.cssd' on 'rhel6m2' succeeded
CRS-2673: Attempting to stop 'ora.gipcd' on 'rhel6m2'
CRS-2677: Stop of 'ora.gipcd' on 'rhel6m2' succeeded
CRS-2793: Shutdown of Oracle High Availability Services-managed resources on 'rhel6m2' has completed
CRS-4133: Oracle High Availability Services has been stopped.
CRS-4123: Starting Oracle High Availability Services-managed resources
CRS-2672: Attempting to start 'ora.mdnsd' on 'rhel6m2'
CRS-2672: Attempting to start 'ora.evmd' on 'rhel6m2'
CRS-2676: Start of 'ora.mdnsd' on 'rhel6m2' succeeded
CRS-2676: Start of 'ora.evmd' on 'rhel6m2' succeeded
CRS-2672: Attempting to start 'ora.gpnpd' on 'rhel6m2'
CRS-2676: Start of 'ora.gpnpd' on 'rhel6m2' succeeded
CRS-2672: Attempting to start 'ora.gipcd' on 'rhel6m2'
CRS-2676: Start of 'ora.gipcd' on 'rhel6m2' succeeded
CRS-2672: Attempting to start 'ora.cssdmonitor' on 'rhel6m2'
CRS-2676: Start of 'ora.cssdmonitor' on 'rhel6m2' succeeded
CRS-2672: Attempting to start 'ora.cssd' on 'rhel6m2'
CRS-2672: Attempting to start 'ora.diskmon' on 'rhel6m2'
CRS-2676: Start of 'ora.diskmon' on 'rhel6m2' succeeded
CRS-2676: Start of 'ora.cssd' on 'rhel6m2' succeeded
CRS-2672: Attempting to start 'ora.cluster_interconnect.haip' on 'rhel6m2'
CRS-2672: Attempting to start 'ora.ctssd' on 'rhel6m2'
CRS-2676: Start of 'ora.ctssd' on 'rhel6m2' succeeded
CRS-2676: Start of 'ora.cluster_interconnect.haip' on 'rhel6m2' succeeded
CRS-2672: Attempting to start 'ora.asm' on 'rhel6m2'
CRS-2676: Start of 'ora.asm' on 'rhel6m2' succeeded
CRS-2672: Attempting to start 'ora.storage' on 'rhel6m2'
CRS-2676: Start of 'ora.storage' on 'rhel6m2' succeeded
CRS-2672: Attempting to start 'ora.crf' on 'rhel6m2'
CRS-2676: Start of 'ora.crf' on 'rhel6m2' succeeded
CRS-2672: Attempting to start 'ora.crsd' on 'rhel6m2'
CRS-2676: Start of 'ora.crsd' on 'rhel6m2' succeeded
CRS-6023: Starting Oracle Cluster Ready Services-managed resources
CRS-2664: Resource 'ora.DATA.dg' is already running on 'rhel6m2'
CRS-2664: Resource 'ora.CLUSTER_DG.dg' is already running on 'rhel6m2'
CRS-2664: Resource 'ora.FLASH.dg' is already running on 'rhel6m2'
CRS-2664: Resource 'ora.GIMR.dg' is already running on 'rhel6m2'
CRS-6017: Processing resource auto-start for servers: rhel6m2
CRS-2672: Attempting to start 'ora.scan1.vip' on 'rhel6m2'
CRS-2672: Attempting to start 'ora.scan2.vip' on 'rhel6m2'
CRS-2672: Attempting to start 'ora.rhel6m2.vip' on 'rhel6m2'
CRS-2672: Attempting to start 'ora.scan3.vip' on 'rhel6m2'
CRS-2672: Attempting to start 'ora.qosmserver' on 'rhel6m2'
CRS-2672: Attempting to start 'ora.ons' on 'rhel6m2'
CRS-2672: Attempting to start 'ora.rhel6m1.vip' on 'rhel6m2'
CRS-2672: Attempting to start 'ora.cvu' on 'rhel6m2'
CRS-2676: Start of 'ora.scan1.vip' on 'rhel6m2' succeeded
CRS-2672: Attempting to start 'ora.MYLISTENER_SCAN1.lsnr' on 'rhel6m2'
CRS-2676: Start of 'ora.scan2.vip' on 'rhel6m2' succeeded
CRS-2672: Attempting to start 'ora.MYLISTENER_SCAN2.lsnr' on 'rhel6m2'
CRS-2676: Start of 'ora.cvu' on 'rhel6m2' succeeded
CRS-2676: Start of 'ora.rhel6m2.vip' on 'rhel6m2' succeeded
CRS-2672: Attempting to start 'ora.MGMTLSNR' on 'rhel6m2'
CRS-2672: Attempting to start 'ora.MYLISTENER.lsnr' on 'rhel6m2'
CRS-2676: Start of 'ora.scan3.vip' on 'rhel6m2' succeeded
CRS-2672: Attempting to start 'ora.MYLISTENER_SCAN3.lsnr' on 'rhel6m2'
CRS-2676: Start of 'ora.rhel6m1.vip' on 'rhel6m2' succeeded
CRS-2676: Start of 'ora.ons' on 'rhel6m2' succeeded
CRS-2676: Start of 'ora.MGMTLSNR' on 'rhel6m2' succeeded
CRS-2676: Start of 'ora.MYLISTENER.lsnr' on 'rhel6m2' succeeded
CRS-2676: Start of 'ora.MYLISTENER_SCAN1.lsnr' on 'rhel6m2' succeeded
CRS-2672: Attempting to start 'ora.std11g2.db' on 'rhel6m2'
CRS-2676: Start of 'ora.MYLISTENER_SCAN2.lsnr' on 'rhel6m2' succeeded
CRS-2676: Start of 'ora.MYLISTENER_SCAN3.lsnr' on 'rhel6m2' succeeded
CRS-2676: Start of 'ora.qosmserver' on 'rhel6m2' succeeded
CRS-2676: Start of 'ora.std11g2.db' on 'rhel6m2' succeeded
CRS-2672: Attempting to start 'ora.std11g2.myservice.svc' on 'rhel6m2'
CRS-2672: Attempting to start 'ora.std11g2.bx.domain.net.svc' on 'rhel6m2'
CRS-2676: Start of 'ora.std11g2.bx.domain.net.svc' on 'rhel6m2' succeeded
CRS-2676: Start of 'ora.std11g2.myservice.svc' on 'rhel6m2' succeeded
CRS-6016: Resource auto-start has completed for server rhel6m2
CRS-6024: Completed start of Oracle Cluster Ready Services-managed resources
CRS-4123: Oracle High Availability Services has been started.
CRS-2791: Starting shutdown of Oracle High Availability Services-managed resources on 'rhel6m2'
CRS-2673: Attempting to stop 'ora.crsd' on 'rhel6m2'
CRS-2790: Starting shutdown of Cluster Ready Services-managed resources on server 'rhel6m2'
CRS-2673: Attempting to stop 'ora.std11g2.bx.domain.net.svc' on 'rhel6m2'
CRS-2673: Attempting to stop 'ora.MGMTLSNR' on 'rhel6m2'
CRS-2673: Attempting to stop 'ora.qosmserver' on 'rhel6m2'
CRS-2673: Attempting to stop 'ora.std11g2.myservice.svc' on 'rhel6m2'
CRS-2677: Stop of 'ora.std11g2.myservice.svc' on 'rhel6m2' succeeded
CRS-2677: Stop of 'ora.std11g2.bx.domain.net.svc' on 'rhel6m2' succeeded
CRS-2673: Attempting to stop 'ora.std11g2.db' on 'rhel6m2'
CRS-2677: Stop of 'ora.std11g2.db' on 'rhel6m2' succeeded
CRS-2673: Attempting to stop 'ora.CLUSTER_DG.dg' on 'rhel6m2'
CRS-2673: Attempting to stop 'ora.GIMR.dg' on 'rhel6m2'
CRS-2673: Attempting to stop 'ora.DATA.dg' on 'rhel6m2'
CRS-2673: Attempting to stop 'ora.FLASH.dg' on 'rhel6m2'
CRS-2673: Attempting to stop 'ora.MYLISTENER.lsnr' on 'rhel6m2'
CRS-2673: Attempting to stop 'ora.MYLISTENER_SCAN1.lsnr' on 'rhel6m2'
CRS-2673: Attempting to stop 'ora.MYLISTENER_SCAN2.lsnr' on 'rhel6m2'
CRS-2673: Attempting to stop 'ora.MYLISTENER_SCAN3.lsnr' on 'rhel6m2'
CRS-2673: Attempting to stop 'ora.cvu' on 'rhel6m2'
CRS-2673: Attempting to stop 'ora.rhel6m1.vip' on 'rhel6m2'
CRS-2677: Stop of 'ora.DATA.dg' on 'rhel6m2' succeeded
CRS-2677: Stop of 'ora.GIMR.dg' on 'rhel6m2' succeeded
CRS-2677: Stop of 'ora.CLUSTER_DG.dg' on 'rhel6m2' succeeded
CRS-2677: Stop of 'ora.FLASH.dg' on 'rhel6m2' succeeded
CRS-2673: Attempting to stop 'ora.asm' on 'rhel6m2'
CRS-2677: Stop of 'ora.MYLISTENER.lsnr' on 'rhel6m2' succeeded
CRS-2677: Stop of 'ora.MYLISTENER_SCAN2.lsnr' on 'rhel6m2' succeeded
CRS-2673: Attempting to stop 'ora.scan2.vip' on 'rhel6m2'
CRS-2677: Stop of 'ora.MYLISTENER_SCAN1.lsnr' on 'rhel6m2' succeeded
CRS-2673: Attempting to stop 'ora.scan1.vip' on 'rhel6m2'
CRS-2677: Stop of 'ora.MYLISTENER_SCAN3.lsnr' on 'rhel6m2' succeeded
CRS-2673: Attempting to stop 'ora.scan3.vip' on 'rhel6m2'
CRS-2677: Stop of 'ora.asm' on 'rhel6m2' succeeded
CRS-2673: Attempting to stop 'ora.ASMNET1LSNR_ASM.lsnr' on 'rhel6m2'
CRS-2677: Stop of 'ora.rhel6m1.vip' on 'rhel6m2' succeeded
CRS-2677: Stop of 'ora.cvu' on 'rhel6m2' succeeded
CRS-2677: Stop of 'ora.qosmserver' on 'rhel6m2' succeeded
CRS-2677: Stop of 'ora.MGMTLSNR' on 'rhel6m2' succeeded
CRS-2673: Attempting to stop 'ora.rhel6m2.vip' on 'rhel6m2'
CRS-2677: Stop of 'ora.ASMNET1LSNR_ASM.lsnr' on 'rhel6m2' succeeded
CRS-2677: Stop of 'ora.scan3.vip' on 'rhel6m2' succeeded
CRS-2677: Stop of 'ora.scan1.vip' on 'rhel6m2' succeeded
CRS-2677: Stop of 'ora.scan2.vip' on 'rhel6m2' succeeded
CRS-2677: Stop of 'ora.rhel6m2.vip' on 'rhel6m2' succeeded
CRS-2673: Attempting to stop 'ora.ons' on 'rhel6m2'
CRS-2677: Stop of 'ora.ons' on 'rhel6m2' succeeded
CRS-2673: Attempting to stop 'ora.net1.network' on 'rhel6m2'
CRS-2677: Stop of 'ora.net1.network' on 'rhel6m2' succeeded
CRS-2792: Shutdown of Cluster Ready Services-managed resources on 'rhel6m2' has completed
CRS-2677: Stop of 'ora.crsd' on 'rhel6m2' succeeded
CRS-2673: Attempting to stop 'ora.storage' on 'rhel6m2'
CRS-2673: Attempting to stop 'ora.crf' on 'rhel6m2'
CRS-2673: Attempting to stop 'ora.drivers.acfs' on 'rhel6m2'
CRS-2673: Attempting to stop 'ora.gpnpd' on 'rhel6m2'
CRS-2673: Attempting to stop 'ora.mdnsd' on 'rhel6m2'
CRS-2677: Stop of 'ora.drivers.acfs' on 'rhel6m2' succeeded
CRS-2677: Stop of 'ora.crf' on 'rhel6m2' succeeded
CRS-2677: Stop of 'ora.gpnpd' on 'rhel6m2' succeeded
CRS-2677: Stop of 'ora.storage' on 'rhel6m2' succeeded
CRS-2673: Attempting to stop 'ora.asm' on 'rhel6m2'
CRS-2677: Stop of 'ora.mdnsd' on 'rhel6m2' succeeded
CRS-2677: Stop of 'ora.asm' on 'rhel6m2' succeeded
CRS-2673: Attempting to stop 'ora.cluster_interconnect.haip' on 'rhel6m2'
CRS-2677: Stop of 'ora.cluster_interconnect.haip' on 'rhel6m2' succeeded
CRS-2673: Attempting to stop 'ora.ctssd' on 'rhel6m2'
CRS-2673: Attempting to stop 'ora.evmd' on 'rhel6m2'
CRS-2677: Stop of 'ora.ctssd' on 'rhel6m2' succeeded
CRS-2677: Stop of 'ora.evmd' on 'rhel6m2' succeeded
CRS-2673: Attempting to stop 'ora.cssd' on 'rhel6m2'
CRS-2677: Stop of 'ora.cssd' on 'rhel6m2' succeeded
CRS-2673: Attempting to stop 'ora.gipcd' on 'rhel6m2'
CRS-2677: Stop of 'ora.gipcd' on 'rhel6m2' succeeded
CRS-2793: Shutdown of Oracle High Availability Services-managed resources on 'rhel6m2' has completed
CRS-4133: Oracle High Availability Services has been stopped.
CRS-4123: Oracle High Availability Services has been started.
CRS-2672: Attempting to start 'ora.evmd' on 'rhel6m2'
CRS-2672: Attempting to start 'ora.mdnsd' on 'rhel6m2'
CRS-2676: Start of 'ora.mdnsd' on 'rhel6m2' succeeded
CRS-2676: Start of 'ora.evmd' on 'rhel6m2' succeeded
CRS-2672: Attempting to start 'ora.gpnpd' on 'rhel6m2'
CRS-2676: Start of 'ora.gpnpd' on 'rhel6m2' succeeded
CRS-2672: Attempting to start 'ora.cssdmonitor' on 'rhel6m2'
CRS-2672: Attempting to start 'ora.gipcd' on 'rhel6m2'
CRS-2676: Start of 'ora.cssdmonitor' on 'rhel6m2' succeeded
CRS-2676: Start of 'ora.gipcd' on 'rhel6m2' succeeded
CRS-2672: Attempting to start 'ora.cssd' on 'rhel6m2'
CRS-2672: Attempting to start 'ora.diskmon' on 'rhel6m2'
CRS-2676: Start of 'ora.diskmon' on 'rhel6m2' succeeded
CRS-2676: Start of 'ora.cssd' on 'rhel6m2' succeeded
CRS-2672: Attempting to start 'ora.crf' on 'rhel6m2'
CRS-2672: Attempting to start 'ora.ctssd' on 'rhel6m2'
CRS-2672: Attempting to start 'ora.cluster_interconnect.haip' on 'rhel6m2'
CRS-2676: Start of 'ora.crf' on 'rhel6m2' succeeded
CRS-2676: Start of 'ora.ctssd' on 'rhel6m2' succeeded
CRS-2676: Start of 'ora.cluster_interconnect.haip' on 'rhel6m2' succeeded
CRS-2672: Attempting to start 'ora.asm' on 'rhel6m2'
CRS-2676: Start of 'ora.asm' on 'rhel6m2' succeeded
CRS-2672: Attempting to start 'ora.storage' on 'rhel6m2'
CRS-2676: Start of 'ora.storage' on 'rhel6m2' succeeded
CRS-2672: Attempting to start 'ora.crsd' on 'rhel6m2'
CRS-2676: Start of 'ora.crsd' on 'rhel6m2' succeeded
CRS-2673: Attempting to stop 'ora.crsd' on 'rhel6m2'
CRS-2677: Stop of 'ora.crsd' on 'rhel6m2' succeeded
CRS-2791: Starting shutdown of Oracle High Availability Services-managed resources on 'rhel6m2'
CRS-2673: Attempting to stop 'ora.mdnsd' on 'rhel6m2'
CRS-2673: Attempting to stop 'ora.storage' on 'rhel6m2'
CRS-2673: Attempting to stop 'ora.crf' on 'rhel6m2'
CRS-2673: Attempting to stop 'ora.gpnpd' on 'rhel6m2'
CRS-2673: Attempting to stop 'ora.drivers.acfs' on 'rhel6m2'
CRS-2677: Stop of 'ora.drivers.acfs' on 'rhel6m2' succeeded
CRS-2677: Stop of 'ora.storage' on 'rhel6m2' succeeded
CRS-2673: Attempting to stop 'ora.asm' on 'rhel6m2'
CRS-2677: Stop of 'ora.mdnsd' on 'rhel6m2' succeeded
CRS-2677: Stop of 'ora.crf' on 'rhel6m2' succeeded
CRS-2677: Stop of 'ora.gpnpd' on 'rhel6m2' succeeded
CRS-2677: Stop of 'ora.asm' on 'rhel6m2' succeeded
CRS-2673: Attempting to stop 'ora.cluster_interconnect.haip' on 'rhel6m2'
CRS-2677: Stop of 'ora.cluster_interconnect.haip' on 'rhel6m2' succeeded
CRS-2673: Attempting to stop 'ora.ctssd' on 'rhel6m2'
CRS-2673: Attempting to stop 'ora.evmd' on 'rhel6m2'
CRS-2677: Stop of 'ora.ctssd' on 'rhel6m2' succeeded
CRS-2677: Stop of 'ora.evmd' on 'rhel6m2' succeeded
CRS-2673: Attempting to stop 'ora.cssd' on 'rhel6m2'
CRS-2677: Stop of 'ora.cssd' on 'rhel6m2' succeeded
CRS-2673: Attempting to stop 'ora.gipcd' on 'rhel6m2'
CRS-2677: Stop of 'ora.gipcd' on 'rhel6m2' succeeded
CRS-2793: Shutdown of Oracle High Availability Services-managed resources on 'rhel6m2' has completed
CRS-4133: Oracle High Availability Services has been stopped.
CRS-4123: Oracle High Availability Services has been started.
CRS-2672: Attempting to start 'ora.mdnsd' on 'rhel6m2'
CRS-2676: Start of 'ora.mdnsd' on 'rhel6m2' succeeded
CRS-2672: Attempting to start 'ora.gpnpd' on 'rhel6m2'
CRS-2676: Start of 'ora.gpnpd' on 'rhel6m2' succeeded
CRS-2672: Attempting to start 'ora.cssdmonitor' on 'rhel6m2'
CRS-2672: Attempting to start 'ora.gipcd' on 'rhel6m2'
CRS-2676: Start of 'ora.cssdmonitor' on 'rhel6m2' succeeded
CRS-2676: Start of 'ora.gipcd' on 'rhel6m2' succeeded
CRS-2672: Attempting to start 'ora.cssd' on 'rhel6m2'
CRS-2672: Attempting to start 'ora.diskmon' on 'rhel6m2'
CRS-2676: Start of 'ora.diskmon' on 'rhel6m2' succeeded
CRS-2676: Start of 'ora.cssd' on 'rhel6m2' succeeded
CRS-2672: Attempting to start 'ora.drivers.acfs' on 'rhel6m2'
CRS-2679: Attempting to clean 'ora.cluster_interconnect.haip' on 'rhel6m2'
CRS-2672: Attempting to start 'ora.ctssd' on 'rhel6m2'
CRS-2681: Clean of 'ora.cluster_interconnect.haip' on 'rhel6m2' succeeded
CRS-2672: Attempting to start 'ora.cluster_interconnect.haip' on 'rhel6m2'
CRS-2674: Start of 'ora.drivers.acfs' on 'rhel6m2' failed
CRS-2676: Start of 'ora.ctssd' on 'rhel6m2' succeeded
CRS-2676: Start of 'ora.cluster_interconnect.haip' on 'rhel6m2' succeeded
CRS-2672: Attempting to start 'ora.asm' on 'rhel6m2'
CRS-2676: Start of 'ora.asm' on 'rhel6m2' succeeded
2017/09/15 14:11:02 CLSRSC-338: Successfully downgraded OCR to version 11.2.0.4.0
CRS-2672: Attempting to start 'ora.crsd' on 'rhel6m2'
CRS-2676: Start of 'ora.crsd' on 'rhel6m2' succeeded
2017/09/15 14:13:37 CLSRSC-4006: Removing Oracle Trace File Analyzer (TFA) Collector.
2017/09/15 14:14:07 CLSRSC-4007: Successfully removed Oracle Trace File Analyzer (TFA) Collector.
2017/09/15 14:14:08 CLSRSC-4001: Installing Oracle Trace File Analyzer (TFA) Collector.
2017/09/15 14:14:47 CLSRSC-4002: Successfully installed Oracle Trace File Analyzer (TFA) Collector.
2017/09/15 14:14:47 CLSRSC-591: successfully downgraded Oracle Clusterware stack on this node
2017/09/15 14:14:48 CLSRSC-640: To complete the downgrade operation, ensure that the node inventory on all nodes points to the configured Grid Infrastructure home '/opt/app/11.2.0/grid4'.
2017/09/15 14:14:49 CLSRSC-592: Run 'crsctl start crs' from home /opt/app/11.2.0/grid4 on each node to complete downgrade.

As mentioned earlier the downgrade process detect the last node automatically and lines below were observed in the log file generated

2017-09-15 14:03:32: Current node: rhel6m2 should be the last node to downgrade.
2017-09-15 14:05:18: Executing the step [asm_DowngradeCurrentNode_step_1] to downgrade ASM on the last node
2017-09-15 14:05:18: Performing ASM downgrade on the last node ...
2017-09-15 14:09:17: Executing the step [ocr_downgradeLastNode_step_1] to downgrade OCR on the last node.
2017-09-15 14:09:46: Executing step downgrade ACFS on the last node
2017-09-15 14:10:59: Executing the step [ocr_downgradeLastNode_step_2] to downgrade OCR on the last node.

7. Next is to upgrade the oraInventory by setting CRS=false on 12.2 GI home and CRS=ture on 11.2 GI home.

cd /opt/app/12.2.0/grid/oui/bin/
$ ./runInstaller -nowait -waitforcompletion -ignoreSysPrereqs -updateNodeList -silent CRS=false ORACLE_HOME=/opt/app/12.2.0/grid  "CLUSTER_NODES=rhel6m1,rhel6m2" -doNotUpdateNodeList
Starting Oracle Universal Installer...

Checking swap space: must be greater than 500 MB.   Actual 4011 MB    Passed
The inventory pointer is located at /etc/oraInst.loc
'UpdateNodeList' was successful.

cd /opt/app/11.2.0/grid4/oui/bin
$ ./runInstaller -nowait -waitforcompletion -ignoreSysPrereqs -updateNodeList  -silent CRS=true ORACLE_HOME=/opt/app/11.2.0/grid4
Starting Oracle Universal Installer...

Checking swap space: must be greater than 500 MB.   Actual 4011 MB    Passed
The inventory pointer is located at /etc/oraInst.loc
The inventory is located at /opt/app/oraInventory
'UpdateNodeList' was successful.

Verify this by checking in the inventory.xml

<HOME NAME="Ora11g_gridinfrahome2" LOC="/opt/app/11.2.0/grid4" TYPE="O" IDX="3" CRS="true">
   <NODE_LIST>
      <NODE NAME="rhel6m1"/>
      <NODE NAME="rhel6m2"/>
   </NODE_LIST>
</HOME>

<HOME NAME="OraGI12Home1" LOC="/opt/app/12.2.0/grid" TYPE="O" IDX="5"/>

8. Check if /etc/oratab has entries for ASM instance with the correct GI home. After the downgrade ASM instance should be associated with 11.2.0.4 GI HOME.

9. Start the cluster stack using the 11.2.0.4 GI Home.

which crsctl
/opt/app/11.2.0/grid4/bin/crsctl
crsctl start crs

9. All the changes made to OCR location after the upgrade are lost once the downgrade is done. As mentioned previously OCR was moved to disk group +CLUSTER_DG after upgrade. But after the downgrade OCR location goes back to what it was before the upgrade.

ocrcheck
Status of Oracle Cluster Registry is as follows :
         Version                  :          3
         Total space (kbytes)     :     262120
         Used space (kbytes)      :       3512
         Available space (kbytes) :     258608
         ID                       : 1487892601
         Device/File Name         :      +GIMR

However the changes made to the vote disk location doesn't revert backup after the downgrade and remain the same

$ crsctl query css votedisk
##  STATE    File Universal Id                File Name Disk group
--  -----    -----------------                --------- ---------
 1. ONLINE   880f0769cd344ff3bf158b0ee82a91dc (/dev/sdc1) [CLUSTER_DG]
 2. ONLINE   3e1484cd24c74fbfbfc6c3a1f371d4d3 (/dev/sdd1) [CLUSTER_DG]
 3. ONLINE   9604775aeca64f3ebf36df1ff70f6222 (/dev/sdb1) [CLUSTER_DG]
Located 3 voting disk(s).

OCR backup location to goes back to default location on local node

ocrconfig -showbackup

rhel6m1     2017/09/15 09:30:32     /opt/app/11.2.0/grid4/cdata/rhel6m-cluster/backup00.ocr
rhel6m1     2017/09/15 05:30:32     /opt/app/11.2.0/grid4/cdata/rhel6m-cluster/backup01.ocr

10. Verify there's no reference to mgmtdb in 11.2

 crsctl stat res -t
--------------------------------------------------------------------------------
NAME           TARGET  STATE        SERVER                   STATE_DETAILS
--------------------------------------------------------------------------------
Local Resources
--------------------------------------------------------------------------------
ora.CLUSTER_DG.dg
               ONLINE  ONLINE       rhel6m1
               ONLINE  ONLINE       rhel6m2
ora.DATA.dg
               ONLINE  ONLINE       rhel6m1
               ONLINE  ONLINE       rhel6m2
ora.FLASH.dg
               ONLINE  ONLINE       rhel6m1
               ONLINE  ONLINE       rhel6m2
ora.GIMR.dg
               ONLINE  ONLINE       rhel6m1
               ONLINE  ONLINE       rhel6m2
ora.MYLISTENER.lsnr
               ONLINE  ONLINE       rhel6m1
               ONLINE  ONLINE       rhel6m2
ora.asm
               ONLINE  ONLINE       rhel6m1                  Started
               ONLINE  ONLINE       rhel6m2                  Started
ora.gsd
               OFFLINE OFFLINE      rhel6m1
               OFFLINE OFFLINE      rhel6m2
ora.net1.network
               ONLINE  ONLINE       rhel6m1
               ONLINE  ONLINE       rhel6m2
ora.ons
               ONLINE  ONLINE       rhel6m1
               ONLINE  ONLINE       rhel6m2
ora.registry.acfs
               ONLINE  ONLINE       rhel6m1
               ONLINE  ONLINE       rhel6m2
--------------------------------------------------------------------------------
Cluster Resources
--------------------------------------------------------------------------------
ora.MYLISTENER_SCAN1.lsnr
      1        ONLINE  ONLINE       rhel6m2
ora.MYLISTENER_SCAN2.lsnr
      1        ONLINE  ONLINE       rhel6m1
ora.MYLISTENER_SCAN3.lsnr
      1        ONLINE  ONLINE       rhel6m1
ora.cvu
      1        ONLINE  ONLINE       rhel6m1
ora.oc4j
      1        ONLINE  ONLINE       rhel6m1
ora.rhel6m1.vip
      1        ONLINE  ONLINE       rhel6m1
ora.rhel6m2.vip
      1        ONLINE  ONLINE       rhel6m2
ora.scan1.vip
      1        ONLINE  ONLINE       rhel6m2
ora.scan2.vip
      1        ONLINE  ONLINE       rhel6m1
ora.scan3.vip
      1        ONLINE  ONLINE       rhel6m1
ora.std11g2.db
      1        ONLINE  ONLINE       rhel6m1                  Open
      2        ONLINE  ONLINE       rhel6m2                  Open
ora.std11g2.myservice.svc
      1        ONLINE  ONLINE       rhel6m1
      2        ONLINE  ONLINE       rhel6m2
ora.std11g2.bx.domain.net.svc
      1        ONLINE  ONLINE       rhel6m2
      2        ONLINE  ONLINE       rhel6m1

Check ASM is configured with default listener configuration not with ASMNET1LSNR_ASM (which is part of flex)

srvctl config asm
ASM home: /opt/app/11.2.0/grid4
ASM listener: MYLISTENER

Finally verify the cluster active version.

crsctl query crs activeversion
Oracle Clusterware active version on the cluster is [11.2.0.4.0]

crsctl query crs softwareversion -all
Oracle Clusterware version on node [rhel6m1] is [11.2.0.4.0]
Oracle Clusterware version on node [rhel6m2] is [11.2.0.4.0]

Related posts
Downgrade Grid Infrastructure from 12.1.0.2 to 11.2.0.4
Downgrade Grid Infrastructure from 11.2.0.4 to 11.2.0.3
Downgrade Grid Infrastructure from 11.2.0.4 to 11.1.0.7

Useful metalink notes
How to Upgrade to/Downgrade from Grid Infrastructure 12.2 and Known Issues [ID 2240959.1]

Downgrade Grid Infrastructure from 12.1.0.2 to 11.2.0.4

This post list steps for downgrading grid infrastructure from 12.1.0.2 to 11.2.0.4. The grid infrastructure upgrade has been successful on all nodes. If upgrade to 12.1.0.2 had failed on any of the nodes then follow oracle documentation for additional steps/options required for such cases. Only the GI was upgraded to 12.1.0.2 and the database remained on 11.2.0.4.

[grid@rhel6m1 ~]$  crsctl query crs activeversion
Oracle Clusterware active version on the cluster is [12.1.0.2.0]
[grid@rhel6m1 ~]$ crsctl query crs softwareversion
Oracle Clusterware version on node [rhel6m1] is [12.1.0.2.0]

[root@rhel6m1 grid]# ocrcheck
Status of Oracle Cluster Registry is as follows :
         Version                  :          4
         Total space (kbytes)     :     409568
         Used space (kbytes)      :       9068
         Available space (kbytes) :     400500
         ID                       : 2072206343
         Device/File Name         : +CLUSTER_DG

The environment used here is the same environment used to upgrade from 11.2.0.3 to 11.2.0.4. As such the OCR-node (node containing the backup of the OCR created during the upgrade to 12.1.0.2) contains both 11.2.0.3 and 11.2.0.4 OCR backups but with different permissions.

[grid@rhel6m1 cdata]$ ls -lrt /opt/app/12.1.0/grid2/cdata/
total 266644
drwxr-xr-x. 2 grid oinstall      4096 Mar 26 17:22 localhost
-rw-r--r--. 1 grid oinstall    132453 Mar 26 17:34 ocr11.2.0.3.0
-rw-------. 1 root root        130412 Mar 26 17:44 ocr11.2.0.4.0
drwxr-xr-x. 2 grid oinstall      4096 Mar 26 17:46 rhel6m1
-rw-------. 1 root oinstall 503484416 Mar 26 17:54 rhel6m1.olr
drwxrwxr-x. 2 grid oinstall      4096 Mar 26 17:58 rhel6m-cluster

During the downgrade the the higher version is detected and restored (no harm even if the ocr11.2.0.3.0 is removed before the downgrade).
Also before the downgrade commands are run make sure only the 12.1 related binaries are in the PATH variable (crsctl, srvctl) and ORACLE_HOME points to 12.1. At times the downgrade failed with

Died at /opt/app/12.1.0/grid2/crs/install/crsdeconfig.pm line 2336.
The command '/opt/app/12.1.0/grid2/perl/bin/perl -I/opt/app/12.1.0/grid2/perl/lib -I/opt/app/12.1.0/grid2/crs/install /opt/app/12.1.0/grid2/crs/install/rootcrs.pl -downgrade -lastnode' execution failed

for which solution according to 1917922.1 is to run the downgrade command from a location where grid user has write access to. In this case it was run from grid user home so MOS note solution may not be applicable.
Another time the downgrade on the OCR-node failed with

Using configuration parameter file: /opt/app/12.1.0/grid2/crs/install/crsconfig_params
2015-04-01 10:43:00.047
CLSD: An error occurred while attempting to generate a full name. Logging may not be active for this process
Additional diagnostics: CLSU-00100: operating system function: sclsdgcwd failed with error data: -1
CLSU-00103: error location: sclsdgcwd5
(:CLSD00183:)

MOS notes 1961344.1 and 1911258.1 gives causes and solutions for this but neither case was valid in this case as the permission of directories mentioned on these MOS were already set to correct values. These issues went away after removing from PATH any references to 11.2 binaries.
To begin the downgrade run the rootcrs.sh with downgrade option on all but the OCR-node. In this case the rhel6m1 is the OCR-node and downgrade first run on rhel6m2.

[root@rhel6m2 grid]# /opt/app/12.1.0/grid2/crs/install/rootcrs.sh -downgrade
Using configuration parameter file: /opt/app/12.1.0/grid2/crs/install/crsconfig_params
CRS-2791: Starting shutdown of Oracle High Availability Services-managed resources on 'rhel6m2'
CRS-2673: Attempting to stop 'ora.crsd' on 'rhel6m2'
CRS-2790: Starting shutdown of Cluster Ready Services-managed resources on 'rhel6m2'
CRS-2673: Attempting to stop 'ora.CLUSTER_DG.dg' on 'rhel6m2'
CRS-2673: Attempting to stop 'ora.MYSCANLISTENER_SCAN1.lsnr' on 'rhel6m2'
CRS-2673: Attempting to stop 'ora.std11g2.asa.domain.net.svc' on 'rhel6m2'
CRS-2673: Attempting to stop 'ora.oc4j' on 'rhel6m2'
CRS-2677: Stop of 'ora.CLUSTER_DG.dg' on 'rhel6m2' succeeded
CRS-2677: Stop of 'ora.std11g2.asa.domain.net.svc' on 'rhel6m2' succeeded
CRS-2673: Attempting to stop 'ora.std11g2.db' on 'rhel6m2'
CRS-2673: Attempting to stop 'ora.MYLISTENER.lsnr' on 'rhel6m2'
CRS-2677: Stop of 'ora.MYSCANLISTENER_SCAN1.lsnr' on 'rhel6m2' succeeded
CRS-2673: Attempting to stop 'ora.scan1.vip' on 'rhel6m2'
CRS-2677: Stop of 'ora.MYLISTENER.lsnr' on 'rhel6m2' succeeded
CRS-2673: Attempting to stop 'ora.rhel6m2.vip' on 'rhel6m2'
CRS-2677: Stop of 'ora.std11g2.db' on 'rhel6m2' succeeded
CRS-2673: Attempting to stop 'ora.FLASH.dg' on 'rhel6m2'
CRS-2673: Attempting to stop 'ora.DATA.dg' on 'rhel6m2'
CRS-2677: Stop of 'ora.scan1.vip' on 'rhel6m2' succeeded
CRS-2672: Attempting to start 'ora.scan1.vip' on 'rhel6m1'
CRS-2677: Stop of 'ora.DATA.dg' on 'rhel6m2' succeeded
CRS-2677: Stop of 'ora.rhel6m2.vip' on 'rhel6m2' succeeded
CRS-2672: Attempting to start 'ora.rhel6m2.vip' on 'rhel6m1'
CRS-2676: Start of 'ora.scan1.vip' on 'rhel6m1' succeeded
CRS-2672: Attempting to start 'ora.MYSCANLISTENER_SCAN1.lsnr' on 'rhel6m1'
CRS-2677: Stop of 'ora.FLASH.dg' on 'rhel6m2' succeeded
CRS-2673: Attempting to stop 'ora.asm' on 'rhel6m2'
CRS-2677: Stop of 'ora.asm' on 'rhel6m2' succeeded
CRS-2676: Start of 'ora.rhel6m2.vip' on 'rhel6m1' succeeded
CRS-2676: Start of 'ora.MYSCANLISTENER_SCAN1.lsnr' on 'rhel6m1' succeeded
CRS-2677: Stop of 'ora.oc4j' on 'rhel6m2' succeeded
CRS-2672: Attempting to start 'ora.oc4j' on 'rhel6m1'
CRS-2676: Start of 'ora.oc4j' on 'rhel6m1' succeeded
CRS-2673: Attempting to stop 'ora.ons' on 'rhel6m2'
CRS-2677: Stop of 'ora.ons' on 'rhel6m2' succeeded
CRS-2673: Attempting to stop 'ora.net1.network' on 'rhel6m2'
CRS-2677: Stop of 'ora.net1.network' on 'rhel6m2' succeeded
CRS-2792: Shutdown of Cluster Ready Services-managed resources on 'rhel6m2' has completed
CRS-2677: Stop of 'ora.crsd' on 'rhel6m2' succeeded
CRS-2673: Attempting to stop 'ora.ctssd' on 'rhel6m2'
CRS-2673: Attempting to stop 'ora.evmd' on 'rhel6m2'
CRS-2673: Attempting to stop 'ora.storage' on 'rhel6m2'
CRS-2673: Attempting to stop 'ora.mdnsd' on 'rhel6m2'
CRS-2673: Attempting to stop 'ora.gpnpd' on 'rhel6m2'
CRS-2673: Attempting to stop 'ora.drivers.acfs' on 'rhel6m2'
CRS-2677: Stop of 'ora.storage' on 'rhel6m2' succeeded
CRS-2673: Attempting to stop 'ora.asm' on 'rhel6m2'
CRS-2677: Stop of 'ora.drivers.acfs' on 'rhel6m2' succeeded
CRS-2677: Stop of 'ora.ctssd' on 'rhel6m2' succeeded
CRS-2677: Stop of 'ora.mdnsd' on 'rhel6m2' succeeded
CRS-2677: Stop of 'ora.evmd' on 'rhel6m2' succeeded
CRS-2677: Stop of 'ora.gpnpd' on 'rhel6m2' succeeded
CRS-2677: Stop of 'ora.asm' on 'rhel6m2' succeeded
CRS-2673: Attempting to stop 'ora.cluster_interconnect.haip' on 'rhel6m2'
CRS-2677: Stop of 'ora.cluster_interconnect.haip' on 'rhel6m2' succeeded
CRS-2673: Attempting to stop 'ora.cssd' on 'rhel6m2'
CRS-2677: Stop of 'ora.cssd' on 'rhel6m2' succeeded
CRS-2673: Attempting to stop 'ora.crf' on 'rhel6m2'
CRS-2677: Stop of 'ora.crf' on 'rhel6m2' succeeded
CRS-2673: Attempting to stop 'ora.gipcd' on 'rhel6m2'
CRS-2677: Stop of 'ora.gipcd' on 'rhel6m2' succeeded
CRS-2793: Shutdown of Oracle High Availability Services-managed resources on 'rhel6m2' has completed
CRS-4133: Oracle High Availability Services has been stopped.
2015/04/01 12:07:02 CLSRSC-4001: Installing Oracle Trace File Analyzer (TFA) Collector.

2015/04/01 12:07:02 CLSRSC-4002: Successfully installed Oracle Trace File Analyzer (TFA) Collector.

Successfully downgraded Oracle Clusterware stack on this node

Since this is a two node RAC the downgrade command is run with lastnode option on rhel6m1 (OCR-node). This will remove the GI management repository, downgrade the OCR

[root@rhel6m1 grid]# /opt/app/12.1.0/grid2/crs/install/rootcrs.sh -downgrade -lastnode
Using configuration parameter file: /opt/app/12.1.0/grid2/crs/install/crsconfig_params
CRS-2791: Starting shutdown of Oracle High Availability Services-managed resources on 'rhel6m1'
CRS-2673: Attempting to stop 'ora.crsd' on 'rhel6m1'
CRS-2790: Starting shutdown of Cluster Ready Services-managed resources on 'rhel6m1'
CRS-2673: Attempting to stop 'ora.MYLISTENER.lsnr' on 'rhel6m1'
CRS-2673: Attempting to stop 'ora.std11g2.asa.domain.net.svc' on 'rhel6m1'
CRS-2673: Attempting to stop 'ora.MYSCANLISTENER_SCAN1.lsnr' on 'rhel6m1'
CRS-2673: Attempting to stop 'ora.rhel6m2.vip' on 'rhel6m1'
CRS-2673: Attempting to stop 'ora.oc4j' on 'rhel6m1'
CRS-2673: Attempting to stop 'ora.mgmtdb' on 'rhel6m1'
CRS-2673: Attempting to stop 'ora.cvu' on 'rhel6m1'
CRS-2677: Stop of 'ora.cvu' on 'rhel6m1' succeeded
CRS-2677: Stop of 'ora.MYSCANLISTENER_SCAN1.lsnr' on 'rhel6m1' succeeded
CRS-2673: Attempting to stop 'ora.scan1.vip' on 'rhel6m1'
CRS-2677: Stop of 'ora.MYLISTENER.lsnr' on 'rhel6m1' succeeded
CRS-2677: Stop of 'ora.std11g2.asa.domain.net.svc' on 'rhel6m1' succeeded
CRS-2673: Attempting to stop 'ora.std11g2.db' on 'rhel6m1'
CRS-2677: Stop of 'ora.std11g2.db' on 'rhel6m1' succeeded
CRS-2673: Attempting to stop 'ora.DATA.dg' on 'rhel6m1'
CRS-2673: Attempting to stop 'ora.FLASH.dg' on 'rhel6m1'
CRS-2677: Stop of 'ora.rhel6m2.vip' on 'rhel6m1' succeeded
CRS-2673: Attempting to stop 'ora.rhel6m1.vip' on 'rhel6m1'
CRS-2677: Stop of 'ora.DATA.dg' on 'rhel6m1' succeeded
CRS-2677: Stop of 'ora.scan1.vip' on 'rhel6m1' succeeded
CRS-2677: Stop of 'ora.rhel6m1.vip' on 'rhel6m1' succeeded
CRS-2677: Stop of 'ora.mgmtdb' on 'rhel6m1' succeeded
CRS-2673: Attempting to stop 'ora.CLUSTER_DG.dg' on 'rhel6m1'
CRS-2673: Attempting to stop 'ora.MGMTLSNR' on 'rhel6m1'
CRS-2677: Stop of 'ora.FLASH.dg' on 'rhel6m1' succeeded
CRS-2677: Stop of 'ora.MGMTLSNR' on 'rhel6m1' succeeded
CRS-2677: Stop of 'ora.CLUSTER_DG.dg' on 'rhel6m1' succeeded
CRS-2673: Attempting to stop 'ora.asm' on 'rhel6m1'
CRS-2677: Stop of 'ora.asm' on 'rhel6m1' succeeded
CRS-2677: Stop of 'ora.oc4j' on 'rhel6m1' succeeded
CRS-2673: Attempting to stop 'ora.ons' on 'rhel6m1'
CRS-2677: Stop of 'ora.ons' on 'rhel6m1' succeeded
CRS-2673: Attempting to stop 'ora.net1.network' on 'rhel6m1'
CRS-2677: Stop of 'ora.net1.network' on 'rhel6m1' succeeded
CRS-2792: Shutdown of Cluster Ready Services-managed resources on 'rhel6m1' has completed
CRS-2677: Stop of 'ora.crsd' on 'rhel6m1' succeeded
CRS-2673: Attempting to stop 'ora.storage' on 'rhel6m1'
CRS-2673: Attempting to stop 'ora.mdnsd' on 'rhel6m1'
CRS-2673: Attempting to stop 'ora.gpnpd' on 'rhel6m1'
CRS-2673: Attempting to stop 'ora.drivers.acfs' on 'rhel6m1'
CRS-2677: Stop of 'ora.storage' on 'rhel6m1' succeeded
CRS-2677: Stop of 'ora.drivers.acfs' on 'rhel6m1' succeeded
CRS-2673: Attempting to stop 'ora.crf' on 'rhel6m1'
CRS-2673: Attempting to stop 'ora.ctssd' on 'rhel6m1'
CRS-2673: Attempting to stop 'ora.evmd' on 'rhel6m1'
CRS-2673: Attempting to stop 'ora.asm' on 'rhel6m1'
CRS-2677: Stop of 'ora.mdnsd' on 'rhel6m1' succeeded
CRS-2677: Stop of 'ora.gpnpd' on 'rhel6m1' succeeded
CRS-2677: Stop of 'ora.crf' on 'rhel6m1' succeeded
CRS-2677: Stop of 'ora.ctssd' on 'rhel6m1' succeeded
CRS-2677: Stop of 'ora.evmd' on 'rhel6m1' succeeded
CRS-2677: Stop of 'ora.asm' on 'rhel6m1' succeeded
CRS-2673: Attempting to stop 'ora.cluster_interconnect.haip' on 'rhel6m1'
CRS-2677: Stop of 'ora.cluster_interconnect.haip' on 'rhel6m1' succeeded
CRS-2673: Attempting to stop 'ora.cssd' on 'rhel6m1'
CRS-2677: Stop of 'ora.cssd' on 'rhel6m1' succeeded
CRS-2673: Attempting to stop 'ora.gipcd' on 'rhel6m1'
CRS-2677: Stop of 'ora.gipcd' on 'rhel6m1' succeeded
CRS-2793: Shutdown of Oracle High Availability Services-managed resources on 'rhel6m1' has completed
CRS-4133: Oracle High Availability Services has been stopped.

ASM downgrade operation succeeded

CRS-2791: Starting shutdown of Oracle High Availability Services-managed resources on 'rhel6m1'
CRS-2673: Attempting to stop 'ora.crsd' on 'rhel6m1'
CRS-2677: Stop of 'ora.crsd' on 'rhel6m1' succeeded
CRS-2673: Attempting to stop 'ora.gpnpd' on 'rhel6m1'
CRS-2673: Attempting to stop 'ora.ctssd' on 'rhel6m1'
CRS-2673: Attempting to stop 'ora.evmd' on 'rhel6m1'
CRS-2673: Attempting to stop 'ora.storage' on 'rhel6m1'
CRS-2673: Attempting to stop 'ora.crf' on 'rhel6m1'
CRS-2673: Attempting to stop 'ora.drivers.acfs' on 'rhel6m1'
CRS-2673: Attempting to stop 'ora.mdnsd' on 'rhel6m1'
CRS-2677: Stop of 'ora.storage' on 'rhel6m1' succeeded
CRS-2673: Attempting to stop 'ora.asm' on 'rhel6m1'
CRS-2677: Stop of 'ora.drivers.acfs' on 'rhel6m1' succeeded
CRS-2677: Stop of 'ora.ctssd' on 'rhel6m1' succeeded
CRS-2677: Stop of 'ora.evmd' on 'rhel6m1' succeeded
CRS-2677: Stop of 'ora.gpnpd' on 'rhel6m1' succeeded
CRS-2677: Stop of 'ora.crf' on 'rhel6m1' succeeded
CRS-2677: Stop of 'ora.mdnsd' on 'rhel6m1' succeeded
CRS-2677: Stop of 'ora.asm' on 'rhel6m1' succeeded
CRS-2673: Attempting to stop 'ora.cluster_interconnect.haip' on 'rhel6m1'
CRS-2677: Stop of 'ora.cluster_interconnect.haip' on 'rhel6m1' succeeded
CRS-2673: Attempting to stop 'ora.cssd' on 'rhel6m1'
CRS-2677: Stop of 'ora.cssd' on 'rhel6m1' succeeded
CRS-2673: Attempting to stop 'ora.gipcd' on 'rhel6m1'
CRS-2677: Stop of 'ora.gipcd' on 'rhel6m1' succeeded
CRS-2793: Shutdown of Oracle High Availability Services-managed resources on 'rhel6m1' has completed
CRS-4133: Oracle High Availability Services has been stopped.
CRS-4123: Oracle High Availability Services has been started.
CRS-2791: Starting shutdown of Oracle High Availability Services-managed resources on 'rhel6m1'
CRS-2673: Attempting to stop 'ora.drivers.acfs' on 'rhel6m1'
CRS-2677: Stop of 'ora.drivers.acfs' on 'rhel6m1' succeeded
CRS-2793: Shutdown of Oracle High Availability Services-managed resources on 'rhel6m1' has completed
CRS-4133: Oracle High Availability Services has been stopped.
CRS-4123: Starting Oracle High Availability Services-managed resources
CRS-2672: Attempting to start 'ora.mdnsd' on 'rhel6m1'
CRS-2672: Attempting to start 'ora.evmd' on 'rhel6m1'
CRS-2676: Start of 'ora.mdnsd' on 'rhel6m1' succeeded
CRS-2676: Start of 'ora.evmd' on 'rhel6m1' succeeded
CRS-2672: Attempting to start 'ora.gpnpd' on 'rhel6m1'
CRS-2676: Start of 'ora.gpnpd' on 'rhel6m1' succeeded
CRS-2672: Attempting to start 'ora.gipcd' on 'rhel6m1'
CRS-2676: Start of 'ora.gipcd' on 'rhel6m1' succeeded
CRS-2672: Attempting to start 'ora.cssdmonitor' on 'rhel6m1'
CRS-2676: Start of 'ora.cssdmonitor' on 'rhel6m1' succeeded
CRS-2672: Attempting to start 'ora.cssd' on 'rhel6m1'
CRS-2672: Attempting to start 'ora.diskmon' on 'rhel6m1'
CRS-2676: Start of 'ora.diskmon' on 'rhel6m1' succeeded
CRS-2676: Start of 'ora.cssd' on 'rhel6m1' succeeded
CRS-2672: Attempting to start 'ora.cluster_interconnect.haip' on 'rhel6m1'
CRS-2672: Attempting to start 'ora.ctssd' on 'rhel6m1'
CRS-2676: Start of 'ora.ctssd' on 'rhel6m1' succeeded
CRS-2676: Start of 'ora.cluster_interconnect.haip' on 'rhel6m1' succeeded
CRS-2672: Attempting to start 'ora.asm' on 'rhel6m1'
CRS-2676: Start of 'ora.asm' on 'rhel6m1' succeeded
CRS-2672: Attempting to start 'ora.storage' on 'rhel6m1'
CRS-2676: Start of 'ora.storage' on 'rhel6m1' succeeded
CRS-2672: Attempting to start 'ora.crf' on 'rhel6m1'
CRS-2676: Start of 'ora.crf' on 'rhel6m1' succeeded
CRS-2672: Attempting to start 'ora.crsd' on 'rhel6m1'
CRS-2676: Start of 'ora.crsd' on 'rhel6m1' succeeded
CRS-6023: Starting Oracle Cluster Ready Services-managed resources
CRS-6017: Processing resource auto-start for servers: rhel6m1
CRS-6016: Resource auto-start has completed for server rhel6m1
CRS-6024: Completed start of Oracle Cluster Ready Services-managed resources
CRS-4123: Oracle High Availability Services has been started.
Starting the Grid Infrastructure Management Repository database
Using PFILE='/opt/app/12.1.0/grid2/dbs/init-dropmgmtdb.ora'
Command output:
SQL*Plus: Release 12.1.0.2.0 Production on Wed Apr 1 12:12:12 2015 Copyright (c) 1982, 2014, Oracle. All rights reserved. Connected to an idle instance. SQL> ORACLE instance started. Total System Global Area 788529152 bytes Fixed Size 2929352 bytes Variable Size 314576184 bytes Database Buffers 465567744 bytes Redo Buffers 5455872 bytes Database mounted. SQL> Disconnected from Oracle Database 12c Enterprise Edition Release 12.1.0.2.0 - 64bit Production With the Partitioning, Automatic Storage Management and Advanced Analytics options
Starting the Grid Infrastructure Management Repository database succeeded
Creating new pfile for the Management Repository database

SQL*Plus: Release 12.1.0.2.0 Production on Wed Apr 1 12:12:25 2015

Copyright (c) 1982, 2014, Oracle.  All rights reserved.


Connected to:
Oracle Database 12c Enterprise Edition Release 12.1.0.2.0 - 64bit Production
With the Partitioning, Automatic Storage Management and Advanced Analytics options

SQL>
File created.

SQL> Disconnected from Oracle Database 12c Enterprise Edition Release 12.1.0.2.0 - 64bit Production
With the Partitioning, Automatic Storage Management and Advanced Analytics options
Succeeded in creating PFILE '/opt/app/12.1.0/grid2/dbs/init-dropmgmtdbSAVED.ora'
Dropping the Grid Infrastructure Management Repository database
Dropping the Grid Infrastructure Management Repository database succeeded
CRS-2791: Starting shutdown of Oracle High Availability Services-managed resources on 'rhel6m1'
CRS-2673: Attempting to stop 'ora.crsd' on 'rhel6m1'
CRS-2790: Starting shutdown of Cluster Ready Services-managed resources on 'rhel6m1'
CRS-2673: Attempting to stop 'ora.CLUSTER_DG.dg' on 'rhel6m1'
CRS-2677: Stop of 'ora.CLUSTER_DG.dg' on 'rhel6m1' succeeded
CRS-2673: Attempting to stop 'ora.DATA.dg' on 'rhel6m1'
CRS-2673: Attempting to stop 'ora.FLASH.dg' on 'rhel6m1'
CRS-2677: Stop of 'ora.DATA.dg' on 'rhel6m1' succeeded
CRS-2677: Stop of 'ora.FLASH.dg' on 'rhel6m1' succeeded
CRS-2673: Attempting to stop 'ora.asm' on 'rhel6m1'
CRS-2677: Stop of 'ora.asm' on 'rhel6m1' succeeded
CRS-2792: Shutdown of Cluster Ready Services-managed resources on 'rhel6m1' has completed
CRS-2677: Stop of 'ora.crsd' on 'rhel6m1' succeeded
CRS-2673: Attempting to stop 'ora.evmd' on 'rhel6m1'
CRS-2673: Attempting to stop 'ora.drivers.acfs' on 'rhel6m1'
CRS-2673: Attempting to stop 'ora.ctssd' on 'rhel6m1'
CRS-2673: Attempting to stop 'ora.storage' on 'rhel6m1'
CRS-2673: Attempting to stop 'ora.mdnsd' on 'rhel6m1'
CRS-2673: Attempting to stop 'ora.gpnpd' on 'rhel6m1'
CRS-2677: Stop of 'ora.drivers.acfs' on 'rhel6m1' succeeded
CRS-2677: Stop of 'ora.storage' on 'rhel6m1' succeeded
CRS-2673: Attempting to stop 'ora.asm' on 'rhel6m1'
CRS-2677: Stop of 'ora.evmd' on 'rhel6m1' succeeded
CRS-2677: Stop of 'ora.ctssd' on 'rhel6m1' succeeded
CRS-2677: Stop of 'ora.mdnsd' on 'rhel6m1' succeeded
CRS-2677: Stop of 'ora.gpnpd' on 'rhel6m1' succeeded
CRS-2677: Stop of 'ora.asm' on 'rhel6m1' succeeded
CRS-2673: Attempting to stop 'ora.cluster_interconnect.haip' on 'rhel6m1'
CRS-2677: Stop of 'ora.cluster_interconnect.haip' on 'rhel6m1' succeeded
CRS-2673: Attempting to stop 'ora.cssd' on 'rhel6m1'
CRS-2677: Stop of 'ora.cssd' on 'rhel6m1' succeeded
CRS-2673: Attempting to stop 'ora.crf' on 'rhel6m1'
CRS-2677: Stop of 'ora.crf' on 'rhel6m1' succeeded
CRS-2673: Attempting to stop 'ora.gipcd' on 'rhel6m1'
CRS-2677: Stop of 'ora.gipcd' on 'rhel6m1' succeeded
CRS-2793: Shutdown of Oracle High Availability Services-managed resources on 'rhel6m1' has completed
CRS-4133: Oracle High Availability Services has been stopped.
CRS-4123: Oracle High Availability Services has been started.
CRS-2672: Attempting to start 'ora.evmd' on 'rhel6m1'
CRS-2672: Attempting to start 'ora.mdnsd' on 'rhel6m1'
CRS-2676: Start of 'ora.mdnsd' on 'rhel6m1' succeeded
CRS-2676: Start of 'ora.evmd' on 'rhel6m1' succeeded
CRS-2672: Attempting to start 'ora.gpnpd' on 'rhel6m1'
CRS-2676: Start of 'ora.gpnpd' on 'rhel6m1' succeeded
CRS-2672: Attempting to start 'ora.cssdmonitor' on 'rhel6m1'
CRS-2672: Attempting to start 'ora.gipcd' on 'rhel6m1'
CRS-2676: Start of 'ora.cssdmonitor' on 'rhel6m1' succeeded
CRS-2676: Start of 'ora.gipcd' on 'rhel6m1' succeeded
CRS-2672: Attempting to start 'ora.cssd' on 'rhel6m1'
CRS-2672: Attempting to start 'ora.diskmon' on 'rhel6m1'
CRS-2676: Start of 'ora.diskmon' on 'rhel6m1' succeeded
CRS-2676: Start of 'ora.cssd' on 'rhel6m1' succeeded
CRS-2672: Attempting to start 'ora.crf' on 'rhel6m1'
CRS-2672: Attempting to start 'ora.ctssd' on 'rhel6m1'
CRS-2672: Attempting to start 'ora.cluster_interconnect.haip' on 'rhel6m1'
CRS-2676: Start of 'ora.crf' on 'rhel6m1' succeeded
CRS-2676: Start of 'ora.ctssd' on 'rhel6m1' succeeded
CRS-2676: Start of 'ora.cluster_interconnect.haip' on 'rhel6m1' succeeded
CRS-2672: Attempting to start 'ora.asm' on 'rhel6m1'
CRS-2676: Start of 'ora.asm' on 'rhel6m1' succeeded
CRS-2672: Attempting to start 'ora.storage' on 'rhel6m1'
CRS-2676: Start of 'ora.storage' on 'rhel6m1' succeeded
CRS-2672: Attempting to start 'ora.crsd' on 'rhel6m1'
CRS-2676: Start of 'ora.crsd' on 'rhel6m1' succeeded
CRS-2673: Attempting to stop 'ora.crsd' on 'rhel6m1'
CRS-2677: Stop of 'ora.crsd' on 'rhel6m1' succeeded
CRS-2791: Starting shutdown of Oracle High Availability Services-managed resources on 'rhel6m1'
CRS-2673: Attempting to stop 'ora.ctssd' on 'rhel6m1'
CRS-2673: Attempting to stop 'ora.evmd' on 'rhel6m1'
CRS-2673: Attempting to stop 'ora.storage' on 'rhel6m1'
CRS-2673: Attempting to stop 'ora.drivers.acfs' on 'rhel6m1'
CRS-2673: Attempting to stop 'ora.gpnpd' on 'rhel6m1'
CRS-2673: Attempting to stop 'ora.mdnsd' on 'rhel6m1'
CRS-2677: Stop of 'ora.storage' on 'rhel6m1' succeeded
CRS-2673: Attempting to stop 'ora.asm' on 'rhel6m1'
CRS-2677: Stop of 'ora.drivers.acfs' on 'rhel6m1' succeeded
CRS-2677: Stop of 'ora.ctssd' on 'rhel6m1' succeeded
CRS-2677: Stop of 'ora.evmd' on 'rhel6m1' succeeded
CRS-2677: Stop of 'ora.gpnpd' on 'rhel6m1' succeeded
CRS-2677: Stop of 'ora.mdnsd' on 'rhel6m1' succeeded
CRS-2677: Stop of 'ora.asm' on 'rhel6m1' succeeded
CRS-2673: Attempting to stop 'ora.cluster_interconnect.haip' on 'rhel6m1'
CRS-2677: Stop of 'ora.cluster_interconnect.haip' on 'rhel6m1' succeeded
CRS-2673: Attempting to stop 'ora.cssd' on 'rhel6m1'
CRS-2677: Stop of 'ora.cssd' on 'rhel6m1' succeeded
CRS-2673: Attempting to stop 'ora.crf' on 'rhel6m1'
CRS-2677: Stop of 'ora.crf' on 'rhel6m1' succeeded
CRS-2673: Attempting to stop 'ora.gipcd' on 'rhel6m1'
CRS-2677: Stop of 'ora.gipcd' on 'rhel6m1' succeeded
CRS-2793: Shutdown of Oracle High Availability Services-managed resources on 'rhel6m1' has completed
CRS-4133: Oracle High Availability Services has been stopped.
CRS-4123: Oracle High Availability Services has been started.
CRS-2672: Attempting to start 'ora.mdnsd' on 'rhel6m1'
CRS-2676: Start of 'ora.mdnsd' on 'rhel6m1' succeeded
CRS-2672: Attempting to start 'ora.gpnpd' on 'rhel6m1'
CRS-2676: Start of 'ora.gpnpd' on 'rhel6m1' succeeded
CRS-2672: Attempting to start 'ora.cssdmonitor' on 'rhel6m1'
CRS-2672: Attempting to start 'ora.gipcd' on 'rhel6m1'
CRS-2676: Start of 'ora.cssdmonitor' on 'rhel6m1' succeeded
CRS-2676: Start of 'ora.gipcd' on 'rhel6m1' succeeded
CRS-2672: Attempting to start 'ora.cssd' on 'rhel6m1'
CRS-2672: Attempting to start 'ora.diskmon' on 'rhel6m1'
CRS-2676: Start of 'ora.diskmon' on 'rhel6m1' succeeded
CRS-2676: Start of 'ora.cssd' on 'rhel6m1' succeeded
CRS-2672: Attempting to start 'ora.drivers.acfs' on 'rhel6m1'
CRS-2679: Attempting to clean 'ora.cluster_interconnect.haip' on 'rhel6m1'
CRS-2672: Attempting to start 'ora.ctssd' on 'rhel6m1'
CRS-2681: Clean of 'ora.cluster_interconnect.haip' on 'rhel6m1' succeeded
CRS-2672: Attempting to start 'ora.cluster_interconnect.haip' on 'rhel6m1'
CRS-2674: Start of 'ora.drivers.acfs' on 'rhel6m1' failed
CRS-2676: Start of 'ora.ctssd' on 'rhel6m1' succeeded
CRS-2676: Start of 'ora.cluster_interconnect.haip' on 'rhel6m1' succeeded
CRS-2672: Attempting to start 'ora.asm' on 'rhel6m1'
CRS-2676: Start of 'ora.asm' on 'rhel6m1' succeeded
2015/04/01 12:15:49 CLSRSC-338: Successfully downgraded OCR to version 11.2.0.4.0

CRS-2672: Attempting to start 'ora.crsd' on 'rhel6m1'
CRS-2676: Start of 'ora.crsd' on 'rhel6m1' succeeded
2015/04/01 12:20:01 CLSRSC-4001: Installing Oracle Trace File Analyzer (TFA) Collector.

2015/04/01 12:20:01 CLSRSC-4002: Successfully installed Oracle Trace File Analyzer (TFA) Collector.

Successfully downgraded Oracle Clusterware stack on this node
Run '/opt/app/11.2.0/grid4/bin/crsctl start crs' on all nodes to complete downgrade

Before starting the cluster with 11.2 update the inventory by setting crs=true for 11.2 GI home. At this point 12.1 GI home will have crs=true.

<HOME NAME="Ora11g_gridinfrahome2" LOC="/opt/app/11.2.0/grid4" TYPE="O" IDX="4">
   <NODE_LIST>
      <NODE NAME="rhel6m1"/>
      <NODE NAME="rhel6m2"/>
   </NODE_LIST>
</HOME>
<HOME NAME="OraGI12Home1" LOC="/opt/app/12.1.0/grid2" TYPE="O" IDX="5" CRS="true">
   <NODE_LIST>
      <NODE NAME="rhel6m1"/>
      <NODE NAME="rhel6m2"/>
   </NODE_LIST>
</HOME>

Run the following to update the inventory

cd /opt/app/12.1.0/grid2/oui/bin/
[grid@rhel6m1 bin]$ ./runInstaller -nowait -waitforcompletion -ignoreSysPrereqs -updateNodeList -silent CRS=false ORACLE_HOME=/opt/app/12.1.0/grid2
Starting Oracle Universal Installer...
Checking swap space: must be greater than 500 MB.   Actual 4095 MB    Passed
The inventory pointer is located at /etc/oraInst.loc
'UpdateNodeList' was successful.

[grid@rhel6m1 bin]$ ./runInstaller -nowait -waitforcompletion -ignoreSysPrereqs -updateNodeList -silent CRS=true ORACLE_HOME=/opt/app/11.2.0/grid4
Starting Oracle Universal Installer...
Checking swap space: must be greater than 500 MB.   Actual 4095 MB    Passed
The inventory pointer is located at /etc/oraInst.loc
'UpdateNodeList' was successful.

Verify the inventory update

<HOME NAME="Ora11g_gridinfrahome2" LOC="/opt/app/11.2.0/grid4" TYPE="O" IDX="4" CRS="true">
   <NODE_LIST>
      <NODE NAME="rhel6m1"/>
      <NODE NAME="rhel6m2"/>
   </NODE_LIST>
</HOME>
<HOME NAME="OraGI12Home1" LOC="/opt/app/12.1.0/grid2" TYPE="O" IDX="5">
   <NODE_LIST>
      <NODE NAME="rhel6m1"/>
      <NODE NAME="rhel6m2"/>
   </NODE_LIST>
</HOME>

Also make sure the contents in /etc/init.d/ohasd and /etc/init.d/init.ohasd refers the 11.2 as the ORA_CRS_HOME and there are no references to 12.1. Few times the last downgrade node (rhel6m1) did contain references to 12.1 even though downgrade command had successfully completed. However these files on the other node had the correct references to 11.2.

[root@rhel6m1 grid]#  cat /etc/init.d/ohasd | grep ORA_CRS_HOME
ORA_CRS_HOME=/opt/app/11.2.0/grid4
export ORA_CRS_HOME

[root@rhel6m1 grid]#  cat /etc/init.d/init.ohasd | grep ORA_CRS_HOME
ORA_CRS_HOME=/opt/app/11.2.0/grid4
export ORA_CRS_HOME
PERL="/opt/app/11.2.0/grid4/perl/bin/perl -I${ORA_CRS_HOME}/perl/lib"

Also as part of the downgrade /etc/oratab is modified. All references to MGMTDB are moved and ASM instances homes are set to 11.2. But this only happens on last node (rhel6m1). For other nodes /etc/oratab did not have any ASM related entry. Manually add this ASM related entry to /etc/oratab

+ASM2:/opt/app/11.2.0/grid4:N

Before starting the crs unset any references to 12.1 binaries from the environment (PATH, ORACLE_HOME variables). Make sure crsctl use to start the crs comes from 11.2 GI home.

[grid@rhel6m1 bin]$ . oraenv
ORACLE_SID = [+ASM1] ? +ASM1
The Oracle base remains unchanged with value /opt/app/oracle
[grid@rhel6m1 bin]$ which crsctl
/opt/app/11.2.0/grid4/bin/crsctl

Start the crs on all nodes.

# crsctl start crs

crsctl query crs activeversion -f
Oracle Clusterware active version on the cluster is [11.2.0.4.0]. The cluster upgrade state is [NORMAL].

crsctl query crs softwareversion
Oracle Clusterware version on node [rhel6m1] is [11.2.0.4.0]

[root@rhel6m1 grid]# ocrcheck
Status of Oracle Cluster Registry is as follows :
         Version                  :          3
         Total space (kbytes)     :     262120
         Used space (kbytes)      :       3356
         Available space (kbytes) :     258764
         ID                       : 2072206343
         Device/File Name         : +CLUSTER_DG

This concludes the steps to successfully downgrading GI from 12.1.0.2 to 11.2.0.4.

There could be occasions the downgrade command successfully completes but start of crs fails. Symptoms in this case include unable to discover any voting disks crsctl query css votedisk doesn't return any vote disk infomration and ocssd.log will have entries similar to

2015-03-31 14:00:09.618: [    CSSD][898090752]clssnmvDiskVerify: Successful discovery of 0 disks
2015-03-31 14:00:09.618: [    CSSD][898090752]clssnmCompleteInitVFDiscovery: Completing initial voting file discovery
2015-03-31 14:00:09.618: [    CSSD][898090752]clssnmvFindInitialConfigs: No voting files found

Other times includes corrupted OCR with ocssd.log having entries similar to

2015-03-26 11:33:05.633: [ CRSMAIN][3817551648] Initialing cluclu context...
[  OCRMAS][3776734976]th_calc_av:8': Failed in vsnupr. Incorrect SV stored in OCR. Key [SYSTEM.version.hostnames.] Value []
2015-03-26 11:33:06.618: [  OCRSRV][3776734976]th_upgrade:9 Shutdown CacheMaster. prev AV [186647552] new calc av [186647552] my sv [186647552]

No root cause was found of these cases and could only assume this may be due to some of the earlier mentioned reasons such as having backups of OCR from previous upgrades i.e. ocr11.2.0.3.0 (But it must be said a successful downgrade was archived while ocr11.2.0.3.0 was in the cdata directory), wrong binaries referred during downgrade due to environment variable settings. The only option to recover from such a situation is to restore the OCR taken while cluster was 11.2.

crsctl stop crs -f # run on all nodes
crsctl start crs -excl -nocrs # run only on one node
ocrconfig -restore /opt/app/11.2.0/grid4/cdata/rhel6m-cluster/backup_20150331_170634.ocr
crsctl replace votedisk +cluster_dg

Useful metalink notes
rootcrs.pl -downgrade -lastnode" Fails if Current Directory Can not be Accessed by Grid User [ID 1917922.1]
SRVCTL commands fails with error CLSU-00100: Operating System function: sclsdgcwd failed with error data: -1 [ID 1961344.1]
Srvctl Status Commands Report CLSU-00100, CLSU-00101 and CLSU-00103 Errors [ID 1911258.1]

Related Posts
Downgrade Grid Infrastructure from 11.2.0.4 to 11.2.0.3
Downgrade Grid Infrastructure from 11.2.0.4 to 11.1.0.7