Sunday, October 1, 2017

Downgrade Grid Infrastructure from 12.2.0.1 to 11.2.0.4

This post list steps for downgrading grid infrastructure from 12.2.0.1 to 11.2.0.4. The upgrade to 12.2 GI was completed successfully on all nodes. No irreversible changes such as compatible.[asm|rdbms] updates have been done as part of the upgrade. Steps to downgrade would be different if the upgrade had failed on some nodes. For more refer Oracle documentation.
1. The current active version of the cluster is 12.2.0.1 and all nodes have the same version.
crsctl query crs activeversion
Oracle Clusterware active version on the cluster is [12.2.0.1.0]

crsctl query crs softwareversion -all
Oracle Clusterware version on node [rhel6m1] is [12.2.0.1.0]
Oracle Clusterware version on node [rhel6m2] is [12.2.0.1.0]
2. As part of the upgrade the OCR and vote disk were moved to ASM disk group with external redundancy called +GIMR to meet the space requirement for 12.2 upgrade. (Refer previous post on 11.2.0.4 to 12.2 GI upgrade for more) After the upgrade the OCR and Vote disk were moved back to a disk group of normal redundancy called +CLUSTER_DG. The +GIMR is used as the OCR backup location by the 12.2 GI.
ocrconfig -showbackuploc
The Oracle Cluster Registry backup location is [+GIMR]

ocrcheck
Status of Oracle Cluster Registry is as follows :
         Version                  :          4
         Total space (kbytes)     :     409568
         Used space (kbytes)      :      12056
         Available space (kbytes) :     397512
         ID                       : 1487892601
         Device/File Name         : +CLUSTER_DG
3. As shown above 12.2 the OCR and OCR backups reside in ASM and are accessible from all nodes. This is different from previous version where OCR backups reside on the local file system by default. This required to find out the "OCR-Node" before the downgrade. In 12.2 backups are in ASM any node can carry out the downgrade, no need to make a particular node the last to down grade. Previous version OCR node had to be the last one to downgrade. The OCR backup taken at the time of the upgrade could be listed with asmcmd.
ASMCMD> pwd
+gimr/rhel6m-cluster/OCRBACKUP

ASMCMD> ls -l *11.2.0.4*
Type       Redund  Striped  Time             Sys  Name
OCRBACKUP  UNPROT  COARSE   SEP 15 14:00:00  Y    rhel6m-cluster_backup11.2.0.4.0.ocr.256.954763871
4. First step in the downgrade to delete the management repository database (MGMTDB). Use DBCA silent option to drop the MGMTDB.
dbca -silent -deleteDatabase -sourceDB -MGMTDB
Connecting to database
4% complete
9% complete
14% complete
19% complete
23% complete
28% complete
47% complete
Updating network configuration files
52% complete
Deleting instance and datafiles
76% complete
100% complete
Look at the log file "/opt/app/oracle/cfgtoollogs/dbca/_mgmtdb.log" for further details.
After the delete verify that there's no ora.mgmtdb resource.
crsctl stat res -t
--------------------------------------------------------------------------------
Name           Target  State        Server                   State details
--------------------------------------------------------------------------------
Local Resources
--------------------------------------------------------------------------------
ora.ASMNET1LSNR_ASM.lsnr
               ONLINE  ONLINE       rhel6m1                  STABLE
               ONLINE  ONLINE       rhel6m2                  STABLE
ora.CLUSTER_DG.dg
               ONLINE  ONLINE       rhel6m1                  STABLE
               ONLINE  ONLINE       rhel6m2                  STABLE
ora.DATA.dg
               ONLINE  ONLINE       rhel6m1                  STABLE
               ONLINE  ONLINE       rhel6m2                  STABLE
ora.FLASH.dg
               ONLINE  ONLINE       rhel6m1                  STABLE
               ONLINE  ONLINE       rhel6m2                  STABLE
ora.GIMR.dg
               ONLINE  ONLINE       rhel6m1                  STABLE
               ONLINE  ONLINE       rhel6m2                  STABLE
ora.MYLISTENER.lsnr
               ONLINE  ONLINE       rhel6m1                  STABLE
               ONLINE  ONLINE       rhel6m2                  STABLE
ora.chad
               ONLINE  OFFLINE      rhel6m1                  STABLE
               ONLINE  OFFLINE      rhel6m2                  STABLE
ora.net1.network
               ONLINE  ONLINE       rhel6m1                  STABLE
               ONLINE  ONLINE       rhel6m2                  STABLE
ora.ons
               ONLINE  ONLINE       rhel6m1                  STABLE
               ONLINE  ONLINE       rhel6m2                  STABLE
ora.proxy_advm
               OFFLINE OFFLINE      rhel6m1                  STABLE
               OFFLINE OFFLINE      rhel6m2                  STABLE
--------------------------------------------------------------------------------
Cluster Resources
--------------------------------------------------------------------------------
ora.MGMTLSNR
      1        ONLINE  ONLINE       rhel6m1                  169.254.31.63 192.16
                                                             8.1.87,STABLE
ora.MYLISTENER_SCAN1.lsnr
      1        ONLINE  ONLINE       rhel6m1                  STABLE
ora.MYLISTENER_SCAN2.lsnr
      1        ONLINE  ONLINE       rhel6m2                  STABLE
ora.MYLISTENER_SCAN3.lsnr
      1        ONLINE  ONLINE       rhel6m2                  STABLE
ora.asm
      1        ONLINE  ONLINE       rhel6m1                  Started,STABLE
      2        ONLINE  ONLINE       rhel6m2                  Started,STABLE
ora.cvu
      1        ONLINE  ONLINE       rhel6m2                  STABLE
ora.qosmserver
      1        ONLINE  ONLINE       rhel6m2                  STABLE
ora.rhel6m1.vip
      1        ONLINE  ONLINE       rhel6m1                  STABLE
ora.rhel6m2.vip
      1        ONLINE  ONLINE       rhel6m2                  STABLE
ora.scan1.vip
      1        ONLINE  ONLINE       rhel6m1                  STABLE
ora.scan2.vip
      1        ONLINE  ONLINE       rhel6m2                  STABLE
ora.scan3.vip
      1        ONLINE  ONLINE       rhel6m2                  STABLE
ora.std11g2.db
      1        ONLINE  ONLINE       rhel6m1                  Open,HOME=/opt/app/o
                                                             racle/product/11.2.0
                                                             /dbhome_4,STABLE
      2        ONLINE  ONLINE       rhel6m2                  Open,HOME=/opt/app/o
                                                             racle/product/11.2.0
                                                             /dbhome_4,STABLE
ora.std11g2.myservice.svc
      1        ONLINE  ONLINE       rhel6m1                  STABLE
      2        ONLINE  ONLINE       rhel6m2                  STABLE
ora.std11g2.bx.domain.net.svc
      1        ONLINE  ONLINE       rhel6m2                  STABLE
      2        ONLINE  ONLINE       rhel6m1                  STABLE
If there's any residual details about mgmtdb or if it is not clearly removed then downgrade could fail.
2017/09/12 14:16:38 CLSRSC-503: Deleting the Grid Infrastructure Management Repository database failed.
Died at /opt/app/12.2.0/grid/crs/install/crsdowngrade.pm line 1143.
The command '/opt/app/12.2.0/grid/perl/bin/perl -I/opt/app/12.2.0/grid/perl/lib -I/opt/app/12.2.0/grid/crs/install /opt/app/12.2.0/grid/crs/install/rootcrs.pl -downgrade' execution failed
5. Run the downgrade from all but one of the nodes. Since this is a 2 node cluster downgrade has to be done one node at a time.
[root@rhel6m1 grid]# /opt/app/12.2.0/grid/crs/install/rootcrs.sh -downgrade
Using configuration parameter file: /opt/app/12.2.0/grid/crs/install/crsconfig_params
The log of current session can be found at:
  /opt/app/oracle/crsdata/rhel6m1/crsconfig/crsdowngrade_rhel6m1_2017-09-15_01-58-55PM.log
CRS-2791: Starting shutdown of Oracle High Availability Services-managed resources on 'rhel6m1'
CRS-2673: Attempting to stop 'ora.crsd' on 'rhel6m1'
CRS-2790: Starting shutdown of Cluster Ready Services-managed resources on server 'rhel6m1'
CRS-2673: Attempting to stop 'ora.std11g2.myservice.svc' on 'rhel6m1'
CRS-2673: Attempting to stop 'ora.std11g2.bx.domain.net.svc' on 'rhel6m1'
CRS-2673: Attempting to stop 'ora.MGMTLSNR' on 'rhel6m1'
CRS-2677: Stop of 'ora.std11g2.myservice.svc' on 'rhel6m1' succeeded
CRS-2677: Stop of 'ora.std11g2.bx.domain.net.svc' on 'rhel6m1' succeeded
CRS-2673: Attempting to stop 'ora.std11g2.db' on 'rhel6m1'
CRS-2677: Stop of 'ora.MGMTLSNR' on 'rhel6m1' succeeded
CRS-2677: Stop of 'ora.std11g2.db' on 'rhel6m1' succeeded
CRS-2673: Attempting to stop 'ora.CLUSTER_DG.dg' on 'rhel6m1'
CRS-2673: Attempting to stop 'ora.GIMR.dg' on 'rhel6m1'
CRS-2673: Attempting to stop 'ora.DATA.dg' on 'rhel6m1'
CRS-2673: Attempting to stop 'ora.FLASH.dg' on 'rhel6m1'
CRS-2673: Attempting to stop 'ora.MYLISTENER.lsnr' on 'rhel6m1'
CRS-2673: Attempting to stop 'ora.MYLISTENER_SCAN1.lsnr' on 'rhel6m1'
CRS-2677: Stop of 'ora.DATA.dg' on 'rhel6m1' succeeded
CRS-2677: Stop of 'ora.FLASH.dg' on 'rhel6m1' succeeded
CRS-2677: Stop of 'ora.CLUSTER_DG.dg' on 'rhel6m1' succeeded
CRS-2677: Stop of 'ora.GIMR.dg' on 'rhel6m1' succeeded
CRS-2673: Attempting to stop 'ora.asm' on 'rhel6m1'
CRS-2677: Stop of 'ora.MYLISTENER.lsnr' on 'rhel6m1' succeeded
CRS-2673: Attempting to stop 'ora.rhel6m1.vip' on 'rhel6m1'
CRS-2677: Stop of 'ora.asm' on 'rhel6m1' succeeded
CRS-2673: Attempting to stop 'ora.ASMNET1LSNR_ASM.lsnr' on 'rhel6m1'
CRS-2677: Stop of 'ora.MYLISTENER_SCAN1.lsnr' on 'rhel6m1' succeeded
CRS-2673: Attempting to stop 'ora.scan1.vip' on 'rhel6m1'
CRS-2677: Stop of 'ora.rhel6m1.vip' on 'rhel6m1' succeeded
CRS-2677: Stop of 'ora.scan1.vip' on 'rhel6m1' succeeded
CRS-2677: Stop of 'ora.ASMNET1LSNR_ASM.lsnr' on 'rhel6m1' succeeded
CRS-2672: Attempting to start 'ora.MGMTLSNR' on 'rhel6m2'
CRS-2672: Attempting to start 'ora.scan1.vip' on 'rhel6m2'
CRS-2672: Attempting to start 'ora.rhel6m1.vip' on 'rhel6m2'
CRS-2676: Start of 'ora.scan1.vip' on 'rhel6m2' succeeded
CRS-2672: Attempting to start 'ora.MYLISTENER_SCAN1.lsnr' on 'rhel6m2'
CRS-2676: Start of 'ora.rhel6m1.vip' on 'rhel6m2' succeeded
CRS-2676: Start of 'ora.MGMTLSNR' on 'rhel6m2' succeeded
CRS-2676: Start of 'ora.MYLISTENER_SCAN1.lsnr' on 'rhel6m2' succeeded
CRS-2673: Attempting to stop 'ora.ons' on 'rhel6m1'
CRS-2677: Stop of 'ora.ons' on 'rhel6m1' succeeded
CRS-2673: Attempting to stop 'ora.net1.network' on 'rhel6m1'
CRS-2677: Stop of 'ora.net1.network' on 'rhel6m1' succeeded
CRS-2792: Shutdown of Cluster Ready Services-managed resources on 'rhel6m1' has completed
CRS-2677: Stop of 'ora.crsd' on 'rhel6m1' succeeded
CRS-2673: Attempting to stop 'ora.storage' on 'rhel6m1'
CRS-2673: Attempting to stop 'ora.crf' on 'rhel6m1'
CRS-2673: Attempting to stop 'ora.drivers.acfs' on 'rhel6m1'
CRS-2673: Attempting to stop 'ora.gpnpd' on 'rhel6m1'
CRS-2673: Attempting to stop 'ora.mdnsd' on 'rhel6m1'
CRS-2677: Stop of 'ora.drivers.acfs' on 'rhel6m1' succeeded
CRS-2677: Stop of 'ora.storage' on 'rhel6m1' succeeded
CRS-2673: Attempting to stop 'ora.asm' on 'rhel6m1'
CRS-2677: Stop of 'ora.crf' on 'rhel6m1' succeeded
CRS-2677: Stop of 'ora.gpnpd' on 'rhel6m1' succeeded
CRS-2677: Stop of 'ora.mdnsd' on 'rhel6m1' succeeded
CRS-2677: Stop of 'ora.asm' on 'rhel6m1' succeeded
CRS-2673: Attempting to stop 'ora.cluster_interconnect.haip' on 'rhel6m1'
CRS-2677: Stop of 'ora.cluster_interconnect.haip' on 'rhel6m1' succeeded
CRS-2673: Attempting to stop 'ora.ctssd' on 'rhel6m1'
CRS-2673: Attempting to stop 'ora.evmd' on 'rhel6m1'
CRS-2677: Stop of 'ora.ctssd' on 'rhel6m1' succeeded
CRS-2677: Stop of 'ora.evmd' on 'rhel6m1' succeeded
CRS-2673: Attempting to stop 'ora.cssd' on 'rhel6m1'
CRS-2677: Stop of 'ora.cssd' on 'rhel6m1' succeeded
CRS-2673: Attempting to stop 'ora.gipcd' on 'rhel6m1'
CRS-2677: Stop of 'ora.gipcd' on 'rhel6m1' succeeded
CRS-2793: Shutdown of Oracle High Availability Services-managed resources on 'rhel6m1' has completed
CRS-4133: Oracle High Availability Services has been stopped.
2017/09/15 14:01:49 CLSRSC-4006: Removing Oracle Trace File Analyzer (TFA) Collector.
2017/09/15 14:02:16 CLSRSC-4007: Successfully removed Oracle Trace File Analyzer (TFA) Collector.
2017/09/15 14:02:16 CLSRSC-4001: Installing Oracle Trace File Analyzer (TFA) Collector.
2017/09/15 14:02:55 CLSRSC-4002: Successfully installed Oracle Trace File Analyzer (TFA) Collector.
2017/09/15 14:02:55 CLSRSC-591: successfully downgraded Oracle Clusterware stack on this node


6. Run the downgrade from the last node. There's no need to specify the "-lastnode" option anymore as with previous downgrades. Specifying this would result in command failing. It is automatically detected as the last node. This could be verified by referring the generated log (output given at the end).
[root@rhel6m2 ~]# /opt/app/12.2.0/grid/crs/install/rootcrs.sh -downgrade
Using configuration parameter file: /opt/app/12.2.0/grid/crs/install/crsconfig_params
The log of current session can be found at:
  /opt/app/oracle/crsdata/rhel6m2/crsconfig/crsdowngrade_rhel6m2_2017-09-15_02-03-24PM.log
CRS-2791: Starting shutdown of Oracle High Availability Services-managed resources on 'rhel6m2'
CRS-2673: Attempting to stop 'ora.crsd' on 'rhel6m2'
CRS-2790: Starting shutdown of Cluster Ready Services-managed resources on server 'rhel6m2'
CRS-2673: Attempting to stop 'ora.qosmserver' on 'rhel6m2'
CRS-2673: Attempting to stop 'ora.std11g2.bx.domain.net.svc' on 'rhel6m2'
CRS-2673: Attempting to stop 'ora.std11g2.myservice.svc' on 'rhel6m2'
CRS-2673: Attempting to stop 'ora.MGMTLSNR' on 'rhel6m2'
CRS-2677: Stop of 'ora.std11g2.bx.domain.net.svc' on 'rhel6m2' succeeded
CRS-2677: Stop of 'ora.std11g2.myservice.svc' on 'rhel6m2' succeeded
CRS-2673: Attempting to stop 'ora.std11g2.db' on 'rhel6m2'
CRS-2677: Stop of 'ora.std11g2.db' on 'rhel6m2' succeeded
CRS-2673: Attempting to stop 'ora.CLUSTER_DG.dg' on 'rhel6m2'
CRS-2673: Attempting to stop 'ora.GIMR.dg' on 'rhel6m2'
CRS-2673: Attempting to stop 'ora.DATA.dg' on 'rhel6m2'
CRS-2673: Attempting to stop 'ora.FLASH.dg' on 'rhel6m2'
CRS-2673: Attempting to stop 'ora.MYLISTENER.lsnr' on 'rhel6m2'
CRS-2673: Attempting to stop 'ora.MYLISTENER_SCAN1.lsnr' on 'rhel6m2'
CRS-2673: Attempting to stop 'ora.MYLISTENER_SCAN2.lsnr' on 'rhel6m2'
CRS-2673: Attempting to stop 'ora.MYLISTENER_SCAN3.lsnr' on 'rhel6m2'
CRS-2673: Attempting to stop 'ora.cvu' on 'rhel6m2'
CRS-2673: Attempting to stop 'ora.rhel6m1.vip' on 'rhel6m2'
CRS-2677: Stop of 'ora.FLASH.dg' on 'rhel6m2' succeeded
CRS-2677: Stop of 'ora.CLUSTER_DG.dg' on 'rhel6m2' succeeded
CRS-2677: Stop of 'ora.DATA.dg' on 'rhel6m2' succeeded
CRS-2677: Stop of 'ora.cvu' on 'rhel6m2' succeeded
CRS-2677: Stop of 'ora.MYLISTENER.lsnr' on 'rhel6m2' succeeded
CRS-2677: Stop of 'ora.MYLISTENER_SCAN1.lsnr' on 'rhel6m2' succeeded
CRS-2673: Attempting to stop 'ora.scan1.vip' on 'rhel6m2'
CRS-2677: Stop of 'ora.MYLISTENER_SCAN3.lsnr' on 'rhel6m2' succeeded
CRS-2673: Attempting to stop 'ora.scan3.vip' on 'rhel6m2'
CRS-2677: Stop of 'ora.MYLISTENER_SCAN2.lsnr' on 'rhel6m2' succeeded
CRS-2673: Attempting to stop 'ora.scan2.vip' on 'rhel6m2'
CRS-2677: Stop of 'ora.GIMR.dg' on 'rhel6m2' succeeded
CRS-2673: Attempting to stop 'ora.asm' on 'rhel6m2'
CRS-2677: Stop of 'ora.rhel6m1.vip' on 'rhel6m2' succeeded
CRS-2677: Stop of 'ora.asm' on 'rhel6m2' succeeded
CRS-2673: Attempting to stop 'ora.ASMNET1LSNR_ASM.lsnr' on 'rhel6m2'
CRS-2677: Stop of 'ora.scan1.vip' on 'rhel6m2' succeeded
CRS-2677: Stop of 'ora.MGMTLSNR' on 'rhel6m2' succeeded
CRS-2673: Attempting to stop 'ora.rhel6m2.vip' on 'rhel6m2'
CRS-2677: Stop of 'ora.qosmserver' on 'rhel6m2' succeeded
CRS-2677: Stop of 'ora.ASMNET1LSNR_ASM.lsnr' on 'rhel6m2' succeeded
CRS-2677: Stop of 'ora.scan3.vip' on 'rhel6m2' succeeded
CRS-2677: Stop of 'ora.scan2.vip' on 'rhel6m2' succeeded
CRS-2677: Stop of 'ora.rhel6m2.vip' on 'rhel6m2' succeeded
CRS-2673: Attempting to stop 'ora.ons' on 'rhel6m2'
CRS-2677: Stop of 'ora.ons' on 'rhel6m2' succeeded
CRS-2673: Attempting to stop 'ora.net1.network' on 'rhel6m2'
CRS-2677: Stop of 'ora.net1.network' on 'rhel6m2' succeeded
CRS-2792: Shutdown of Cluster Ready Services-managed resources on 'rhel6m2' has completed
CRS-2677: Stop of 'ora.crsd' on 'rhel6m2' succeeded
CRS-2673: Attempting to stop 'ora.asm' on 'rhel6m2'
CRS-2673: Attempting to stop 'ora.crf' on 'rhel6m2'
CRS-2673: Attempting to stop 'ora.drivers.acfs' on 'rhel6m2'
CRS-2673: Attempting to stop 'ora.gpnpd' on 'rhel6m2'
CRS-2673: Attempting to stop 'ora.mdnsd' on 'rhel6m2'
CRS-2677: Stop of 'ora.drivers.acfs' on 'rhel6m2' succeeded
CRS-2677: Stop of 'ora.crf' on 'rhel6m2' succeeded
CRS-2677: Stop of 'ora.gpnpd' on 'rhel6m2' succeeded
CRS-2677: Stop of 'ora.mdnsd' on 'rhel6m2' succeeded
CRS-2677: Stop of 'ora.asm' on 'rhel6m2' succeeded
CRS-2673: Attempting to stop 'ora.cluster_interconnect.haip' on 'rhel6m2'
CRS-2677: Stop of 'ora.cluster_interconnect.haip' on 'rhel6m2' succeeded
CRS-2673: Attempting to stop 'ora.ctssd' on 'rhel6m2'
CRS-2673: Attempting to stop 'ora.evmd' on 'rhel6m2'
CRS-2677: Stop of 'ora.ctssd' on 'rhel6m2' succeeded
CRS-2677: Stop of 'ora.evmd' on 'rhel6m2' succeeded
CRS-2673: Attempting to stop 'ora.cssd' on 'rhel6m2'
CRS-2677: Stop of 'ora.cssd' on 'rhel6m2' succeeded
CRS-2673: Attempting to stop 'ora.gipcd' on 'rhel6m2'
CRS-2677: Stop of 'ora.gipcd' on 'rhel6m2' succeeded
CRS-2793: Shutdown of Oracle High Availability Services-managed resources on 'rhel6m2' has completed
CRS-4133: Oracle High Availability Services has been stopped.

ASM downgrade operation succeeded

CRS-2791: Starting shutdown of Oracle High Availability Services-managed resources on 'rhel6m2'
CRS-2673: Attempting to stop 'ora.crsd' on 'rhel6m2'
CRS-2677: Stop of 'ora.crsd' on 'rhel6m2' succeeded
CRS-2673: Attempting to stop 'ora.storage' on 'rhel6m2'
CRS-2673: Attempting to stop 'ora.crf' on 'rhel6m2'
CRS-2673: Attempting to stop 'ora.drivers.acfs' on 'rhel6m2'
CRS-2673: Attempting to stop 'ora.gpnpd' on 'rhel6m2'
CRS-2673: Attempting to stop 'ora.mdnsd' on 'rhel6m2'
CRS-2677: Stop of 'ora.drivers.acfs' on 'rhel6m2' succeeded
CRS-2677: Stop of 'ora.crf' on 'rhel6m2' succeeded
CRS-2677: Stop of 'ora.gpnpd' on 'rhel6m2' succeeded
CRS-2677: Stop of 'ora.storage' on 'rhel6m2' succeeded
CRS-2673: Attempting to stop 'ora.asm' on 'rhel6m2'
CRS-2677: Stop of 'ora.mdnsd' on 'rhel6m2' succeeded
CRS-2677: Stop of 'ora.asm' on 'rhel6m2' succeeded
CRS-2673: Attempting to stop 'ora.cluster_interconnect.haip' on 'rhel6m2'
CRS-2677: Stop of 'ora.cluster_interconnect.haip' on 'rhel6m2' succeeded
CRS-2673: Attempting to stop 'ora.ctssd' on 'rhel6m2'
CRS-2673: Attempting to stop 'ora.evmd' on 'rhel6m2'
CRS-2677: Stop of 'ora.ctssd' on 'rhel6m2' succeeded
CRS-2677: Stop of 'ora.evmd' on 'rhel6m2' succeeded
CRS-2673: Attempting to stop 'ora.cssd' on 'rhel6m2'
CRS-2677: Stop of 'ora.cssd' on 'rhel6m2' succeeded
CRS-2673: Attempting to stop 'ora.gipcd' on 'rhel6m2'
CRS-2677: Stop of 'ora.gipcd' on 'rhel6m2' succeeded
CRS-2793: Shutdown of Oracle High Availability Services-managed resources on 'rhel6m2' has completed
CRS-4133: Oracle High Availability Services has been stopped.
CRS-4123: Starting Oracle High Availability Services-managed resources
CRS-2672: Attempting to start 'ora.mdnsd' on 'rhel6m2'
CRS-2672: Attempting to start 'ora.evmd' on 'rhel6m2'
CRS-2676: Start of 'ora.mdnsd' on 'rhel6m2' succeeded
CRS-2676: Start of 'ora.evmd' on 'rhel6m2' succeeded
CRS-2672: Attempting to start 'ora.gpnpd' on 'rhel6m2'
CRS-2676: Start of 'ora.gpnpd' on 'rhel6m2' succeeded
CRS-2672: Attempting to start 'ora.gipcd' on 'rhel6m2'
CRS-2676: Start of 'ora.gipcd' on 'rhel6m2' succeeded
CRS-2672: Attempting to start 'ora.cssdmonitor' on 'rhel6m2'
CRS-2676: Start of 'ora.cssdmonitor' on 'rhel6m2' succeeded
CRS-2672: Attempting to start 'ora.cssd' on 'rhel6m2'
CRS-2672: Attempting to start 'ora.diskmon' on 'rhel6m2'
CRS-2676: Start of 'ora.diskmon' on 'rhel6m2' succeeded
CRS-2676: Start of 'ora.cssd' on 'rhel6m2' succeeded
CRS-2672: Attempting to start 'ora.cluster_interconnect.haip' on 'rhel6m2'
CRS-2672: Attempting to start 'ora.ctssd' on 'rhel6m2'
CRS-2676: Start of 'ora.ctssd' on 'rhel6m2' succeeded
CRS-2676: Start of 'ora.cluster_interconnect.haip' on 'rhel6m2' succeeded
CRS-2672: Attempting to start 'ora.asm' on 'rhel6m2'
CRS-2676: Start of 'ora.asm' on 'rhel6m2' succeeded
CRS-2672: Attempting to start 'ora.storage' on 'rhel6m2'
CRS-2676: Start of 'ora.storage' on 'rhel6m2' succeeded
CRS-2672: Attempting to start 'ora.crf' on 'rhel6m2'
CRS-2676: Start of 'ora.crf' on 'rhel6m2' succeeded
CRS-2672: Attempting to start 'ora.crsd' on 'rhel6m2'
CRS-2676: Start of 'ora.crsd' on 'rhel6m2' succeeded
CRS-6023: Starting Oracle Cluster Ready Services-managed resources
CRS-2664: Resource 'ora.DATA.dg' is already running on 'rhel6m2'
CRS-2664: Resource 'ora.CLUSTER_DG.dg' is already running on 'rhel6m2'
CRS-2664: Resource 'ora.FLASH.dg' is already running on 'rhel6m2'
CRS-2664: Resource 'ora.GIMR.dg' is already running on 'rhel6m2'
CRS-6017: Processing resource auto-start for servers: rhel6m2
CRS-2672: Attempting to start 'ora.scan1.vip' on 'rhel6m2'
CRS-2672: Attempting to start 'ora.scan2.vip' on 'rhel6m2'
CRS-2672: Attempting to start 'ora.rhel6m2.vip' on 'rhel6m2'
CRS-2672: Attempting to start 'ora.scan3.vip' on 'rhel6m2'
CRS-2672: Attempting to start 'ora.qosmserver' on 'rhel6m2'
CRS-2672: Attempting to start 'ora.ons' on 'rhel6m2'
CRS-2672: Attempting to start 'ora.rhel6m1.vip' on 'rhel6m2'
CRS-2672: Attempting to start 'ora.cvu' on 'rhel6m2'
CRS-2676: Start of 'ora.scan1.vip' on 'rhel6m2' succeeded
CRS-2672: Attempting to start 'ora.MYLISTENER_SCAN1.lsnr' on 'rhel6m2'
CRS-2676: Start of 'ora.scan2.vip' on 'rhel6m2' succeeded
CRS-2672: Attempting to start 'ora.MYLISTENER_SCAN2.lsnr' on 'rhel6m2'
CRS-2676: Start of 'ora.cvu' on 'rhel6m2' succeeded
CRS-2676: Start of 'ora.rhel6m2.vip' on 'rhel6m2' succeeded
CRS-2672: Attempting to start 'ora.MGMTLSNR' on 'rhel6m2'
CRS-2672: Attempting to start 'ora.MYLISTENER.lsnr' on 'rhel6m2'
CRS-2676: Start of 'ora.scan3.vip' on 'rhel6m2' succeeded
CRS-2672: Attempting to start 'ora.MYLISTENER_SCAN3.lsnr' on 'rhel6m2'
CRS-2676: Start of 'ora.rhel6m1.vip' on 'rhel6m2' succeeded
CRS-2676: Start of 'ora.ons' on 'rhel6m2' succeeded
CRS-2676: Start of 'ora.MGMTLSNR' on 'rhel6m2' succeeded
CRS-2676: Start of 'ora.MYLISTENER.lsnr' on 'rhel6m2' succeeded
CRS-2676: Start of 'ora.MYLISTENER_SCAN1.lsnr' on 'rhel6m2' succeeded
CRS-2672: Attempting to start 'ora.std11g2.db' on 'rhel6m2'
CRS-2676: Start of 'ora.MYLISTENER_SCAN2.lsnr' on 'rhel6m2' succeeded
CRS-2676: Start of 'ora.MYLISTENER_SCAN3.lsnr' on 'rhel6m2' succeeded
CRS-2676: Start of 'ora.qosmserver' on 'rhel6m2' succeeded
CRS-2676: Start of 'ora.std11g2.db' on 'rhel6m2' succeeded
CRS-2672: Attempting to start 'ora.std11g2.myservice.svc' on 'rhel6m2'
CRS-2672: Attempting to start 'ora.std11g2.bx.domain.net.svc' on 'rhel6m2'
CRS-2676: Start of 'ora.std11g2.bx.domain.net.svc' on 'rhel6m2' succeeded
CRS-2676: Start of 'ora.std11g2.myservice.svc' on 'rhel6m2' succeeded
CRS-6016: Resource auto-start has completed for server rhel6m2
CRS-6024: Completed start of Oracle Cluster Ready Services-managed resources
CRS-4123: Oracle High Availability Services has been started.
CRS-2791: Starting shutdown of Oracle High Availability Services-managed resources on 'rhel6m2'
CRS-2673: Attempting to stop 'ora.crsd' on 'rhel6m2'
CRS-2790: Starting shutdown of Cluster Ready Services-managed resources on server 'rhel6m2'
CRS-2673: Attempting to stop 'ora.std11g2.bx.domain.net.svc' on 'rhel6m2'
CRS-2673: Attempting to stop 'ora.MGMTLSNR' on 'rhel6m2'
CRS-2673: Attempting to stop 'ora.qosmserver' on 'rhel6m2'
CRS-2673: Attempting to stop 'ora.std11g2.myservice.svc' on 'rhel6m2'
CRS-2677: Stop of 'ora.std11g2.myservice.svc' on 'rhel6m2' succeeded
CRS-2677: Stop of 'ora.std11g2.bx.domain.net.svc' on 'rhel6m2' succeeded
CRS-2673: Attempting to stop 'ora.std11g2.db' on 'rhel6m2'
CRS-2677: Stop of 'ora.std11g2.db' on 'rhel6m2' succeeded
CRS-2673: Attempting to stop 'ora.CLUSTER_DG.dg' on 'rhel6m2'
CRS-2673: Attempting to stop 'ora.GIMR.dg' on 'rhel6m2'
CRS-2673: Attempting to stop 'ora.DATA.dg' on 'rhel6m2'
CRS-2673: Attempting to stop 'ora.FLASH.dg' on 'rhel6m2'
CRS-2673: Attempting to stop 'ora.MYLISTENER.lsnr' on 'rhel6m2'
CRS-2673: Attempting to stop 'ora.MYLISTENER_SCAN1.lsnr' on 'rhel6m2'
CRS-2673: Attempting to stop 'ora.MYLISTENER_SCAN2.lsnr' on 'rhel6m2'
CRS-2673: Attempting to stop 'ora.MYLISTENER_SCAN3.lsnr' on 'rhel6m2'
CRS-2673: Attempting to stop 'ora.cvu' on 'rhel6m2'
CRS-2673: Attempting to stop 'ora.rhel6m1.vip' on 'rhel6m2'
CRS-2677: Stop of 'ora.DATA.dg' on 'rhel6m2' succeeded
CRS-2677: Stop of 'ora.GIMR.dg' on 'rhel6m2' succeeded
CRS-2677: Stop of 'ora.CLUSTER_DG.dg' on 'rhel6m2' succeeded
CRS-2677: Stop of 'ora.FLASH.dg' on 'rhel6m2' succeeded
CRS-2673: Attempting to stop 'ora.asm' on 'rhel6m2'
CRS-2677: Stop of 'ora.MYLISTENER.lsnr' on 'rhel6m2' succeeded
CRS-2677: Stop of 'ora.MYLISTENER_SCAN2.lsnr' on 'rhel6m2' succeeded
CRS-2673: Attempting to stop 'ora.scan2.vip' on 'rhel6m2'
CRS-2677: Stop of 'ora.MYLISTENER_SCAN1.lsnr' on 'rhel6m2' succeeded
CRS-2673: Attempting to stop 'ora.scan1.vip' on 'rhel6m2'
CRS-2677: Stop of 'ora.MYLISTENER_SCAN3.lsnr' on 'rhel6m2' succeeded
CRS-2673: Attempting to stop 'ora.scan3.vip' on 'rhel6m2'
CRS-2677: Stop of 'ora.asm' on 'rhel6m2' succeeded
CRS-2673: Attempting to stop 'ora.ASMNET1LSNR_ASM.lsnr' on 'rhel6m2'
CRS-2677: Stop of 'ora.rhel6m1.vip' on 'rhel6m2' succeeded
CRS-2677: Stop of 'ora.cvu' on 'rhel6m2' succeeded
CRS-2677: Stop of 'ora.qosmserver' on 'rhel6m2' succeeded
CRS-2677: Stop of 'ora.MGMTLSNR' on 'rhel6m2' succeeded
CRS-2673: Attempting to stop 'ora.rhel6m2.vip' on 'rhel6m2'
CRS-2677: Stop of 'ora.ASMNET1LSNR_ASM.lsnr' on 'rhel6m2' succeeded
CRS-2677: Stop of 'ora.scan3.vip' on 'rhel6m2' succeeded
CRS-2677: Stop of 'ora.scan1.vip' on 'rhel6m2' succeeded
CRS-2677: Stop of 'ora.scan2.vip' on 'rhel6m2' succeeded
CRS-2677: Stop of 'ora.rhel6m2.vip' on 'rhel6m2' succeeded
CRS-2673: Attempting to stop 'ora.ons' on 'rhel6m2'
CRS-2677: Stop of 'ora.ons' on 'rhel6m2' succeeded
CRS-2673: Attempting to stop 'ora.net1.network' on 'rhel6m2'
CRS-2677: Stop of 'ora.net1.network' on 'rhel6m2' succeeded
CRS-2792: Shutdown of Cluster Ready Services-managed resources on 'rhel6m2' has completed
CRS-2677: Stop of 'ora.crsd' on 'rhel6m2' succeeded
CRS-2673: Attempting to stop 'ora.storage' on 'rhel6m2'
CRS-2673: Attempting to stop 'ora.crf' on 'rhel6m2'
CRS-2673: Attempting to stop 'ora.drivers.acfs' on 'rhel6m2'
CRS-2673: Attempting to stop 'ora.gpnpd' on 'rhel6m2'
CRS-2673: Attempting to stop 'ora.mdnsd' on 'rhel6m2'
CRS-2677: Stop of 'ora.drivers.acfs' on 'rhel6m2' succeeded
CRS-2677: Stop of 'ora.crf' on 'rhel6m2' succeeded
CRS-2677: Stop of 'ora.gpnpd' on 'rhel6m2' succeeded
CRS-2677: Stop of 'ora.storage' on 'rhel6m2' succeeded
CRS-2673: Attempting to stop 'ora.asm' on 'rhel6m2'
CRS-2677: Stop of 'ora.mdnsd' on 'rhel6m2' succeeded
CRS-2677: Stop of 'ora.asm' on 'rhel6m2' succeeded
CRS-2673: Attempting to stop 'ora.cluster_interconnect.haip' on 'rhel6m2'
CRS-2677: Stop of 'ora.cluster_interconnect.haip' on 'rhel6m2' succeeded
CRS-2673: Attempting to stop 'ora.ctssd' on 'rhel6m2'
CRS-2673: Attempting to stop 'ora.evmd' on 'rhel6m2'
CRS-2677: Stop of 'ora.ctssd' on 'rhel6m2' succeeded
CRS-2677: Stop of 'ora.evmd' on 'rhel6m2' succeeded
CRS-2673: Attempting to stop 'ora.cssd' on 'rhel6m2'
CRS-2677: Stop of 'ora.cssd' on 'rhel6m2' succeeded
CRS-2673: Attempting to stop 'ora.gipcd' on 'rhel6m2'
CRS-2677: Stop of 'ora.gipcd' on 'rhel6m2' succeeded
CRS-2793: Shutdown of Oracle High Availability Services-managed resources on 'rhel6m2' has completed
CRS-4133: Oracle High Availability Services has been stopped.
CRS-4123: Oracle High Availability Services has been started.
CRS-2672: Attempting to start 'ora.evmd' on 'rhel6m2'
CRS-2672: Attempting to start 'ora.mdnsd' on 'rhel6m2'
CRS-2676: Start of 'ora.mdnsd' on 'rhel6m2' succeeded
CRS-2676: Start of 'ora.evmd' on 'rhel6m2' succeeded
CRS-2672: Attempting to start 'ora.gpnpd' on 'rhel6m2'
CRS-2676: Start of 'ora.gpnpd' on 'rhel6m2' succeeded
CRS-2672: Attempting to start 'ora.cssdmonitor' on 'rhel6m2'
CRS-2672: Attempting to start 'ora.gipcd' on 'rhel6m2'
CRS-2676: Start of 'ora.cssdmonitor' on 'rhel6m2' succeeded
CRS-2676: Start of 'ora.gipcd' on 'rhel6m2' succeeded
CRS-2672: Attempting to start 'ora.cssd' on 'rhel6m2'
CRS-2672: Attempting to start 'ora.diskmon' on 'rhel6m2'
CRS-2676: Start of 'ora.diskmon' on 'rhel6m2' succeeded
CRS-2676: Start of 'ora.cssd' on 'rhel6m2' succeeded
CRS-2672: Attempting to start 'ora.crf' on 'rhel6m2'
CRS-2672: Attempting to start 'ora.ctssd' on 'rhel6m2'
CRS-2672: Attempting to start 'ora.cluster_interconnect.haip' on 'rhel6m2'
CRS-2676: Start of 'ora.crf' on 'rhel6m2' succeeded
CRS-2676: Start of 'ora.ctssd' on 'rhel6m2' succeeded
CRS-2676: Start of 'ora.cluster_interconnect.haip' on 'rhel6m2' succeeded
CRS-2672: Attempting to start 'ora.asm' on 'rhel6m2'
CRS-2676: Start of 'ora.asm' on 'rhel6m2' succeeded
CRS-2672: Attempting to start 'ora.storage' on 'rhel6m2'
CRS-2676: Start of 'ora.storage' on 'rhel6m2' succeeded
CRS-2672: Attempting to start 'ora.crsd' on 'rhel6m2'
CRS-2676: Start of 'ora.crsd' on 'rhel6m2' succeeded
CRS-2673: Attempting to stop 'ora.crsd' on 'rhel6m2'
CRS-2677: Stop of 'ora.crsd' on 'rhel6m2' succeeded
CRS-2791: Starting shutdown of Oracle High Availability Services-managed resources on 'rhel6m2'
CRS-2673: Attempting to stop 'ora.mdnsd' on 'rhel6m2'
CRS-2673: Attempting to stop 'ora.storage' on 'rhel6m2'
CRS-2673: Attempting to stop 'ora.crf' on 'rhel6m2'
CRS-2673: Attempting to stop 'ora.gpnpd' on 'rhel6m2'
CRS-2673: Attempting to stop 'ora.drivers.acfs' on 'rhel6m2'
CRS-2677: Stop of 'ora.drivers.acfs' on 'rhel6m2' succeeded
CRS-2677: Stop of 'ora.storage' on 'rhel6m2' succeeded
CRS-2673: Attempting to stop 'ora.asm' on 'rhel6m2'
CRS-2677: Stop of 'ora.mdnsd' on 'rhel6m2' succeeded
CRS-2677: Stop of 'ora.crf' on 'rhel6m2' succeeded
CRS-2677: Stop of 'ora.gpnpd' on 'rhel6m2' succeeded
CRS-2677: Stop of 'ora.asm' on 'rhel6m2' succeeded
CRS-2673: Attempting to stop 'ora.cluster_interconnect.haip' on 'rhel6m2'
CRS-2677: Stop of 'ora.cluster_interconnect.haip' on 'rhel6m2' succeeded
CRS-2673: Attempting to stop 'ora.ctssd' on 'rhel6m2'
CRS-2673: Attempting to stop 'ora.evmd' on 'rhel6m2'
CRS-2677: Stop of 'ora.ctssd' on 'rhel6m2' succeeded
CRS-2677: Stop of 'ora.evmd' on 'rhel6m2' succeeded
CRS-2673: Attempting to stop 'ora.cssd' on 'rhel6m2'
CRS-2677: Stop of 'ora.cssd' on 'rhel6m2' succeeded
CRS-2673: Attempting to stop 'ora.gipcd' on 'rhel6m2'
CRS-2677: Stop of 'ora.gipcd' on 'rhel6m2' succeeded
CRS-2793: Shutdown of Oracle High Availability Services-managed resources on 'rhel6m2' has completed
CRS-4133: Oracle High Availability Services has been stopped.
CRS-4123: Oracle High Availability Services has been started.
CRS-2672: Attempting to start 'ora.mdnsd' on 'rhel6m2'
CRS-2676: Start of 'ora.mdnsd' on 'rhel6m2' succeeded
CRS-2672: Attempting to start 'ora.gpnpd' on 'rhel6m2'
CRS-2676: Start of 'ora.gpnpd' on 'rhel6m2' succeeded
CRS-2672: Attempting to start 'ora.cssdmonitor' on 'rhel6m2'
CRS-2672: Attempting to start 'ora.gipcd' on 'rhel6m2'
CRS-2676: Start of 'ora.cssdmonitor' on 'rhel6m2' succeeded
CRS-2676: Start of 'ora.gipcd' on 'rhel6m2' succeeded
CRS-2672: Attempting to start 'ora.cssd' on 'rhel6m2'
CRS-2672: Attempting to start 'ora.diskmon' on 'rhel6m2'
CRS-2676: Start of 'ora.diskmon' on 'rhel6m2' succeeded
CRS-2676: Start of 'ora.cssd' on 'rhel6m2' succeeded
CRS-2672: Attempting to start 'ora.drivers.acfs' on 'rhel6m2'
CRS-2679: Attempting to clean 'ora.cluster_interconnect.haip' on 'rhel6m2'
CRS-2672: Attempting to start 'ora.ctssd' on 'rhel6m2'
CRS-2681: Clean of 'ora.cluster_interconnect.haip' on 'rhel6m2' succeeded
CRS-2672: Attempting to start 'ora.cluster_interconnect.haip' on 'rhel6m2'
CRS-2674: Start of 'ora.drivers.acfs' on 'rhel6m2' failed
CRS-2676: Start of 'ora.ctssd' on 'rhel6m2' succeeded
CRS-2676: Start of 'ora.cluster_interconnect.haip' on 'rhel6m2' succeeded
CRS-2672: Attempting to start 'ora.asm' on 'rhel6m2'
CRS-2676: Start of 'ora.asm' on 'rhel6m2' succeeded
2017/09/15 14:11:02 CLSRSC-338: Successfully downgraded OCR to version 11.2.0.4.0
CRS-2672: Attempting to start 'ora.crsd' on 'rhel6m2'
CRS-2676: Start of 'ora.crsd' on 'rhel6m2' succeeded
2017/09/15 14:13:37 CLSRSC-4006: Removing Oracle Trace File Analyzer (TFA) Collector.
2017/09/15 14:14:07 CLSRSC-4007: Successfully removed Oracle Trace File Analyzer (TFA) Collector.
2017/09/15 14:14:08 CLSRSC-4001: Installing Oracle Trace File Analyzer (TFA) Collector.
2017/09/15 14:14:47 CLSRSC-4002: Successfully installed Oracle Trace File Analyzer (TFA) Collector.
2017/09/15 14:14:47 CLSRSC-591: successfully downgraded Oracle Clusterware stack on this node
2017/09/15 14:14:48 CLSRSC-640: To complete the downgrade operation, ensure that the node inventory on all nodes points to the configured Grid Infrastructure home '/opt/app/11.2.0/grid4'.
2017/09/15 14:14:49 CLSRSC-592: Run 'crsctl start crs' from home /opt/app/11.2.0/grid4 on each node to complete downgrade.
As mentioned earlier the downgrade process detect the last node automatically and lines below were observed in the log file generated
2017-09-15 14:03:32: Current node: rhel6m2 should be the last node to downgrade.
2017-09-15 14:05:18: Executing the step [asm_DowngradeCurrentNode_step_1] to downgrade ASM on the last node
2017-09-15 14:05:18: Performing ASM downgrade on the last node ...
2017-09-15 14:09:17: Executing the step [ocr_downgradeLastNode_step_1] to downgrade OCR on the last node.
2017-09-15 14:09:46: Executing step downgrade ACFS on the last node
2017-09-15 14:10:59: Executing the step [ocr_downgradeLastNode_step_2] to downgrade OCR on the last node.
7. Next is to upgrade the oraInventory by setting CRS=false on 12.2 GI home and CRS=ture on 11.2 GI home.
cd /opt/app/12.2.0/grid/oui/bin/
$ ./runInstaller -nowait -waitforcompletion -ignoreSysPrereqs -updateNodeList -silent CRS=false ORACLE_HOME=/opt/app/12.2.0/grid  "CLUSTER_NODES=rhel6m1,rhel6m2" -doNotUpdateNodeList
Starting Oracle Universal Installer...

Checking swap space: must be greater than 500 MB.   Actual 4011 MB    Passed
The inventory pointer is located at /etc/oraInst.loc
'UpdateNodeList' was successful.

cd /opt/app/11.2.0/grid4/oui/bin
$ ./runInstaller -nowait -waitforcompletion -ignoreSysPrereqs -updateNodeList  -silent CRS=true ORACLE_HOME=/opt/app/11.2.0/grid4
Starting Oracle Universal Installer...

Checking swap space: must be greater than 500 MB.   Actual 4011 MB    Passed
The inventory pointer is located at /etc/oraInst.loc
The inventory is located at /opt/app/oraInventory
'UpdateNodeList' was successful.
Verify this by checking in the inventory.xml
<HOME NAME="Ora11g_gridinfrahome2" LOC="/opt/app/11.2.0/grid4" TYPE="O" IDX="3" CRS="true">
   <NODE_LIST>
      <NODE NAME="rhel6m1"/>
      <NODE NAME="rhel6m2"/>
   </NODE_LIST>
</HOME>

<HOME NAME="OraGI12Home1" LOC="/opt/app/12.2.0/grid" TYPE="O" IDX="5"/>
8. Check if /etc/oratab has entries for ASM instance with the correct GI home. After the downgrade ASM instance should be associated with 11.2.0.4 GI HOME.

9. Start the cluster stack using the 11.2.0.4 GI Home.
which crsctl
/opt/app/11.2.0/grid4/bin/crsctl
crsctl start crs
9. All the changes made to OCR location after the upgrade are lost once the downgrade is done. As mentioned previously OCR was moved to disk group +CLUSTER_DG after upgrade. But after the downgrade OCR location goes back to what it was before the upgrade.
ocrcheck
Status of Oracle Cluster Registry is as follows :
         Version                  :          3
         Total space (kbytes)     :     262120
         Used space (kbytes)      :       3512
         Available space (kbytes) :     258608
         ID                       : 1487892601
         Device/File Name         :      +GIMR
However the changes made to the vote disk location doesn't revert backup after the downgrade and remain the same
$ crsctl query css votedisk
##  STATE    File Universal Id                File Name Disk group
--  -----    -----------------                --------- ---------
 1. ONLINE   880f0769cd344ff3bf158b0ee82a91dc (/dev/sdc1) [CLUSTER_DG]
 2. ONLINE   3e1484cd24c74fbfbfc6c3a1f371d4d3 (/dev/sdd1) [CLUSTER_DG]
 3. ONLINE   9604775aeca64f3ebf36df1ff70f6222 (/dev/sdb1) [CLUSTER_DG]
Located 3 voting disk(s).
OCR backup location to goes back to default location on local node
ocrconfig -showbackup

rhel6m1     2017/09/15 09:30:32     /opt/app/11.2.0/grid4/cdata/rhel6m-cluster/backup00.ocr
rhel6m1     2017/09/15 05:30:32     /opt/app/11.2.0/grid4/cdata/rhel6m-cluster/backup01.ocr
10. Verify there's no reference to mgmtdb in 11.2
 crsctl stat res -t
--------------------------------------------------------------------------------
NAME           TARGET  STATE        SERVER                   STATE_DETAILS
--------------------------------------------------------------------------------
Local Resources
--------------------------------------------------------------------------------
ora.CLUSTER_DG.dg
               ONLINE  ONLINE       rhel6m1
               ONLINE  ONLINE       rhel6m2
ora.DATA.dg
               ONLINE  ONLINE       rhel6m1
               ONLINE  ONLINE       rhel6m2
ora.FLASH.dg
               ONLINE  ONLINE       rhel6m1
               ONLINE  ONLINE       rhel6m2
ora.GIMR.dg
               ONLINE  ONLINE       rhel6m1
               ONLINE  ONLINE       rhel6m2
ora.MYLISTENER.lsnr
               ONLINE  ONLINE       rhel6m1
               ONLINE  ONLINE       rhel6m2
ora.asm
               ONLINE  ONLINE       rhel6m1                  Started
               ONLINE  ONLINE       rhel6m2                  Started
ora.gsd
               OFFLINE OFFLINE      rhel6m1
               OFFLINE OFFLINE      rhel6m2
ora.net1.network
               ONLINE  ONLINE       rhel6m1
               ONLINE  ONLINE       rhel6m2
ora.ons
               ONLINE  ONLINE       rhel6m1
               ONLINE  ONLINE       rhel6m2
ora.registry.acfs
               ONLINE  ONLINE       rhel6m1
               ONLINE  ONLINE       rhel6m2
--------------------------------------------------------------------------------
Cluster Resources
--------------------------------------------------------------------------------
ora.MYLISTENER_SCAN1.lsnr
      1        ONLINE  ONLINE       rhel6m2
ora.MYLISTENER_SCAN2.lsnr
      1        ONLINE  ONLINE       rhel6m1
ora.MYLISTENER_SCAN3.lsnr
      1        ONLINE  ONLINE       rhel6m1
ora.cvu
      1        ONLINE  ONLINE       rhel6m1
ora.oc4j
      1        ONLINE  ONLINE       rhel6m1
ora.rhel6m1.vip
      1        ONLINE  ONLINE       rhel6m1
ora.rhel6m2.vip
      1        ONLINE  ONLINE       rhel6m2
ora.scan1.vip
      1        ONLINE  ONLINE       rhel6m2
ora.scan2.vip
      1        ONLINE  ONLINE       rhel6m1
ora.scan3.vip
      1        ONLINE  ONLINE       rhel6m1
ora.std11g2.db
      1        ONLINE  ONLINE       rhel6m1                  Open
      2        ONLINE  ONLINE       rhel6m2                  Open
ora.std11g2.myservice.svc
      1        ONLINE  ONLINE       rhel6m1
      2        ONLINE  ONLINE       rhel6m2
ora.std11g2.bx.domain.net.svc
      1        ONLINE  ONLINE       rhel6m2
      2        ONLINE  ONLINE       rhel6m1
Check ASM is configured with default listener configuration not with ASMNET1LSNR_ASM (which is part of flex)
srvctl config asm
ASM home: /opt/app/11.2.0/grid4
ASM listener: MYLISTENER
Finally verify the cluster active version.
crsctl query crs activeversion
Oracle Clusterware active version on the cluster is [11.2.0.4.0]

crsctl query crs softwareversion -all
Oracle Clusterware version on node [rhel6m1] is [11.2.0.4.0]
Oracle Clusterware version on node [rhel6m2] is [11.2.0.4.0]
Related posts
Downgrade Grid Infrastructure from 12.1.0.2 to 11.2.0.4
Downgrade Grid Infrastructure from 11.2.0.4 to 11.2.0.3
Downgrade Grid Infrastructure from 11.2.0.4 to 11.1.0.7

Useful metalink notes
How to Upgrade to/Downgrade from Grid Infrastructure 12.2 and Known Issues [ID 2240959.1]