Upgrading RAC from 11.2.0.4 to 12.1.0.2 - Database

After upgrading grid infrastructure from 11.2.0.4 to 12.1.0.2 the next step is the upgrade of RAC software and the database itself. There are earlier post which upgraded 12.1.0.1 to 12.1.0.2 and also upgraded in data guard configuration and with PDBs. These posts have useful metalink notes that could be beneficial in RAC environment as well. The database software upgrade is an out of place upgrade with role separation. Use orachk or cluvfy to check the pre upgrade status.

cluvfy stage  -pre dbinst -upgrade -src_dbhome /opt/app/oracle/product/11.2.0/dbhome_1 -dest_dbhome /opt/app/oracle/product/12.1.0/dbhome_2 -dest_version 12.1.0.2.0

Create additional user groups (backupdba,kmdba,dgdba) and install the 12.1.0.2 using the software only installation option out of place to current 11.2.0.4 oracle home. One difference that was noted is that with 12.1.0.2 there's no need to manually correct the oracle binary's ownership as before. After the database installation the oracle binaries will have ownership as oracel:oinstall even though the correct ownership should be (for role separated installation) oracle:asmadmin. However this get auto rectified when the DBUA is run.
Before running the DBUA check oracle user has the write permission on the audit folder under $ORACLE_BASE if not grant write permission for oinstall ggroup.

chmod 770 $ORACLE_BASE/audit

The default listener name has been changed on this cluster setup. This missing default listener causes a problem similar to earlier upgrade (from 11.2.0.3 to 12.1.0.1).

To fix this add default listener and start it before DBUA is run.

$ srvctl add listener -listener listener -p 1521
$ srvctl start listener -l listener

To avoid the bug 19518079 apply the patch to 12c oracle home before DBUA is run. However this is not a must and could use other workarounds as well. For more information refere MOS 19518079.8 and 1941383.1
Run preupgrade script to identify any pre-reqs tasks that must be done on the database before the upgrade.

cd /opt/app/oracle/product/12.1.0/dbhome_2/rdbms/admin
SQL> @preupgrd.sql

Run the DBUA to start the database upgrade.

Timezone is also be upgraded at the same time the database.

Summary

Upgrade tasks completion

Upgrade results

Check time timezone information is same on both timezone file and database registry as there was a mismatch between these values in 12.1.0.1 after upgrading from 11.2.0.3

SQL> select * from v$timezone_file;

FILENAME                VERSION     CON_ID
-------------------- ---------- ----------
timezlrg_18.dat              18          0

SQL> select TZ_VERSION from registry$database;

TZ_VERSION
----------
        18

If ASM configuration is referring the default listener modify it before removing the default listener entry added during the upgrade.

[grid@rhel6m2 ~]$ srvctl config asm
ASM home: <CRS home>
Password file:
ASM listener: LISTENER

$ srvctl modify asm -l MYLISTENER
$ srvctl config asm
ASM home: <CRS home>
Password file:
ASM listener: MYLISTENER

$ srvctl stop listener -l listener
$ srvctl remove listener -l listener

If database patches are being applied make sure sqlpatch directory has write permission for oracle user.

chmod 775 /opt/app/oracle/cfgtoollogs/sqlpatch

If the database upgrade is satisfactory upgrade the compatible parameter to new version.

alter system set compatible='12.1.0.2.0' scope=spfile sid='*';

ASM diskgroup compatible parameters could also be upgrade at this time. There are dependencies between the DB compatibility value and ASM's rdbms compatibility value as such consider in which order to update these values.

alter diskgroup data set attribute 'compatible.asm'='12.1.0.2.0';
alter diskgroup flash set attribute 'compatible.asm'='12.1.0.2.0';
alter diskgroup flash set attribute 'compatible.rdbms'='12.1.0.2.0';
alter diskgroup data set attribute 'compatible.rdbms'='12.1.0.2.0';
alter diskgroup cluster_dg set attribute 'compatible.rdbms'='12.1.0.2.0';
alter diskgroup cluster_dg set attribute 'compatible.asm'='12.1.0.2.0';
alter diskgroup flash set attribute 'compatible.advm'='12.1.0.2.0';

Verify that upgrade entry is added to patch history.

ACTION_TIME                    ACTION           VERSION            COMMENTS
------------------------------ ---------------- ------------------ --------------------------
11-JUN-12 05.07.59.601706 PM   APPLY            11.2.0.3           Patchset 11.2.0.2.0
12-JUN-12 02.24.59.156482 PM   APPLY            11.2.0.3           PSU 11.2.0.3.2
19-JUL-12 05.31.51.092475 PM   APPLY            11.2.0.3           PSU 11.2.0.3.3
15-NOV-12 02.59.09.956573 PM   APPLY            11.2.0.3           PSU 11.2.0.3.4
22-JAN-13 01.02.11.445936 PM   APPLY            11.2.0.3           PSU 11.2.0.3.5
24-APR-13 11.59.40.161626 AM   APPLY            11.2.0.3           PSU 11.2.0.3.6
31-JUL-13 02.01.12.758084 PM   APPLY            11.2.0.3           PSU 11.2.0.3.7
09-SEP-13 04.24.23.479935 PM   VIEW INVALIDATE                     view invalidation
09-SEP-13 04.24.23.586103 PM   UPGRADE          11.2.0.4.0         Upgraded from 11.2.0.3.0
09-SEP-13 04.25.02.867598 PM   APPLY            11.2.0.4           Patchset 11.2.0.2.0
08-MAY-14 01.51.36.029968 PM   APPLY            11.2.0.4           PSU 11.2.0.4.2
25-JUL-14 04.16.58.140512 PM   APPLY            11.2.0.4           PSU 11.2.0.4.3
17-NOV-14 04.01.03.481360 PM   APPLY            11.2.0.4           PSU 11.2.0.4.4
22-JAN-15 05.19.08.323577 PM   jvmpsu.sql       11.2.0.4.2OJVMBP   RAN jvmpsu.sql
22-JAN-15 05.19.15.000000 PM   APPLY                               Patch 19877440 applied
22-JAN-15 05.19.15.796773 PM   APPLY            11.2.0.4.2OJVMBP   OJVM PSU post-install
22-JAN-15 05.21.55.225164 PM   APPLY            11.2.0.4           PSU 11.2.0.4.5
22-APR-15 01.05.40.917488 PM   VIEW INVALIDATE                     view invalidation
22-APR-15 01.07.04.188992 PM   UPGRADE          12.1.0.2.0         Upgraded from 11.2.0.4.0
22-APR-15 03.51.53.094015 PM   jvmpsu.sql       12.1.0.2.2OJVMBP   RAN jvmpsu.sql

This conclude the upgrade of RAC DB from 11.2.0.4 to 12.1.0.2.

Useful metalink notes
Oracle Database Upgrade Path Reference List [ID 730365.1]
Oracle 12cR1 Upgrade Companion [ID 1462240.1]
Memory Guard Offlines Database Service Due to Low Free Memory [ID 1929994.1]
DBA_REGISTRY_HISTORY Does Not Get Updated on Database Upgraded to 12.1.0.2 [ID 1941383.1]
Bug 19518079 - DBA_REGISTRY_HISTORY is not updated during upgrade to 12.1.0.2 [ID 19518079.8]
ORA-600 [kwqitnmphe:ltbagi] or ORA-600 [kwqdlprochstentry:ltbagi] on the SYS$SERVICE_METRICS_TAB, Orphan Subscribers [ID 1986078.1]
Master Note For Oracle Database 12c Release 1 (12.1) Database/Client Installation/Upgrade/Migration Standalone Environment (Non-RAC) [ID 1520299.1]
Actions For DST Updates When Upgrading To Or Applying The 12.1.0.2 Patchset [ID 1665676.1]
Things to Consider Before Upgrading to 12.1.0.2 to Avoid Poor Performance or Wrong Results [ID 2034610.1]

Related Posts
Upgrading 11.2.0.3 (11gR2) Database to 12.1.0.1 (12c) Using DBUA
Upgrading from 11.2.0.3 to 12.1.0.1 RAC
Upgrade Oracle Database 12c1 from 12.1.0.1 to 12.1.0.2
Upgrading 12c CDB and PDB from 12.1.0.1 to 12.1.0.2

Upgrading RAC from 11.2.0.4 to 12.1.0.2 - Grid Infrastructure

This post gives the highlights of upgrading grid infrastructure of a two node RAC from 11.2.0.4 to 12.1.0.2. There's an earlier post of upgrading GI for a single instance from 11.2.0.4 to 12.1.0.2. The system upgraded was the same system upgraded from 11.2.0.3 to 11.2.0.4 earlier.
As mentioned in upgrading from 12.1.0.1 to 12.1.0.2 the pre-reqs for upgrading to 12.1.0.2 has changed and most notably the GI management repository is no longer optional. Prior to the upgrade OCR and Vote files were moved to a diskgroup with sufficient space for GI management repository. Rest of the steps are listed assuming these pre-reqs are completed.
1. Before the GI upgrade backup the OCR manually. This could be used in downgrading the GI from 12.1.0.2 to 11.2.0.4 later on.
2. The RAC has been setup with role separation. This requires certain directories having permission so that both grid user and oracle user is able to write and read to them. Make cfgtoollogs writable to both grid and oracle, usually this is writable only for one user and group permission is only read and executable. Without write permission on this directory following error could be seen when upgrading GI.

PRCR-1158 : Directory /opt/app/oracle/cfgtoollogs/asmca in file path does not exist
PRCR-1154 : Failed to create file output stream with file name: /opt/app/oracle/cfgtoollogs/asmca/asmca-150325PM054138.logCouldn't get lock for /opt/app/oracle/cfgtoollogs/asmca/asmca-150325PM054138.log

chmod 775 $ORACLE_BASE/cfgtoollogs

Another directory requires write permission for grid user is the dbca inside cfgtoollogs.

chmod 770 $ORACLE_BASE/cfgtoollogs/dbca

As the GI management repository is a database the admin directory is another directory that requires write permission for grid user. Without this permission creation of GI management repository will fail with following

Cannot create directory "/opt/app/oracle/admin/_mgmtdb/dpdump".

chmod 770 $ORACLE_BASE/admin

These permissions must be set on all the nodes.
3.GI upgrade is an out of place upgrade. Create a new GI home on all nodes.

mkdir -p  /opt/app/12.1.0/grid2
chown -R grid:oinstall 12.1.0
chmod -R 775 12.1.0

4. Before the upgrade check GI active version is 11.2.0.4 and upgrade state

[grid@rhel6m1 grid]$ crsctl query crs releaseversion
Oracle High Availability Services release version on the local node is [11.2.0.4.0]
[grid@rhel6m1 grid]$ crsctl query crs softwareversion
Oracle Clusterware version on node [rhel6m1] is [11.2.0.4.0]
[grid@rhel6m1 grid]$ crsctl query crs activeversion -f
Oracle Clusterware active version on the cluster is [11.2.0.4.0]. The cluster upgrade state is [NORMAL].

5. Verify the pre-reqs with orachk (refer 1268927.2) and cluvfy

orachk -u -o pre

[grid@rhel6m1 grid]$ ./runcluvfy.sh stage -pre crsinst -upgrade -rolling -src_crshome /opt/app/11.2.0/grid4 -dest_crshome /opt/app/12.1.0/grid2 -dest_version 12.1.0.2.0 -fixup -verbose

Performing pre-checks for cluster services setup

Checking node reachability...

Check: Node reachability from node "rhel6m1"
  Destination Node                      Reachable?
  ------------------------------------  ------------------------
  rhel6m1                               yes
  rhel6m2                               yes
Result: Node reachability check passed from node "rhel6m1"


Checking user equivalence...

Check: User equivalence for user "grid"
  Node Name                             Status
  ------------------------------------  ------------------------
  rhel6m2                               passed
  rhel6m1                               passed
Result: User equivalence check passed for user "grid"

Check: Package existence for "cvuqdisk"
  Node Name     Available                 Required                  Status
  ------------  ------------------------  ------------------------  ----------
  rhel6m2       cvuqdisk-1.0.9-1          cvuqdisk-1.0.9-1          passed
  rhel6m1       cvuqdisk-1.0.9-1          cvuqdisk-1.0.9-1          passed
Result: Package existence check passed for "cvuqdisk"

Check: Grid Infrastructure home writeability of path /opt/app/12.1.0/grid2
Grid Infrastructure home check passed

Checking CRS user consistency
Result: CRS user consistency check successful
Checking network configuration consistency.
Result: Check for network configuration consistency passed.
Checking ASM disk size consistency
All ASM disks are correctly sized
Checking if default discovery string is being used by ASM
ASM discovery string "/dev/sd*" is not the default discovery string
Checking if ASM parameter file is in use by an ASM instance on the local node
Result: ASM instance is using parameter file "+CLUSTER_DG/rhel6m-cluster/asmparameterfile/registry.253.785350531" on node "rhel6m1" on which upgrade is requested.

Checking OLR integrity...
Check of existence of OLR configuration file "/etc/oracle/olr.loc" passed
Check of attributes of OLR configuration file "/etc/oracle/olr.loc" passed

WARNING:
This check does not verify the integrity of the OLR contents. Execute 'ocrcheck -local' as a privileged user to verify the contents of OLR.

OLR integrity check passed

Checking node connectivity...

Checking hosts config file...
  Node Name                             Status
  ------------------------------------  ------------------------
  rhel6m1                               passed
  rhel6m2                               passed

Verification of the hosts config file successful


Interface information for node "rhel6m1"
 Name   IP Address      Subnet          Gateway         Def. Gateway    HW Address        MTU
 ------ --------------- --------------- --------------- --------------- ----------------- ------
 eth0   192.168.0.85    192.168.0.0     0.0.0.0         192.168.0.100   08:00:27:29:3C:7A 1500
 eth0   192.168.0.91    192.168.0.0     0.0.0.0         192.168.0.100   08:00:27:29:3C:7A 1500
 eth0   192.168.0.89    192.168.0.0     0.0.0.0         192.168.0.100   08:00:27:29:3C:7A 1500
 eth1   192.168.1.87    192.168.1.0     0.0.0.0         192.168.0.100   08:00:27:AE:A8:83 1500
 eth1   169.254.31.63   169.254.0.0     0.0.0.0         192.168.0.100   08:00:27:AE:A8:83 1500


Interface information for node "rhel6m2"
 Name   IP Address      Subnet          Gateway         Def. Gateway    HW Address        MTU
 ------ --------------- --------------- --------------- --------------- ----------------- ------
 eth0   192.168.0.86    192.168.0.0     0.0.0.0         192.168.0.100   08:00:27:6A:D4:18 1500
 eth0   192.168.0.90    192.168.0.0     0.0.0.0         192.168.0.100   08:00:27:6A:D4:18 1500
 eth1   192.168.1.88    192.168.1.0     0.0.0.0         192.168.0.100   08:00:27:D4:AC:BE 1500
 eth1   169.254.160.199 169.254.0.0     0.0.0.0         192.168.0.100   08:00:27:D4:AC:BE 1500


Check: Node connectivity using interfaces on subnet "192.168.0.0"

Check: Node connectivity of subnet "192.168.0.0"
  Source                          Destination                     Connected?
  ------------------------------  ------------------------------  ----------------
  rhel6m1[192.168.0.89]           rhel6m2[192.168.0.90]           yes
  rhel6m1[192.168.0.89]           rhel6m1[192.168.0.85]           yes
  rhel6m1[192.168.0.89]           rhel6m1[192.168.0.91]           yes
  rhel6m1[192.168.0.89]           rhel6m2[192.168.0.86]           yes
  rhel6m2[192.168.0.90]           rhel6m1[192.168.0.85]           yes
  rhel6m2[192.168.0.90]           rhel6m1[192.168.0.91]           yes
  rhel6m2[192.168.0.90]           rhel6m2[192.168.0.86]           yes
  rhel6m1[192.168.0.85]           rhel6m1[192.168.0.91]           yes
  rhel6m1[192.168.0.85]           rhel6m2[192.168.0.86]           yes
  rhel6m1[192.168.0.91]           rhel6m2[192.168.0.86]           yes
Result: Node connectivity passed for subnet "192.168.0.0" with node(s) rhel6m1,rhel6m2


Check: TCP connectivity of subnet "192.168.0.0"
  Source                          Destination                     Connected?
  ------------------------------  ------------------------------  ----------------
  rhel6m1 : 192.168.0.89          rhel6m1 : 192.168.0.89          passed
  rhel6m2 : 192.168.0.90          rhel6m1 : 192.168.0.89          passed
  rhel6m1 : 192.168.0.85          rhel6m1 : 192.168.0.89          passed
  rhel6m1 : 192.168.0.91          rhel6m1 : 192.168.0.89          passed
  rhel6m2 : 192.168.0.86          rhel6m1 : 192.168.0.89          passed
  rhel6m1 : 192.168.0.89          rhel6m2 : 192.168.0.90          passed
  rhel6m2 : 192.168.0.90          rhel6m2 : 192.168.0.90          passed
  rhel6m1 : 192.168.0.85          rhel6m2 : 192.168.0.90          passed
  rhel6m1 : 192.168.0.91          rhel6m2 : 192.168.0.90          passed
  rhel6m2 : 192.168.0.86          rhel6m2 : 192.168.0.90          passed
  rhel6m1 : 192.168.0.89          rhel6m1 : 192.168.0.85          passed
  rhel6m2 : 192.168.0.90          rhel6m1 : 192.168.0.85          passed
  rhel6m1 : 192.168.0.85          rhel6m1 : 192.168.0.85          passed
  rhel6m1 : 192.168.0.91          rhel6m1 : 192.168.0.85          passed
  rhel6m2 : 192.168.0.86          rhel6m1 : 192.168.0.85          passed
  rhel6m1 : 192.168.0.89          rhel6m1 : 192.168.0.91          passed
  rhel6m2 : 192.168.0.90          rhel6m1 : 192.168.0.91          passed
  rhel6m1 : 192.168.0.85          rhel6m1 : 192.168.0.91          passed
  rhel6m1 : 192.168.0.91          rhel6m1 : 192.168.0.91          passed
  rhel6m2 : 192.168.0.86          rhel6m1 : 192.168.0.91          passed
  rhel6m1 : 192.168.0.89          rhel6m2 : 192.168.0.86          passed
  rhel6m2 : 192.168.0.90          rhel6m2 : 192.168.0.86          passed
  rhel6m1 : 192.168.0.85          rhel6m2 : 192.168.0.86          passed
  rhel6m1 : 192.168.0.91          rhel6m2 : 192.168.0.86          passed
  rhel6m2 : 192.168.0.86          rhel6m2 : 192.168.0.86          passed
Result: TCP connectivity check passed for subnet "192.168.0.0"


Check: Node connectivity using interfaces on subnet "192.168.1.0"

Check: Node connectivity of subnet "192.168.1.0"
  Source                          Destination                     Connected?
  ------------------------------  ------------------------------  ----------------
  rhel6m1[192.168.1.87]           rhel6m2[192.168.1.88]           yes
Result: Node connectivity passed for subnet "192.168.1.0" with node(s) rhel6m1,rhel6m2


Check: TCP connectivity of subnet "192.168.1.0"
  Source                          Destination                     Connected?
  ------------------------------  ------------------------------  ----------------
  rhel6m1 : 192.168.1.87          rhel6m1 : 192.168.1.87          passed
  rhel6m2 : 192.168.1.88          rhel6m1 : 192.168.1.87          passed
  rhel6m1 : 192.168.1.87          rhel6m2 : 192.168.1.88          passed
  rhel6m2 : 192.168.1.88          rhel6m2 : 192.168.1.88          passed
Result: TCP connectivity check passed for subnet "192.168.1.0"

Checking subnet mask consistency...
Subnet mask consistency check passed for subnet "192.168.0.0".
Subnet mask consistency check passed for subnet "192.168.1.0".
Subnet mask consistency check passed.

Result: Node connectivity check passed

Checking multicast communication...

Checking subnet "192.168.1.0" for multicast communication with multicast group "224.0.0.251"...
Check of subnet "192.168.1.0" for multicast communication with multicast group "224.0.0.251" passed.

Check of multicast communication passed.
Task ASM Integrity check started...


Starting check to see if ASM is running on all cluster nodes...

ASM Running check passed. ASM is running on all specified nodes

Confirming that at least one ASM disk group is configured...
Disk Group Check passed. At least one Disk Group configured

Task ASM Integrity check passed...

Checking OCR integrity...
Disks "+CLUSTER_DG" are managed by ASM.

OCR integrity check passed

Check: Total memory
  Node Name     Available                 Required                  Status
  ------------  ------------------------  ------------------------  ----------
  rhel6m2       3.9992GB (4193496.0KB)    4GB (4194304.0KB)         failed
  rhel6m1       3.9992GB (4193496.0KB)    4GB (4194304.0KB)         failed
Result: Total memory check failed

Check: Available memory
  Node Name     Available                 Required                  Status
  ------------  ------------------------  ------------------------  ----------
  rhel6m2       2.6916GB (2822336.0KB)    50MB (51200.0KB)          passed
  rhel6m1       2.4122GB (2529368.0KB)    50MB (51200.0KB)          passed
Result: Available memory check passed

Check: Swap space
  Node Name     Available                 Required                  Status
  ------------  ------------------------  ------------------------  ----------
  rhel6m2       4GB (4194296.0KB)         3.9992GB (4193496.0KB)    passed
  rhel6m1       4GB (4194296.0KB)         3.9992GB (4193496.0KB)    passed
Result: Swap space check passed

Check: Free disk space for "rhel6m2:/usr,rhel6m2:/var,rhel6m2:/etc,rhel6m2:/opt/app/11.2.0/grid4,rhel6m2:/sbin,rhel6m2:/tmp"
  Path              Node Name     Mount point   Available     Required      Status
  ----------------  ------------  ------------  ------------  ------------  ------------
  /usr              rhel6m2       /             11.7715GB     7.9635GB      passed
  /var              rhel6m2       /             11.7715GB     7.9635GB      passed
  /etc              rhel6m2       /             11.7715GB     7.9635GB      passed
  /opt/app/11.2.0/grid4  rhel6m2       /             11.7715GB     7.9635GB      passed
  /sbin             rhel6m2       /             11.7715GB     7.9635GB      passed
  /tmp              rhel6m2       /             11.7715GB     7.9635GB      passed
Result: Free disk space check passed for "rhel6m2:/usr,rhel6m2:/var,rhel6m2:/etc,rhel6m2:/opt/app/11.2.0/grid4,rhel6m2:/sbin,rhel6m2:/tmp"

Check: Free disk space for "rhel6m1:/usr,rhel6m1:/var,rhel6m1:/etc,rhel6m1:/opt/app/11.2.0/grid4,rhel6m1:/sbin,rhel6m1:/tmp"
  Path              Node Name     Mount point   Available     Required      Status
  ----------------  ------------  ------------  ------------  ------------  ------------
  /usr              rhel6m1       /             10.072GB      7.9635GB      passed
  /var              rhel6m1       /             10.072GB      7.9635GB      passed
  /etc              rhel6m1       /             10.072GB      7.9635GB      passed
  /opt/app/11.2.0/grid4  rhel6m1       /             10.072GB      7.9635GB      passed
  /sbin             rhel6m1       /             10.072GB      7.9635GB      passed
  /tmp              rhel6m1       /             10.072GB      7.9635GB      passed
Result: Free disk space check passed for "rhel6m1:/usr,rhel6m1:/var,rhel6m1:/etc,rhel6m1:/opt/app/11.2.0/grid4,rhel6m1:/sbin,rhel6m1:/tmp"

Check: User existence for "grid"
  Node Name     Status                    Comment
  ------------  ------------------------  ------------------------
  rhel6m2       passed                    exists(502)
  rhel6m1       passed                    exists(502)

Checking for multiple users with UID value 502
Result: Check for multiple users with UID value 502 passed
Result: User existence check passed for "grid"

Check: Group existence for "oinstall"
  Node Name     Status                    Comment
  ------------  ------------------------  ------------------------
  rhel6m2       passed                    exists
  rhel6m1       passed                    exists
Result: Group existence check passed for "oinstall"

Check: Group existence for "dba"
  Node Name     Status                    Comment
  ------------  ------------------------  ------------------------
  rhel6m2       passed                    exists
  rhel6m1       passed                    exists
Result: Group existence check passed for "dba"

Check: Membership of user "grid" in group "oinstall" [as Primary]
  Node Name         User Exists   Group Exists  User in Group  Primary       Status
  ----------------  ------------  ------------  ------------  ------------  ------------
  rhel6m2           yes           yes           yes           yes           passed
  rhel6m1           yes           yes           yes           yes           passed
Result: Membership check for user "grid" in group "oinstall" [as Primary] passed

Check: Membership of user "grid" in group "dba"
  Node Name         User Exists   Group Exists  User in Group  Status
  ----------------  ------------  ------------  ------------  ----------------
  rhel6m2           yes           yes           yes           passed
  rhel6m1           yes           yes           yes           passed
Result: Membership check for user "grid" in group "dba" passed

Check: Run level
  Node Name     run level                 Required                  Status
  ------------  ------------------------  ------------------------  ----------
  rhel6m2       3                         3,5                       passed
  rhel6m1       3                         3,5                       passed
Result: Run level check passed

Check: Hard limits for "maximum open file descriptors"
  Node Name         Type          Available     Required      Status
  ----------------  ------------  ------------  ------------  ----------------
  rhel6m2           hard          65536         65536         passed
  rhel6m1           hard          65536         65536         passed
Result: Hard limits check passed for "maximum open file descriptors"

Check: Soft limits for "maximum open file descriptors"
  Node Name         Type          Available     Required      Status
  ----------------  ------------  ------------  ------------  ----------------
  rhel6m2           soft          1024          1024          passed
  rhel6m1           soft          1024          1024          passed
Result: Soft limits check passed for "maximum open file descriptors"

Check: Hard limits for "maximum user processes"
  Node Name         Type          Available     Required      Status
  ----------------  ------------  ------------  ------------  ----------------
  rhel6m2           hard          16384         16384         passed
  rhel6m1           hard          16384         16384         passed
Result: Hard limits check passed for "maximum user processes"

Check: Soft limits for "maximum user processes"
  Node Name         Type          Available     Required      Status
  ----------------  ------------  ------------  ------------  ----------------
  rhel6m2           soft          2047          2047          passed
  rhel6m1           soft          2047          2047          passed
Result: Soft limits check passed for "maximum user processes"

There are no oracle patches required for home "/opt/app/11.2.0/grid4".

There are no oracle patches required for home "/opt/app/11.2.0/grid4".

Checking for suitability of source home "/opt/app/11.2.0/grid4" for upgrading to version "12.1.0.2.0".
Result: Source home "/opt/app/11.2.0/grid4" is suitable for upgrading to version "12.1.0.2.0".

Check: System architecture
  Node Name     Available                 Required                  Status
  ------------  ------------------------  ------------------------  ----------
  rhel6m2       x86_64                    x86_64                    passed
  rhel6m1       x86_64                    x86_64                    passed
Result: System architecture check passed

Check: Kernel version
  Node Name     Available                 Required                  Status
  ------------  ------------------------  ------------------------  ----------
  rhel6m2       2.6.32-220.el6.x86_64     2.6.32                    passed
  rhel6m1       2.6.32-220.el6.x86_64     2.6.32                    passed
Result: Kernel version check passed

Check: Kernel parameter for "semmsl"
  Node Name         Current       Configured    Required      Status        Comment
  ----------------  ------------  ------------  ------------  ------------  ------------
  rhel6m1           3010          3010          250           passed
  rhel6m2           3010          3010          250           passed
Result: Kernel parameter check passed for "semmsl"

Check: Kernel parameter for "semmns"
  Node Name         Current       Configured    Required      Status        Comment
  ----------------  ------------  ------------  ------------  ------------  ------------
  rhel6m1           385280        385280        32000         passed
  rhel6m2           385280        385280        32000         passed
Result: Kernel parameter check passed for "semmns"

Check: Kernel parameter for "semopm"
  Node Name         Current       Configured    Required      Status        Comment
  ----------------  ------------  ------------  ------------  ------------  ------------
  rhel6m1           3010          3010          100           passed
  rhel6m2           3010          3010          100           passed
Result: Kernel parameter check passed for "semopm"

Check: Kernel parameter for "semmni"
  Node Name         Current       Configured    Required      Status        Comment
  ----------------  ------------  ------------  ------------  ------------  ------------
  rhel6m1           128           128           128           passed
  rhel6m2           128           128           128           passed
Result: Kernel parameter check passed for "semmni"

Check: Kernel parameter for "shmmax"
  Node Name         Current       Configured    Required      Status        Comment
  ----------------  ------------  ------------  ------------  ------------  ------------
  rhel6m1           68719476736   68719476736   2008657920    passed
  rhel6m2           68719476736   68719476736   2008657920    passed
Result: Kernel parameter check passed for "shmmax"

Check: Kernel parameter for "shmmni"
  Node Name         Current       Configured    Required      Status        Comment
  ----------------  ------------  ------------  ------------  ------------  ------------
  rhel6m1           4096          4096          4096          passed
  rhel6m2           4096          4096          4096          passed
Result: Kernel parameter check passed for "shmmni"

Check: Kernel parameter for "shmall"
  Node Name         Current       Configured    Required      Status        Comment
  ----------------  ------------  ------------  ------------  ------------  ------------
  rhel6m1           4294967296    4294967296    392316        passed
  rhel6m2           4294967296    4294967296    392316        passed
Result: Kernel parameter check passed for "shmall"

Check: Kernel parameter for "file-max"
  Node Name         Current       Configured    Required      Status        Comment
  ----------------  ------------  ------------  ------------  ------------  ------------
  rhel6m1           6815744       6815744       6815744       passed
  rhel6m2           6815744       6815744       6815744       passed
Result: Kernel parameter check passed for "file-max"

Check: Kernel parameter for "ip_local_port_range"
  Node Name         Current       Configured    Required      Status        Comment
  ----------------  ------------  ------------  ------------  ------------  ------------
  rhel6m1           between 9000 & 65500  between 9000 & 65500  between 9000 & 65535  passed
  rhel6m2           between 9000 & 65500  between 9000 & 65500  between 9000 & 65535  passed
Result: Kernel parameter check passed for "ip_local_port_range"

Check: Kernel parameter for "rmem_default"
  Node Name         Current       Configured    Required      Status        Comment
  ----------------  ------------  ------------  ------------  ------------  ------------
  rhel6m1           4194304       4194304       262144        passed
  rhel6m2           4194304       4194304       262144        passed
Result: Kernel parameter check passed for "rmem_default"

Check: Kernel parameter for "rmem_max"
  Node Name         Current       Configured    Required      Status        Comment
  ----------------  ------------  ------------  ------------  ------------  ------------
  rhel6m1           4194304       4194304       4194304       passed
  rhel6m2           4194304       4194304       4194304       passed
Result: Kernel parameter check passed for "rmem_max"

Check: Kernel parameter for "wmem_default"
  Node Name         Current       Configured    Required      Status        Comment
  ----------------  ------------  ------------  ------------  ------------  ------------
  rhel6m1           1048576       1048576       262144        passed
  rhel6m2           1048576       1048576       262144        passed
Result: Kernel parameter check passed for "wmem_default"

Check: Kernel parameter for "wmem_max"
  Node Name         Current       Configured    Required      Status        Comment
  ----------------  ------------  ------------  ------------  ------------  ------------
  rhel6m1           2097152       2097152       1048576       passed
  rhel6m2           2097152       2097152       1048576       passed
Result: Kernel parameter check passed for "wmem_max"

Check: Kernel parameter for "aio-max-nr"
  Node Name         Current       Configured    Required      Status        Comment
  ----------------  ------------  ------------  ------------  ------------  ------------
  rhel6m1           3145728       3145728       1048576       passed
  rhel6m2           3145728       3145728       1048576       passed
Result: Kernel parameter check passed for "aio-max-nr"

Check: Kernel parameter for "panic_on_oops"
  Node Name         Current       Configured    Required      Status        Comment
  ----------------  ------------  ------------  ------------  ------------  ------------
  rhel6m1           1             1             1             passed
  rhel6m2           1             1             1             passed
Result: Kernel parameter check passed for "panic_on_oops"

Check: Package existence for "binutils"
  Node Name     Available                 Required                  Status
  ------------  ------------------------  ------------------------  ----------
  rhel6m2       binutils-2.20.51.0.2-5.28.el6  binutils-2.20.51.0.2      passed
  rhel6m1       binutils-2.20.51.0.2-5.28.el6  binutils-2.20.51.0.2      passed
Result: Package existence check passed for "binutils"

Check: Package existence for "compat-libcap1"
  Node Name     Available                 Required                  Status
  ------------  ------------------------  ------------------------  ----------
  rhel6m2       compat-libcap1-1.10-1     compat-libcap1-1.10       passed
  rhel6m1       compat-libcap1-1.10-1     compat-libcap1-1.10       passed
Result: Package existence check passed for "compat-libcap1"

Check: Package existence for "compat-libstdc++-33(x86_64)"
  Node Name     Available                 Required                  Status
  ------------  ------------------------  ------------------------  ----------
  rhel6m2       compat-libstdc++-33(x86_64)-3.2.3-69.el6  compat-libstdc++-33(x86_64)-3.2.3  passed
  rhel6m1       compat-libstdc++-33(x86_64)-3.2.3-69.el6  compat-libstdc++-33(x86_64)-3.2.3  passed
Result: Package existence check passed for "compat-libstdc++-33(x86_64)"

Check: Package existence for "libgcc(x86_64)"
  Node Name     Available                 Required                  Status
  ------------  ------------------------  ------------------------  ----------
  rhel6m2       libgcc(x86_64)-4.4.6-3.el6  libgcc(x86_64)-4.4.4      passed
  rhel6m1       libgcc(x86_64)-4.4.6-3.el6  libgcc(x86_64)-4.4.4      passed
Result: Package existence check passed for "libgcc(x86_64)"

Check: Package existence for "libstdc++(x86_64)"
  Node Name     Available                 Required                  Status
  ------------  ------------------------  ------------------------  ----------
  rhel6m2       libstdc++(x86_64)-4.4.6-3.el6  libstdc++(x86_64)-4.4.4   passed
  rhel6m1       libstdc++(x86_64)-4.4.6-3.el6  libstdc++(x86_64)-4.4.4   passed
Result: Package existence check passed for "libstdc++(x86_64)"

Check: Package existence for "libstdc++-devel(x86_64)"
  Node Name     Available                 Required                  Status
  ------------  ------------------------  ------------------------  ----------
  rhel6m2       libstdc++-devel(x86_64)-4.4.6-3.el6  libstdc++-devel(x86_64)-4.4.4  passed
  rhel6m1       libstdc++-devel(x86_64)-4.4.6-3.el6  libstdc++-devel(x86_64)-4.4.4  passed
Result: Package existence check passed for "libstdc++-devel(x86_64)"

Check: Package existence for "sysstat"
  Node Name     Available                 Required                  Status
  ------------  ------------------------  ------------------------  ----------
  rhel6m2       sysstat-9.0.4-18.el6      sysstat-9.0.4             passed
  rhel6m1       sysstat-9.0.4-18.el6      sysstat-9.0.4             passed
Result: Package existence check passed for "sysstat"

Check: Package existence for "gcc"
  Node Name     Available                 Required                  Status
  ------------  ------------------------  ------------------------  ----------
  rhel6m2       gcc-4.4.6-3.el6           gcc-4.4.4                 passed
  rhel6m1       gcc-4.4.6-3.el6           gcc-4.4.4                 passed
Result: Package existence check passed for "gcc"

Check: Package existence for "gcc-c++"
  Node Name     Available                 Required                  Status
  ------------  ------------------------  ------------------------  ----------
  rhel6m2       gcc-c++-4.4.6-3.el6       gcc-c++-4.4.4             passed
  rhel6m1       gcc-c++-4.4.6-3.el6       gcc-c++-4.4.4             passed
Result: Package existence check passed for "gcc-c++"

Check: Package existence for "ksh"
  Node Name     Available                 Required                  Status
  ------------  ------------------------  ------------------------  ----------
  rhel6m2       ksh                       ksh                       passed
  rhel6m1       ksh                       ksh                       passed
Result: Package existence check passed for "ksh"

Check: Package existence for "make"
  Node Name     Available                 Required                  Status
  ------------  ------------------------  ------------------------  ----------
  rhel6m2       make-3.81-19.el6          make-3.81                 passed
  rhel6m1       make-3.81-19.el6          make-3.81                 passed
Result: Package existence check passed for "make"

Check: Package existence for "glibc(x86_64)"
  Node Name     Available                 Required                  Status
  ------------  ------------------------  ------------------------  ----------
  rhel6m2       glibc(x86_64)-2.12-1.47.el6  glibc(x86_64)-2.12        passed
  rhel6m1       glibc(x86_64)-2.12-1.47.el6  glibc(x86_64)-2.12        passed
Result: Package existence check passed for "glibc(x86_64)"

Check: Package existence for "glibc-devel(x86_64)"
  Node Name     Available                 Required                  Status
  ------------  ------------------------  ------------------------  ----------
  rhel6m2       glibc-devel(x86_64)-2.12-1.47.el6  glibc-devel(x86_64)-2.12  passed
  rhel6m1       glibc-devel(x86_64)-2.12-1.47.el6  glibc-devel(x86_64)-2.12  passed
Result: Package existence check passed for "glibc-devel(x86_64)"

Check: Package existence for "libaio(x86_64)"
  Node Name     Available                 Required                  Status
  ------------  ------------------------  ------------------------  ----------
  rhel6m2       libaio(x86_64)-0.3.107-10.el6  libaio(x86_64)-0.3.107    passed
  rhel6m1       libaio(x86_64)-0.3.107-10.el6  libaio(x86_64)-0.3.107    passed
Result: Package existence check passed for "libaio(x86_64)"

Check: Package existence for "libaio-devel(x86_64)"
  Node Name     Available                 Required                  Status
  ------------  ------------------------  ------------------------  ----------
  rhel6m2       libaio-devel(x86_64)-0.3.107-10.el6  libaio-devel(x86_64)-0.3.107  passed
  rhel6m1       libaio-devel(x86_64)-0.3.107-10.el6  libaio-devel(x86_64)-0.3.107  passed
Result: Package existence check passed for "libaio-devel(x86_64)"

Check: Package existence for "nfs-utils"
  Node Name     Available                 Required                  Status
  ------------  ------------------------  ------------------------  ----------
  rhel6m2       nfs-utils-1.2.3-15.el6    nfs-utils-1.2.3-15        passed
  rhel6m1       nfs-utils-1.2.3-15.el6    nfs-utils-1.2.3-15        passed
Result: Package existence check passed for "nfs-utils"

Checking for multiple users with UID value 0
Result: Check for multiple users with UID value 0 passed

Check: Current group ID
Result: Current group ID check passed

Starting check for consistency of primary group of root user
  Node Name                             Status
  ------------------------------------  ------------------------
  rhel6m2                               passed
  rhel6m1                               passed

Check for consistency of root user's primary group passed

Starting Clock synchronization checks using Network Time Protocol(NTP)...

Checking existence of NTP configuration file "/etc/ntp.conf" across nodes
  Node Name                             File exists?
  ------------------------------------  ------------------------
  rhel6m2                               no
  rhel6m1                               no
Network Time Protocol(NTP) configuration file not found on any of the nodes. Oracle Cluster Time Synchronization Service(CTSS) can be used instead of NTP for time synchronization on the cluster nodes
No NTP Daemons or Services were found to be running

Result: Clock synchronization check using Network Time Protocol(NTP) passed

Checking Core file name pattern consistency...
Core file name pattern consistency check passed.

Checking to make sure user "grid" is not in "root" group
  Node Name     Status                    Comment
  ------------  ------------------------  ------------------------
  rhel6m2       passed                    does not exist
  rhel6m1       passed                    does not exist
Result: User "grid" is not part of "root" group. Check passed

Check default user file creation mask
  Node Name     Available                 Required                  Comment
  ------------  ------------------------  ------------------------  ----------
  rhel6m2       0022                      0022                      passed
  rhel6m1       0022                      0022                      passed
Result: Default user file creation mask check passed
Checking integrity of file "/etc/resolv.conf" across nodes

Checking the file "/etc/resolv.conf" to make sure only one of 'domain' and 'search' entries is defined
"domain" and "search" entries do not coexist in any "/etc/resolv.conf" file
Checking if 'domain' entry in file "/etc/resolv.conf" is consistent across the nodes...
"domain" entry does not exist in any "/etc/resolv.conf" file
Checking if 'search' entry in file "/etc/resolv.conf" is consistent across the nodes...
Checking file "/etc/resolv.conf" to make sure that only one 'search' entry is defined
More than one "search" entry does not exist in any "/etc/resolv.conf" file
All nodes have same "search" order defined in file "/etc/resolv.conf"
Checking DNS response time for an unreachable node
  Node Name                             Status
  ------------------------------------  ------------------------
  rhel6m1                               passed
  rhel6m2                               passed
The DNS response time for an unreachable node is within acceptable limit on all nodes
checking DNS response from all servers in "/etc/resolv.conf"
checking response for name "rhel6m2" from each of the name servers specified in "/etc/resolv.conf"
  Node Name     Source                    Comment                   Status
  ------------  ------------------------  ------------------------  ----------
  rhel6m2       10.10.10.10                IPv4                      passed
  rhel6m2       10.10.10.11                IPv4                      passed
checking response for name "rhel6m1" from each of the name servers specified in "/etc/resolv.conf"
  Node Name     Source                    Comment                   Status
  ------------  ------------------------  ------------------------  ----------
  rhel6m1       10.10.10.10                IPv4                      passed
  rhel6m1       10.10.10.11                IPv4                      passed

Check for integrity of file "/etc/resolv.conf" passed


UDev attributes check for OCR locations started...
Result: UDev attributes check passed for OCR locations


UDev attributes check for Voting Disk locations started...
Result: UDev attributes check passed for Voting Disk locations

Check: Time zone consistency
Result: Time zone consistency check passed
Checking VIP configuration.
Checking VIP Subnet configuration.
Check for VIP Subnet configuration passed.
Checking VIP reachability
Check for VIP reachability passed.

Checking Oracle Cluster Voting Disk configuration...

Oracle Cluster Voting Disk configuration check passed

Clusterware version consistency passed.

Checking integrity of name service switch configuration file "/etc/nsswitch.conf" ...
Checking if "hosts" entry in file "/etc/nsswitch.conf" is consistent across nodes...
Checking file "/etc/nsswitch.conf" to make sure that only one "hosts" entry is defined
More than one "hosts" entry does not exist in any "/etc/nsswitch.conf" file
All nodes have same "hosts" entry defined in file "/etc/nsswitch.conf"
Check for integrity of name service switch configuration file "/etc/nsswitch.conf" passed


Checking daemon "avahi-daemon" is not configured and running

Check: Daemon "avahi-daemon" not configured
  Node Name     Configured                Status
  ------------  ------------------------  ------------------------
  rhel6m2       no                        passed
  rhel6m1       no                        passed
Daemon not configured check passed for process "avahi-daemon"

Check: Daemon "avahi-daemon" not running
  Node Name     Running?                  Status
  ------------  ------------------------  ------------------------
  rhel6m2       no                        passed
  rhel6m1       no                        passed
Daemon not running check passed for process "avahi-daemon"

Starting check for Network interface bonding status of private interconnect network interfaces ...

Check for Network interface bonding status of private interconnect network interfaces passed

Starting check for /dev/shm mounted as temporary file system ...

Check for /dev/shm mounted as temporary file system passed

Starting check for /boot mount ...

Check for /boot mount passed

Starting check for zeroconf check ...

Check for zeroconf check passed

Pre-check for cluster services setup was unsuccessful on all the nodes.

Reason for failure is the physical memory requirement is not exactly being 4GB, this is ignorable and upgrade could continue.



6. Run the installer and select upgrade GI to continue the upgrade on OUI.

7. When prompted run the rootupgrade.sh scripts on each node. Output below is from running rootupgrade.sh script on first node

[root@rhel6m1 ~]# /opt/app/12.1.0/grid2/rootupgrade.sh
Performing root user operation.

Entries will be added to the /etc/oratab file as needed by
Database Configuration Assistant when a database is created
Finished running generic part of root script.
Now product-specific root actions will be performed.
Using configuration parameter file: /opt/app/12.1.0/grid2/crs/install/crsconfig_params
2015/03/26 17:32:22 CLSRSC-4015: Performing install or upgrade action for Oracle Trace File Analyzer (TFA) Collector.

2015/03/26 17:32:25 CLSRSC-4003: Successfully patched Oracle Trace File Analyzer (TFA) Collector.

2015/03/26 17:32:34 CLSRSC-464: Starting retrieval of the cluster configuration data

2015/03/26 17:32:58 CLSRSC-465: Retrieval of the cluster configuration data has successfully completed.

2015/03/26 17:32:58 CLSRSC-363: User ignored prerequisites during installation

2015/03/26 17:33:28 CLSRSC-515: Starting OCR manual backup.

2015/03/26 17:33:34 CLSRSC-516: OCR manual backup successful.

2015/03/26 17:33:46 CLSRSC-468: Setting Oracle Clusterware and ASM to rolling migration mode

2015/03/26 17:33:46 CLSRSC-482: Running command: '/opt/app/12.1.0/grid2/bin/asmca -silent -upgradeNodeASM -nonRolling false -oldCRSHome /opt/app/11.2.0/grid4 -oldCRSVersion 11.2.0.4.0 -nodeNumber 1 -firstNode true -startRolling true'

ASM configuration upgraded in local node successfully.

2015/03/26 17:33:59 CLSRSC-469: Successfully set Oracle Clusterware and ASM to rolling migration mode

2015/03/26 17:34:00 CLSRSC-466: Starting shutdown of the current Oracle Grid Infrastructure stack

2015/03/26 17:34:28 CLSRSC-467: Shutdown of the current Oracle Grid Infrastructure stack has successfully completed.

OLR initialization - successful
2015/03/26 17:38:13 CLSRSC-329: Replacing Clusterware entries in file 'oracle-ohasd.conf'

CRS-4133: Oracle High Availability Services has been stopped.
CRS-4123: Oracle High Availability Services has been started.
2015/03/26 17:44:21 CLSRSC-472: Attempting to export the OCR

2015/03/26 17:44:21 CLSRSC-482: Running command: 'ocrconfig -upgrade grid oinstall'

2015/03/26 17:44:48 CLSRSC-473: Successfully exported the OCR

2015/03/26 17:44:55 CLSRSC-486:
 At this stage of upgrade, the OCR has changed.
 Any attempt to downgrade the cluster after this point will require a complete cluster outage to restore the OCR.

2015/03/26 17:44:55 CLSRSC-541:
 To downgrade the cluster:
 1. All nodes that have been upgraded must be downgraded.

2015/03/26 17:44:55 CLSRSC-542:
 2. Before downgrading the last node, the Grid Infrastructure stack on all other cluster nodes must be down.

2015/03/26 17:44:55 CLSRSC-543:
 3. The downgrade command must be run on the node rhel6m2 with the '-lastnode' option to restore global configuration data.

2015/03/26 17:45:21 CLSRSC-343: Successfully started Oracle Clusterware stack

clscfg: EXISTING configuration version 5 detected.
clscfg: version 5 is 11g Release 2.
Successfully taken the backup of node specific configuration in OCR.
Successfully accumulated necessary OCR keys.
Creating OCR keys for user 'root', privgrp 'root'..
Operation successful.
2015/03/26 17:45:51 CLSRSC-474: Initiating upgrade of resource types

2015/03/26 17:46:30 CLSRSC-482: Running command: 'upgrade model  -s 11.2.0.4.0 -d 12.1.0.2.0 -p first'

2015/03/26 17:46:31 CLSRSC-475: Upgrade of resource types successfully initiated.

2015/03/26 17:46:46 CLSRSC-325: Configure Oracle Grid Infrastructure for a Cluster ... succeeded

Running the rootupgrade.sh on the second node (last node)

[root@rhel6m2 ~]# /opt/app/12.1.0/grid2/rootupgrade.sh
Performing root user operation.

Entries will be added to the /etc/oratab file as needed by
Database Configuration Assistant when a database is created
Finished running generic part of root script.
Now product-specific root actions will be performed.
Using configuration parameter file: /opt/app/12.1.0/grid2/crs/install/crsconfig_params
2015/03/26 17:48:12 CLSRSC-4015: Performing install or upgrade action for Oracle Trace File Analyzer (TFA) Collector.

2015/03/26 17:48:15 CLSRSC-4003: Successfully patched Oracle Trace File Analyzer (TFA) Collector.

2015/03/26 17:48:18 CLSRSC-464: Starting retrieval of the cluster configuration data

2015/03/26 17:48:34 CLSRSC-465: Retrieval of the cluster configuration data has successfully completed.

2015/03/26 17:48:34 CLSRSC-363: User ignored prerequisites during installation

ASM configuration upgraded in local node successfully.

2015/03/26 17:48:59 CLSRSC-466: Starting shutdown of the current Oracle Grid Infrastructure stack

2015/03/26 17:49:31 CLSRSC-467: Shutdown of the current Oracle Grid Infrastructure stack has successfully completed.

OLR initialization - successful
2015/03/26 17:50:07 CLSRSC-329: Replacing Clusterware entries in file 'oracle-ohasd.conf'

CRS-4133: Oracle High Availability Services has been stopped.
CRS-4123: Oracle High Availability Services has been started.
2015/03/26 17:55:00 CLSRSC-343: Successfully started Oracle Clusterware stack

clscfg: EXISTING configuration version 5 detected.
clscfg: version 5 is 12c Release 1.
Successfully taken the backup of node specific configuration in OCR.
Successfully accumulated necessary OCR keys.
Creating OCR keys for user 'root', privgrp 'root'..
Operation successful.
Start upgrade invoked..
2015/03/26 17:55:29 CLSRSC-478: Setting Oracle Clusterware active version on the last node to be upgraded

2015/03/26 17:55:29 CLSRSC-482: Running command: '/opt/app/12.1.0/grid2/bin/crsctl set crs activeversion'

Started to upgrade the Oracle Clusterware. This operation may take a few minutes.
Started to upgrade the OCR.
Started to upgrade the CSS.
The CSS was successfully upgraded.
Started to upgrade Oracle ASM.
Started to upgrade the CRS.
The CRS was successfully upgraded.
Successfully upgraded the Oracle Clusterware.
Oracle Clusterware operating version was successfully set to 12.1.0.2.0
2015/03/26 17:58:19 CLSRSC-479: Successfully set Oracle Clusterware active version

2015/03/26 17:58:28 CLSRSC-476: Finishing upgrade of resource types

2015/03/26 17:58:42 CLSRSC-482: Running command: 'upgrade model  -s 11.2.0.4.0 -d 12.1.0.2.0 -p last'

2015/03/26 17:58:42 CLSRSC-477: Successfully completed upgrade of resource types

2015/03/26 17:59:32 CLSRSC-325: Configure Oracle Grid Infrastructure for a Cluster ... succeeded

8. Once the rootupgrade.sh are run the GI management repository will be created.

9. Once the OUI is completed verify the active version is upgraded to 12.1.0.2

[grid@rhel6m1 ~]$ crsctl query crs softwareversion
Oracle Clusterware version on node [rhel6m1] is [12.1.0.2.0]
[grid@rhel6m1 ~]$ crsctl query crs activeversion -f
Oracle Clusterware active version on the cluster is [12.1.0.2.0]. The cluster upgrade state is [NORMAL].

10. Use

cluvfy stage -post crsinst
or
orachk -u -o post

to check for any post upgrade issues. If satisfied with the upgrade uninstall the 11.2.0.4 version of grid infrastructure software. This conclude the upgrade of grid infrastructure from 11.2.0.4 to 12.1.0.2.

Related Posts
Upgrading Grid Infrastructure Used for Single Instance from 11.2.0.4 to 12.1.0.2
Upgrading RAC from 12.1.0.1 to 12.1.0.2 - Grid Infrastructure
Upgrading 12c CDB and PDB from 12.1.0.1 to 12.1.0.2
Upgrading from 11gR2 (11.2.0.3) to 12c (12.1.0.1) Grid Infrastructure
Upgrade Oracle Database 12c1 from 12.1.0.1 to 12.1.0.2
Upgrading from 10.2.0.4 to 10.2.0.5 (Clusterware, RAC, ASM)
Upgrade from 10.2.0.5 to 11.2.0.3 (Clusterware, RAC, ASM)
Upgrade from 11.1.0.7 to 11.2.0.3 (Clusterware, ASM & RAC)
Upgrade from 11.1.0.7 to 11.2.0.4 (Clusterware, ASM & RAC)
Upgrading from 11.1.0.7 to 11.2.0.3 with Transient Logical Standby
Upgrading from 11.2.0.1 to 11.2.0.3 with in-place upgrade for RAC
In-place upgrade from 11.2.0.2 to 11.2.0.3
Upgrading from 11.2.0.2 to 11.2.0.3 with Physical Standby - 1
Upgrading from 11.2.0.2 to 11.2.0.3 with Physical Standby - 2
Upgrading from 11gR2 (11.2.0.3) to 12c (12.1.0.1) Grid Infrastructure


Update 24 April 2015
It seems even though the GI gets upgraded the TFA does not and continue to run out of the old 11.2.0.4 GI home. Even on the init script (init.tfa) the TFA home is listed inside the 11.2.0.4 GI home.

more /etc/init.d/init.tfa | grep grid
TFA_HOME=/opt/app/11.2.0/grid4/tfa/rhel6m1/tfa_home

While this old TFA is present even unlocking and patching the new 12c GI home will not install the TFA in the 12c home even though output says upgrading or installing TFA.

/opt/app/12.1.0/grid2/crs/install/rootcrs.pl -patch
Using configuration parameter file: /opt/app/12.1.0/grid2/crs/install/crsconfig_params
2015/04/24 17:58:36 CLSRSC-4015: Performing install or upgrade action for Oracle Trace File Analyzer (TFA) Collector.

2015/04/24 17:58:36 CLSRSC-4003: Successfully patched Oracle Trace File Analyzer (TFA) Collector.

There are several ways to fix this if running of tfa is desired. One is to uninstall the 11.2.0.4 GI home or TFA from 11.2.0.4 and then unlock and patch the 12c home. This has to be done on all nodes.

./tfactl uninstall

TFA will be Uninstalled on Node rhel6m1:

Removing TFA from rhel6m1 only
Please remove TFA locally on any other configured nodes

Notifying Other Nodes about TFA Uninstall...
Sleeping for 10 seconds...
Stopping TFA in rhel6m1...
Shutting down TFA
oracle-tfa stop/waiting
. . . . .
Killing TFA running with pid 1214
. . .
Successfully shutdown TFA..

Deleting TFA support files on rhel6m1:
Removing /opt/app/oracle/tfa/rhel6m1/database...
Removing /opt/app/oracle/tfa/rhel6m1/log...
Removing /opt/app/oracle/tfa/rhel6m1/output...
Removing /opt/app/oracle/tfa/rhel6m1...
Removing /opt/app/oracle/tfa...
Removing /etc/init.d/init.tfa...
Removing /opt/app/11.2.0/grid4/bin/tfactl...
Removing /opt/app/11.2.0/grid4/tfa/bin...
Removing /opt/app/11.2.0/grid4/tfa/rhel6m1...

/opt/app/12.1.0/grid2/crs/install/rootcrs.pl -patch
Using configuration parameter file: /opt/app/12.1.0/grid2/crs/install/crsconfig_params
2015/04/24 18:06:27 CLSRSC-4015: Performing install or upgrade action for Oracle Trace File Analyzer (TFA) Collector.

2015/04/24 18:06:58 CLSRSC-4003: Successfully patched Oracle Trace File Analyzer (TFA) Collector.

# cat  /etc/init.d/init.tfa | grep grid
TFA_HOME=/opt/app/12.1.0/grid2/tfa/rhel6m1/tfa_home

If unlocking/patching the GI home is not an option then uninstall the TFA on old 11.2 home and reinstall specifying 12c home. TFA installation is available from MOS 1513912.2.

Update 09 August 2016
There's a GI related issue for environments upgraded from 11.2 to 12.1 and after applying Oracle Grid Infrastructure Patch Set Update 12.1.0.2.160719 (Jul2016), scan listener will fail to start after applying this patch. This is only applicable to environments upgrade to 12.1 from 11.2. No issue for environments starting at 12.1. For solution refer
SCAN Listener or local listener fails to start after applying Patch 23273629 - Oracle Grid Infrastructure Patch Set Update 12.1.0.2.160719 (Jul2016) [ID 2166451.1]

Useful Metalink Notes
On Linux, When Installing or Upgrading GI to 12.1.0.2, the Root Script on Remote Nodes May Fail with "error while loading shared libraries" Messages [ID 2032832.1]
How to Upgrade to/Downgrade from Grid Infrastructure 12c and Known Issues [ID 1579762.1]
Grid Infrastructure root script (root.sh etc) fails as remote node missing binaries [ID 1991928.1]

[INS-30508] Invalid ASM disks

A test system was failing with [INS-30508] Invalid ASM disks when GI was being installed (same error observed on both 11.2.0.4 and 12.1.0.2).

The ASM block device permission were set on /etc/rc.local instead of via udev rules.

ls -l /dev/sd*1
brw-rw----. 1 grid asmadmin 8, 17 Mar 27 15:34 /dev/sdb1
brw-rw----. 1 grid asmadmin 8, 33 Mar 27 15:34 /dev/sdc1

But these permissions are changed to ownership of root when the "change discovery path" is run.

ls -l /dev/sd*1
brw-rw----. 1 root disk 8, 17 Mar 27 15:40 /dev/sdb1
brw-rw----. 1 root disk 8, 33 Mar 27 15:40 /dev/sdc1

It seems permission is changed at the end of the discovery. The installation log has the kfod output.

INFO: Executing [/tmp/OraInstall2015-03-27_03-43-42PM/ext/bin/kfod, nohdr=true, verbose=true, disks=all, status=true, op=disks, asm_diskstring='/dev/sd*']
 with environment variables {XFILESEARCHPATH=/usr/dt/app-defaults/%L/Dt, SYSFONT=latarcyrheb-sun16, LANG=en_US.UTF-8, _=/home/grid/grid/install/.oui, GS_LIB=, KDE_MULTIHEAD=false, G_BROKEN_FILENAMES=1, SSH_AGENT_PID=11532, HOME=/home/grid, QT_PLUGIN_PATH=/home/grid/.kde/lib64/kde4/plugins/:/usr/lib64/kde4/plugins/, WINDOWID=2097177, LD_LIBRARY_PATH=/tmp/OraInstall2015-03-27_03-43-42PM/ext/bin/../lib, KDE_SESSION_VERSION=4, SSH_ASKPASS=/usr/bin/ksshaskpass, SSH_AUTH_SOCK=/tmp/ssh-TgemA11531/agent.11531, KONSOLE_DBUS_SERVICE=:1.29, CVS_RSH=ssh, GPG_AGENT_INFO=/tmp/gpg-KYVdjy/S.gpg-agent:11557:1, SHLVL=5, XCURSOR_THEME=default, LANGUAGE=, HISTCONTROL=ignoredups, KDE_FULL_SESSION=true, KDE_IS_PRELINKED=1, PROFILEHOME=/home/grid, MODULEPATH=/usr/share/Modules/modulefiles:/etc/modulefiles, QTLIB=/usr/lib64/qt-3.3/lib, LS_COLORS=rs=0:di=01;34:ln=01;36:mh=00:pi=40;33:so=01;35:do=01;35:bd=40;33;01:cd=40;33;01:or=40;31;01:mi=01;05;37;41:su=37;41:sg=30;43:ca=30;41:tw=30;42:ow=34;42:st=37;44:ex=01;32:*.tar=01;31:*.tgz=01;31:*.arj=01;31:*.taz=01;31:*.lzh=01;31:*.lzma=01;31:*.tlz=01;31:*.txz=01;31:*.zip=01;31:*.z=01;31:*.Z=01;31:*.dz=01;31:*.gz=01;31:*.lz=01;31:*.xz=01;31:*.bz2=01;31:*.tbz=01;31:*.tbz2=01;31:*.bz=01;31:*.tz=01;31:*.deb=01;31:*.rpm=01;31:*.jar=01;31:*.rar=01;31:*.ace=01;31:*.zoo=01;31:*.cpio=01;31:*.7z=01;31:*.rz=01;31:*.jpg=01;35:*.jpeg=01;35:*.gif=01;35:*.bmp=01;35:*.pbm=01;35:*.pgm=01;35:*.ppm=01;35:*.tga=01;35:*.xbm=01;35:*.xpm=01;35:*.tif=01;35:*.tiff=01;35:*.png=01;35:*.svg=01;35:*.svgz=01;35:*.mng=01;35:*.pcx=01;35:*.mov=01;35:*.mpg=01;35:*.mpeg=01;35:*.m2v=01;35:*.mkv=01;35:*.ogm=01;35:*.mp4=01;35:*.m4v=01;35:*.mp4v=01;35:*.vob=01;35:*.qt=01;35:*.nuv=01;35:*.wmv=01;35:*.asf=01;35:*.rm=01;35:*.rmvb=01;35:*.flc=01;35:*.avi=01;35:*.fli=01;35:*.flv=01;35:*.gl=01;35:*.dl=01;35:*.xcf=01;35:*.xwd=01;35:*.yuv=01;35:*.cgm=01;35:*.emf=01;35:*.axv=01;35:*.anx=01;35:*.ogv=01;35:*.ogx=01;35:*.aac=01;36:*.au=01;36:*.flac=01;36:*.mid=01;36:*.midi=01;36:*.mka=01;36:*.mp3=01;36:*.mpc=01;36:*.ogg=01;36:*.ra=01;36:*.wav=01;36:*.axa=01;36:*.oga=01;36:*.spx=01;36:*.xspf=01;36:, KDEDIRS=/usr, LC_ALL=, HISTSIZE=1000, LOADEDMODULES=, module=() {  eval `/usr/bin/modulecmd bash $*`
}, TERM=xterm, KDE_SESSION_UID=502, DISPLAY=:1, PWD=/home/grid/grid, USER=grid, NLSPATH=/usr/dt/lib/nls/msg/%L/%N.cat, QTINC=/usr/lib64/qt-3.3/include, CLASSPATH=, XDG_DATA_DIRS=/usr/share/kde-settings/kde-profile/default/share:/usr/local/share:/usr/share, HOSTNAME=rhel6m1.domain.net, MODULESHOME=/usr/share/Modules, SESSION_MANAGER=local/unix:@/tmp/.ICE-unix/11585,unix/unix:/tmp/.ICE-unix/11585, GI_HOME=/opt/app/oracle/product/11.2.0/grid_1, LESSOPEN=|/usr/bin/lesspipe.sh %s, ORACLE_HOME=/tmp/OraInstall2015-03-27_03-43-42PM/ext/bin, LOGNAME=grid, SHELL=/bin/bash, COLORFGBG=15;0, DBUS_SESSION_BUS_ADDRESS=unix:abstract=/tmp/dbus-FHqO2yewGg,guid=022c95b15b2c5a6396b3e9b9000004cd, XDG_SESSION_COOKIE=e7248f2c4d732e53900d83b20000000c-1427470249.835456-1664171820, GTK2_RC_FILES=/etc/gtk-2.0/gtkrc:/home/grid/.gtkrc-2.0::/home/grid/.kde/share/config/gtkrc-2.0, KONSOLE_DBUS_SESSION=/Sessions/1, MAIL=/var/spool/mail/grid, VNCDESKTOP=rhel6m1.domain.net:1 (grid), QTDIR=/usr/lib64/qt-3.3, JAVA_HOME=, XDG_MENU_PREFIX=kde4-, GTK_RC_FILES=/etc/gtk/gtkrc:/home/grid/.gtkrc::/home/grid/.kde/share/config/gtkrc, PATH=/usr/bin:/usr/ccs/bin:/opt/app/oracle/product/11.2.0/grid_1/bin:/usr/lib64/qt-3.3/bin:/usr/local/bin:/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/sbin:/home/grid/bin:/home/grid/grid/install}
INFO: Starting Output Reader Threads for process /tmp/OraInstall2015-03-27_03-43-42PM/ext/bin/kfod
INFO: Parsing 10236 CANDIDATE /dev/sdb1 grid     asmadmin
INFO: The process /tmp/OraInstall2015-03-27_03-43-42PM/ext/bin/kfod exited with code 0
INFO: Waiting for output processor threads to exit.
INFO: Parsing 10236 CANDIDATE /dev/sdc1 grid     asmadmin

From this output it seems that kfod discovers the candidate disks and they appear on the OUI. But since the permission has changed at the end of the command clicking next after selecting disks for the ASM diskgroup results in OUI giving the [INS-30508] Invalid ASM disks error.
One of the workaround is to change the permission manually and click the next button or better option is to have udev rules created instead of putting permissions on /etc/rc.local.

Useful metalink notes
[INS-30508] Invalid ASM disks [ID 1941922.1]
Oracle Universal Installer (OUI) in silent mode fails with INS-30508 [ID 987393.1]

Downgrade Grid Infrastructure from 12.1.0.2 to 11.2.0.4

This post list steps for downgrading grid infrastructure from 12.1.0.2 to 11.2.0.4. The grid infrastructure upgrade has been successful on all nodes. If upgrade to 12.1.0.2 had failed on any of the nodes then follow oracle documentation for additional steps/options required for such cases. Only the GI was upgraded to 12.1.0.2 and the database remained on 11.2.0.4.

[grid@rhel6m1 ~]$  crsctl query crs activeversion
Oracle Clusterware active version on the cluster is [12.1.0.2.0]
[grid@rhel6m1 ~]$ crsctl query crs softwareversion
Oracle Clusterware version on node [rhel6m1] is [12.1.0.2.0]

[root@rhel6m1 grid]# ocrcheck
Status of Oracle Cluster Registry is as follows :
         Version                  :          4
         Total space (kbytes)     :     409568
         Used space (kbytes)      :       9068
         Available space (kbytes) :     400500
         ID                       : 2072206343
         Device/File Name         : +CLUSTER_DG

The environment used here is the same environment used to upgrade from 11.2.0.3 to 11.2.0.4. As such the OCR-node (node containing the backup of the OCR created during the upgrade to 12.1.0.2) contains both 11.2.0.3 and 11.2.0.4 OCR backups but with different permissions.

[grid@rhel6m1 cdata]$ ls -lrt /opt/app/12.1.0/grid2/cdata/
total 266644
drwxr-xr-x. 2 grid oinstall      4096 Mar 26 17:22 localhost
-rw-r--r--. 1 grid oinstall    132453 Mar 26 17:34 ocr11.2.0.3.0
-rw-------. 1 root root        130412 Mar 26 17:44 ocr11.2.0.4.0
drwxr-xr-x. 2 grid oinstall      4096 Mar 26 17:46 rhel6m1
-rw-------. 1 root oinstall 503484416 Mar 26 17:54 rhel6m1.olr
drwxrwxr-x. 2 grid oinstall      4096 Mar 26 17:58 rhel6m-cluster

During the downgrade the the higher version is detected and restored (no harm even if the ocr11.2.0.3.0 is removed before the downgrade).
Also before the downgrade commands are run make sure only the 12.1 related binaries are in the PATH variable (crsctl, srvctl) and ORACLE_HOME points to 12.1. At times the downgrade failed with

Died at /opt/app/12.1.0/grid2/crs/install/crsdeconfig.pm line 2336.
The command '/opt/app/12.1.0/grid2/perl/bin/perl -I/opt/app/12.1.0/grid2/perl/lib -I/opt/app/12.1.0/grid2/crs/install /opt/app/12.1.0/grid2/crs/install/rootcrs.pl -downgrade -lastnode' execution failed

for which solution according to 1917922.1 is to run the downgrade command from a location where grid user has write access to. In this case it was run from grid user home so MOS note solution may not be applicable.
Another time the downgrade on the OCR-node failed with

Using configuration parameter file: /opt/app/12.1.0/grid2/crs/install/crsconfig_params
2015-04-01 10:43:00.047
CLSD: An error occurred while attempting to generate a full name. Logging may not be active for this process
Additional diagnostics: CLSU-00100: operating system function: sclsdgcwd failed with error data: -1
CLSU-00103: error location: sclsdgcwd5
(:CLSD00183:)

MOS notes 1961344.1 and 1911258.1 gives causes and solutions for this but neither case was valid in this case as the permission of directories mentioned on these MOS were already set to correct values. These issues went away after removing from PATH any references to 11.2 binaries.
To begin the downgrade run the rootcrs.sh with downgrade option on all but the OCR-node. In this case the rhel6m1 is the OCR-node and downgrade first run on rhel6m2.

[root@rhel6m2 grid]# /opt/app/12.1.0/grid2/crs/install/rootcrs.sh -downgrade
Using configuration parameter file: /opt/app/12.1.0/grid2/crs/install/crsconfig_params
CRS-2791: Starting shutdown of Oracle High Availability Services-managed resources on 'rhel6m2'
CRS-2673: Attempting to stop 'ora.crsd' on 'rhel6m2'
CRS-2790: Starting shutdown of Cluster Ready Services-managed resources on 'rhel6m2'
CRS-2673: Attempting to stop 'ora.CLUSTER_DG.dg' on 'rhel6m2'
CRS-2673: Attempting to stop 'ora.MYSCANLISTENER_SCAN1.lsnr' on 'rhel6m2'
CRS-2673: Attempting to stop 'ora.std11g2.asa.domain.net.svc' on 'rhel6m2'
CRS-2673: Attempting to stop 'ora.oc4j' on 'rhel6m2'
CRS-2677: Stop of 'ora.CLUSTER_DG.dg' on 'rhel6m2' succeeded
CRS-2677: Stop of 'ora.std11g2.asa.domain.net.svc' on 'rhel6m2' succeeded
CRS-2673: Attempting to stop 'ora.std11g2.db' on 'rhel6m2'
CRS-2673: Attempting to stop 'ora.MYLISTENER.lsnr' on 'rhel6m2'
CRS-2677: Stop of 'ora.MYSCANLISTENER_SCAN1.lsnr' on 'rhel6m2' succeeded
CRS-2673: Attempting to stop 'ora.scan1.vip' on 'rhel6m2'
CRS-2677: Stop of 'ora.MYLISTENER.lsnr' on 'rhel6m2' succeeded
CRS-2673: Attempting to stop 'ora.rhel6m2.vip' on 'rhel6m2'
CRS-2677: Stop of 'ora.std11g2.db' on 'rhel6m2' succeeded
CRS-2673: Attempting to stop 'ora.FLASH.dg' on 'rhel6m2'
CRS-2673: Attempting to stop 'ora.DATA.dg' on 'rhel6m2'
CRS-2677: Stop of 'ora.scan1.vip' on 'rhel6m2' succeeded
CRS-2672: Attempting to start 'ora.scan1.vip' on 'rhel6m1'
CRS-2677: Stop of 'ora.DATA.dg' on 'rhel6m2' succeeded
CRS-2677: Stop of 'ora.rhel6m2.vip' on 'rhel6m2' succeeded
CRS-2672: Attempting to start 'ora.rhel6m2.vip' on 'rhel6m1'
CRS-2676: Start of 'ora.scan1.vip' on 'rhel6m1' succeeded
CRS-2672: Attempting to start 'ora.MYSCANLISTENER_SCAN1.lsnr' on 'rhel6m1'
CRS-2677: Stop of 'ora.FLASH.dg' on 'rhel6m2' succeeded
CRS-2673: Attempting to stop 'ora.asm' on 'rhel6m2'
CRS-2677: Stop of 'ora.asm' on 'rhel6m2' succeeded
CRS-2676: Start of 'ora.rhel6m2.vip' on 'rhel6m1' succeeded
CRS-2676: Start of 'ora.MYSCANLISTENER_SCAN1.lsnr' on 'rhel6m1' succeeded
CRS-2677: Stop of 'ora.oc4j' on 'rhel6m2' succeeded
CRS-2672: Attempting to start 'ora.oc4j' on 'rhel6m1'
CRS-2676: Start of 'ora.oc4j' on 'rhel6m1' succeeded
CRS-2673: Attempting to stop 'ora.ons' on 'rhel6m2'
CRS-2677: Stop of 'ora.ons' on 'rhel6m2' succeeded
CRS-2673: Attempting to stop 'ora.net1.network' on 'rhel6m2'
CRS-2677: Stop of 'ora.net1.network' on 'rhel6m2' succeeded
CRS-2792: Shutdown of Cluster Ready Services-managed resources on 'rhel6m2' has completed
CRS-2677: Stop of 'ora.crsd' on 'rhel6m2' succeeded
CRS-2673: Attempting to stop 'ora.ctssd' on 'rhel6m2'
CRS-2673: Attempting to stop 'ora.evmd' on 'rhel6m2'
CRS-2673: Attempting to stop 'ora.storage' on 'rhel6m2'
CRS-2673: Attempting to stop 'ora.mdnsd' on 'rhel6m2'
CRS-2673: Attempting to stop 'ora.gpnpd' on 'rhel6m2'
CRS-2673: Attempting to stop 'ora.drivers.acfs' on 'rhel6m2'
CRS-2677: Stop of 'ora.storage' on 'rhel6m2' succeeded
CRS-2673: Attempting to stop 'ora.asm' on 'rhel6m2'
CRS-2677: Stop of 'ora.drivers.acfs' on 'rhel6m2' succeeded
CRS-2677: Stop of 'ora.ctssd' on 'rhel6m2' succeeded
CRS-2677: Stop of 'ora.mdnsd' on 'rhel6m2' succeeded
CRS-2677: Stop of 'ora.evmd' on 'rhel6m2' succeeded
CRS-2677: Stop of 'ora.gpnpd' on 'rhel6m2' succeeded
CRS-2677: Stop of 'ora.asm' on 'rhel6m2' succeeded
CRS-2673: Attempting to stop 'ora.cluster_interconnect.haip' on 'rhel6m2'
CRS-2677: Stop of 'ora.cluster_interconnect.haip' on 'rhel6m2' succeeded
CRS-2673: Attempting to stop 'ora.cssd' on 'rhel6m2'
CRS-2677: Stop of 'ora.cssd' on 'rhel6m2' succeeded
CRS-2673: Attempting to stop 'ora.crf' on 'rhel6m2'
CRS-2677: Stop of 'ora.crf' on 'rhel6m2' succeeded
CRS-2673: Attempting to stop 'ora.gipcd' on 'rhel6m2'
CRS-2677: Stop of 'ora.gipcd' on 'rhel6m2' succeeded
CRS-2793: Shutdown of Oracle High Availability Services-managed resources on 'rhel6m2' has completed
CRS-4133: Oracle High Availability Services has been stopped.
2015/04/01 12:07:02 CLSRSC-4001: Installing Oracle Trace File Analyzer (TFA) Collector.

2015/04/01 12:07:02 CLSRSC-4002: Successfully installed Oracle Trace File Analyzer (TFA) Collector.

Successfully downgraded Oracle Clusterware stack on this node

Since this is a two node RAC the downgrade command is run with lastnode option on rhel6m1 (OCR-node). This will remove the GI management repository, downgrade the OCR

[root@rhel6m1 grid]# /opt/app/12.1.0/grid2/crs/install/rootcrs.sh -downgrade -lastnode
Using configuration parameter file: /opt/app/12.1.0/grid2/crs/install/crsconfig_params
CRS-2791: Starting shutdown of Oracle High Availability Services-managed resources on 'rhel6m1'
CRS-2673: Attempting to stop 'ora.crsd' on 'rhel6m1'
CRS-2790: Starting shutdown of Cluster Ready Services-managed resources on 'rhel6m1'
CRS-2673: Attempting to stop 'ora.MYLISTENER.lsnr' on 'rhel6m1'
CRS-2673: Attempting to stop 'ora.std11g2.asa.domain.net.svc' on 'rhel6m1'
CRS-2673: Attempting to stop 'ora.MYSCANLISTENER_SCAN1.lsnr' on 'rhel6m1'
CRS-2673: Attempting to stop 'ora.rhel6m2.vip' on 'rhel6m1'
CRS-2673: Attempting to stop 'ora.oc4j' on 'rhel6m1'
CRS-2673: Attempting to stop 'ora.mgmtdb' on 'rhel6m1'
CRS-2673: Attempting to stop 'ora.cvu' on 'rhel6m1'
CRS-2677: Stop of 'ora.cvu' on 'rhel6m1' succeeded
CRS-2677: Stop of 'ora.MYSCANLISTENER_SCAN1.lsnr' on 'rhel6m1' succeeded
CRS-2673: Attempting to stop 'ora.scan1.vip' on 'rhel6m1'
CRS-2677: Stop of 'ora.MYLISTENER.lsnr' on 'rhel6m1' succeeded
CRS-2677: Stop of 'ora.std11g2.asa.domain.net.svc' on 'rhel6m1' succeeded
CRS-2673: Attempting to stop 'ora.std11g2.db' on 'rhel6m1'
CRS-2677: Stop of 'ora.std11g2.db' on 'rhel6m1' succeeded
CRS-2673: Attempting to stop 'ora.DATA.dg' on 'rhel6m1'
CRS-2673: Attempting to stop 'ora.FLASH.dg' on 'rhel6m1'
CRS-2677: Stop of 'ora.rhel6m2.vip' on 'rhel6m1' succeeded
CRS-2673: Attempting to stop 'ora.rhel6m1.vip' on 'rhel6m1'
CRS-2677: Stop of 'ora.DATA.dg' on 'rhel6m1' succeeded
CRS-2677: Stop of 'ora.scan1.vip' on 'rhel6m1' succeeded
CRS-2677: Stop of 'ora.rhel6m1.vip' on 'rhel6m1' succeeded
CRS-2677: Stop of 'ora.mgmtdb' on 'rhel6m1' succeeded
CRS-2673: Attempting to stop 'ora.CLUSTER_DG.dg' on 'rhel6m1'
CRS-2673: Attempting to stop 'ora.MGMTLSNR' on 'rhel6m1'
CRS-2677: Stop of 'ora.FLASH.dg' on 'rhel6m1' succeeded
CRS-2677: Stop of 'ora.MGMTLSNR' on 'rhel6m1' succeeded
CRS-2677: Stop of 'ora.CLUSTER_DG.dg' on 'rhel6m1' succeeded
CRS-2673: Attempting to stop 'ora.asm' on 'rhel6m1'
CRS-2677: Stop of 'ora.asm' on 'rhel6m1' succeeded
CRS-2677: Stop of 'ora.oc4j' on 'rhel6m1' succeeded
CRS-2673: Attempting to stop 'ora.ons' on 'rhel6m1'
CRS-2677: Stop of 'ora.ons' on 'rhel6m1' succeeded
CRS-2673: Attempting to stop 'ora.net1.network' on 'rhel6m1'
CRS-2677: Stop of 'ora.net1.network' on 'rhel6m1' succeeded
CRS-2792: Shutdown of Cluster Ready Services-managed resources on 'rhel6m1' has completed
CRS-2677: Stop of 'ora.crsd' on 'rhel6m1' succeeded
CRS-2673: Attempting to stop 'ora.storage' on 'rhel6m1'
CRS-2673: Attempting to stop 'ora.mdnsd' on 'rhel6m1'
CRS-2673: Attempting to stop 'ora.gpnpd' on 'rhel6m1'
CRS-2673: Attempting to stop 'ora.drivers.acfs' on 'rhel6m1'
CRS-2677: Stop of 'ora.storage' on 'rhel6m1' succeeded
CRS-2677: Stop of 'ora.drivers.acfs' on 'rhel6m1' succeeded
CRS-2673: Attempting to stop 'ora.crf' on 'rhel6m1'
CRS-2673: Attempting to stop 'ora.ctssd' on 'rhel6m1'
CRS-2673: Attempting to stop 'ora.evmd' on 'rhel6m1'
CRS-2673: Attempting to stop 'ora.asm' on 'rhel6m1'
CRS-2677: Stop of 'ora.mdnsd' on 'rhel6m1' succeeded
CRS-2677: Stop of 'ora.gpnpd' on 'rhel6m1' succeeded
CRS-2677: Stop of 'ora.crf' on 'rhel6m1' succeeded
CRS-2677: Stop of 'ora.ctssd' on 'rhel6m1' succeeded
CRS-2677: Stop of 'ora.evmd' on 'rhel6m1' succeeded
CRS-2677: Stop of 'ora.asm' on 'rhel6m1' succeeded
CRS-2673: Attempting to stop 'ora.cluster_interconnect.haip' on 'rhel6m1'
CRS-2677: Stop of 'ora.cluster_interconnect.haip' on 'rhel6m1' succeeded
CRS-2673: Attempting to stop 'ora.cssd' on 'rhel6m1'
CRS-2677: Stop of 'ora.cssd' on 'rhel6m1' succeeded
CRS-2673: Attempting to stop 'ora.gipcd' on 'rhel6m1'
CRS-2677: Stop of 'ora.gipcd' on 'rhel6m1' succeeded
CRS-2793: Shutdown of Oracle High Availability Services-managed resources on 'rhel6m1' has completed
CRS-4133: Oracle High Availability Services has been stopped.

ASM downgrade operation succeeded

CRS-2791: Starting shutdown of Oracle High Availability Services-managed resources on 'rhel6m1'
CRS-2673: Attempting to stop 'ora.crsd' on 'rhel6m1'
CRS-2677: Stop of 'ora.crsd' on 'rhel6m1' succeeded
CRS-2673: Attempting to stop 'ora.gpnpd' on 'rhel6m1'
CRS-2673: Attempting to stop 'ora.ctssd' on 'rhel6m1'
CRS-2673: Attempting to stop 'ora.evmd' on 'rhel6m1'
CRS-2673: Attempting to stop 'ora.storage' on 'rhel6m1'
CRS-2673: Attempting to stop 'ora.crf' on 'rhel6m1'
CRS-2673: Attempting to stop 'ora.drivers.acfs' on 'rhel6m1'
CRS-2673: Attempting to stop 'ora.mdnsd' on 'rhel6m1'
CRS-2677: Stop of 'ora.storage' on 'rhel6m1' succeeded
CRS-2673: Attempting to stop 'ora.asm' on 'rhel6m1'
CRS-2677: Stop of 'ora.drivers.acfs' on 'rhel6m1' succeeded
CRS-2677: Stop of 'ora.ctssd' on 'rhel6m1' succeeded
CRS-2677: Stop of 'ora.evmd' on 'rhel6m1' succeeded
CRS-2677: Stop of 'ora.gpnpd' on 'rhel6m1' succeeded
CRS-2677: Stop of 'ora.crf' on 'rhel6m1' succeeded
CRS-2677: Stop of 'ora.mdnsd' on 'rhel6m1' succeeded
CRS-2677: Stop of 'ora.asm' on 'rhel6m1' succeeded
CRS-2673: Attempting to stop 'ora.cluster_interconnect.haip' on 'rhel6m1'
CRS-2677: Stop of 'ora.cluster_interconnect.haip' on 'rhel6m1' succeeded
CRS-2673: Attempting to stop 'ora.cssd' on 'rhel6m1'
CRS-2677: Stop of 'ora.cssd' on 'rhel6m1' succeeded
CRS-2673: Attempting to stop 'ora.gipcd' on 'rhel6m1'
CRS-2677: Stop of 'ora.gipcd' on 'rhel6m1' succeeded
CRS-2793: Shutdown of Oracle High Availability Services-managed resources on 'rhel6m1' has completed
CRS-4133: Oracle High Availability Services has been stopped.
CRS-4123: Oracle High Availability Services has been started.
CRS-2791: Starting shutdown of Oracle High Availability Services-managed resources on 'rhel6m1'
CRS-2673: Attempting to stop 'ora.drivers.acfs' on 'rhel6m1'
CRS-2677: Stop of 'ora.drivers.acfs' on 'rhel6m1' succeeded
CRS-2793: Shutdown of Oracle High Availability Services-managed resources on 'rhel6m1' has completed
CRS-4133: Oracle High Availability Services has been stopped.
CRS-4123: Starting Oracle High Availability Services-managed resources
CRS-2672: Attempting to start 'ora.mdnsd' on 'rhel6m1'
CRS-2672: Attempting to start 'ora.evmd' on 'rhel6m1'
CRS-2676: Start of 'ora.mdnsd' on 'rhel6m1' succeeded
CRS-2676: Start of 'ora.evmd' on 'rhel6m1' succeeded
CRS-2672: Attempting to start 'ora.gpnpd' on 'rhel6m1'
CRS-2676: Start of 'ora.gpnpd' on 'rhel6m1' succeeded
CRS-2672: Attempting to start 'ora.gipcd' on 'rhel6m1'
CRS-2676: Start of 'ora.gipcd' on 'rhel6m1' succeeded
CRS-2672: Attempting to start 'ora.cssdmonitor' on 'rhel6m1'
CRS-2676: Start of 'ora.cssdmonitor' on 'rhel6m1' succeeded
CRS-2672: Attempting to start 'ora.cssd' on 'rhel6m1'
CRS-2672: Attempting to start 'ora.diskmon' on 'rhel6m1'
CRS-2676: Start of 'ora.diskmon' on 'rhel6m1' succeeded
CRS-2676: Start of 'ora.cssd' on 'rhel6m1' succeeded
CRS-2672: Attempting to start 'ora.cluster_interconnect.haip' on 'rhel6m1'
CRS-2672: Attempting to start 'ora.ctssd' on 'rhel6m1'
CRS-2676: Start of 'ora.ctssd' on 'rhel6m1' succeeded
CRS-2676: Start of 'ora.cluster_interconnect.haip' on 'rhel6m1' succeeded
CRS-2672: Attempting to start 'ora.asm' on 'rhel6m1'
CRS-2676: Start of 'ora.asm' on 'rhel6m1' succeeded
CRS-2672: Attempting to start 'ora.storage' on 'rhel6m1'
CRS-2676: Start of 'ora.storage' on 'rhel6m1' succeeded
CRS-2672: Attempting to start 'ora.crf' on 'rhel6m1'
CRS-2676: Start of 'ora.crf' on 'rhel6m1' succeeded
CRS-2672: Attempting to start 'ora.crsd' on 'rhel6m1'
CRS-2676: Start of 'ora.crsd' on 'rhel6m1' succeeded
CRS-6023: Starting Oracle Cluster Ready Services-managed resources
CRS-6017: Processing resource auto-start for servers: rhel6m1
CRS-6016: Resource auto-start has completed for server rhel6m1
CRS-6024: Completed start of Oracle Cluster Ready Services-managed resources
CRS-4123: Oracle High Availability Services has been started.
Starting the Grid Infrastructure Management Repository database
Using PFILE='/opt/app/12.1.0/grid2/dbs/init-dropmgmtdb.ora'
Command output:
SQL*Plus: Release 12.1.0.2.0 Production on Wed Apr 1 12:12:12 2015 Copyright (c) 1982, 2014, Oracle. All rights reserved. Connected to an idle instance. SQL> ORACLE instance started. Total System Global Area 788529152 bytes Fixed Size 2929352 bytes Variable Size 314576184 bytes Database Buffers 465567744 bytes Redo Buffers 5455872 bytes Database mounted. SQL> Disconnected from Oracle Database 12c Enterprise Edition Release 12.1.0.2.0 - 64bit Production With the Partitioning, Automatic Storage Management and Advanced Analytics options
Starting the Grid Infrastructure Management Repository database succeeded
Creating new pfile for the Management Repository database

SQL*Plus: Release 12.1.0.2.0 Production on Wed Apr 1 12:12:25 2015

Copyright (c) 1982, 2014, Oracle.  All rights reserved.


Connected to:
Oracle Database 12c Enterprise Edition Release 12.1.0.2.0 - 64bit Production
With the Partitioning, Automatic Storage Management and Advanced Analytics options

SQL>
File created.

SQL> Disconnected from Oracle Database 12c Enterprise Edition Release 12.1.0.2.0 - 64bit Production
With the Partitioning, Automatic Storage Management and Advanced Analytics options
Succeeded in creating PFILE '/opt/app/12.1.0/grid2/dbs/init-dropmgmtdbSAVED.ora'
Dropping the Grid Infrastructure Management Repository database
Dropping the Grid Infrastructure Management Repository database succeeded
CRS-2791: Starting shutdown of Oracle High Availability Services-managed resources on 'rhel6m1'
CRS-2673: Attempting to stop 'ora.crsd' on 'rhel6m1'
CRS-2790: Starting shutdown of Cluster Ready Services-managed resources on 'rhel6m1'
CRS-2673: Attempting to stop 'ora.CLUSTER_DG.dg' on 'rhel6m1'
CRS-2677: Stop of 'ora.CLUSTER_DG.dg' on 'rhel6m1' succeeded
CRS-2673: Attempting to stop 'ora.DATA.dg' on 'rhel6m1'
CRS-2673: Attempting to stop 'ora.FLASH.dg' on 'rhel6m1'
CRS-2677: Stop of 'ora.DATA.dg' on 'rhel6m1' succeeded
CRS-2677: Stop of 'ora.FLASH.dg' on 'rhel6m1' succeeded
CRS-2673: Attempting to stop 'ora.asm' on 'rhel6m1'
CRS-2677: Stop of 'ora.asm' on 'rhel6m1' succeeded
CRS-2792: Shutdown of Cluster Ready Services-managed resources on 'rhel6m1' has completed
CRS-2677: Stop of 'ora.crsd' on 'rhel6m1' succeeded
CRS-2673: Attempting to stop 'ora.evmd' on 'rhel6m1'
CRS-2673: Attempting to stop 'ora.drivers.acfs' on 'rhel6m1'
CRS-2673: Attempting to stop 'ora.ctssd' on 'rhel6m1'
CRS-2673: Attempting to stop 'ora.storage' on 'rhel6m1'
CRS-2673: Attempting to stop 'ora.mdnsd' on 'rhel6m1'
CRS-2673: Attempting to stop 'ora.gpnpd' on 'rhel6m1'
CRS-2677: Stop of 'ora.drivers.acfs' on 'rhel6m1' succeeded
CRS-2677: Stop of 'ora.storage' on 'rhel6m1' succeeded
CRS-2673: Attempting to stop 'ora.asm' on 'rhel6m1'
CRS-2677: Stop of 'ora.evmd' on 'rhel6m1' succeeded
CRS-2677: Stop of 'ora.ctssd' on 'rhel6m1' succeeded
CRS-2677: Stop of 'ora.mdnsd' on 'rhel6m1' succeeded
CRS-2677: Stop of 'ora.gpnpd' on 'rhel6m1' succeeded
CRS-2677: Stop of 'ora.asm' on 'rhel6m1' succeeded
CRS-2673: Attempting to stop 'ora.cluster_interconnect.haip' on 'rhel6m1'
CRS-2677: Stop of 'ora.cluster_interconnect.haip' on 'rhel6m1' succeeded
CRS-2673: Attempting to stop 'ora.cssd' on 'rhel6m1'
CRS-2677: Stop of 'ora.cssd' on 'rhel6m1' succeeded
CRS-2673: Attempting to stop 'ora.crf' on 'rhel6m1'
CRS-2677: Stop of 'ora.crf' on 'rhel6m1' succeeded
CRS-2673: Attempting to stop 'ora.gipcd' on 'rhel6m1'
CRS-2677: Stop of 'ora.gipcd' on 'rhel6m1' succeeded
CRS-2793: Shutdown of Oracle High Availability Services-managed resources on 'rhel6m1' has completed
CRS-4133: Oracle High Availability Services has been stopped.
CRS-4123: Oracle High Availability Services has been started.
CRS-2672: Attempting to start 'ora.evmd' on 'rhel6m1'
CRS-2672: Attempting to start 'ora.mdnsd' on 'rhel6m1'
CRS-2676: Start of 'ora.mdnsd' on 'rhel6m1' succeeded
CRS-2676: Start of 'ora.evmd' on 'rhel6m1' succeeded
CRS-2672: Attempting to start 'ora.gpnpd' on 'rhel6m1'
CRS-2676: Start of 'ora.gpnpd' on 'rhel6m1' succeeded
CRS-2672: Attempting to start 'ora.cssdmonitor' on 'rhel6m1'
CRS-2672: Attempting to start 'ora.gipcd' on 'rhel6m1'
CRS-2676: Start of 'ora.cssdmonitor' on 'rhel6m1' succeeded
CRS-2676: Start of 'ora.gipcd' on 'rhel6m1' succeeded
CRS-2672: Attempting to start 'ora.cssd' on 'rhel6m1'
CRS-2672: Attempting to start 'ora.diskmon' on 'rhel6m1'
CRS-2676: Start of 'ora.diskmon' on 'rhel6m1' succeeded
CRS-2676: Start of 'ora.cssd' on 'rhel6m1' succeeded
CRS-2672: Attempting to start 'ora.crf' on 'rhel6m1'
CRS-2672: Attempting to start 'ora.ctssd' on 'rhel6m1'
CRS-2672: Attempting to start 'ora.cluster_interconnect.haip' on 'rhel6m1'
CRS-2676: Start of 'ora.crf' on 'rhel6m1' succeeded
CRS-2676: Start of 'ora.ctssd' on 'rhel6m1' succeeded
CRS-2676: Start of 'ora.cluster_interconnect.haip' on 'rhel6m1' succeeded
CRS-2672: Attempting to start 'ora.asm' on 'rhel6m1'
CRS-2676: Start of 'ora.asm' on 'rhel6m1' succeeded
CRS-2672: Attempting to start 'ora.storage' on 'rhel6m1'
CRS-2676: Start of 'ora.storage' on 'rhel6m1' succeeded
CRS-2672: Attempting to start 'ora.crsd' on 'rhel6m1'
CRS-2676: Start of 'ora.crsd' on 'rhel6m1' succeeded
CRS-2673: Attempting to stop 'ora.crsd' on 'rhel6m1'
CRS-2677: Stop of 'ora.crsd' on 'rhel6m1' succeeded
CRS-2791: Starting shutdown of Oracle High Availability Services-managed resources on 'rhel6m1'
CRS-2673: Attempting to stop 'ora.ctssd' on 'rhel6m1'
CRS-2673: Attempting to stop 'ora.evmd' on 'rhel6m1'
CRS-2673: Attempting to stop 'ora.storage' on 'rhel6m1'
CRS-2673: Attempting to stop 'ora.drivers.acfs' on 'rhel6m1'
CRS-2673: Attempting to stop 'ora.gpnpd' on 'rhel6m1'
CRS-2673: Attempting to stop 'ora.mdnsd' on 'rhel6m1'
CRS-2677: Stop of 'ora.storage' on 'rhel6m1' succeeded
CRS-2673: Attempting to stop 'ora.asm' on 'rhel6m1'
CRS-2677: Stop of 'ora.drivers.acfs' on 'rhel6m1' succeeded
CRS-2677: Stop of 'ora.ctssd' on 'rhel6m1' succeeded
CRS-2677: Stop of 'ora.evmd' on 'rhel6m1' succeeded
CRS-2677: Stop of 'ora.gpnpd' on 'rhel6m1' succeeded
CRS-2677: Stop of 'ora.mdnsd' on 'rhel6m1' succeeded
CRS-2677: Stop of 'ora.asm' on 'rhel6m1' succeeded
CRS-2673: Attempting to stop 'ora.cluster_interconnect.haip' on 'rhel6m1'
CRS-2677: Stop of 'ora.cluster_interconnect.haip' on 'rhel6m1' succeeded
CRS-2673: Attempting to stop 'ora.cssd' on 'rhel6m1'
CRS-2677: Stop of 'ora.cssd' on 'rhel6m1' succeeded
CRS-2673: Attempting to stop 'ora.crf' on 'rhel6m1'
CRS-2677: Stop of 'ora.crf' on 'rhel6m1' succeeded
CRS-2673: Attempting to stop 'ora.gipcd' on 'rhel6m1'
CRS-2677: Stop of 'ora.gipcd' on 'rhel6m1' succeeded
CRS-2793: Shutdown of Oracle High Availability Services-managed resources on 'rhel6m1' has completed
CRS-4133: Oracle High Availability Services has been stopped.
CRS-4123: Oracle High Availability Services has been started.
CRS-2672: Attempting to start 'ora.mdnsd' on 'rhel6m1'
CRS-2676: Start of 'ora.mdnsd' on 'rhel6m1' succeeded
CRS-2672: Attempting to start 'ora.gpnpd' on 'rhel6m1'
CRS-2676: Start of 'ora.gpnpd' on 'rhel6m1' succeeded
CRS-2672: Attempting to start 'ora.cssdmonitor' on 'rhel6m1'
CRS-2672: Attempting to start 'ora.gipcd' on 'rhel6m1'
CRS-2676: Start of 'ora.cssdmonitor' on 'rhel6m1' succeeded
CRS-2676: Start of 'ora.gipcd' on 'rhel6m1' succeeded
CRS-2672: Attempting to start 'ora.cssd' on 'rhel6m1'
CRS-2672: Attempting to start 'ora.diskmon' on 'rhel6m1'
CRS-2676: Start of 'ora.diskmon' on 'rhel6m1' succeeded
CRS-2676: Start of 'ora.cssd' on 'rhel6m1' succeeded
CRS-2672: Attempting to start 'ora.drivers.acfs' on 'rhel6m1'
CRS-2679: Attempting to clean 'ora.cluster_interconnect.haip' on 'rhel6m1'
CRS-2672: Attempting to start 'ora.ctssd' on 'rhel6m1'
CRS-2681: Clean of 'ora.cluster_interconnect.haip' on 'rhel6m1' succeeded
CRS-2672: Attempting to start 'ora.cluster_interconnect.haip' on 'rhel6m1'
CRS-2674: Start of 'ora.drivers.acfs' on 'rhel6m1' failed
CRS-2676: Start of 'ora.ctssd' on 'rhel6m1' succeeded
CRS-2676: Start of 'ora.cluster_interconnect.haip' on 'rhel6m1' succeeded
CRS-2672: Attempting to start 'ora.asm' on 'rhel6m1'
CRS-2676: Start of 'ora.asm' on 'rhel6m1' succeeded
2015/04/01 12:15:49 CLSRSC-338: Successfully downgraded OCR to version 11.2.0.4.0

CRS-2672: Attempting to start 'ora.crsd' on 'rhel6m1'
CRS-2676: Start of 'ora.crsd' on 'rhel6m1' succeeded
2015/04/01 12:20:01 CLSRSC-4001: Installing Oracle Trace File Analyzer (TFA) Collector.

2015/04/01 12:20:01 CLSRSC-4002: Successfully installed Oracle Trace File Analyzer (TFA) Collector.

Successfully downgraded Oracle Clusterware stack on this node
Run '/opt/app/11.2.0/grid4/bin/crsctl start crs' on all nodes to complete downgrade

Before starting the cluster with 11.2 update the inventory by setting crs=true for 11.2 GI home. At this point 12.1 GI home will have crs=true.

<HOME NAME="Ora11g_gridinfrahome2" LOC="/opt/app/11.2.0/grid4" TYPE="O" IDX="4">
   <NODE_LIST>
      <NODE NAME="rhel6m1"/>
      <NODE NAME="rhel6m2"/>
   </NODE_LIST>
</HOME>
<HOME NAME="OraGI12Home1" LOC="/opt/app/12.1.0/grid2" TYPE="O" IDX="5" CRS="true">
   <NODE_LIST>
      <NODE NAME="rhel6m1"/>
      <NODE NAME="rhel6m2"/>
   </NODE_LIST>
</HOME>

Run the following to update the inventory

cd /opt/app/12.1.0/grid2/oui/bin/
[grid@rhel6m1 bin]$ ./runInstaller -nowait -waitforcompletion -ignoreSysPrereqs -updateNodeList -silent CRS=false ORACLE_HOME=/opt/app/12.1.0/grid2
Starting Oracle Universal Installer...
Checking swap space: must be greater than 500 MB.   Actual 4095 MB    Passed
The inventory pointer is located at /etc/oraInst.loc
'UpdateNodeList' was successful.

[grid@rhel6m1 bin]$ ./runInstaller -nowait -waitforcompletion -ignoreSysPrereqs -updateNodeList -silent CRS=true ORACLE_HOME=/opt/app/11.2.0/grid4
Starting Oracle Universal Installer...
Checking swap space: must be greater than 500 MB.   Actual 4095 MB    Passed
The inventory pointer is located at /etc/oraInst.loc
'UpdateNodeList' was successful.

Verify the inventory update

<HOME NAME="Ora11g_gridinfrahome2" LOC="/opt/app/11.2.0/grid4" TYPE="O" IDX="4" CRS="true">
   <NODE_LIST>
      <NODE NAME="rhel6m1"/>
      <NODE NAME="rhel6m2"/>
   </NODE_LIST>
</HOME>
<HOME NAME="OraGI12Home1" LOC="/opt/app/12.1.0/grid2" TYPE="O" IDX="5">
   <NODE_LIST>
      <NODE NAME="rhel6m1"/>
      <NODE NAME="rhel6m2"/>
   </NODE_LIST>
</HOME>

Also make sure the contents in /etc/init.d/ohasd and /etc/init.d/init.ohasd refers the 11.2 as the ORA_CRS_HOME and there are no references to 12.1. Few times the last downgrade node (rhel6m1) did contain references to 12.1 even though downgrade command had successfully completed. However these files on the other node had the correct references to 11.2.

[root@rhel6m1 grid]#  cat /etc/init.d/ohasd | grep ORA_CRS_HOME
ORA_CRS_HOME=/opt/app/11.2.0/grid4
export ORA_CRS_HOME

[root@rhel6m1 grid]#  cat /etc/init.d/init.ohasd | grep ORA_CRS_HOME
ORA_CRS_HOME=/opt/app/11.2.0/grid4
export ORA_CRS_HOME
PERL="/opt/app/11.2.0/grid4/perl/bin/perl -I${ORA_CRS_HOME}/perl/lib"

Also as part of the downgrade /etc/oratab is modified. All references to MGMTDB are moved and ASM instances homes are set to 11.2. But this only happens on last node (rhel6m1). For other nodes /etc/oratab did not have any ASM related entry. Manually add this ASM related entry to /etc/oratab

+ASM2:/opt/app/11.2.0/grid4:N

Before starting the crs unset any references to 12.1 binaries from the environment (PATH, ORACLE_HOME variables). Make sure crsctl use to start the crs comes from 11.2 GI home.

[grid@rhel6m1 bin]$ . oraenv
ORACLE_SID = [+ASM1] ? +ASM1
The Oracle base remains unchanged with value /opt/app/oracle
[grid@rhel6m1 bin]$ which crsctl
/opt/app/11.2.0/grid4/bin/crsctl

Start the crs on all nodes.

# crsctl start crs

crsctl query crs activeversion -f
Oracle Clusterware active version on the cluster is [11.2.0.4.0]. The cluster upgrade state is [NORMAL].

crsctl query crs softwareversion
Oracle Clusterware version on node [rhel6m1] is [11.2.0.4.0]

[root@rhel6m1 grid]# ocrcheck
Status of Oracle Cluster Registry is as follows :
         Version                  :          3
         Total space (kbytes)     :     262120
         Used space (kbytes)      :       3356
         Available space (kbytes) :     258764
         ID                       : 2072206343
         Device/File Name         : +CLUSTER_DG

This concludes the steps to successfully downgrading GI from 12.1.0.2 to 11.2.0.4.

There could be occasions the downgrade command successfully completes but start of crs fails. Symptoms in this case include unable to discover any voting disks crsctl query css votedisk doesn't return any vote disk infomration and ocssd.log will have entries similar to

2015-03-31 14:00:09.618: [    CSSD][898090752]clssnmvDiskVerify: Successful discovery of 0 disks
2015-03-31 14:00:09.618: [    CSSD][898090752]clssnmCompleteInitVFDiscovery: Completing initial voting file discovery
2015-03-31 14:00:09.618: [    CSSD][898090752]clssnmvFindInitialConfigs: No voting files found

Other times includes corrupted OCR with ocssd.log having entries similar to

2015-03-26 11:33:05.633: [ CRSMAIN][3817551648] Initialing cluclu context...
[  OCRMAS][3776734976]th_calc_av:8': Failed in vsnupr. Incorrect SV stored in OCR. Key [SYSTEM.version.hostnames.] Value []
2015-03-26 11:33:06.618: [  OCRSRV][3776734976]th_upgrade:9 Shutdown CacheMaster. prev AV [186647552] new calc av [186647552] my sv [186647552]

No root cause was found of these cases and could only assume this may be due to some of the earlier mentioned reasons such as having backups of OCR from previous upgrades i.e. ocr11.2.0.3.0 (But it must be said a successful downgrade was archived while ocr11.2.0.3.0 was in the cdata directory), wrong binaries referred during downgrade due to environment variable settings. The only option to recover from such a situation is to restore the OCR taken while cluster was 11.2.

crsctl stop crs -f # run on all nodes
crsctl start crs -excl -nocrs # run only on one node
ocrconfig -restore /opt/app/11.2.0/grid4/cdata/rhel6m-cluster/backup_20150331_170634.ocr
crsctl replace votedisk +cluster_dg

Useful metalink notes
rootcrs.pl -downgrade -lastnode" Fails if Current Directory Can not be Accessed by Grid User [ID 1917922.1]
SRVCTL commands fails with error CLSU-00100: Operating System function: sclsdgcwd failed with error data: -1 [ID 1961344.1]
Srvctl Status Commands Report CLSU-00100, CLSU-00101 and CLSU-00103 Errors [ID 1911258.1]

Related Posts
Downgrade Grid Infrastructure from 11.2.0.4 to 11.2.0.3
Downgrade Grid Infrastructure from 11.2.0.4 to 11.1.0.7