Thursday, September 14, 2017

Upgrading RAC from 11.2.0.4 to 12.2.0.1 - Grid Infrastructure

This post gives the highlights of upgrading grid infrastructure of a two node RAC from 11.2.0.4 to 12.2.0.1. There's an earlier post of upgrading GI from 11.2.0.4 to 12.1.0.2.
When upgrading to Oracle Grid Infrastructure 12c Release 2 (12.2), the upgrade is to an Oracle Standalone Cluster configuration with Flex cluster option. As mentioned in 12.2 RAC install, GNS no longer necessary to have flex cluster. If OCR and vote disks are stored in non-ASM then these need to be migrated to ASM before the upgrade to 12.2.
1. Before the GI upgrade backup the OCR manually. This could be used in downgrading the GI from 12.2.0.1 to 11.2.0.4 later on.
2. The RAC has been setup with role separation. This requires certain directories having permission so that both grid user and oracle user is able to write and read to them. Make cfgtoollogs writable to both grid and oracle, usually this is writable only for one user and group permission is only read and executable. Without write permission on this directory following error could be seen when upgrading GI.
WARNING:  [Apr 12, 2017 3:20:13 PM] Skipping line: [FATAL] [DBT-00007] User does not have the appropiate write privileges.
INFO:  [Apr 12, 2017 3:20:13 PM] End of argument passing to stdin
INFO:  [Apr 12, 2017 3:20:13 PM] Read:    CAUSE: User does not have the required write privileges on "/opt/app/oracle/cfgtoollogs/dbca" .
INFO:  [Apr 12, 2017 3:20:13 PM]    CAUSE: User does not have the required write privileges on "/opt/app/oracle/cfgtoollogs/dbca" .
WARNING:  [Apr 12, 2017 3:20:13 PM] Skipping line:    CAUSE: User does not have the required write privileges on "/opt/app/oracle/cfgtoollogs/dbca" .

chmod 775 $ORACLE_BASE/cfgtoollogs
Another directory requires write permission for grid user is the dbca inside cfgtoollogs.
chmod 770 $ORACLE_BASE/cfgtoollogs/dbca
As the GI management repository is a database the admin directory is another directory that requires write permission for grid user. Without this permission creation of GI management repository will fail with following
INFO:  [Apr 12, 2017 3:27:01 PM] Read: [FATAL] [DBT-06608] The specified audit file destination (/opt/app/oracle/admin/_mgmtdb/adump) is not writable.
INFO:  [Apr 12, 2017 3:27:01 PM] [FATAL] [DBT-06608] The specified audit file destination (/opt/app/oracle/admin/_mgmtdb/adump) is not writable.
WARNING:  [Apr 12, 2017 3:27:01 PM] Skipping line: [FATAL] [DBT-06608] The specified audit file destination (/opt/app/oracle/admin/_mgmtdb/adump) is not writable.

chmod 770 $ORACLE_BASE/admin
These permissions must be set on all the nodes to avoid above error when MGMTDB migrates from node to node during rolling restarts.
3. Once upgraded the GI would be a standalone flex cluster. But this doesn't require setting up GNS (confirmed by an Oracle SR) before the upgrade. It was similar for 12.2 standalone cluster new installation. As with other 12c upgrades the OCR and vote disks must be moved to ASM before the upgrade. Also 12.2 is supported on RHEL 6.4 or above. So if the OS is a RHEL 5 then OS upgrade would be required.

4. The current versions of the GI are
[grid@rhel6m1 ~]$  crsctl query crs activeversion
Oracle Clusterware active version on the cluster is [11.2.0.4.0]
[grid@rhel6m1 ~]$ crsctl query crs releaseversion
Oracle High Availability Services release version on the local node is [11.2.0.4.0]
[grid@rhel6m1 ~]$  crsctl query crs softwareversion -all
Oracle Clusterware version on node [rhel6m1] is [11.2.0.4.0]
Oracle Clusterware version on node [rhel6m2] is [11.2.0.4.0]
5. Check upgrade readiness by running orachk
./orachk -u -o pre
The GI home must be patched to at least PSU 11.2.0.4.161018 before the upgrade. This will be checked during the upgrade readiness. The GI home had PSU later than this applied on the GI home. In-spite of the this following patches were flagged as needed.
 WARNING => One or more dependent start or stop resources are missing from the OCR.
 WARNING => system disk group does not have enough free space to create management database
 WARNING => Oracle patch 19855835 is not applied on RDBMS_HOME /opt/app/oracle/product/11.2.0/dbhome_4
 WARNING => Oracle patch 20898997 is not applied on RDBMS_HOME /opt/app/oracle/product/11.2.0/dbhome_4
 WARNING => Oracle patch 20348910 is not applied on RDBMS_HOME /opt/app/oracle/product/11.2.0/dbhome_4
 WARNING => Oracle Patch 23186035 is not applied on RDBMS home /opt/app/oracle/product/11.2.0/dbhome_4
The dependency warning is due to GNS dependency on the database resouce, even though this GI setup didn't have any GNS resource.
ora.scan2.vip = FOUND
ora.ons = FOUND
ora.scan3.vip = FOUND
ora.net1.network = FOUND
ora.gns = MISSING
ora.DATA.dg = FOUND
ora.FLASH.dg = FOUND
ora.scan1.vip = FOUND

START_DEPENDENCIES=hard(ora.DATA.dg,ora.FLASH.dg) weak(type:ora.listener.type,global:type:ora.scan_listener.type,uniform:ora.ons,
global:ora.gns) pullup(ora.DATA.dg,ora.FLASH.dg)
This warning could be resolved by removing the GNS dependency
$ srvctl stop database -d std11g2
# crsctl modify resource ora.std11g2.db -attr "START_DEPENDENCIES='hard(ora.DATA.dg,ora.FLASH.dg) weak(type:ora.listener.type,global:type:ora.scan_listener.type,uniform:ora.ons) pullup(ora.DATA.dg,ora.FLASH.dg)'"

START_DEPENDENCIES=hard(ora.DATA.dg,ora.FLASH.dg) weak(type:ora.listener.type,global:type:ora.scan_listener.type,uniform:ora.ons) pullup(ora.DATA.dg,ora.FLASH.dg)
$ srvctl start database -d std11g2
Download the flagged patches from MOS and apply. Following is a short description of what each patch does
 23186035 - This patch will ensure RDBMS Homes of lower version work with 12.2 GI Home.
 20348910 - Will address ORA-22308 during upgrade / downgrade. - Patches to apply before upgrading Oracle GI and DB to 12.2.0.1 (Doc ID 2180188.1) 
 20898997 - Will address ORA-22308 / ORA-600 [8153] or similar during upgrade / downgrade.
 19855835 - This is one of the top popular issues reported in 12.1.0.2. Applying this patch proactively would help avoiding issues related to slow upgrade if there is a large amount of data in the stats history tables.
The 11.2.0.4 GI setup had a 15G disk group (with 14G free) with normal redundancy for OCR and vote disks.
NAME                           TYPE   V   TOTAL_MB    FREE_MB
------------------------------ ------ - ---------- ----------
CLUSTER_DG                     NORMAL Y      15342      14516
DATA                           EXTERN N      10236       5417
FLASH                          EXTERN N      10236       2547
This amount of free space is not sufficient for a 12.2 upgrade. The readiness report will give a warning as well and mentions
System diskgroup CLUSTER_DG has 14 GB of free space and it requires 35 of free space. - There is insufficient space available in the OCR/voting disk diskgroup  to house the MGMTDB. Should you choose to create the MGMTDB during the upgrade process the creation will fail due to space constraints.  
Please add space to the diskgroup.  The Space requirement for the MGMTDB is 35 GB for clusters up to 4 nodes, for each node above 4 add 500MB/node.
However setting a 39GB disk group with normal redundancy would pass the orachk readiness test but actual GI upgrade would fail.
NAME                             TOTAL_MB    FREE_MB
------------------------------ ---------- ----------
CLUSTER_DG                          15342      14516
DATA                                10236       5387
FLASH                               10236       1503
CLUSTER_DG2                         39933      39648
Following images shows error message that would be given during actual GI upgrade for two situations explained above.
As shown on the image the expected size for the disk group to host both OCR and MGMTDB is 66GB. However as per GI install and upgrade document for a normal redundancy disk group the space requirement is 78GB (for standalone cluster). For external redundancy disk group the space requirement is 39GB. Two options are available in this case, if normal redundancy is preferred for both OCR and MGMTDB then create a disk group of at least 78GB. As part of the upgrade testing a 92GB normal redundancy disk group was created and upgraded succeed without issue.
NAME                             TOTAL_MB    FREE_MB
------------------------------ ---------- ----------
CLUSTER_DG                          15342      14612
DATA                                10236       5387
FLASH                               10236       1489
CLUSTER_DG3                         92154      91230
However if there's space constraint then create a external redundancy disk group of 40GB.
NAME                             TOTAL_MB    FREE_MB
------------------------------ ---------- ----------
CLUSTER_DG                          15342      14711
DATA                                10236       5163
FLASH                               10236       1776
GIMR                                40952       40902
Move the OCR and vote disk to it before the upgrade (for the duration of the upgrade OCR and Vote will be in external redundancy disk group). Once the upgrade complete move the OCR and vote disk back to normal redundancy disk group, leaving only the MGMTDB in the external redundancy disk group. It's valid to have these two in different disk group. In fact for a 12.2 fresh installations OUI presents the option to have a different disk group for MGMTDB.

6. Once GI upgrade readiness tasks are complete create out-of-place directories for GI home. Unzip the 12.2 GI binaries to the GI home. This only needs to be done in one of the nodes. During the upgrade process the software is copied and installed on all other nodes in the cluster.
$ unset ORACLE_BASE
$ unset ORACLE_HOME
$ unset ORACLE_SID

mkdir -p /opt/app/12.2.0/grid
cp ~/linuxx64_12201_grid_home.zip /opt/app/12.2.0/grid
cd /opt/app/12.2.0/grid
unzip linuxx64_12201_grid_home.zip 




7. Run the cluvfy from the new GI home.
[grid@rhel6m1 grid]$ ./runcluvfy.sh stage -pre crsinst -upgrade -rolling -src_crshome /opt/app/11.2.0/grid4 -dest_crshome /opt/app/12.2.0/grid -dest_version 12.2.0.1.0 -fixup -verbose

Verifying Physical Memory ...
  Node Name     Available                 Required                  Status
  ------------  ------------------------  ------------------------  ----------
  rhel6m2       7.6863GB (8059712.0KB)    8GB (8388608.0KB)         passed
  rhel6m1       7.6863GB (8059712.0KB)    8GB (8388608.0KB)         passed
Verifying Physical Memory ...PASSED
Verifying Available Physical Memory ...
  Node Name     Available                 Required                  Status
  ------------  ------------------------  ------------------------  ----------
  rhel6m2       6.3133GB (6620016.0KB)    50MB (51200.0KB)          passed
  rhel6m1       6.0682GB (6362944.0KB)    50MB (51200.0KB)          passed
Verifying Available Physical Memory ...PASSED
Verifying Swap Size ...
  Node Name     Available                 Required                  Status
  ------------  ------------------------  ------------------------  ----------
  rhel6m2       4GB (4194296.0KB)         7.6863GB (8059712.0KB)    failed
  rhel6m1       4GB (4194296.0KB)         7.6863GB (8059712.0KB)    failed
Verifying Swap Size ...FAILED (PRVF-7573)
Verifying Free Space: rhel6m2:/usr,rhel6m2:/var,rhel6m2:/etc,rhel6m2:/sbin,rhel6m2:/tmp ...
  Path              Node Name     Mount point   Available     Required      Status
  ----------------  ------------  ------------  ------------  ------------  ------------
  /usr              rhel6m2       /             15.415GB      25MB          passed
  /var              rhel6m2       /             15.415GB      5MB           passed
  /etc              rhel6m2       /             15.415GB      25MB          passed
  /sbin             rhel6m2       /             15.415GB      10MB          passed
  /tmp              rhel6m2       /             15.415GB      1GB           passed
Verifying Free Space: rhel6m2:/usr,rhel6m2:/var,rhel6m2:/etc,rhel6m2:/sbin,rhel6m2:/tmp ...PASSED
Verifying Free Space: rhel6m1:/usr,rhel6m1:/var,rhel6m1:/etc,rhel6m1:/sbin,rhel6m1:/tmp ...
  Path              Node Name     Mount point   Available     Required      Status
  ----------------  ------------  ------------  ------------  ------------  ------------
  /usr              rhel6m1       /             5.6611GB      25MB          passed
  /var              rhel6m1       /             5.6611GB      5MB           passed
  /etc              rhel6m1       /             5.6611GB      25MB          passed
  /sbin             rhel6m1       /             5.6611GB      10MB          passed
  /tmp              rhel6m1       /             5.6611GB      1GB           passed
Verifying Free Space: rhel6m1:/usr,rhel6m1:/var,rhel6m1:/etc,rhel6m1:/sbin,rhel6m1:/tmp ...PASSED
Verifying User Existence: grid ...
  Node Name     Status                    Comment
  ------------  ------------------------  ------------------------
  rhel6m2       passed                    exists(502)
  rhel6m1       passed                    exists(502)

  Verifying Users With Same UID: 502 ...PASSED
Verifying User Existence: grid ...PASSED
Verifying Group Existence: asmadmin ...
  Node Name     Status                    Comment
  ------------  ------------------------  ------------------------
  rhel6m2       passed                    exists
  rhel6m1       passed                    exists
Verifying Group Existence: asmadmin ...PASSED
Verifying Group Existence: asmoper ...
  Node Name     Status                    Comment
  ------------  ------------------------  ------------------------
  rhel6m2       passed                    exists
  rhel6m1       passed                    exists
Verifying Group Existence: asmoper ...PASSED
Verifying Group Existence: asmdba ...
  Node Name     Status                    Comment
  ------------  ------------------------  ------------------------
  rhel6m2       passed                    exists
  rhel6m1       passed                    exists
Verifying Group Existence: asmdba ...PASSED
Verifying Group Existence: oinstall ...
  Node Name     Status                    Comment
  ------------  ------------------------  ------------------------
  rhel6m2       passed                    exists
  rhel6m1       passed                    exists
Verifying Group Existence: oinstall ...PASSED
Verifying Group Membership: asmdba ...
  Node Name         User Exists   Group Exists  User in Group  Status
  ----------------  ------------  ------------  ------------  ----------------
  rhel6m2           yes           yes           yes           passed
  rhel6m1           yes           yes           yes           passed
Verifying Group Membership: asmdba ...PASSED
Verifying Group Membership: asmadmin ...
  Node Name         User Exists   Group Exists  User in Group  Status
  ----------------  ------------  ------------  ------------  ----------------
  rhel6m2           yes           yes           yes           passed
  rhel6m1           yes           yes           yes           passed
Verifying Group Membership: asmadmin ...PASSED
Verifying Group Membership: oinstall(Primary) ...
  Node Name         User Exists   Group Exists  User in Group  Primary       Status
  ----------------  ------------  ------------  ------------  ------------  ------------
  rhel6m2           yes           yes           yes           yes           passed
  rhel6m1           yes           yes           yes           yes           passed
Verifying Group Membership: oinstall(Primary) ...PASSED
Verifying Group Membership: asmoper ...
  Node Name         User Exists   Group Exists  User in Group  Status
  ----------------  ------------  ------------  ------------  ----------------
  rhel6m2           yes           yes           yes           passed
  rhel6m1           yes           yes           yes           passed
Verifying Group Membership: asmoper ...PASSED
Verifying Run Level ...
  Node Name     run level                 Required                  Status
  ------------  ------------------------  ------------------------  ----------
  rhel6m2       3                         3,5                       passed
  rhel6m1       3                         3,5                       passed
Verifying Run Level ...PASSED
Verifying Hard Limit: maximum open file descriptors ...
  Node Name         Type          Available     Required      Status
  ----------------  ------------  ------------  ------------  ----------------
  rhel6m2           hard          65536         65536         passed
  rhel6m1           hard          65536         65536         passed
Verifying Hard Limit: maximum open file descriptors ...PASSED
Verifying Soft Limit: maximum open file descriptors ...
  Node Name         Type          Available     Required      Status
  ----------------  ------------  ------------  ------------  ----------------
  rhel6m2           soft          1024          1024          passed
  rhel6m1           soft          1024          1024          passed
Verifying Soft Limit: maximum open file descriptors ...PASSED
Verifying Hard Limit: maximum user processes ...
  Node Name         Type          Available     Required      Status
  ----------------  ------------  ------------  ------------  ----------------
  rhel6m2           hard          16384         16384         passed
  rhel6m1           hard          16384         16384         passed
Verifying Hard Limit: maximum user processes ...PASSED
Verifying Soft Limit: maximum user processes ...
  Node Name         Type          Available     Required      Status
  ----------------  ------------  ------------  ------------  ----------------
  rhel6m2           soft          2047          2047          passed
  rhel6m1           soft          2047          2047          passed
Verifying Soft Limit: maximum user processes ...PASSED
Verifying Soft Limit: maximum stack size ...
  Node Name         Type          Available     Required      Status
  ----------------  ------------  ------------  ------------  ----------------
  rhel6m2           soft          10240         10240         passed
  rhel6m1           soft          10240         10240         passed
Verifying Soft Limit: maximum stack size ...PASSED
Verifying Oracle patch:17617807 ...
  Node Name     Applied                   Required                  Comment
  ------------  ------------------------  ------------------------  ----------
  rhel6m1       17617807                  17617807                  passed
  rhel6m2       17617807                  17617807                  passed
Verifying Oracle patch:17617807 ...PASSED
Verifying Oracle patch:21255373 ...
  Node Name     Applied                   Required                  Comment
  ------------  ------------------------  ------------------------  ----------
  rhel6m1       21255373                  21255373                  passed
  rhel6m2       21255373                  21255373                  passed
Verifying Oracle patch:21255373 ...PASSED
Verifying This test checks that the source home "/opt/app/11.2.0/grid4" is suitable for upgrading to version "12.2.0.1.0". ...PASSED
Verifying Architecture ...
  Node Name     Available                 Required                  Status
  ------------  ------------------------  ------------------------  ----------
  rhel6m2       x86_64                    x86_64                    passed
  rhel6m1       x86_64                    x86_64                    passed
Verifying Architecture ...PASSED
Verifying OS Kernel Version ...
  Node Name     Available                 Required                  Status
  ------------  ------------------------  ------------------------  ----------
  rhel6m2       2.6.32-358.el6.x86_64     2.6.32                    passed
  rhel6m1       2.6.32-358.el6.x86_64     2.6.32                    passed
Verifying OS Kernel Version ...PASSED
Verifying OS Kernel Parameter: semmsl ...
  Node Name         Current       Configured    Required      Status        Comment
  ----------------  ------------  ------------  ------------  ------------  ------------
  rhel6m1           3010          3010          250           passed
  rhel6m2           3010          3010          250           passed
Verifying OS Kernel Parameter: semmsl ...PASSED
Verifying OS Kernel Parameter: semmns ...
  Node Name         Current       Configured    Required      Status        Comment
  ----------------  ------------  ------------  ------------  ------------  ------------
  rhel6m1           385280        385280        32000         passed
  rhel6m2           385280        385280        32000         passed
Verifying OS Kernel Parameter: semmns ...PASSED
Verifying OS Kernel Parameter: semopm ...
  Node Name         Current       Configured    Required      Status        Comment
  ----------------  ------------  ------------  ------------  ------------  ------------
  rhel6m1           3010          3010          100           passed
  rhel6m2           3010          3010          100           passed
Verifying OS Kernel Parameter: semopm ...PASSED
Verifying OS Kernel Parameter: semmni ...
  Node Name         Current       Configured    Required      Status        Comment
  ----------------  ------------  ------------  ------------  ------------  ------------
  rhel6m1           128           128           128           passed
  rhel6m2           128           128           128           passed
Verifying OS Kernel Parameter: semmni ...PASSED
Verifying OS Kernel Parameter: shmmax ...
  Node Name         Current       Configured    Required      Status        Comment
  ----------------  ------------  ------------  ------------  ------------  ------------
  rhel6m1           68719476736   68719476736   4126572544    passed
  rhel6m2           68719476736   68719476736   4126572544    passed
Verifying OS Kernel Parameter: shmmax ...PASSED
Verifying OS Kernel Parameter: shmmni ...
  Node Name         Current       Configured    Required      Status        Comment
  ----------------  ------------  ------------  ------------  ------------  ------------
  rhel6m1           4096          4096          4096          passed
  rhel6m2           4096          4096          4096          passed
Verifying OS Kernel Parameter: shmmni ...PASSED
Verifying OS Kernel Parameter: shmall ...
  Node Name         Current       Configured    Required      Status        Comment
  ----------------  ------------  ------------  ------------  ------------  ------------
  rhel6m1           4294967296    4294967296    805971        passed
  rhel6m2           4294967296    4294967296    805971        passed
Verifying OS Kernel Parameter: shmall ...PASSED
Verifying OS Kernel Parameter: file-max ...
  Node Name         Current       Configured    Required      Status        Comment
  ----------------  ------------  ------------  ------------  ------------  ------------
  rhel6m1           6815744       6815744       6815744       passed
  rhel6m2           6815744       6815744       6815744       passed
Verifying OS Kernel Parameter: file-max ...PASSED
Verifying OS Kernel Parameter: ip_local_port_range ...
  Node Name         Current       Configured    Required      Status        Comment
  ----------------  ------------  ------------  ------------  ------------  ------------
  rhel6m1           between 9000 & 65500  between 9000 & 65500  between 9000 & 65535  passed
  rhel6m2           between 9000 & 65500  between 9000 & 65500  between 9000 & 65535  passed
Verifying OS Kernel Parameter: ip_local_port_range ...PASSED
Verifying OS Kernel Parameter: rmem_default ...
  Node Name         Current       Configured    Required      Status        Comment
  ----------------  ------------  ------------  ------------  ------------  ------------
  rhel6m1           4194304       4194304       262144        passed
  rhel6m2           4194304       4194304       262144        passed
Verifying OS Kernel Parameter: rmem_default ...PASSED
Verifying OS Kernel Parameter: rmem_max ...
  Node Name         Current       Configured    Required      Status        Comment
  ----------------  ------------  ------------  ------------  ------------  ------------
  rhel6m1           4194304       4194304       4194304       passed
  rhel6m2           4194304       4194304       4194304       passed
Verifying OS Kernel Parameter: rmem_max ...PASSED
Verifying OS Kernel Parameter: wmem_default ...
  Node Name         Current       Configured    Required      Status        Comment
  ----------------  ------------  ------------  ------------  ------------  ------------
  rhel6m1           1048576       1048576       262144        passed
  rhel6m2           1048576       1048576       262144        passed
Verifying OS Kernel Parameter: wmem_default ...PASSED
Verifying OS Kernel Parameter: wmem_max ...
  Node Name         Current       Configured    Required      Status        Comment
  ----------------  ------------  ------------  ------------  ------------  ------------
  rhel6m1           2097152       2097152       1048576       passed
  rhel6m2           2097152       2097152       1048576       passed
Verifying OS Kernel Parameter: wmem_max ...PASSED
Verifying OS Kernel Parameter: aio-max-nr ...
  Node Name         Current       Configured    Required      Status        Comment
  ----------------  ------------  ------------  ------------  ------------  ------------
  rhel6m1           3145728       3145728       1048576       passed
  rhel6m2           3145728       3145728       1048576       passed
Verifying OS Kernel Parameter: aio-max-nr ...PASSED
Verifying OS Kernel Parameter: panic_on_oops ...
  Node Name         Current       Configured    Required      Status        Comment
  ----------------  ------------  ------------  ------------  ------------  ------------
  rhel6m1           1             undefined     1             passed
  rhel6m2           1             undefined     1             passed
Verifying OS Kernel Parameter: panic_on_oops ...PASSED
Verifying Package: binutils-2.20.51.0.2 ...
  Node Name     Available                 Required                  Status
  ------------  ------------------------  ------------------------  ----------
  rhel6m2       binutils-2.20.51.0.2-5.36.el6  binutils-2.20.51.0.2      passed
  rhel6m1       binutils-2.20.51.0.2-5.36.el6  binutils-2.20.51.0.2      passed
Verifying Package: binutils-2.20.51.0.2 ...PASSED
Verifying Package: compat-libcap1-1.10 ...
  Node Name     Available                 Required                  Status
  ------------  ------------------------  ------------------------  ----------
  rhel6m2       compat-libcap1-1.10-1     compat-libcap1-1.10       passed
  rhel6m1       compat-libcap1-1.10-1     compat-libcap1-1.10       passed
Verifying Package: compat-libcap1-1.10 ...PASSED
Verifying Package: compat-libstdc++-33-3.2.3 (x86_64) ...
  Node Name     Available                 Required                  Status
  ------------  ------------------------  ------------------------  ----------
  rhel6m2       compat-libstdc++-33(x86_64)-3.2.3-69.el6  compat-libstdc++-33(x86_64)-3.2.3  passed
  rhel6m1       compat-libstdc++-33(x86_64)-3.2.3-69.el6  compat-libstdc++-33(x86_64)-3.2.3  passed
Verifying Package: compat-libstdc++-33-3.2.3 (x86_64) ...PASSED
Verifying Package: libgcc-4.4.7 (x86_64) ...
  Node Name     Available                 Required                  Status
  ------------  ------------------------  ------------------------  ----------
  rhel6m2       libgcc(x86_64)-4.4.7-3.el6  libgcc(x86_64)-4.4.7      passed
  rhel6m1       libgcc(x86_64)-4.4.7-3.el6  libgcc(x86_64)-4.4.7      passed
Verifying Package: libgcc-4.4.7 (x86_64) ...PASSED
Verifying Package: libstdc++-4.4.7 (x86_64) ...
  Node Name     Available                 Required                  Status
  ------------  ------------------------  ------------------------  ----------
  rhel6m2       libstdc++(x86_64)-4.4.7-3.el6  libstdc++(x86_64)-4.4.7   passed
  rhel6m1       libstdc++(x86_64)-4.4.7-3.el6  libstdc++(x86_64)-4.4.7   passed
Verifying Package: libstdc++-4.4.7 (x86_64) ...PASSED
Verifying Package: libstdc++-devel-4.4.7 (x86_64) ...
  Node Name     Available                 Required                  Status
  ------------  ------------------------  ------------------------  ----------
  rhel6m2       libstdc++-devel(x86_64)-4.4.7-3.el6  libstdc++-devel(x86_64)-4.4.7  passed
  rhel6m1       libstdc++-devel(x86_64)-4.4.7-3.el6  libstdc++-devel(x86_64)-4.4.7  passed
Verifying Package: libstdc++-devel-4.4.7 (x86_64) ...PASSED
Verifying Package: sysstat-9.0.4 ...
  Node Name     Available                 Required                  Status
  ------------  ------------------------  ------------------------  ----------
  rhel6m2       sysstat-9.0.4-20.el6      sysstat-9.0.4             passed
  rhel6m1       sysstat-9.0.4-20.el6      sysstat-9.0.4             passed
Verifying Package: sysstat-9.0.4 ...PASSED
Verifying Package: gcc-4.4.7 ...
  Node Name     Available                 Required                  Status
  ------------  ------------------------  ------------------------  ----------
  rhel6m2       gcc-4.4.7-3.el6           gcc-4.4.7                 passed
  rhel6m1       gcc-4.4.7-3.el6           gcc-4.4.7                 passed
Verifying Package: gcc-4.4.7 ...PASSED
Verifying Package: gcc-c++-4.4.7 ...
  Node Name     Available                 Required                  Status
  ------------  ------------------------  ------------------------  ----------
  rhel6m2       gcc-c++-4.4.7-3.el6       gcc-c++-4.4.7             passed
  rhel6m1       gcc-c++-4.4.7-3.el6       gcc-c++-4.4.7             passed
Verifying Package: gcc-c++-4.4.7 ...PASSED
Verifying Package: ksh ...
  Node Name     Available                 Required                  Status
  ------------  ------------------------  ------------------------  ----------
  rhel6m2       ksh                       ksh                       passed
  rhel6m1       ksh                       ksh                       passed
Verifying Package: ksh ...PASSED
Verifying Package: make-3.81 ...
  Node Name     Available                 Required                  Status
  ------------  ------------------------  ------------------------  ----------
  rhel6m2       make-3.81-20.el6          make-3.81                 passed
  rhel6m1       make-3.81-20.el6          make-3.81                 passed
Verifying Package: make-3.81 ...PASSED
Verifying Package: glibc-2.12 (x86_64) ...
  Node Name     Available                 Required                  Status
  ------------  ------------------------  ------------------------  ----------
  rhel6m2       glibc(x86_64)-2.12-1.107.el6  glibc(x86_64)-2.12        passed
  rhel6m1       glibc(x86_64)-2.12-1.107.el6  glibc(x86_64)-2.12        passed
Verifying Package: glibc-2.12 (x86_64) ...PASSED
Verifying Package: glibc-devel-2.12 (x86_64) ...
  Node Name     Available                 Required                  Status
  ------------  ------------------------  ------------------------  ----------
  rhel6m2       glibc-devel(x86_64)-2.12-1.107.el6  glibc-devel(x86_64)-2.12  passed
  rhel6m1       glibc-devel(x86_64)-2.12-1.107.el6  glibc-devel(x86_64)-2.12  passed
Verifying Package: glibc-devel-2.12 (x86_64) ...PASSED
Verifying Package: libaio-0.3.107 (x86_64) ...
  Node Name     Available                 Required                  Status
  ------------  ------------------------  ------------------------  ----------
  rhel6m2       libaio(x86_64)-0.3.107-10.el6  libaio(x86_64)-0.3.107    passed
  rhel6m1       libaio(x86_64)-0.3.107-10.el6  libaio(x86_64)-0.3.107    passed
Verifying Package: libaio-0.3.107 (x86_64) ...PASSED
Verifying Package: libaio-devel-0.3.107 (x86_64) ...
  Node Name     Available                 Required                  Status
  ------------  ------------------------  ------------------------  ----------
  rhel6m2       libaio-devel(x86_64)-0.3.107-10.el6  libaio-devel(x86_64)-0.3.107  passed
  rhel6m1       libaio-devel(x86_64)-0.3.107-10.el6  libaio-devel(x86_64)-0.3.107  passed
Verifying Package: libaio-devel-0.3.107 (x86_64) ...PASSED
Verifying Package: nfs-utils-1.2.3-15 ...
  Node Name     Available                 Required                  Status
  ------------  ------------------------  ------------------------  ----------
  rhel6m2       nfs-utils-1.2.3-36.el6    nfs-utils-1.2.3-15        passed
  rhel6m1       nfs-utils-1.2.3-36.el6    nfs-utils-1.2.3-15        passed
Verifying Package: nfs-utils-1.2.3-15 ...PASSED
Verifying Package: smartmontools-5.43-1 ...
  Node Name     Available                 Required                  Status
  ------------  ------------------------  ------------------------  ----------
  rhel6m2       smartmontools-5.43-1.el6  smartmontools-5.43-1      passed
  rhel6m1       smartmontools-5.43-1.el6  smartmontools-5.43-1      passed
Verifying Package: smartmontools-5.43-1 ...PASSED
Verifying Package: net-tools-1.60-110 ...
  Node Name     Available                 Required                  Status
  ------------  ------------------------  ------------------------  ----------
  rhel6m2       net-tools-1.60-110.el6_2  net-tools-1.60-110        passed
  rhel6m1       net-tools-1.60-110.el6_2  net-tools-1.60-110        passed
Verifying Package: net-tools-1.60-110 ...PASSED
Verifying Users With Same UID: 0 ...PASSED
Verifying Current Group ID ...PASSED
Verifying Root user consistency ...
  Node Name                             Status
  ------------------------------------  ------------------------
  rhel6m2                               passed
  rhel6m1                               passed
Verifying Root user consistency ...PASSED
Verifying selectivity of ASM discovery string ...PASSED
Verifying ASM spare parameters ...PASSED
Verifying Disk group ASM compatibility setting ...PASSED
Verifying Package: cvuqdisk-1.0.10-1 ...
  Node Name     Available                 Required                  Status
  ------------  ------------------------  ------------------------  ----------
  rhel6m2       cvuqdisk-1.0.10-1         cvuqdisk-1.0.10-1         passed
  rhel6m1       cvuqdisk-1.0.10-1         cvuqdisk-1.0.10-1         passed
Verifying Package: cvuqdisk-1.0.10-1 ...PASSED
Verifying Node Connectivity ...
  Verifying Hosts File ...
  Node Name                             Status
  ------------------------------------  ------------------------
  rhel6m1                               passed
  rhel6m2                               passed
  Verifying Hosts File ...PASSED

Interface information for node "rhel6m2"

 Name   IP Address      Subnet          Gateway         Def. Gateway    HW Address        MTU
 ------ --------------- --------------- --------------- --------------- ----------------- ------
 eth0   192.168.0.94    192.168.0.0     0.0.0.0         192.168.0.100   08:00:27:7E:61:A9 1500
 eth0   192.168.0.98    192.168.0.0     0.0.0.0         192.168.0.100   08:00:27:7E:61:A9 1500
 eth0   192.168.0.125   192.168.0.0     0.0.0.0         192.168.0.100   08:00:27:7E:61:A9 1500
 eth1   192.168.1.88    192.168.1.0     0.0.0.0         192.168.0.100   08:00:27:69:2C:B6 1500

Interface information for node "rhel6m1"

 Name   IP Address      Subnet          Gateway         Def. Gateway    HW Address        MTU
 ------ --------------- --------------- --------------- --------------- ----------------- ------
 eth0   192.168.0.93    192.168.0.0     0.0.0.0         192.168.0.100   08:00:27:AC:F3:CC 1500
 eth0   192.168.0.135   192.168.0.0     0.0.0.0         192.168.0.100   08:00:27:AC:F3:CC 1500
 eth0   192.168.0.145   192.168.0.0     0.0.0.0         192.168.0.100   08:00:27:AC:F3:CC 1500
 eth0   192.168.0.97    192.168.0.0     0.0.0.0         192.168.0.100   08:00:27:AC:F3:CC 1500
 eth1   192.168.1.87    192.168.1.0     0.0.0.0         192.168.0.100   08:00:27:A3:C4:6F 1500

Check: MTU consistency on the private interfaces of subnet "192.168.1.0"

  Node              Name          IP Address    Subnet        MTU
  ----------------  ------------  ------------  ------------  ----------------
  rhel6m2           eth1          192.168.1.88  192.168.1.0   1500
  rhel6m1           eth1          192.168.1.87  192.168.1.0   1500

Check: MTU consistency of the subnet "192.168.0.0".

  Node              Name          IP Address    Subnet        MTU
  ----------------  ------------  ------------  ------------  ----------------
  rhel6m2           eth0          192.168.0.94  192.168.0.0   1500
  rhel6m2           eth0          192.168.0.98  192.168.0.0   1500
  rhel6m2           eth0          192.168.0.125  192.168.0.0   1500
  rhel6m1           eth0          192.168.0.93  192.168.0.0   1500
  rhel6m1           eth0          192.168.0.135  192.168.0.0   1500
  rhel6m1           eth0          192.168.0.145  192.168.0.0   1500
  rhel6m1           eth0          192.168.0.97  192.168.0.0   1500

  Source                          Destination                     Connected?
  ------------------------------  ------------------------------  ----------------
  rhel6m1[eth1:192.168.1.87]      rhel6m2[eth1:192.168.1.88]      yes

  Source                          Destination                     Connected?
  ------------------------------  ------------------------------  ----------------
  rhel6m1[eth0:192.168.0.93]      rhel6m2[eth0:192.168.0.98]      yes
  rhel6m1[eth0:192.168.0.93]      rhel6m2[eth0:192.168.0.125]     yes
  rhel6m1[eth0:192.168.0.93]      rhel6m2[eth0:192.168.0.94]      yes
  rhel6m1[eth0:192.168.0.93]      rhel6m1[eth0:192.168.0.135]     yes
  rhel6m1[eth0:192.168.0.93]      rhel6m1[eth0:192.168.0.145]     yes
  rhel6m1[eth0:192.168.0.93]      rhel6m1[eth0:192.168.0.97]      yes
  rhel6m2[eth0:192.168.0.98]      rhel6m2[eth0:192.168.0.125]     yes
  rhel6m2[eth0:192.168.0.98]      rhel6m2[eth0:192.168.0.94]      yes
  rhel6m2[eth0:192.168.0.98]      rhel6m1[eth0:192.168.0.135]     yes
  rhel6m2[eth0:192.168.0.98]      rhel6m1[eth0:192.168.0.145]     yes
  rhel6m2[eth0:192.168.0.98]      rhel6m1[eth0:192.168.0.97]      yes
  rhel6m2[eth0:192.168.0.125]     rhel6m2[eth0:192.168.0.94]      yes
  rhel6m2[eth0:192.168.0.125]     rhel6m1[eth0:192.168.0.135]     yes
  rhel6m2[eth0:192.168.0.125]     rhel6m1[eth0:192.168.0.145]     yes
  rhel6m2[eth0:192.168.0.125]     rhel6m1[eth0:192.168.0.97]      yes
  rhel6m2[eth0:192.168.0.94]      rhel6m1[eth0:192.168.0.135]     yes
  rhel6m2[eth0:192.168.0.94]      rhel6m1[eth0:192.168.0.145]     yes
  rhel6m2[eth0:192.168.0.94]      rhel6m1[eth0:192.168.0.97]      yes
  rhel6m1[eth0:192.168.0.135]     rhel6m1[eth0:192.168.0.145]     yes
  rhel6m1[eth0:192.168.0.135]     rhel6m1[eth0:192.168.0.97]      yes
  rhel6m1[eth0:192.168.0.145]     rhel6m1[eth0:192.168.0.97]      yes
  Verifying Check that maximum (MTU) size packet goes through subnet ...PASSED
  Verifying subnet mask consistency for subnet "192.168.1.0" ...PASSED
  Verifying subnet mask consistency for subnet "192.168.0.0" ...PASSED
Verifying Node Connectivity ...PASSED
Verifying Multicast check ...
Checking subnet "192.168.1.0" for multicast communication with multicast group "224.0.0.251"
Verifying Multicast check ...PASSED
Verifying OCR Integrity ...PASSED
Verifying Network Time Protocol (NTP) ...
  Verifying '/etc/ntp.conf' ...
  Node Name                             File exists?
  ------------------------------------  ------------------------
  rhel6m2                               no
  rhel6m1                               no

  Verifying '/etc/ntp.conf' ...PASSED
  Verifying '/var/run/ntpd.pid' ...
  Node Name                             File exists?
  ------------------------------------  ------------------------
  rhel6m2                               no
  rhel6m1                               no

  Verifying '/var/run/ntpd.pid' ...PASSED
Verifying Network Time Protocol (NTP) ...PASSED
Verifying Same core file name pattern ...PASSED
Verifying User Mask ...
  Node Name     Available                 Required                  Comment
  ------------  ------------------------  ------------------------  ----------
  rhel6m2       0022                      0022                      passed
  rhel6m1       0022                      0022                      passed
Verifying User Mask ...PASSED
Verifying User Not In Group "root": grid ...
  Node Name     Status                    Comment
  ------------  ------------------------  ------------------------
  rhel6m2       passed                    does not exist
  rhel6m1       passed                    does not exist
Verifying User Not In Group "root": grid ...PASSED
Verifying Time zone consistency ...PASSED
Verifying VIP Subnet configuration check ...PASSED
Verifying Voting Disk ...PASSED
Verifying resolv.conf Integrity ...
  Verifying (Linux) resolv.conf Integrity ...
  Node Name                             Status
  ------------------------------------  ------------------------
  rhel6m1                               passed
  rhel6m2                               passed

  checking response for name "rhel6m2" from each of the name servers specified
  in "/etc/resolv.conf"

  Node Name     Source                    Comment                   Status
  ------------  ------------------------  ------------------------  ----------
  rhel6m2       192.168.0.66              IPv4                      passed
  rhel6m2       10.10.10.1                IPv4                      passed

  checking response for name "rhel6m1" from each of the name servers specified
  in "/etc/resolv.conf"

  Node Name     Source                    Comment                   Status
  ------------  ------------------------  ------------------------  ----------
  rhel6m1       192.168.0.66              IPv4                      passed
  rhel6m1       10.10.10.1                IPv4                      passed
  Verifying (Linux) resolv.conf Integrity ...PASSED
Verifying resolv.conf Integrity ...PASSED
Verifying DNS/NIS name service ...PASSED
Verifying Daemon "avahi-daemon" not configured and running ...
  Node Name     Configured                Status
  ------------  ------------------------  ------------------------
  rhel6m2       no                        passed
  rhel6m1       no                        passed

  Node Name     Running?                  Status
  ------------  ------------------------  ------------------------
  rhel6m2       no                        passed
  rhel6m1       no                        passed
Verifying Daemon "avahi-daemon" not configured and running ...PASSED
Verifying Daemon "proxyt" not configured and running ...
  Node Name     Configured                Status
  ------------  ------------------------  ------------------------
  rhel6m2       no                        passed
  rhel6m1       no                        passed

  Node Name     Running?                  Status
  ------------  ------------------------  ------------------------
  rhel6m2       no                        passed
  rhel6m1       no                        passed
Verifying Daemon "proxyt" not configured and running ...PASSED
Verifying loopback network interface address ...PASSED
Verifying Grid Infrastructure home path: /opt/app/12.2.0/grid ...
  Verifying '/opt/app/12.2.0/grid' ...PASSED
Verifying Grid Infrastructure home path: /opt/app/12.2.0/grid ...PASSED
Verifying ACFS Driver Checks ...PASSED
Verifying Privileged group consistency for upgrade ...PASSED
Verifying CRS user Consistency for upgrade ...PASSED
Verifying Clusterware Version Consistency ...PASSED
Verifying Check incorrectly sized ASM Disks ...PASSED
Verifying that default ASM disk discovery string is in use ...PASSED
Verifying Network configuration consistency checks ...PASSED
Verifying File system mount options for path GI_HOME ...PASSED
Verifying /boot mount ...PASSED
Verifying OLR Integrity ...PASSED
Verifying Verify that the ASM instance was configured using an existing ASM parameter file. ...PASSED
Verifying User Equivalence ...PASSED
Verifying /dev/shm mounted as temporary file system ...PASSED
Verifying File system mount options for path /var ...PASSED
Verifying zeroconf check ...PASSED
Verifying ASM Filter Driver configuration ...PASSED

Pre-check for cluster services setup was unsuccessful.
Checks did not pass for the following nodes:
        rhel6m2,rhel6m1


Failures were encountered during execution of CVU verification request "stage -pre crsinst".

Verifying Swap Size ...FAILED
rhel6m2: PRVF-7573 : Sufficient swap size is not available on node "rhel6m2"
         [Required = 7.6863GB (8059712.0KB) ; Found = 4GB (4194296.0KB)]

rhel6m1: PRVF-7573 : Sufficient swap size is not available on node "rhel6m1"
         [Required = 7.6863GB (8059712.0KB) ; Found = 4GB (4194296.0KB)]


CVU operation performed:      stage -pre crsinst
Date:                         Apr 11, 2017 10:28:40 AM
CVU home:                     /opt/app/12.2.0/grid/
User:                         grid
The failure is due to insufficient Swap size. On a test setup this is ignorable.

8. Run the gridSetup from the new GI home to begin the upgrade.
./gridSetup.sh
9. Run the rootupgrade.sh when prompted. Following output shows running rootupgrade on first node.
[root@rhel6m1 grid]# /opt/app/12.2.0/grid/rootupgrade.sh
Performing root user operation.

The following environment variables are set as:
    ORACLE_OWNER= grid
    ORACLE_HOME=  /opt/app/12.2.0/grid

Enter the full pathname of the local bin directory: [/usr/local/bin]:
The file "dbhome" already exists in /usr/local/bin.  Overwrite it? (y/n)
[n]:
The file "oraenv" already exists in /usr/local/bin.  Overwrite it? (y/n)
[n]:
The file "coraenv" already exists in /usr/local/bin.  Overwrite it? (y/n)
[n]:

Entries will be added to the /etc/oratab file as needed by
Database Configuration Assistant when a database is created
Finished running generic part of root script.
Now product-specific root actions will be performed.
Relinking oracle with rac_on option
Using configuration parameter file: /opt/app/12.2.0/grid/crs/install/crsconfig_params
The log of current session can be found at:
  /opt/app/oracle/crsdata/rhel6m1/crsconfig/rootcrs_rhel6m1_2017-04-12_02-31-52PM.log
2017/04/12 14:31:57 CLSRSC-595: Executing upgrade step 1 of 19: 'UpgradeTFA'.
2017/04/12 14:31:58 CLSRSC-4015: Performing install or upgrade action for Oracle Trace File Analyzer (TFA) Collector.
2017/04/12 14:33:31 CLSRSC-4003: Successfully patched Oracle Trace File Analyzer (TFA) Collector.
2017/04/12 14:33:31 CLSRSC-595: Executing upgrade step 2 of 19: 'ValidateEnv'.
2017/04/12 14:33:50 CLSRSC-595: Executing upgrade step 3 of 19: 'GenSiteGUIDs'.
2017/04/12 14:33:52 CLSRSC-595: Executing upgrade step 4 of 19: 'GetOldConfig'.
2017/04/12 14:33:52 CLSRSC-464: Starting retrieval of the cluster configuration data
2017/04/12 14:34:06 CLSRSC-515: Starting OCR manual backup.
2017/04/12 14:34:12 CLSRSC-516: OCR manual backup successful.
2017/04/12 14:34:36 CLSRSC-486:
 At this stage of upgrade, the OCR has changed.
 Any attempt to downgrade the cluster after this point will require a complete cluster outage to restore the OCR.
2017/04/12 14:34:37 CLSRSC-541:
 To downgrade the cluster:
 1. All nodes that have been upgraded must be downgraded.
2017/04/12 14:34:37 CLSRSC-542:
 2. Before downgrading the last node, the Grid Infrastructure stack on all other cluster nodes must be down.
2017/04/12 14:34:37 CLSRSC-615:
 3. The last node to downgrade cannot be a Leaf node.
2017/04/12 14:34:51 CLSRSC-465: Retrieval of the cluster configuration data has successfully completed.
2017/04/12 14:34:51 CLSRSC-595: Executing upgrade step 5 of 19: 'UpgPrechecks'.
2017/04/12 14:34:57 CLSRSC-363: User ignored prerequisites during installation
2017/04/12 14:35:14 CLSRSC-595: Executing upgrade step 6 of 19: 'SaveParamFile'.
2017/04/12 14:35:29 CLSRSC-595: Executing upgrade step 7 of 19: 'SetupOSD'.
2017/04/12 14:35:42 CLSRSC-595: Executing upgrade step 8 of 19: 'PreUpgrade'.
2017/04/12 14:35:49 CLSRSC-468: Setting Oracle Clusterware and ASM to rolling migration mode
2017/04/12 14:35:49 CLSRSC-482: Running command: '/opt/app/12.2.0/grid/bin/asmca -silent -upgradeNodeASM -nonRolling false -oldCRSHome /opt/app/11.2.0/grid4 -oldCRSVersion 11.2.0.4.0 -firstNode true -startRolling true '

ASM configuration upgraded in local node successfully.

2017/04/12 14:36:01 CLSRSC-469: Successfully set Oracle Clusterware and ASM to rolling migration mode
2017/04/12 14:36:19 CLSRSC-466: Starting shutdown of the current Oracle Grid Infrastructure stack
2017/04/12 14:39:40 CLSRSC-467: Shutdown of the current Oracle Grid Infrastructure stack has successfully completed.
2017/04/12 14:39:43 CLSRSC-595: Executing upgrade step 9 of 19: 'CheckCRSConfig'.
2017/04/12 14:39:44 CLSRSC-595: Executing upgrade step 10 of 19: 'UpgradeOLR'.
2017/04/12 14:39:55 CLSRSC-595: Executing upgrade step 11 of 19: 'ConfigCHMOS'.
2017/04/12 14:39:55 CLSRSC-595: Executing upgrade step 12 of 19: 'InstallAFD'.
2017/04/12 14:40:07 CLSRSC-595: Executing upgrade step 13 of 19: 'createOHASD'.
2017/04/12 14:40:18 CLSRSC-595: Executing upgrade step 14 of 19: 'ConfigOHASD'.
2017/04/12 14:40:34 CLSRSC-329: Replacing Clusterware entries in file 'oracle-ohasd.conf'
2017/04/12 14:41:05 CLSRSC-595: Executing upgrade step 15 of 19: 'InstallACFS'.
CRS-2791: Starting shutdown of Oracle High Availability Services-managed resources on 'rhel6m1'
CRS-2793: Shutdown of Oracle High Availability Services-managed resources on 'rhel6m1' has completed
CRS-4133: Oracle High Availability Services has been stopped.
CRS-4123: Oracle High Availability Services has been started.
2017/04/12 14:42:04 CLSRSC-595: Executing upgrade step 16 of 19: 'InstallKA'.
2017/04/12 14:42:15 CLSRSC-595: Executing upgrade step 17 of 19: 'UpgradeCluster'.
CRS-2791: Starting shutdown of Oracle High Availability Services-managed resources on 'rhel6m1'
CRS-2793: Shutdown of Oracle High Availability Services-managed resources on 'rhel6m1' has completed
CRS-4133: Oracle High Availability Services has been stopped.
CRS-4123: Oracle High Availability Services has been started.
clscfg: EXISTING configuration version 5 detected.
clscfg: version 5 is 11g Release 2.
Successfully taken the backup of node specific configuration in OCR.
Successfully accumulated necessary OCR keys.
Creating OCR keys for user 'root', privgrp 'root'..
Operation successful.
CRS-2791: Starting shutdown of Oracle High Availability Services-managed resources on 'rhel6m1'
CRS-2673: Attempting to stop 'ora.crsd' on 'rhel6m1'
CRS-2677: Stop of 'ora.crsd' on 'rhel6m1' succeeded
CRS-2673: Attempting to stop 'ora.asm' on 'rhel6m1'
CRS-2673: Attempting to stop 'ora.crf' on 'rhel6m1'
CRS-2673: Attempting to stop 'ora.drivers.acfs' on 'rhel6m1'
CRS-2673: Attempting to stop 'ora.gpnpd' on 'rhel6m1'
CRS-2673: Attempting to stop 'ora.mdnsd' on 'rhel6m1'
CRS-2677: Stop of 'ora.drivers.acfs' on 'rhel6m1' succeeded
CRS-2677: Stop of 'ora.crf' on 'rhel6m1' succeeded
CRS-2677: Stop of 'ora.gpnpd' on 'rhel6m1' succeeded
CRS-2677: Stop of 'ora.mdnsd' on 'rhel6m1' succeeded
CRS-2677: Stop of 'ora.asm' on 'rhel6m1' succeeded
CRS-2673: Attempting to stop 'ora.cluster_interconnect.haip' on 'rhel6m1'
CRS-2677: Stop of 'ora.cluster_interconnect.haip' on 'rhel6m1' succeeded
CRS-2673: Attempting to stop 'ora.ctssd' on 'rhel6m1'
CRS-2673: Attempting to stop 'ora.evmd' on 'rhel6m1'
CRS-2677: Stop of 'ora.ctssd' on 'rhel6m1' succeeded
CRS-2677: Stop of 'ora.evmd' on 'rhel6m1' succeeded
CRS-2673: Attempting to stop 'ora.cssd' on 'rhel6m1'
CRS-2677: Stop of 'ora.cssd' on 'rhel6m1' succeeded
CRS-2673: Attempting to stop 'ora.gipcd' on 'rhel6m1'
CRS-2677: Stop of 'ora.gipcd' on 'rhel6m1' succeeded
CRS-2793: Shutdown of Oracle High Availability Services-managed resources on 'rhel6m1' has completed
CRS-4133: Oracle High Availability Services has been stopped.
CRS-4123: Oracle High Availability Services has been started.
2017/04/12 14:56:43 CLSRSC-343: Successfully started Oracle Clusterware stack
2017/04/12 14:56:43 CLSRSC-595: Executing upgrade step 18 of 19: 'UpgradeNode'.
2017/04/12 14:56:50 CLSRSC-474: Initiating upgrade of resource types
2017/04/12 14:58:44 CLSRSC-482: Running command: 'srvctl upgrade model -s 11.2.0.4.0 -d 12.2.0.1.0 -p first'
2017/04/12 14:58:44 CLSRSC-475: Upgrade of resource types successfully initiated.
2017/04/12 14:59:05 CLSRSC-595: Executing upgrade step 19 of 19: 'PostUpgrade'.
2017/04/12 14:59:23 CLSRSC-325: Configure Oracle Grid Infrastructure for a Cluster ... succeeded
After the complete clusterware on first node is updated by clusterwide active version still remains the 11.2.0.4
[root@rhel6m1 grid]# crsctl query crs activeversion
Oracle Clusterware active version on the cluster is [11.2.0.4.0]
[root@rhel6m1 grid]# crsctl query crs releaseversion
Oracle High Availability Services release version on the local node is [12.2.0.1.0]
[root@rhel6m1 grid]# crsctl query crs softwareversion -all
Oracle Clusterware version on node [rhel6m1] is [12.2.0.1.0]
Oracle Clusterware version on node [rhel6m2] is [11.2.0.4.0]
Following output shows the rootupgrade run on second node.
[root@rhel6m2 ~]# /opt/app/12.2.0/grid/rootupgrade.sh
Performing root user operation.

The following environment variables are set as:
    ORACLE_OWNER= grid
    ORACLE_HOME=  /opt/app/12.2.0/grid

Enter the full pathname of the local bin directory: [/usr/local/bin]:
The file "dbhome" already exists in /usr/local/bin.  Overwrite it? (y/n)
[n]:
The file "oraenv" already exists in /usr/local/bin.  Overwrite it? (y/n)
[n]:
The file "coraenv" already exists in /usr/local/bin.  Overwrite it? (y/n)
[n]:

Entries will be added to the /etc/oratab file as needed by
Database Configuration Assistant when a database is created
Finished running generic part of root script.
Now product-specific root actions will be performed.
Relinking oracle with rac_on option
Using configuration parameter file: /opt/app/12.2.0/grid/crs/install/crsconfig_params
The log of current session can be found at:
  /opt/app/oracle/crsdata/rhel6m2/crsconfig/rootcrs_rhel6m2_2017-04-12_03-00-25PM.log
2017/04/12 15:00:30 CLSRSC-595: Executing upgrade step 1 of 19: 'UpgradeTFA'.
2017/04/12 15:00:30 CLSRSC-4015: Performing install or upgrade action for Oracle Trace File Analyzer (TFA) Collector.
2017/04/12 15:02:06 CLSRSC-4003: Successfully patched Oracle Trace File Analyzer (TFA) Collector.
2017/04/12 15:02:06 CLSRSC-595: Executing upgrade step 2 of 19: 'ValidateEnv'.
2017/04/12 15:02:08 CLSRSC-595: Executing upgrade step 3 of 19: 'GenSiteGUIDs'.
2017/04/12 15:02:09 CLSRSC-595: Executing upgrade step 4 of 19: 'GetOldConfig'.
2017/04/12 15:02:09 CLSRSC-464: Starting retrieval of the cluster configuration data
2017/04/12 15:02:25 CLSRSC-465: Retrieval of the cluster configuration data has successfully completed.
2017/04/12 15:02:25 CLSRSC-595: Executing upgrade step 5 of 19: 'UpgPrechecks'.
2017/04/12 15:02:26 CLSRSC-363: User ignored prerequisites during installation
2017/04/12 15:02:29 CLSRSC-595: Executing upgrade step 6 of 19: 'SaveParamFile'.
2017/04/12 15:02:33 CLSRSC-595: Executing upgrade step 7 of 19: 'SetupOSD'.
2017/04/12 15:02:35 CLSRSC-595: Executing upgrade step 8 of 19: 'PreUpgrade'.

ASM configuration upgraded in local node successfully.

2017/04/12 15:02:42 CLSRSC-466: Starting shutdown of the current Oracle Grid Infrastructure stack
2017/04/12 15:06:05 CLSRSC-467: Shutdown of the current Oracle Grid Infrastructure stack has successfully completed.
2017/04/12 15:06:11 CLSRSC-595: Executing upgrade step 9 of 19: 'CheckCRSConfig'.
2017/04/12 15:06:12 CLSRSC-595: Executing upgrade step 10 of 19: 'UpgradeOLR'.
2017/04/12 15:06:14 CLSRSC-595: Executing upgrade step 11 of 19: 'ConfigCHMOS'.
2017/04/12 15:06:14 CLSRSC-595: Executing upgrade step 12 of 19: 'InstallAFD'.
2017/04/12 15:06:15 CLSRSC-595: Executing upgrade step 13 of 19: 'createOHASD'.
2017/04/12 15:06:18 CLSRSC-595: Executing upgrade step 14 of 19: 'ConfigOHASD'.
2017/04/12 15:06:33 CLSRSC-329: Replacing Clusterware entries in file 'oracle-ohasd.conf'
2017/04/12 15:06:55 CLSRSC-595: Executing upgrade step 15 of 19: 'InstallACFS'.
CRS-2791: Starting shutdown of Oracle High Availability Services-managed resources on 'rhel6m2'
CRS-2793: Shutdown of Oracle High Availability Services-managed resources on 'rhel6m2' has completed
CRS-4133: Oracle High Availability Services has been stopped.
CRS-4123: Oracle High Availability Services has been started.
2017/04/12 15:07:40 CLSRSC-595: Executing upgrade step 16 of 19: 'InstallKA'.
2017/04/12 15:07:41 CLSRSC-595: Executing upgrade step 17 of 19: 'UpgradeCluster'.
CRS-2791: Starting shutdown of Oracle High Availability Services-managed resources on 'rhel6m2'
CRS-2793: Shutdown of Oracle High Availability Services-managed resources on 'rhel6m2' has completed
CRS-4133: Oracle High Availability Services has been stopped.
CRS-4123: Oracle High Availability Services has been started.
2017/04/12 15:08:39 CLSRSC-343: Successfully started Oracle Clusterware stack
clscfg: EXISTING configuration version 5 detected.
clscfg: version 5 is 12c Release 2.
Successfully taken the backup of node specific configuration in OCR.
Successfully accumulated necessary OCR keys.
Creating OCR keys for user 'root', privgrp 'root'..
Operation successful.
2017/04/12 15:09:02 CLSRSC-595: Executing upgrade step 18 of 19: 'UpgradeNode'.
Start upgrade invoked..
2017/04/12 15:09:36 CLSRSC-478: Setting Oracle Clusterware active version on the last node to be upgraded
2017/04/12 15:09:36 CLSRSC-482: Running command: '/opt/app/12.2.0/grid/bin/crsctl set crs activeversion'
Started to upgrade the active version of Oracle Clusterware. This operation may take a few minutes.
Started to upgrade the OCR.
Started to upgrade CSS.
CSS was successfully upgraded.
Started to upgrade Oracle ASM.
Started to upgrade CRS.
CRS was successfully upgraded.
Successfully upgraded the active version of Oracle Clusterware.
Oracle Clusterware active version was successfully set to 12.2.0.1.0.
2017/04/12 15:11:43 CLSRSC-479: Successfully set Oracle Clusterware active version
2017/04/12 15:11:55 CLSRSC-476: Finishing upgrade of resource types
2017/04/12 15:13:12 CLSRSC-482: Running command: 'srvctl upgrade model -s 11.2.0.4.0 -d 12.2.0.1.0 -p last'
2017/04/12 15:13:12 CLSRSC-477: Successfully completed upgrade of resource types
2017/04/12 15:14:00 CLSRSC-595: Executing upgrade step 19 of 19: 'PostUpgrade'.
2017/04/12 15:14:24 CLSRSC-325: Configure Oracle Grid Infrastructure for a Cluster ... succeeded
After the second (in this case last node) is upgraded the active version is upgraded to 12.2.0.1
[grid@rhel6m2 ~]$ crsctl query crs activeversion
Oracle Clusterware active version on the cluster is [12.2.0.1.0]
[grid@rhel6m2 ~]$ crsctl query crs releaseversion
Oracle High Availability Services release version on the local node is [12.2.0.1.0]
[grid@rhel6m2 ~]$ crsctl query crs softwareversion -all
Oracle Clusterware version on node [rhel6m1] is [12.2.0.1.0]
Oracle Clusterware version on node [rhel6m2] is [12.2.0.1.0]
10. After the root scripts are run the upgrade process complete any remaining tasks and concludes.

11. At this time the GI is upgraded to 12.2. The cluster mode would be listed as flex (which could be veriried through crsctl or asmcmd)
[grid@rhel6m1 ~]$ crsctl get cluster mode status
Cluster is running in "flex" mode

ASMCMD> showclustermode
ASM cluster : Flex mode enabled
As stated earlier, even though cluster is a flex cluster there's no GNS setup
[grid@rhel6m1 ~]$ srvctl config gns
PRKF-1110 : Neither GNS server nor GNS client is configured on this cluster
All the nodes have the "hub" role
[grid@rhel6m1 ~]$ crsctl get node role config -all
Node 'rhel6m1' configured role is 'hub'
Node 'rhel6m2' configured role is 'hub'
12. If as part of the upgrade OCR and Vote disks were moved to a external redundnacy disk group, move them back to normal redundnacy diskgroup.
[root@rhel6m1 app]# ocrcheck
Status of Oracle Cluster Registry is as follows :
         Version                  :          4
         Total space (kbytes)     :     409568
         Used space (kbytes)      :      12056
         Available space (kbytes) :     397512
         ID                       : 1487892601
         Device/File Name         :      +GIMR
                                    Device/File integrity check succeeded

                                    Device/File not configured

                                    Device/File not configured

                                    Device/File not configured

                                    Device/File not configured

         Cluster registry integrity check succeeded

         Logical corruption check succeeded

[root@rhel6m1 app]# ocrconfig -add +CLUSTER_DG
[root@rhel6m1 app]# ocrconfig -delete +GIMR

[grid@rhel6m1 app]$ crsctl replace votedisk +CLUSTER_DG
Successful addition of voting disk ddf6f87af94d4f2bbf8a1c5597b2f6fd.
Successful addition of voting disk 152250cb7ef94f3abf1afebc489071fa.
Successful addition of voting disk 87eeffff539b4f9ebf0a5d1932fca9b4.
Successful deletion of voting disk 7917ab02c3e54f5cbf1453f6ba7b1bc7.
Successfully replaced voting disk group with +CLUSTER_DG.
CRS-4266: Voting file(s) successfully replaced
[grid@rhel6m1 app]$ crsctl query css votedisk
##  STATE    File Universal Id                File Name Disk group
--  -----    -----------------                --------- ---------
 1. ONLINE   ddf6f87af94d4f2bbf8a1c5597b2f6fd (/dev/sdc1) [CLUSTER_DG]
 2. ONLINE   152250cb7ef94f3abf1afebc489071fa (/dev/sdd1) [CLUSTER_DG]
 3. ONLINE   87eeffff539b4f9ebf0a5d1932fca9b4 (/dev/sdb1) [CLUSTER_DG]
Located 3 voting disk(s).
After this the disk group where MGMTDB is located will be used as the location for OCR backups.
[root@rhel6m1 app]#  ocrconfig -showbackuploc
The Oracle Cluster Registry backup location is [+GIMR]

ocrconfig -showbackup manual

rhel6m2     2017/07/14 11:20:31     +GIMR:/rhel6m-cluster/OCRBACKUP/backup_20170714_112031.ocr.289.949317631     0

rhel6m2     2017/07/13 17:34:43     +GIMR:/rhel6m-cluster/OCRBACKUP/backup_20170713_173443.ocr.258.949253683     0
13. Size the manage repository size based on retention and number of nodes
oclumon manage -repos changerepossize 4000
14. Finally run the cluvfy post crsinst
[grid@rhel6m1 ~]$ cluvfy stage -post crsinst -n all

Verifying Node Connectivity ...
  Verifying Hosts File ...PASSED
  Verifying Check that maximum (MTU) size packet goes through subnet ...PASSED
  Verifying subnet mask consistency for subnet "192.168.1.0" ...PASSED
  Verifying subnet mask consistency for subnet "192.168.0.0" ...PASSED
Verifying Node Connectivity ...PASSED
Verifying Multicast check ...PASSED
Verifying ASM filter driver configuration consistency ...PASSED
Verifying Time zone consistency ...PASSED
Verifying Cluster Manager Integrity ...PASSED
Verifying User Mask ...PASSED
Verifying Cluster Integrity ...PASSED
Verifying OCR Integrity ...PASSED
Verifying CRS Integrity ...
  Verifying Clusterware Version Consistency ...PASSED
Verifying CRS Integrity ...PASSED
Verifying Node Application Existence ...PASSED
Verifying Single Client Access Name (SCAN) ...
  Verifying DNS/NIS name service 'rac-scan.domain.net' ...
    Verifying Name Service Switch Configuration File Integrity ...PASSED
  Verifying DNS/NIS name service 'rac-scan.domain.net' ...PASSED
Verifying Single Client Access Name (SCAN) ...PASSED
Verifying OLR Integrity ...PASSED
Verifying Voting Disk ...PASSED
Verifying ASM Integrity ...
  Verifying Node Connectivity ...
    Verifying Hosts File ...PASSED
    Verifying Check that maximum (MTU) size packet goes through subnet ...PASSED
    Verifying subnet mask consistency for subnet "192.168.1.0" ...PASSED
    Verifying subnet mask consistency for subnet "192.168.0.0" ...PASSED
  Verifying Node Connectivity ...PASSED
Verifying ASM Integrity ...PASSED
Verifying Device Checks for ASM ...
  Verifying Access Control List check ...PASSED
Verifying Device Checks for ASM ...PASSED
Verifying ASM disk group free space ...PASSED
Verifying I/O scheduler ...
  Verifying Package: cvuqdisk-1.0.10-1 ...PASSED
Verifying I/O scheduler ...PASSED
Verifying User Not In Group "root": grid ...PASSED
Verifying Clock Synchronization ...PASSED
Verifying VIP Subnet configuration check ...PASSED
Verifying Network configuration consistency checks ...PASSED
Verifying File system mount options for path GI_HOME ...PASSED

Post-check for cluster services setup was successful.

CVU operation performed:      stage -post crsinst
Date:                         Apr 13, 2017 2:06:17 PM
CVU home:                     /opt/app/12.2.0/grid/
User:                         grid
Related Posts
Upgrading RAC from 11.2.0.4 to 12.1.0.2 - Grid Infrastructure
Upgrading Grid Infrastructure Used for Single Instance from 11.2.0.4 to 12.1.0.2
Upgrading RAC from 12.1.0.1 to 12.1.0.2 - Grid Infrastructure
Upgrading 12c CDB and PDB from 12.1.0.1 to 12.1.0.2
Upgrading from 11gR2 (11.2.0.3) to 12c (12.1.0.1) Grid Infrastructure
Upgrade Oracle Database 12c1 from 12.1.0.1 to 12.1.0.2
Upgrading RAC from 11.2.0.3 to 11.2.0.4 - Grid Infrastructure
Upgrading from 10.2.0.4 to 10.2.0.5 (Clusterware, RAC, ASM)
Upgrade from 10.2.0.5 to 11.2.0.3 (Clusterware, RAC, ASM)
Upgrade from 11.1.0.7 to 11.2.0.3 (Clusterware, ASM & RAC)
Upgrade from 11.1.0.7 to 11.2.0.4 (Clusterware, ASM & RAC)
Upgrading from 11.1.0.7 to 11.2.0.3 with Transient Logical Standby
Upgrading from 11.2.0.1 to 11.2.0.3 with in-place upgrade for RAC
In-place upgrade from 11.2.0.2 to 11.2.0.3
Upgrading from 11.2.0.2 to 11.2.0.3 with Physical Standby - 1
Upgrading from 11.2.0.2 to 11.2.0.3 with Physical Standby - 2
Upgrading from 11gR2 (11.2.0.3) to 12c (12.1.0.1) Grid Infrastructure