Friday, September 6, 2013

Upgrading RAC from 11.2.0.3 to 11.2.0.4 - Grid Infrastructure

11.2.0.4 patchset is available under patch number 13390677. This post list the steps related to upgrading from 11.2.0.3 to 11.2.0.4. It's not a comprehensive post of "how to upgrade" and list only major steps and things to look out for during the upgrade. At the time of this post, there's not much metalink notes related to 11.2.0.4 upgrade. However step is almost same as previous posts that showed upgrading to 11.2.0.3 and some of the information in metalink notes applicable to 11.2.0.3 are still useful.
There few new features introduced with 11.2.0.4 grid infrastructure. Some of these "features" were already available as separate software packages and now being bundled with Grid and Database software. Trace File Analyzer (1513912.1) comes as part of GI and automatically installed. During the upgrade this is installed when rootupgrade.sh is run. It will be added to upstart (or inittab) to start at boot time. With this on a 11.2.0.4 installation there will be two Oracle related processes added to init.
14907 ?        Ss     0:00 /bin/sh /etc/init.d/init.tfa run
16456 ?        Ss     0:00 /bin/sh /etc/init.d/init.ohasd run
RACcheck (1268927.1, 1457357.1) is another software that is bundled in 11.2.0.4 database software. For upgrades RACcheck is useful to check the upgrade readiness of the RAC (1457357.1) which was available from 11.2.0.3 onwards.
Oracle documentation says that installing clusterware files on block or raw devices is not supported. This was also the behavior from 11.2.0.1 as well, that block or raw devices are not supported when a fresh installation is done. But upgrading from a system that uses block or raw devices was not an issue. It's not clear why this is specifically mentioned in de-supported section of 11.2.0.4 and if this is now applicable for upgrades as well. (cannot upgrade to 12c if cluster files are in block or raw devices). However under the 11.2 section of the latest documentation it still says "Installing Oracle Clusterware files on raw or block devices is no longer supported, unless an existing system is being upgraded." Upgrading to 11.2.0.4 when cluster files are on block or raw devices will be verified later is verified here. System used for this post was the one earlier converted from standard edition to enterprise edition and had cluster files in ASM.
Refer new feature guide for complete list of features available with 11.2.0.4.
The system used is 11.2.0.3 RAC system on RHEL 6 and has listener names (scan listener and listener) and ports changed to non-default values. During that upgrade this caused issues at the end of the execution of the rootupgrade.sh script on the last node. The rootupgrade.sh seem to be looking for default listener name and without it some of the steps seem to fail. However output also shows that it is able to detect non-default listener names. It's not sure if any step actually failed or not as ultimately cluster version was upgraded to 11.2.0.4. Below is an few lines of output during rootupgrade.sh execution on the last node.
...
[main] [ 2013-09-06 12:07:20.698 BST ] [OracleHome.getVersion:1003]  ORACLE_HOME=/opt/app/11.2.0/grid4
[main] [ 2013-09-06 12:07:20.807 BST ] [OracleHome.getVersion:1008]  /opt/app/11.2.0/grid4/bin/sqlplus Banner:

SQL*Plus: Release 11.2.0.4.0 Production

[main] [ 2013-09-06 12:07:20.808 BST ] [OracleHome.getVersion:1026]  Current version from sqlplus: 11.2.0.4.0
[main] [ 2013-09-06 12:07:21.050 BST ] [NetworkUtils.checkListenerStatus:495]  Checking default listener status
[main] [ 2013-09-06 12:07:21.050 BST ] [NetworkUtils.checkListenerStatus:500]  Checking listener status: LISTENER
[main] [ 2013-09-06 12:07:21.070 BST ] [NetworkUtils.checkListenerStatus:506]  lsnrct status output

LSNRCTL for Linux: Version 11.2.0.4.0 - Production on 06-SEP-2013 12:07:21

Copyright (c) 1991, 2013, Oracle.  All rights reserved.

Connecting to (ADDRESS=(PROTOCOL=tcp)(HOST=)(PORT=1521))
TNS-12541: TNS:no listener
 TNS-12560: TNS:protocol adapter error
  TNS-00511: No listener
   Linux Error: 111: Connection refused
[main] [ 2013-09-06 12:07:21.082 BST ] [NetworkUtils.getOneLocalListenerProtocolAddress:3085]  lsnr MYSCANLISTENER_SCAN1
[main] [ 2013-09-06 12:07:21.139 BST ] [InventoryUtil.getOUIInvSession:347]  setting OUI READ level to ACCESSLEVEL_READ_LOCKLESS
...
[main] [ 2013-09-06 12:07:21.722 BST ] [OCR.isCluster:968]  Bolean result = true
[main] [ 2013-09-06 12:07:21.790 BST ] [CRSNative.isEntityRegistered:737]  entity: ora.MYSCANLISTENER_SCAN1.lsnr, type: 1, registered: true
[main] [ 2013-09-06 12:07:21.817 BST ] [CRSNative.isEntityRegistered:737]  entity: ora.MYSCANLISTENER_SCAN1.lsnr, type: 1, registered: true
[main] [ 2013-09-06 12:07:21.818 BST ] [CRSNative.getStat:1447]  About to call crs stat: name: ora.MYSCANLISTENER_SCAN1.lsnr, type: 1, node: null
[main] [ 2013-09-06 12:07:21.818 BST ] [CRSNative.getStat:1449]  looking for following attributes:
[main] [ 2013-09-06 12:07:21.819 BST ] [CRSNative.getStat:1451]                 'TYPE'
[main] [ 2013-09-06 12:07:21.848 BST ] [CRSNative.getStat:1464]  crs found 1 attributes
[main] [ 2013-09-06 12:07:21.849 BST ] [CRSNative.getStat:1474]         Name: 'TYPE'; Value: 'ora.scan_listener.type'
[main] [ 2013-09-06 12:07:21.850 BST ] [NodeAppsFactoryImpl.getListener:1140]  Resource type is ora.scan_listener.type
[main] [ 2013-09-06 12:07:21.876 BST ] [NetworkUtils.getOneLocalListenerProtocolAddress:3099]  Error: Error while getting listener address list
[main] [ 2013-09-06 12:07:21.878 BST ] [NetworkUtils.getOneLocalListenerProtocolAddress:3100]  PRCN-2066 : Failed to retrieve cluster listener for MYSCANLISTENER_SCAN1 because it has the type of Single Client Access Name Listener
...
[ 2013-09-06 12:07:22.379 BST ] [RuntimeExec.runCommand:192]  Returning from RunTimeExec.runCommand
[main] [ 2013-09-06 12:07:22.416 BST ] [CRSNative.isEntityRegistered:737]  entity: ora.MYLISTENER.lsnr, type: 1, registered: true
[main] [ 2013-09-06 12:07:22.448 BST ] [CRSNative.isEntityRegistered:737]  entity: ora.MYLISTENER.lsnr, type: 1, registered: true
[main] [ 2013-09-06 12:07:22.448 BST ] [CRSNative.getStat:1447]  About to call crs stat: name: ora.MYLISTENER.lsnr, type: 1, node: null
[main] [ 2013-09-06 12:07:22.449 BST ] [CRSNative.getStat:1449]  looking for following attributes:
[main] [ 2013-09-06 12:07:22.449 BST ] [CRSNative.getStat:1451]                 'TYPE'
[main] [ 2013-09-06 12:07:22.481 BST ] [CRSNative.getStat:1464]  crs found 1 attributes
[main] [ 2013-09-06 12:07:22.482 BST ] [CRSNative.getStat:1474]         Name: 'TYPE'; Value: 'ora.listener.type'
[main] [ 2013-09-06 12:07:22.483 BST ] [NodeAppsFactoryImpl.getListener:1140]  Resource type is ora.listener.type
[main] [ 2013-09-06 12:07:22.483 BST ] [CRSNative.getStat:1447]  About to call crs stat: name: ora.MYLISTENER.lsnr, type: 1, node: null
[main] [ 2013-09-06 12:07:22.485 BST ] [CRSNative.getStat:1449]  looking for following attributes:
[main] [ 2013-09-06 12:07:22.486 BST ] [CRSNative.getStat:1451]                 'ENDPOINTS'
[main] [ 2013-09-06 12:07:22.515 BST ] [CRSNative.getStat:1464]  crs found 1 attributes
[main] [ 2013-09-06 12:07:22.515 BST ] [CRSNative.getStat:1474]         Name: 'ENDPOINTS'; Value: 'TCP:9120'
[main] [ 2013-09-06 12:07:22.518 BST ] [EndpointsImpl.:276]  Get listener endpoints from TCP:9120
[main] [ 2013-09-06 12:07:22.518 BST ] [EndpointsImpl.:302]  Token count is 1
[main] [ 2013-09-06 12:07:22.519 BST ] [EndpointsImpl.:324]  LsnrProtocol type = TCP, values = 9120
[main] [ 2013-09-06 12:07:22.521 BST ] [EndpointsImpl$LsnrProtocol.addPortValue:86]  Calling parseInt to check 9120
[main] [ 2013-09-06 12:07:22.522 BST ] [EndpointsImpl$LsnrProtocol.addPortValue:97]  Get port number 9120
...
[main] [ 2013-09-06 12:07:22.532 BST ] [NetworkUtils.getOneLocalListenerProtocolAddress:3116]  addr (ADDRESS=(PROTOCOL=TCP)(HOST=rhel6m2.domain.net)(PORT=9120))
[main] [ 2013-09-06 12:07:22.534 BST ] [NetworkUtils.getLocalListenerAddresses:1310]  listener[1]=MYLISTENER address=(ADDRESS=(PROTOCOL=TCP)(HOST=rhel6m2.domain.net)(PORT=9120))
...
[main] [ 2013-09-06 12:07:30.536 BST ] [CRSNativeResult.addComp:162]  add comp: name ora.LISTENER.lsnr, rc 210, msg CRS-0210: Could not find resource 'ora.LISTENER.lsnr'.
[main] [ 2013-09-06 12:07:30.539 BST ] [CRSNative.isEntityRegistered:737]  entity: ora.LISTENER.lsnr, type: 1, registered: false
ASM upgrade has finished on last node.
null
Configuring ASM instance in EM failed. The following error occured:
PRCR-1001 : Resource ora.LISTENER.lsnr does not exist
Configure Oracle Grid Infrastructure for a Cluster ... succeeded
To be on the safe side if non-default listener name/port is used it's best to create a default listener and start it before the upgrade and remove once the upgrade is completed. Same problem was there upgrading to 12c using DBUA
[grid@rhel6m1 admin]$ srvctl add listener -listener listener -p 1521
[grid@rhel6m1 admin]$ srvctl start listener -l listener
[grid@rhel6m1 admin]$ lsnrctl status
Running RACcheck post upgrade check (run as oracle user)
./raccheck -u -o post
on the existing 11.2.0.3 system gives the following warning on the resulting html file."The default setting for memory_target in 11.2.0.4 and 12.1 versions has been increased to 1024M from the previous default of 256M. Oracle also advises increasing memory_target to 1024M as a minimum in version 11.2 as a proactive measure to match the new defaults. No patch is required in order to increase the value of memory_target." This fact is also noted in Bug 14292825 - ORA-4031 in ASM as default memory parameters values for 11.2 ASM instances low (Doc ID 14292825.8). So it's best to increase the memory target for the ASM instances before the upgrade. (update 2015/12/03. Seems there's further changes to minimum amount, doc 1363369.1 suggest minimum value of 1536M.)
SQL> show parameter memory

NAME                                 TYPE        VALUE
------------------------------------ ----------- ------------------------------
memory_max_target                    big integer 272M
memory_target                        big integer 272M
SQL> alter system set memory_max_target=1024M scope=spfile sid='*';
System altered.

SQL> alter system set memory_target=1024M scope=spfile sid='*';
System altered.
Restart cluster stack on each node
crsctl stop cluster -n rhel6m1
crsctl start cluster -n rhel6m1

SQL> show parameter memory
NAME                                 TYPE        VALUE
------------------------------------ ----------- ------------------------------
memory_max_target                    big integer 1G
memory_target                        big integer 1G
Use both RACCheck pre upgrade check
./raccheck -u -o pre
and cluvfy to check the upgrade readiness to 11.2.0.4. Cluvfy output is shown below. Since the system is RHEL 6 the pdksh issue is there and cluvfy fails. Manually verify that all other checks have passed except for pdksh package check.
[grid@rhel6m1 grid]$ ./runcluvfy.sh stage -pre crsinst -upgrade -n rhel6m1,rhel6m2 -rolling -src_crshome /opt/app/11.2.0/grid -dest_crshome /opt/app/11.2.0/grid4 -dest_version 11.2.0.4.0 -fixup -fixupdir /home/grid/fixupscript -verbose

Performing pre-checks for cluster services setup

Checking node reachability...

Check: Node reachability from node "rhel6m1"
  Destination Node                      Reachable?
  ------------------------------------  ------------------------
  rhel6m1                               yes
  rhel6m2                               yes
Result: Node reachability check passed from node "rhel6m1"


Checking user equivalence...

Check: User equivalence for user "grid"
  Node Name                             Status
  ------------------------------------  ------------------------
  rhel6m2                               passed
  rhel6m1                               passed
Result: User equivalence check passed for user "grid"

Checking CRS user consistency
Result: CRS user consistency check successful

Checking node connectivity...

Checking hosts config file...
  Node Name                             Status
  ------------------------------------  ------------------------
  rhel6m2                               passed
  rhel6m1                               passed

Verification of the hosts config file successful


Interface information for node "rhel6m2"
 Name   IP Address      Subnet          Gateway         Def. Gateway    HW Address        MTU
 ------ --------------- --------------- --------------- --------------- ----------------- ------
 eth0   192.168.0.86    192.168.0.0     0.0.0.0         192.168.0.100   08:00:27:6A:D4:18 1500
 eth0   192.168.0.90    192.168.0.0     0.0.0.0         192.168.0.100   08:00:27:6A:D4:18 1500
 eth0   192.168.0.91    192.168.0.0     0.0.0.0         192.168.0.100   08:00:27:6A:D4:18 1500
 eth1   192.168.1.88    192.168.1.0     0.0.0.0         192.168.0.100   08:00:27:D4:AC:BE 1500
 eth1   169.254.160.199 169.254.0.0     0.0.0.0         192.168.0.100   08:00:27:D4:AC:BE 1500


Interface information for node "rhel6m1"
 Name   IP Address      Subnet          Gateway         Def. Gateway    HW Address        MTU
 ------ --------------- --------------- --------------- --------------- ----------------- ------
 eth0   192.168.0.85    192.168.0.0     0.0.0.0         192.168.0.100   08:00:27:44:1B:94 1500
 eth0   192.168.0.89    192.168.0.0     0.0.0.0         192.168.0.100   08:00:27:44:1B:94 1500
 eth1   192.168.1.87    192.168.1.0     0.0.0.0         192.168.0.100   08:00:27:E6:AB:27 1500
 eth1   169.254.31.63   169.254.0.0     0.0.0.0         192.168.0.100   08:00:27:E6:AB:27 1500


Check: Node connectivity for interface "eth0"
  Source                          Destination                     Connected?
  ------------------------------  ------------------------------  ----------------
  rhel6m2[192.168.0.86]           rhel6m2[192.168.0.90]           yes
  rhel6m2[192.168.0.86]           rhel6m2[192.168.0.91]           yes
  rhel6m2[192.168.0.86]           rhel6m1[192.168.0.85]           yes
  rhel6m2[192.168.0.86]           rhel6m1[192.168.0.89]           yes
  rhel6m2[192.168.0.90]           rhel6m2[192.168.0.91]           yes
  rhel6m2[192.168.0.90]           rhel6m1[192.168.0.85]           yes
  rhel6m2[192.168.0.90]           rhel6m1[192.168.0.89]           yes
  rhel6m2[192.168.0.91]           rhel6m1[192.168.0.85]           yes
  rhel6m2[192.168.0.91]           rhel6m1[192.168.0.89]           yes
  rhel6m1[192.168.0.85]           rhel6m1[192.168.0.89]           yes
Result: Node connectivity passed for interface "eth0"


Check: TCP connectivity of subnet "192.168.0.0"
  Source                          Destination                     Connected?
  ------------------------------  ------------------------------  ----------------
  rhel6m1:192.168.0.85            rhel6m2:192.168.0.86            passed
  rhel6m1:192.168.0.85            rhel6m2:192.168.0.90            passed
  rhel6m1:192.168.0.85            rhel6m2:192.168.0.91            passed
  rhel6m1:192.168.0.85            rhel6m1:192.168.0.89            passed
Result: TCP connectivity check passed for subnet "192.168.0.0"


Check: Node connectivity for interface "eth1"
  Source                          Destination                     Connected?
  ------------------------------  ------------------------------  ----------------
  rhel6m2[192.168.1.88]           rhel6m1[192.168.1.87]           yes
Result: Node connectivity passed for interface "eth1"


Check: TCP connectivity of subnet "192.168.1.0"
  Source                          Destination                     Connected?
  ------------------------------  ------------------------------  ----------------
  rhel6m1:192.168.1.87            rhel6m2:192.168.1.88            passed
Result: TCP connectivity check passed for subnet "192.168.1.0"

Checking subnet mask consistency...
Subnet mask consistency check passed for subnet "192.168.0.0".
Subnet mask consistency check passed for subnet "192.168.1.0".
Subnet mask consistency check passed.

Result: Node connectivity check passed

Checking multicast communication...

Checking subnet "192.168.0.0" for multicast communication with multicast group "230.0.1.0"...
Check of subnet "192.168.0.0" for multicast communication with multicast group "230.0.1.0" passed.

Checking subnet "192.168.1.0" for multicast communication with multicast group "230.0.1.0"...
Check of subnet "192.168.1.0" for multicast communication with multicast group "230.0.1.0" passed.

Check of multicast communication passed.

Checking OCR integrity...

OCR integrity check passed

Checking ASMLib configuration.
  Node Name                             Status
  ------------------------------------  ------------------------
  rhel6m2                               passed
  rhel6m1                               passed
Result: Check for ASMLib configuration passed.

Check: Total memory
  Node Name     Available                 Required                  Status
  ------------  ------------------------  ------------------------  ----------
  rhel6m2       3.7414GB (3923160.0KB)    1.5GB (1572864.0KB)       passed
  rhel6m1       3.7414GB (3923160.0KB)    1.5GB (1572864.0KB)       passed
Result: Total memory check passed

Check: Available memory
  Node Name     Available                 Required                  Status
  ------------  ------------------------  ------------------------  ----------
  rhel6m2       2.2198GB (2327612.0KB)    50MB (51200.0KB)          passed
  rhel6m1       2.2859GB (2396920.0KB)    50MB (51200.0KB)          passed
Result: Available memory check passed

Check: Swap space
  Node Name     Available                 Required                  Status
  ------------  ------------------------  ------------------------  ----------
  rhel6m2       4GB (4194296.0KB)         3.7414GB (3923160.0KB)    passed
  rhel6m1       4GB (4194296.0KB)         3.7414GB (3923160.0KB)    passed
Result: Swap space check passed

Check: Free disk space for "rhel6m2:/opt/app/11.2.0/grid4,rhel6m2:/tmp"
  Path              Node Name     Mount point   Available     Required      Status
  ----------------  ------------  ------------  ------------  ------------  ------------
  /opt/app/11.2.0/grid4  rhel6m2       /             13.3555GB     7.5GB         passed
  /tmp              rhel6m2       /             13.3555GB     7.5GB         passed
Result: Free disk space check passed for "rhel6m2:/opt/app/11.2.0/grid4,rhel6m2:/tmp"

Check: Free disk space for "rhel6m1:/opt/app/11.2.0/grid4,rhel6m1:/tmp"
  Path              Node Name     Mount point   Available     Required      Status
  ----------------  ------------  ------------  ------------  ------------  ------------
  /opt/app/11.2.0/grid4  rhel6m1       /             11.6013GB     7.5GB         passed
  /tmp              rhel6m1       /             11.6013GB     7.5GB         passed
Result: Free disk space check passed for "rhel6m1:/opt/app/11.2.0/grid4,rhel6m1:/tmp"

Check: User existence for "grid"
  Node Name     Status                    Comment
  ------------  ------------------------  ------------------------
  rhel6m2       passed                    exists(502)
  rhel6m1       passed                    exists(502)

Checking for multiple users with UID value 502
Result: Check for multiple users with UID value 502 passed
Result: User existence check passed for "grid"

Check: Group existence for "oinstall"
  Node Name     Status                    Comment
  ------------  ------------------------  ------------------------
  rhel6m2       passed                    exists
  rhel6m1       passed                    exists
Result: Group existence check passed for "oinstall"

Check: Membership of user "grid" in group "oinstall" [as Primary]
  Node Name         User Exists   Group Exists  User in Group  Primary       Status
  ----------------  ------------  ------------  ------------  ------------  ------------
  rhel6m2           yes           yes           yes           yes           passed
  rhel6m1           yes           yes           yes           yes           passed
Result: Membership check for user "grid" in group "oinstall" [as Primary] passed

Check: Run level
  Node Name     run level                 Required                  Status
  ------------  ------------------------  ------------------------  ----------
  rhel6m2       3                         3,5                       passed
  rhel6m1       3                         3,5                       passed
Result: Run level check passed

Check: Hard limits for "maximum open file descriptors"
  Node Name         Type          Available     Required      Status
  ----------------  ------------  ------------  ------------  ----------------
  rhel6m2           hard          65536         65536         passed
  rhel6m1           hard          65536         65536         passed
Result: Hard limits check passed for "maximum open file descriptors"

Check: Soft limits for "maximum open file descriptors"
  Node Name         Type          Available     Required      Status
  ----------------  ------------  ------------  ------------  ----------------
  rhel6m2           soft          1024          1024          passed
  rhel6m1           soft          1024          1024          passed
Result: Soft limits check passed for "maximum open file descriptors"

Check: Hard limits for "maximum user processes"
  Node Name         Type          Available     Required      Status
  ----------------  ------------  ------------  ------------  ----------------
  rhel6m2           hard          16384         16384         passed
  rhel6m1           hard          16384         16384         passed
Result: Hard limits check passed for "maximum user processes"

Check: Soft limits for "maximum user processes"
  Node Name         Type          Available     Required      Status
  ----------------  ------------  ------------  ------------  ----------------
  rhel6m2           soft          2047          2047          passed
  rhel6m1           soft          2047          2047          passed
Result: Soft limits check passed for "maximum user processes"

There are no oracle patches required for home "/opt/app/11.2.0/grid".

There are no oracle patches required for home "/opt/app/11.2.0/grid4".

Check: System architecture
  Node Name     Available                 Required                  Status
  ------------  ------------------------  ------------------------  ----------
  rhel6m2       x86_64                    x86_64                    passed
  rhel6m1       x86_64                    x86_64                    passed
Result: System architecture check passed

Check: Kernel version
  Node Name     Available                 Required                  Status
  ------------  ------------------------  ------------------------  ----------
  rhel6m2       2.6.32-220.el6.x86_64     2.6.9                     passed
  rhel6m1       2.6.32-220.el6.x86_64     2.6.9                     passed
Result: Kernel version check passed

Check: Kernel parameter for "semmsl"
  Node Name         Current       Configured    Required      Status        Comment
  ----------------  ------------  ------------  ------------  ------------  ------------
  rhel6m2           3010          3010          250           passed
  rhel6m1           3010          3010          250           passed
Result: Kernel parameter check passed for "semmsl"

Check: Kernel parameter for "semmns"
  Node Name         Current       Configured    Required      Status        Comment
  ----------------  ------------  ------------  ------------  ------------  ------------
  rhel6m2           385280        385280        32000         passed
  rhel6m1           385280        385280        32000         passed
Result: Kernel parameter check passed for "semmns"

Check: Kernel parameter for "semopm"
  Node Name         Current       Configured    Required      Status        Comment
  ----------------  ------------  ------------  ------------  ------------  ------------
  rhel6m2           3010          3010          100           passed
  rhel6m1           3010          3010          100           passed
Result: Kernel parameter check passed for "semopm"

Check: Kernel parameter for "semmni"
  Node Name         Current       Configured    Required      Status        Comment
  ----------------  ------------  ------------  ------------  ------------  ------------
  rhel6m2           128           128           128           passed
  rhel6m1           128           128           128           passed
Result: Kernel parameter check passed for "semmni"

Check: Kernel parameter for "shmmax"
  Node Name         Current       Configured    Required      Status        Comment
  ----------------  ------------  ------------  ------------  ------------  ------------
  rhel6m2           68719476736   68719476736   2008657920    passed
  rhel6m1           68719476736   68719476736   2008657920    passed
Result: Kernel parameter check passed for "shmmax"

Check: Kernel parameter for "shmmni"
  Node Name         Current       Configured    Required      Status        Comment
  ----------------  ------------  ------------  ------------  ------------  ------------
  rhel6m2           4096          4096          4096          passed
  rhel6m1           4096          4096          4096          passed
Result: Kernel parameter check passed for "shmmni"

Check: Kernel parameter for "shmall"
  Node Name         Current       Configured    Required      Status        Comment
  ----------------  ------------  ------------  ------------  ------------  ------------
  rhel6m2           4294967296    4294967296    2097152       passed
  rhel6m1           4294967296    4294967296    2097152       passed
Result: Kernel parameter check passed for "shmall"

Check: Kernel parameter for "file-max"
  Node Name         Current       Configured    Required      Status        Comment
  ----------------  ------------  ------------  ------------  ------------  ------------
  rhel6m2           6815744       6815744       6815744       passed
  rhel6m1           6815744       6815744       6815744       passed
Result: Kernel parameter check passed for "file-max"

Check: Kernel parameter for "ip_local_port_range"
  Node Name         Current       Configured    Required      Status        Comment
  ----------------  ------------  ------------  ------------  ------------  ------------
  rhel6m2           between 9000.0 & 65500.0  between 9000.0 & 65500.0  between 9000.0 & 65500.0  passed
  rhel6m1           between 9000.0 & 65500.0  between 9000.0 & 65500.0  between 9000.0 & 65500.0  passed
Result: Kernel parameter check passed for "ip_local_port_range"

Check: Kernel parameter for "rmem_default"
  Node Name         Current       Configured    Required      Status        Comment
  ----------------  ------------  ------------  ------------  ------------  ------------
  rhel6m2           4194304       4194304       262144        passed
  rhel6m1           4194304       4194304       262144        passed
Result: Kernel parameter check passed for "rmem_default"

Check: Kernel parameter for "rmem_max"
  Node Name         Current       Configured    Required      Status        Comment
  ----------------  ------------  ------------  ------------  ------------  ------------
  rhel6m2           4194304       4194304       4194304       passed
  rhel6m1           4194304       4194304       4194304       passed
Result: Kernel parameter check passed for "rmem_max"

Check: Kernel parameter for "wmem_default"
  Node Name         Current       Configured    Required      Status        Comment
  ----------------  ------------  ------------  ------------  ------------  ------------
  rhel6m2           1048576       1048576       262144        passed
  rhel6m1           1048576       1048576       262144        passed
Result: Kernel parameter check passed for "wmem_default"

Check: Kernel parameter for "wmem_max"
  Node Name         Current       Configured    Required      Status        Comment
  ----------------  ------------  ------------  ------------  ------------  ------------
  rhel6m2           2097152       2097152       1048576       passed
  rhel6m1           2097152       2097152       1048576       passed
Result: Kernel parameter check passed for "wmem_max"

Check: Kernel parameter for "aio-max-nr"
  Node Name         Current       Configured    Required      Status        Comment
  ----------------  ------------  ------------  ------------  ------------  ------------
  rhel6m2           3145728       3145728       1048576       passed
  rhel6m1           3145728       3145728       1048576       passed
Result: Kernel parameter check passed for "aio-max-nr"

Check: Package existence for "make"
  Node Name     Available                 Required                  Status
  ------------  ------------------------  ------------------------  ----------
  rhel6m2       make-3.81-19.el6          make-3.80                 passed
  rhel6m1       make-3.81-19.el6          make-3.80                 passed
Result: Package existence check passed for "make"

Check: Package existence for "binutils"
  Node Name     Available                 Required                  Status
  ------------  ------------------------  ------------------------  ----------
  rhel6m2       binutils-2.20.51.0.2-5.28.el6  binutils-2.15.92.0.2      passed
  rhel6m1       binutils-2.20.51.0.2-5.28.el6  binutils-2.15.92.0.2      passed
Result: Package existence check passed for "binutils"

Check: Package existence for "gcc(x86_64)"
  Node Name     Available                 Required                  Status
  ------------  ------------------------  ------------------------  ----------
  rhel6m2       gcc(x86_64)-4.4.6-3.el6   gcc(x86_64)-3.4.6         passed
  rhel6m1       gcc(x86_64)-4.4.6-3.el6   gcc(x86_64)-3.4.6         passed
Result: Package existence check passed for "gcc(x86_64)"

Check: Package existence for "libaio(x86_64)"
  Node Name     Available                 Required                  Status
  ------------  ------------------------  ------------------------  ----------
  rhel6m2       libaio(x86_64)-0.3.107-10.el6  libaio(x86_64)-0.3.105    passed
  rhel6m1       libaio(x86_64)-0.3.107-10.el6  libaio(x86_64)-0.3.105    passed
Result: Package existence check passed for "libaio(x86_64)"

Check: Package existence for "glibc(x86_64)"
  Node Name     Available                 Required                  Status
  ------------  ------------------------  ------------------------  ----------
  rhel6m2       glibc(x86_64)-2.12-1.47.el6  glibc(x86_64)-2.3.4-2.41  passed
  rhel6m1       glibc(x86_64)-2.12-1.47.el6  glibc(x86_64)-2.3.4-2.41  passed
Result: Package existence check passed for "glibc(x86_64)"

Check: Package existence for "compat-libstdc++-33(x86_64)"
  Node Name     Available                 Required                  Status
  ------------  ------------------------  ------------------------  ----------
  rhel6m2       compat-libstdc++-33(x86_64)-3.2.3-69.el6  compat-libstdc++-33(x86_64)-3.2.3  passed
  rhel6m1       compat-libstdc++-33(x86_64)-3.2.3-69.el6  compat-libstdc++-33(x86_64)-3.2.3  passed
Result: Package existence check passed for "compat-libstdc++-33(x86_64)"

Check: Package existence for "elfutils-libelf(x86_64)"
  Node Name     Available                 Required                  Status
  ------------  ------------------------  ------------------------  ----------
  rhel6m2       elfutils-libelf(x86_64)-0.152-1.el6  elfutils-libelf(x86_64)-0.97  passed
  rhel6m1       elfutils-libelf(x86_64)-0.152-1.el6  elfutils-libelf(x86_64)-0.97  passed
Result: Package existence check passed for "elfutils-libelf(x86_64)"

Check: Package existence for "elfutils-libelf-devel"
  Node Name     Available                 Required                  Status
  ------------  ------------------------  ------------------------  ----------
  rhel6m2       elfutils-libelf-devel-0.152-1.el6  elfutils-libelf-devel-0.97  passed
  rhel6m1       elfutils-libelf-devel-0.152-1.el6  elfutils-libelf-devel-0.97  passed
Result: Package existence check passed for "elfutils-libelf-devel"

Check: Package existence for "glibc-common"
  Node Name     Available                 Required                  Status
  ------------  ------------------------  ------------------------  ----------
  rhel6m2       glibc-common-2.12-1.47.el6  glibc-common-2.3.4        passed
  rhel6m1       glibc-common-2.12-1.47.el6  glibc-common-2.3.4        passed
Result: Package existence check passed for "glibc-common"

Check: Package existence for "glibc-devel(x86_64)"
  Node Name     Available                 Required                  Status
  ------------  ------------------------  ------------------------  ----------
  rhel6m2       glibc-devel(x86_64)-2.12-1.47.el6  glibc-devel(x86_64)-2.3.4  passed
  rhel6m1       glibc-devel(x86_64)-2.12-1.47.el6  glibc-devel(x86_64)-2.3.4  passed
Result: Package existence check passed for "glibc-devel(x86_64)"

Check: Package existence for "glibc-headers"
  Node Name     Available                 Required                  Status
  ------------  ------------------------  ------------------------  ----------
  rhel6m2       glibc-headers-2.12-1.47.el6  glibc-headers-2.3.4       passed
  rhel6m1       glibc-headers-2.12-1.47.el6  glibc-headers-2.3.4       passed
Result: Package existence check passed for "glibc-headers"

Check: Package existence for "gcc-c++(x86_64)"
  Node Name     Available                 Required                  Status
  ------------  ------------------------  ------------------------  ----------
  rhel6m2       gcc-c++(x86_64)-4.4.6-3.el6  gcc-c++(x86_64)-3.4.6     passed
  rhel6m1       gcc-c++(x86_64)-4.4.6-3.el6  gcc-c++(x86_64)-3.4.6     passed
Result: Package existence check passed for "gcc-c++(x86_64)"

Check: Package existence for "libaio-devel(x86_64)"
  Node Name     Available                 Required                  Status
  ------------  ------------------------  ------------------------  ----------
  rhel6m2       libaio-devel(x86_64)-0.3.107-10.el6  libaio-devel(x86_64)-0.3.105  passed
  rhel6m1       libaio-devel(x86_64)-0.3.107-10.el6  libaio-devel(x86_64)-0.3.105  passed
Result: Package existence check passed for "libaio-devel(x86_64)"

Check: Package existence for "libgcc(x86_64)"
  Node Name     Available                 Required                  Status
  ------------  ------------------------  ------------------------  ----------
  rhel6m2       libgcc(x86_64)-4.4.6-3.el6  libgcc(x86_64)-3.4.6      passed
  rhel6m1       libgcc(x86_64)-4.4.6-3.el6  libgcc(x86_64)-3.4.6      passed
Result: Package existence check passed for "libgcc(x86_64)"

Check: Package existence for "libstdc++(x86_64)"
  Node Name     Available                 Required                  Status
  ------------  ------------------------  ------------------------  ----------
  rhel6m2       libstdc++(x86_64)-4.4.6-3.el6  libstdc++(x86_64)-3.4.6   passed
  rhel6m1       libstdc++(x86_64)-4.4.6-3.el6  libstdc++(x86_64)-3.4.6   passed
Result: Package existence check passed for "libstdc++(x86_64)"

Check: Package existence for "libstdc++-devel(x86_64)"
  Node Name     Available                 Required                  Status
  ------------  ------------------------  ------------------------  ----------
  rhel6m2       libstdc++-devel(x86_64)-4.4.6-3.el6  libstdc++-devel(x86_64)-3.4.6  passed
  rhel6m1       libstdc++-devel(x86_64)-4.4.6-3.el6  libstdc++-devel(x86_64)-3.4.6  passed
Result: Package existence check passed for "libstdc++-devel(x86_64)"

Check: Package existence for "sysstat"
  Node Name     Available                 Required                  Status
  ------------  ------------------------  ------------------------  ----------
  rhel6m2       sysstat-9.0.4-18.el6      sysstat-5.0.5             passed
  rhel6m1       sysstat-9.0.4-18.el6      sysstat-5.0.5             passed
Result: Package existence check passed for "sysstat"

Check: Package existence for "pdksh"
  Node Name     Available                 Required                  Status
  ------------  ------------------------  ------------------------  ----------
  rhel6m2       missing                   pdksh-5.2.14              failed
  rhel6m1       missing                   pdksh-5.2.14              failed
Result: Package existence check failed for "pdksh"

Check: Package existence for "expat(x86_64)"
  Node Name     Available                 Required                  Status
  ------------  ------------------------  ------------------------  ----------
  rhel6m2       expat(x86_64)-2.0.1-9.1.el6  expat(x86_64)-1.95.7      passed
  rhel6m1       expat(x86_64)-2.0.1-9.1.el6  expat(x86_64)-1.95.7      passed
Result: Package existence check passed for "expat(x86_64)"

Checking for multiple users with UID value 0
Result: Check for multiple users with UID value 0 passed

Check: Current group ID
Result: Current group ID check passed

Starting check for consistency of primary group of root user
  Node Name                             Status
  ------------------------------------  ------------------------
  rhel6m2                               passed
  rhel6m1                               passed

Check for consistency of root user's primary group passed

Check: Package existence for "cvuqdisk"
  Node Name     Available                 Required                  Status
  ------------  ------------------------  ------------------------  ----------
  rhel6m2       cvuqdisk-1.0.9-1          cvuqdisk-1.0.9-1          passed
  rhel6m1       cvuqdisk-1.0.9-1          cvuqdisk-1.0.9-1          passed
Result: Package existence check passed for "cvuqdisk"

Starting Clock synchronization checks using Network Time Protocol(NTP)...

NTP Configuration file check started...
Network Time Protocol(NTP) configuration file not found on any of the nodes. Oracle Cluster Time Synchronization Service(CTSS) can be used instead of NTP for time synchronization on the cluster nodes
No NTP Daemons or Services were found to be running

Result: Clock synchronization check using Network Time Protocol(NTP) passed

Checking Core file name pattern consistency...
Core file name pattern consistency check passed.

Checking to make sure user "grid" is not in "root" group
  Node Name     Status                    Comment
  ------------  ------------------------  ------------------------
  rhel6m2       passed                    does not exist
  rhel6m1       passed                    does not exist
Result: User "grid" is not part of "root" group. Check passed

Check default user file creation mask
  Node Name     Available                 Required                  Comment
  ------------  ------------------------  ------------------------  ----------
  rhel6m2       0022                      0022                      passed
  rhel6m1       0022                      0022                      passed
Result: Default user file creation mask check passed
Checking consistency of file "/etc/resolv.conf" across nodes

Checking the file "/etc/resolv.conf" to make sure only one of domain and search entries is defined
File "/etc/resolv.conf" does not have both domain and search entries defined
Checking if domain entry in file "/etc/resolv.conf" is consistent across the nodes...
domain entry in file "/etc/resolv.conf" is consistent across nodes
Checking if search entry in file "/etc/resolv.conf" is consistent across the nodes...
search entry in file "/etc/resolv.conf" is consistent across nodes
Checking file "/etc/resolv.conf" to make sure that only one search entry is defined
All nodes have one search entry defined in file "/etc/resolv.conf"
Checking all nodes to make sure that search entry is "domain.net" as found on node "rhel6m2"
All nodes of the cluster have same value for 'search'
Checking DNS response time for an unreachable node
  Node Name                             Status
  ------------------------------------  ------------------------
  rhel6m2                               passed
  rhel6m1                               passed
The DNS response time for an unreachable node is within acceptable limit on all nodes

File "/etc/resolv.conf" is consistent across nodes


UDev attributes check for OCR locations started...
Result: UDev attributes check passed for OCR locations


UDev attributes check for Voting Disk locations started...
Result: UDev attributes check passed for Voting Disk locations

Check: Time zone consistency
Result: Time zone consistency check passed
Checking VIP configuration.
Checking VIP Subnet configuration.
Check for VIP Subnet configuration passed.
Checking VIP reachability
Check for VIP reachability passed.

Checking Oracle Cluster Voting Disk configuration...

ASM Running check passed. ASM is running on all specified nodes

Oracle Cluster Voting Disk configuration check passed

Clusterware version consistency passed

Pre-check for cluster services setup was unsuccessful on all the nodes.
Manually create the new out of place directory to be used for the 11.2.0.4 grid installation. Since the parent directories are owned by root the grid user installation process will not be able to create the new directory.
Create the directory on all the nodes of the cluster
# cd /opt/app/11.2.0
# mkdir grid4
# chown grid:oinstall grid4
# chmod 775 grid4



Start the GI upgrade by executing runInstaller.

When prompted execute the rootupgrade.sh script on each node.
It was also observed that COST related files were copied to the new GI HOME during the upgrade and doesn't require any manual intervention. However once the upgrade is completed and before the old GI home is removed change the wallet location in the sqlnet.ora file inside $ORACLE_HOME/network/admin. Upgrade is a rolling upgrade and script will bring down cluster stack on the node and once upgrade is finish bring it back up. Other nodes are open for use during this time.
[root@rhel6m1 grid]# /opt/app/11.2.0/grid4/rootupgrade.sh
Performing root user operation for Oracle 11g

The following environment variables are set as:
    ORACLE_OWNER= grid
    ORACLE_HOME=  /opt/app/11.2.0/grid4

Enter the full pathname of the local bin directory: [/usr/local/bin]:
The contents of "dbhome" have not changed. No need to overwrite.
The contents of "oraenv" have not changed. No need to overwrite.
The contents of "coraenv" have not changed. No need to overwrite.

Entries will be added to the /etc/oratab file as needed by
Database Configuration Assistant when a database is created
Finished running generic part of root script.
Now product-specific root actions will be performed.
Using configuration parameter file: /opt/app/11.2.0/grid4/crs/install/crsconfig_params
Creating trace directory
Installing Trace File Analyzer

ASM upgrade has started on first node.

CRS-2791: Starting shutdown of Oracle High Availability Services-managed resources on 'rhel6m1'
CRS-2673: Attempting to stop 'ora.crsd' on 'rhel6m1'
CRS-2790: Starting shutdown of Cluster Ready Services-managed resources on 'rhel6m1'
CRS-2673: Attempting to stop 'ora.CLUSTER_DG.dg' on 'rhel6m1'
CRS-2673: Attempting to stop 'ora.registry.acfs' on 'rhel6m1'
CRS-2673: Attempting to stop 'ora.std11g2.asa.domain.net.svc' on 'rhel6m1'
CRS-2677: Stop of 'ora.std11g2.asa.domain.net.svc' on 'rhel6m1' succeeded
CRS-2673: Attempting to stop 'ora.std11g2.db' on 'rhel6m1'
CRS-2673: Attempting to stop 'ora.LISTENER.lsnr' on 'rhel6m1'
CRS-2673: Attempting to stop 'ora.MYLISTENER.lsnr' on 'rhel6m1'
CRS-2677: Stop of 'ora.LISTENER.lsnr' on 'rhel6m1' succeeded
CRS-2677: Stop of 'ora.MYLISTENER.lsnr' on 'rhel6m1' succeeded
CRS-2673: Attempting to stop 'ora.rhel6m1.vip' on 'rhel6m1'
CRS-2677: Stop of 'ora.registry.acfs' on 'rhel6m1' succeeded
CRS-2677: Stop of 'ora.std11g2.db' on 'rhel6m1' succeeded
CRS-2673: Attempting to stop 'ora.DATA.dg' on 'rhel6m1'
CRS-2673: Attempting to stop 'ora.FLASH.dg' on 'rhel6m1'
CRS-2677: Stop of 'ora.rhel6m1.vip' on 'rhel6m1' succeeded
CRS-2672: Attempting to start 'ora.rhel6m1.vip' on 'rhel6m2'
CRS-2677: Stop of 'ora.DATA.dg' on 'rhel6m1' succeeded
CRS-2677: Stop of 'ora.FLASH.dg' on 'rhel6m1' succeeded
CRS-2676: Start of 'ora.rhel6m1.vip' on 'rhel6m2' succeeded
CRS-2677: Stop of 'ora.CLUSTER_DG.dg' on 'rhel6m1' succeeded
CRS-2673: Attempting to stop 'ora.asm' on 'rhel6m1'
CRS-2677: Stop of 'ora.asm' on 'rhel6m1' succeeded
CRS-2673: Attempting to stop 'ora.ons' on 'rhel6m1'
CRS-2677: Stop of 'ora.ons' on 'rhel6m1' succeeded
CRS-2673: Attempting to stop 'ora.net1.network' on 'rhel6m1'
CRS-2677: Stop of 'ora.net1.network' on 'rhel6m1' succeeded
CRS-2792: Shutdown of Cluster Ready Services-managed resources on 'rhel6m1' has completed
CRS-2677: Stop of 'ora.crsd' on 'rhel6m1' succeeded
CRS-2673: Attempting to stop 'ora.crf' on 'rhel6m1'
CRS-2673: Attempting to stop 'ora.ctssd' on 'rhel6m1'
CRS-2673: Attempting to stop 'ora.evmd' on 'rhel6m1'
CRS-2673: Attempting to stop 'ora.asm' on 'rhel6m1'
CRS-2673: Attempting to stop 'ora.mdnsd' on 'rhel6m1'
CRS-2673: Attempting to stop 'ora.drivers.acfs' on 'rhel6m1'
CRS-2677: Stop of 'ora.crf' on 'rhel6m1' succeeded
CRS-2677: Stop of 'ora.evmd' on 'rhel6m1' succeeded
CRS-2677: Stop of 'ora.mdnsd' on 'rhel6m1' succeeded
CRS-2677: Stop of 'ora.ctssd' on 'rhel6m1' succeeded
CRS-2677: Stop of 'ora.asm' on 'rhel6m1' succeeded
CRS-2673: Attempting to stop 'ora.cluster_interconnect.haip' on 'rhel6m1'
CRS-2677: Stop of 'ora.cluster_interconnect.haip' on 'rhel6m1' succeeded
CRS-2673: Attempting to stop 'ora.cssd' on 'rhel6m1'
CRS-2677: Stop of 'ora.drivers.acfs' on 'rhel6m1' succeeded
CRS-2677: Stop of 'ora.cssd' on 'rhel6m1' succeeded
CRS-2673: Attempting to stop 'ora.gipcd' on 'rhel6m1'
CRS-2677: Stop of 'ora.gipcd' on 'rhel6m1' succeeded
CRS-2673: Attempting to stop 'ora.gpnpd' on 'rhel6m1'
CRS-2677: Stop of 'ora.gpnpd' on 'rhel6m1' succeeded
CRS-2793: Shutdown of Oracle High Availability Services-managed resources on 'rhel6m1' has completed
CRS-4133: Oracle High Availability Services has been stopped.
OLR initialization - successful
Replacing Clusterware entries in upstart
clscfg: EXISTING configuration version 5 detected.
clscfg: version 5 is 11g Release 2.
Successfully accumulated necessary OCR keys.
Creating OCR keys for user 'root', privgrp 'root'..
Operation successful.
Configure Oracle Grid Infrastructure for a Cluster ... succeeded
Active version still remain the lower version until all the nodes are upgraded.
[grid@rhel6m1 ~]$ crsctl query crs releaseversion
Oracle High Availability Services release version on the local node is [11.2.0.4.0]
[grid@rhel6m1 ~]$ crsctl query crs softwareversion
Oracle Clusterware version on node [rhel6m1] is [11.2.0.4.0]
[grid@rhel6m1 ~]$ crsctl query crs activeversion
Oracle Clusterware active version on the cluster is [11.2.0.3.0]
At this time /etc/oratab will also be upgraded to reflect the new GI_HOME
[grid@rhel6m1 ~]$ cat /etc/oratab
#Backup file is  /opt/app/oracle/product/11.2.0/dbhome_2/srvm/admin/oratab.bak.rhel6m1 line added by Agent
#
+ASM1:/opt/app/11.2.0/grid4:N           # line added by Agent
Executing on the last node will result in unusual amount of logging (compared to previous upgrades) after the Oracle Clusterware operating version was successfully set to 11.2.0.4.0 line. Not all the lines of this out put is shown below.
[root@rhel6m2 grid]#  /opt/app/11.2.0/grid4/rootupgrade.sh
Performing root user operation for Oracle 11g

The following environment variables are set as:
    ORACLE_OWNER= grid
    ORACLE_HOME=  /opt/app/11.2.0/grid4

Enter the full pathname of the local bin directory: [/usr/local/bin]:
The contents of "dbhome" have not changed. No need to overwrite.
The contents of "oraenv" have not changed. No need to overwrite.
The contents of "coraenv" have not changed. No need to overwrite.

Entries will be added to the /etc/oratab file as needed by
Database Configuration Assistant when a database is created
Finished running generic part of root script.
Now product-specific root actions will be performed.
Using configuration parameter file: /opt/app/11.2.0/grid4/crs/install/crsconfig_params
Creating trace directory
Installing Trace File Analyzer
CRS-2791: Starting shutdown of Oracle High Availability Services-managed resources on 'rhel6m2'
CRS-2673: Attempting to stop 'ora.crsd' on 'rhel6m2'
CRS-2790: Starting shutdown of Cluster Ready Services-managed resources on 'rhel6m2'
CRS-2673: Attempting to stop 'ora.CLUSTER_DG.dg' on 'rhel6m2'
CRS-2673: Attempting to stop 'ora.registry.acfs' on 'rhel6m2'
CRS-2673: Attempting to stop 'ora.std11g2.asa.domain.net.svc' on 'rhel6m2'
CRS-2677: Stop of 'ora.std11g2.asa.domain.net.svc' on 'rhel6m2' succeeded
CRS-2673: Attempting to stop 'ora.std11g2.db' on 'rhel6m2'
CRS-2673: Attempting to stop 'ora.LISTENER.lsnr' on 'rhel6m2'
CRS-2673: Attempting to stop 'ora.MYLISTENER.lsnr' on 'rhel6m2'
CRS-2677: Stop of 'ora.LISTENER.lsnr' on 'rhel6m2' succeeded
CRS-2677: Stop of 'ora.MYLISTENER.lsnr' on 'rhel6m2' succeeded
CRS-2673: Attempting to stop 'ora.rhel6m2.vip' on 'rhel6m2'
CRS-2677: Stop of 'ora.registry.acfs' on 'rhel6m2' succeeded
CRS-2677: Stop of 'ora.std11g2.db' on 'rhel6m2' succeeded
CRS-2673: Attempting to stop 'ora.DATA.dg' on 'rhel6m2'
CRS-2673: Attempting to stop 'ora.FLASH.dg' on 'rhel6m2'
CRS-2677: Stop of 'ora.rhel6m2.vip' on 'rhel6m2' succeeded
CRS-2672: Attempting to start 'ora.rhel6m2.vip' on 'rhel6m2'
CRS-2677: Stop of 'ora.DATA.dg' on 'rhel6m2' succeeded
CRS-2677: Stop of 'ora.FLASH.dg' on 'rhel6m2' succeeded
CRS-2676: Start of 'ora.rhel6m2.vip' on 'rhel6m2' succeeded
CRS-2677: Stop of 'ora.CLUSTER_DG.dg' on 'rhel6m2' succeeded
CRS-2673: Attempting to stop 'ora.asm' on 'rhel6m2'
CRS-2677: Stop of 'ora.asm' on 'rhel6m2' succeeded
CRS-2673: Attempting to stop 'ora.ons' on 'rhel6m2'
CRS-2677: Stop of 'ora.ons' on 'rhel6m2' succeeded
CRS-2673: Attempting to stop 'ora.net1.network' on 'rhel6m2'
CRS-2677: Stop of 'ora.net1.network' on 'rhel6m2' succeeded
CRS-2792: Shutdown of Cluster Ready Services-managed resources on 'rhel6m2' has completed
CRS-2677: Stop of 'ora.crsd' on 'rhel6m2' succeeded
CRS-2673: Attempting to stop 'ora.crf' on 'rhel6m2'
CRS-2673: Attempting to stop 'ora.ctssd' on 'rhel6m2'
CRS-2673: Attempting to stop 'ora.evmd' on 'rhel6m2'
CRS-2673: Attempting to stop 'ora.asm' on 'rhel6m2'
CRS-2673: Attempting to stop 'ora.mdnsd' on 'rhel6m2'
CRS-2673: Attempting to stop 'ora.drivers.acfs' on 'rhel6m2'
CRS-2677: Stop of 'ora.crf' on 'rhel6m2' succeeded
CRS-2677: Stop of 'ora.evmd' on 'rhel6m2' succeeded
CRS-2677: Stop of 'ora.mdnsd' on 'rhel6m2' succeeded
CRS-2677: Stop of 'ora.ctssd' on 'rhel6m2' succeeded
CRS-2677: Stop of 'ora.asm' on 'rhel6m2' succeeded
CRS-2673: Attempting to stop 'ora.cluster_interconnect.haip' on 'rhel6m2'
CRS-2677: Stop of 'ora.cluster_interconnect.haip' on 'rhel6m2' succeeded
CRS-2673: Attempting to stop 'ora.cssd' on 'rhel6m2'
CRS-2677: Stop of 'ora.drivers.acfs' on 'rhel6m2' succeeded
CRS-2677: Stop of 'ora.cssd' on 'rhel6m2' succeeded
CRS-2673: Attempting to stop 'ora.gipcd' on 'rhel6m2'
CRS-2677: Stop of 'ora.gipcd' on 'rhel6m2' succeeded
CRS-2673: Attempting to stop 'ora.gpnpd' on 'rhel6m2'
CRS-2677: Stop of 'ora.gpnpd' on 'rhel6m2' succeeded
CRS-2793: Shutdown of Oracle High Availability Services-managed resources on 'rhel6m2' has completed
CRS-4133: Oracle High Availability Services has been stopped.
OLR initialization - successful
Replacing Clusterware entries in upstart
clscfg: EXISTING configuration version 5 detected.
clscfg: version 5 is 11g Release 2.
Successfully accumulated necessary OCR keys.
Creating OCR keys for user 'root', privgrp 'root'..
Operation successful.
Started to upgrade the Oracle Clusterware. This operation may take a few minutes.
Started to upgrade the CSS.
Started to upgrade the CRS.
The CRS was successfully upgraded.
Successfully upgraded the Oracle Clusterware.
Oracle Clusterware operating version was successfully set to 11.2.0.4.0
...
...
[main] [ 2013-09-06 15:03:50.561 BST ] [RuntimeExec.runCommand:170]  RunTimeExec: error>
[main] [ 2013-09-06 15:03:50.562 BST ] [RuntimeExec.runCommand:192]  Returning from RunTimeExec.runCommand
[main] [ 2013-09-06 15:03:50.572 BST ] [Utils.getLocalHost:481]  Hostname retrieved: rhel6m2.domain.net, returned: rhel6m2
[main] [ 2013-09-06 15:03:50.574 BST ] [Utils.getLocalHost:481]  Hostname retrieved: rhel6m2.domain.net, returned: rhel6m2
[main] [ 2013-09-06 15:03:51.585 BST ] [SQLEngine.done:2189]  Done called

ASM upgrade has finished on last node.

Configure Oracle Grid Infrastructure for a Cluster ... succeeded
Once the last node is also upgraded the active version is updated
[grid@rhel6m2 ~]$ crsctl query crs activeversion
Oracle Clusterware active version on the cluster is [11.2.0.4.0]
Click OK on the configuration dialog to continue with rest of the upgrade steps.

If the 11.2.0.3 GI HOME is no longer needed deatch it from the inventory and remove
./runInstaller -silent -detachHome ORACLE_HOME=/opt/app/11.2.0/grid 
rm -rf /opt/app/11.2.0/grid
To check the post upgrade health of the cluster use
cluvfy stage -post crsinst -n rhel6m1,rhel6m2 -verbose
or/and
./raccheck -u -o post
.
This conclude the grid infrastructure part of the RAC upgrade. Next step is to upgrade the database software and finally the databsae.

Related Posts
Upgrading from 10.2.0.4 to 10.2.0.5 (Clusterware, RAC, ASM)
Upgrade from 10.2.0.5 to 11.2.0.3 (Clusterware, RAC, ASM)
Upgrade from 11.1.0.7 to 11.2.0.3 (Clusterware, ASM & RAC)
Upgrade from 11.1.0.7 to 11.2.0.4 (Clusterware, ASM & RAC)
Upgrading from 11.1.0.7 to 11.2.0.3 with Transient Logical Standby
Upgrading from 11.2.0.1 to 11.2.0.3 with in-place upgrade for RAC
In-place upgrade from 11.2.0.2 to 11.2.0.3
Upgrading from 11.2.0.2 to 11.2.0.3 with Physical Standby - 1
Upgrading from 11.2.0.2 to 11.2.0.3 with Physical Standby - 2
Upgrading from 11gR2 (11.2.0.3) to 12c (12.1.0.1) Grid Infrastructure
Upgrading RAC from 11.2.0.4 to 12.1.0.2 - Grid Infrastructure