Moving ASMLib disk to block devices (Non-ASMLib) in 11gR2 Standalone

This post list steps to move ASMLib devices to block devices for a standalone 11gR2 installation with role separation created earlier.
1. As the grid user login to ASM instance and create a pfile from the ASM spfile

sqlplus  / as sysasm
SQL> create pfile='/home/grid/asmpfile.ora' from spfile;

File created.

$ more asmpfile.ora
+ASM.__oracle_base='/opt/app/oracle'#ORACLE_BASE set from in memory value
+ASM.asm_diskgroups='FLASH','DATA'#Manual Mount
*.asm_power_limit=1
*.diagnostic_dest='/opt/app/oracle'
*.instance_type='asm'
*.large_pool_size=12M
*.remote_login_passwordfile='EXCLUSIVE'

2. Current ASMLib disks are

SQL> select name,label,path from v$asm_disk;

NAME    LABEL   PATH
------- ------- -----------
DATA    DATA    ORCL:DATA
FLASH   FLASH   ORCL:FLASH

3. Before ASMLib disks could be moved the database must be stopped. This is required as ASM diskgroups will be dismounted later on.

srvctl stop database -d stdalone

4. Dismount and mount the flash diskgroup and monitor the ASM alert log which will show the ASM disk(s) used with the diskgroup. This information is useful to verify if the new block device disk assignment is working.

SQL> alter diskgroup flash dismount;

Diskgroup altered.

From the alert log

NOTE: cache deleting context for group FLASH 2/0x9328489f
Thu Jun 14 11:57:49 2012
NOTE: diskgroup resource ora.FLASH.dg is offline
SUCCESS: alter diskgroup flash dismount
NOTE: diskgroup resource ora.FLASH.dg is updated

When mounting

SQL> alter diskgroup flash mount

From alert log

NOTE: cache registered group FLASH number=2 incarn=0xed0848a6
NOTE: cache began mount (first) of group FLASH number=2 incarn=0xed0848a6
NOTE: Assigning number (2,0) to disk (ORCL:FLASH)
Thu Jun 14 11:58:18 2012
NOTE: GMON heartbeating for grp 2

5. Identify the block device used to create the ASMLib disk by querying the ASMLib disk

# /etc/init.d/oracleasm querydisk -d FLASH
Disk "FLASH" is a valid ASM disk on device [8, 33]

Block device could be identified with

ls -l /dev/sd*
brw-r----- 1 root disk 8,  0 Jun 14 11:47 /dev/sda
brw-r----- 1 root disk 8,  1 Jun 14 11:48 /dev/sda1
brw-r----- 1 root disk 8,  2 Jun 14 11:47 /dev/sda2
brw-r----- 1 root disk 8,  3 Jun 14 11:48 /dev/sda3
brw-r----- 1 root disk 8, 16 Jun 14 11:47 /dev/sdb
brw-r----- 1 root disk 8, 17 Jun 14 11:48 /dev/sdb1
brw-r----- 1 root disk 8, 32 Jun 14 11:47 /dev/sdc
brw-r----- 1 root disk 8, 33 Jun 14 11:48 /dev/sdc1

The block device used to create the FLASH ASMLib disk is /dev/sdc1.

6. Set the permission on the block device to grid:asmadmin

# chown grid:asmadmin /dev/sdc1
# ls -l /dev/sdc*
brw-r----- 1 root disk     8, 32 Jun 14 11:47 /dev/sdc
brw-r----- 1 grid asmadmin 8, 33 Jun 14 11:48 /dev/sdc1

7. Dismount the FLASH diskgroup again and set the asm_diskstring to include the block device. Still the DATA diskgroup is using ASMLib therefore ORCL* is also set. This mix setting is alright for testing purposes but it's better not to run production environments in the mix mode (both ASMLib and Block devices).

SQL> alter system set asm_diskstring ='ORCL:DATA','/dev/sdc1' SCOPE=MEMORY;

Mount the FLASH diskgroup and monitor the ASM alert log and verify block device is used

SQL> ALTER DISKGROUP flash mount

On the alert log

NOTE: cache registered group FLASH number=2 incarn=0x3d4848aa
NOTE: cache began mount (first) of group FLASH number=2 incarn=0x3d4848aa
NOTE: Assigning number (2,0) to disk (/dev/sdc1)
Thu Jun 14 12:01:15 2012
NOTE: GMON heartbeating for grp 2
GMON querying group 2 at 18 for pid 19, osid 3268
NOTE: cache opening disk 0 of grp 2: FLASH path:/dev/sdc1

Further verification from the v$asm_disk view

SQL> select name,label,path from v$asm_disk;

NAME    LABEL   PATH
------- ------- -----------
DATA    DATA    ORCL:DATA
FLASH   FLASH   /dev/sdc1

8. Do the same with other diskgroup

# /etc/init.d/oracleasm querydisk -d DATA
Disk "DATA" is a valid ASM disk on device [8, 17]

Set the block device permissions

# chown grid:asmadmin /dev/sdb1
# ls -l /dev/sdb*
brw-r----- 1 root disk     8, 16 Jun 14 11:47 /dev/sdb
brw-r----- 1 grid asmadmin 8, 17 Jun 14 11:48 /dev/sdb1

9. Dismount all diskgroups. Trying to change the asm_diskstring while diskgroups are mounted will throw the following error

SQL> alter system set asm_diskstring ='/dev/sdb1','/dev/sdc1' scope=memory;
alter system set asm_diskstring ='/dev/sdb1','/dev/sdc1' scope=memory
*
ERROR at line 1:
ORA-02097: parameter cannot be modified because specified value is invalid
ORA-15014: path 'ORCL:DATA' is not in the discovery set

Once all diskgroups are dismounted no issue.

SQL> alter diskgroup data dismount;
Diskgroup altered.

SQL> alter diskgroup flash dismount;
Diskgroup altered.

SQL> alter system set asm_diskstring ='/dev/sdb1','/dev/sdc1' scope=both;

Mount each diskgroup and verify the block device is used. Following from the alert log

Thu Jun 14 12:04:51 2012
SQL> alter diskgroup data mount
NOTE: cache registered group DATA number=1 incarn=0xa8c848b4
NOTE: cache began mount (first) of group DATA number=1 incarn=0xa8c848b4
NOTE: Assigning number (1,0) to disk (/dev/sdb1)

Further verification from v$asm_disk

SQL> select name,label,path from v$asm_disk;

NAME    LABEL   PATH
------- ------- -----------
DATA    DATA    /dev/sdb1
FLASH   FLASH   /dev/sdc1

10. Now that it's been verified that block devices are used for ASM create udev rules.

# ASM DATA
KERNEL=="sdb[1]", OWNER="grid", GROUP="asmadmin", MODE="660"

# ASM FLASH
KERNEL=="sdc[1]", OWNER="grid", GROUP="asmadmin", MODE="660"

11. Stop the high availability stack

crsctl stop has

and unload the oracleasm modules

# lsmod  | grep oracleasm
oracleasm              84136  1

# /etc/init.d/oracleasm stop
Dropping Oracle ASMLib disks:                              [  OK  ]
Shutting down the Oracle ASMLib driver:                    [  OK  ]

# lsmod  | grep oracleasm

disable loading of oracleasm when server is rebooted

# chkconfig oracleasm off

Restart the high availability service

crsctl start has
CRS-4123: Oracle High Availability Services has been started.

and check all ASM starting with block devices without an issue. When the restart is complete check the ASM alert log

SQL> ALTER DISKGROUP ALL MOUNT /* asm agent call crs *//* {0:0:2} */
NOTE: Diskgroups listed in ASM_DISKGROUPS are
         DATA
         FLASH
NOTE: cache registered group DATA number=1 incarn=0xa71a3a36
NOTE: cache began mount (first) of group DATA number=1 incarn=0xa71a3a36
NOTE: cache registered group FLASH number=2 incarn=0xa71a3a37
NOTE: cache began mount (first) of group FLASH number=2 incarn=0xa71a3a37
Errors in file /opt/app/oracle/diag/asm/+asm/+ASM/trace/+ASM_rbal_4010.trc:
ORA-15183: ASMLIB initialization error [driver/agent not installed]
WARNING: FAILED to load library: /opt/oracle/extapi/64/asm/orcl/1/libasm.so
NOTE: Assigning number (1,0) to disk (/dev/sdb1)
NOTE: Assigning number (2,0) to disk (/dev/sdc1)

and/or using kfod tool to confirm block devices are being used correctly

kfod asm_diskstring='/dev/sd*' disk=all
--------------------------------------------------------------------------------
 Disk          Size Path                                     User     Group
================================================================================
   1:      10236 Mb /dev/sdb1                                grid     asmadmin
   2:      10236 Mb /dev/sdc1                                grid     asmadmin
--------------------------------------------------------------------------------
ORACLE_SID ORACLE_HOME
================================================================================
      +ASM /opt/app/oracle/product/11.2.0/grid

12. If possible restart the server and check all changes (udev rules, disk permissions) and are working without issue and as the final step uninstall the ASMLib

# rpm -e oracleasmlib-2.0.4-1.el5
# rpm -e oracleasm-2.6.18-194.el5-2.0.5-1.el5
# rpm -e oracleasm-support-2.1.3-1.el5

Related Post
Migrating block devices using ASM instance to ASMLib
Moving ASMLib disk to block devices (Non-ASMLib) in 11gR1 RAC

Useful Metalink Notes
How To Migrate ASMLIB Devices To Block Devices (Non-ASMLIB)? [ID 567508.1]

Implementing Class of Secure Transport (COST) to Restrict Instance Registration in Oracle 11gR2 SE RAC (Solution mentioned in Oracle Security Alert for CVE-2012-1675)

This post is related to implementing the solution for security vulnerability mentioned in Oracle Security Alert for CVE-2012-1675. There is another post but that is related to version 11gR1 and this one is specific to 11gR2.
The metalink note related to applying the solution in a RAC environment is Using Class of Secure Transport (COST) to Restrict Instance Registration in Oracle RAC [ID 1340831.1]
However this solution fails when tried on a standard edition RAC environment as standard edition Oracle binaries are not linked with TCPS protocol. The standard edition RAC environment created for RHEL 6 is used in this case.
All the steps in section 1 in (1340831.1) up to step 1.4 work same on SE as on EE edition and problem will be encountered trying to implement step 1.5.
As oracle user (since this is a role separated environment) try to set TCPS protocol for the remote listener using scan ips.

show parameter remote_listener

NAME                                 TYPE        VALUE
------------------------------------ ----------- ------------------------------
remote_listener                      string      rhel6m-scan:1521

This environment only used one scan ip (not recommended by Oracle)

srvctl config scan
SCAN name: rhel6m-scan, Network: 1/192.168.0.0/255.255.255.0/eth0
SCAN VIP name: scan1, IP: /rhel6m-scan/192.168.0.91

Trying to set TCPS will give the following error

SQL> alter system set remote_listener='(ADDRESS_LIST=(ADDRESS=(PROTOCOL=TCPS)(HOST=192.168.0.91)(PORT=1523)))' scope=both sid='*';
alter system set remote_listener='(ADDRESS_LIST=(ADDRESS=(PROTOCOL=TCPS)(HOST=192.168.0.91)(PORT=1523)))' scope=both sid='*'
*
ERROR at line 1:
ORA-02097: parameter cannot be modified because specified value is invalid
ORA-00119: invalid specification for system parameter REMOTE_LISTENER
ORA-00130: invalid listener address
'(ADDRESS=(PROTOCOL=TCPS)(HOST=192.168.0.91)(PORT=1523))'

Error messages here doesn't help much as it doesn't specifically say which parameter is invalid unlike in 11gR1 which mentioned invalid protocol. It could be tested indeed it's the protocol that is causing the problem by testing the same command with TCP instead of TCPS, in this case it would work without an error

SQL> alter system set remote_listener='(ADDRESS_LIST=(ADDRESS=(PROTOCOL=TCP)(HOST=192.168.0.91)(PORT=1523)))' scope=both sid='*';

System altered.

Solution is to relink the binaries with tcps same as before. Stop all processes running out of the ORACLE_HOME

srvctl stop home -o $ORACLE_HOME -s status -n `hostname -s`

Change into $ORACLE_HOME/lib and make a backup of the existing libntcps11.a file and copy the libntcps11_ee.a.dbl

cp libntcps11.a libntcps11.a.orig
cp libntcps11_ee.a.dbl libntcps11.a

Execute relink all and monitor the log file

$ relink all
writing relink log to: /opt/app/oracle/product/11.2.0/dbhome_1/install/relink.log

Relinking ends with following text

-L/opt/app/oracle/product/11.2.0/dbhome_1/lib
test ! -f /opt/app/oracle/product/11.2.0/dbhome_1/bin/oracle ||\
           mv -f /opt/app/oracle/product/11.2.0/dbhome_1/bin/oracle /opt/app/oracle/product/11.2.0/dbhome_1/bin/oracleO
mv /opt/app/oracle/product/11.2.0/dbhome_1/rdbms/lib/oracle /opt/app/oracle/product/11.2.0/dbhome_1/bin/oracle
chmod 6751 /opt/app/oracle/product/11.2.0/dbhome_1/bin/oracle

It is important to relink all the oracle homes in the cluster if not the error will still persist

SQL> alter system set remote_listener='(ADDRESS_LIST=(ADDRESS=(PROTOCOL=TCPS)(HOST=192.168.0.91)(PORT=1523)))' scope=both sid='*';
alter system set remote_listener='(ADDRESS_LIST=(ADDRESS=(PROTOCOL=TCPS)(HOST=192.168.0.91)(PORT=1523)))' scope=both sid='*'
*
ERROR at line 1:
ORA-32008: error while processing parameter update at instance std11g21
ORA-02097: parameter cannot be modified because specified value is invalid
ORA-00119: invalid specification for system parameter REMOTE_LISTENER
ORA-00130: invalid listener address
'(ADDRESS=(PROTOCOL=TCPS)(HOST=192.168.0.91)(PORT=1523))'

But to confirm relinking has worked the above command could be executed at instance level

SQL> alter system set remote_listener='(ADDRESS_LIST=(ADDRESS=(PROTOCOL=TCPS)(HOST=192.168.0.91)(PORT=1523)))' scope=both sid='std11g22';

System altered.

Once all oracle homes are relinked execute the original command.

SQL> alter system set remote_listener='(ADDRESS_LIST=(ADDRESS=(PROTOCOL=TCPS)(HOST=192.168.0.91)(PORT=1523)))' scope=both sid='*';

System altered.

Rest of the steps in (1340831.1) could be executed without any additional work.
If the libntcps11_ee.a.dbl file is missing it could obtained by extracting the

database/stage/Components/oracle.network.rsf/11.2.0.3.0/1/DataFiles/filegroup5.jar

Related Post
Using Class of Secure Transport (COST) to Restrict Instance Registration in Oracle SE RAC Fails (Solution mentioned in Oracle Security Alert for CVE-2012-1675)




Update on 11th November 2015
For database version 11.2.0.4 Oracle has introduced "Valid Node Checking for Registration (VNCR)" as an alternative for implementing COST, if the sole reason for implementing cost is preventing remote registration. MOS note 1340831.1 has been updated with this information. VNCR related information could be found on following docs.
How to Enable VNCR on RAC Database to Register only Local Instances [ID 1914282.1]
Valid Node Checking For Registration (VNCR) [ID 1600630.1]

Installing 11gR2 (11.2.0.3) RAC with Role Separation on RHEL 6

Installing GI with role separation in RHEL 6 is identical to installing on OEL 6. This blog post is not meant to be an extensive how to install GI on RHEL 6, it will highlight the key aspects relating to RHEL 6 and role separation. There's also a post on installing standalone server with role separation on RHEL 6.
1. According to GI installation guide for RHEL 6 the kernel version must be 2.6.32-71.el6.x86_64 or later or if the RHEL 6 kernel is with unbreakable enterprise linux kernel then it must be 2.6.32-100.28.5.el6.x86_64 or later. In this case the kernel version is

uname -r
2.6.32-220.el6.x86_64

2. Create OS groups and users needed for role separation. Oracle user will own database software and grid user will own the clusterware.

create groups and users
groupadd -g 501 oinstall
groupadd -g 502 dba
groupadd -g 503 oper
groupadd -g 505 asmadmin
groupadd -g 506 asmdba
groupadd -g 507 asmoper

useradd -u 501 -g oinstall -G dba,oper,asmdba oracle
useradd -u 502 -g oinstall -G asmadmin,asmdba,asmoper,dba grid

grid user should be part of dba group as per metalink note 1084186.1

3. Create user equivalence for both grid user and oracle user.

$ /usr/bin/ssh-keygen -t rsa
Generating public/private rsa key pair.
Enter file in which to save the key (/home/grid/.ssh/id_rsa):
Created directory '/home/grid/.ssh'.
Enter passphrase (empty for no passphrase):
Enter same passphrase again:
Your identification has been saved in /home/grid/.ssh/id_rsa.
Your public key has been saved in /home/grid/.ssh/id_rsa.pub.
...
...

4. Create directory structure for clusterware and Oracle base and set with relevant ownership and permission.

# mkdir -p /opt/app/11.2.0/grid
# chown grid:oinstall /opt/app/11.2.0/grid
# chmod 775 /opt/app/11.2.0/grid

oracle base

# mkdir -p /opt/app/oracle
# chown oracle:oinstall /opt/app/oracle
# chmod 775 /opt/app/oracle

oraInventory

# mkdir -p /opt/app/oraInventory
# chown grid:oinstall /opt/app/oraInventory
# chmod 775 /opt/app/oraInventory

5. Verify pre-req package list is installed. Follow GI installation guide and metalink note Installing Oracle 11gR2 RDBMS on RHEL6 or OL6 64-bit (x86-64) [ID 1441282.1]

6.Set resource limit for both oracle and grid user in /etc/security/limits.conf file

oracle soft nproc 2047
oracle hard nproc 16384
oracle soft nofile 1024
oracle hard nofile 65536
oracle soft stack 10240
oracle hard stack 32768

grid soft nproc 2047
grid hard nproc 16384
grid soft nofile 1024
grid hard nofile 65536
grid soft stack 10240
grid hard stack 32768

7. Disable x11 forward for grid and oracle user by adding the following to each user's .ssh/config file

Host *
 ForwardX11 no

8. Disable tty on .bashrc for both grid and oracle user

if [ -t 0 ]; then
 stty intr ^C
fi

and also set umask value in .bashrc for grid and oracle user

umask 022

9. As per metalink note (1089399.1) for RHEL6/OEL6 to get oracleasm libraries it would require a UEL account . For this installation block devices will be used for ASM diskgroups. Setup udev rules for block devices with following ownerships and permissions.

#ASM OCR
KERNEL=="sdb[1]", OWNER="grid", GROUP="asmadmin", MODE="660"
KERNEL=="sdc[1]", OWNER="grid", GROUP="asmadmin", MODE="660"
KERNEL=="sdd[1]", OWNER="grid", GROUP="asmadmin", MODE="660"

# ASM DATA
KERNEL=="sde[1]", OWNER="grid", GROUP="asmadmin", MODE="660"

# ASM FLASH
KERNEL=="sdf[1]", OWNER="grid", GROUP="asmadmin", MODE="660"

If the permissions are incorrect this will be flagged during OUI pre-req check.

10. Edit the cvu_config file and set CV_ASSUME_DISTID=OEL6 (more on 1454982.1 and explained here). This will prevent pdksh package check being flagged as failed when running the installer using OUI. Still running the cluster verification will flag this as failed but this could be ignored.

./runcluvfy.sh stage -pre crsinst -n rhel6m1,rhel6m2  -fixup  -verbose

Performing pre-checks for cluster services setup

Checking node reachability...

Check: Node reachability from node "rhel6m1"
  Destination Node                      Reachable?
  ------------------------------------  ------------------------
  rhel6m1                               yes
  rhel6m2                               yes
Result: Node reachability check passed from node "rhel6m1"


Checking user equivalence...

Check: User equivalence for user "grid"
  Node Name                             Status
  ------------------------------------  ------------------------
  rhel6m2                               passed
  rhel6m1                               passed
Result: User equivalence check passed for user "grid"

Checking node connectivity...

Checking hosts config file...
  Node Name                             Status
  ------------------------------------  ------------------------
  rhel6m2                               passed
  rhel6m1                               passed

Verification of the hosts config file successful


Interface information for node "rhel6m2"
 Name   IP Address      Subnet          Gateway         Def. Gateway    HW Address        MTU
 ------ --------------- --------------- --------------- --------------- ----------------- ------
 eth0   192.168.0.86    192.168.0.0     0.0.0.0         192.168.0.100   08:00:27:8C:E5:85 1500
 eth0   192.168.0.90    192.168.0.0     0.0.0.0         192.168.0.100   08:00:27:8C:E5:85 1500
 eth1   192.168.1.88    192.168.1.0     0.0.0.0         192.168.0.100   08:00:27:E7:2E:BB 1500


Interface information for node "rhel6m1"
 Name   IP Address      Subnet          Gateway         Def. Gateway    HW Address        MTU
 ------ --------------- --------------- --------------- --------------- ----------------- ------
 eth0   192.168.0.85    192.168.0.0     0.0.0.0         192.168.0.100   08:00:27:FF:7A:38 1500
 eth0   192.168.0.89    192.168.0.0     0.0.0.0         192.168.0.100   08:00:27:FF:7A:38 1500
 eth1   192.168.1.87    192.168.1.0     0.0.0.0         192.168.0.100   08:00:27:F7:AB:A0 1500


Check: Node connectivity of subnet "192.168.0.0"
  Source                          Destination                     Connected?
  ------------------------------  ------------------------------  ----------------
  rhel6m2[192.168.0.86]           rhel6m2[192.168.0.90]           yes
  rhel6m2[192.168.0.86]           rhel6m1[192.168.0.85]           yes
  rhel6m2[192.168.0.86]           rhel6m1[192.168.0.89]           yes
  rhel6m2[192.168.0.90]           rhel6m1[192.168.0.85]           yes
  rhel6m2[192.168.0.90]           rhel6m1[192.168.0.89]           yes
  rhel6m1[192.168.0.85]           rhel6m1[192.168.0.89]           yes
Result: Node connectivity passed for subnet "192.168.0.0" with node(s) rhel6m2,rhel6m1


Check: TCP connectivity of subnet "192.168.0.0"
  Source                          Destination                     Connected?
  ------------------------------  ------------------------------  ----------------
  rhel6m1:192.168.0.85            rhel6m2:192.168.0.86            passed
  rhel6m1:192.168.0.85            rhel6m2:192.168.0.90            passed
  rhel6m1:192.168.0.85            rhel6m1:192.168.0.89            passed
Result: TCP connectivity check passed for subnet "192.168.0.0"


Check: Node connectivity of subnet "192.168.1.0"
  Source                          Destination                     Connected?
  ------------------------------  ------------------------------  ----------------
  rhel6m2[192.168.1.88]           rhel6m1[192.168.1.87]           yes
Result: Node connectivity passed for subnet "192.168.1.0" with node(s) rhel6m2,rhel6m1


Check: TCP connectivity of subnet "192.168.1.0"
  Source                          Destination                     Connected?
  ------------------------------  ------------------------------  ----------------
  rhel6m1:192.168.1.87            rhel6m2:192.168.1.88            passed
Result: TCP connectivity check passed for subnet "192.168.1.0"


Interfaces found on subnet "192.168.0.0" that are likely candidates for VIP are:
rhel6m2 eth0:192.168.0.86 eth0:192.168.0.90
rhel6m1 eth0:192.168.0.85 eth0:192.168.0.89

Interfaces found on subnet "192.168.1.0" that are likely candidates for a private interconnect are:
rhel6m2 eth1:192.168.1.88
rhel6m1 eth1:192.168.1.87
Checking subnet mask consistency...
Subnet mask consistency check passed for subnet "192.168.0.0".
Subnet mask consistency check passed for subnet "192.168.1.0".
Subnet mask consistency check passed.

Result: Node connectivity check passed

Checking multicast communication...

Checking subnet "192.168.0.0" for multicast communication with multicast group "230.0.1.0"...
Check of subnet "192.168.0.0" for multicast communication with multicast group "230.0.1.0" passed.

Checking subnet "192.168.1.0" for multicast communication with multicast group "230.0.1.0"...
Check of subnet "192.168.1.0" for multicast communication with multicast group "230.0.1.0" passed.

Check of multicast communication passed.

Checking ASMLib configuration.
  Node Name                             Status
  ------------------------------------  ------------------------
  rhel6m2                               passed
  rhel6m1                               passed
Result: Check for ASMLib configuration passed.

Check: Total memory
  Node Name     Available                 Required                  Status
  ------------  ------------------------  ------------------------  ----------
  rhel6m2       1.9597GB (2054884.0KB)    1.5GB (1572864.0KB)       passed
  rhel6m1       1.9597GB (2054884.0KB)    1.5GB (1572864.0KB)       passed
Result: Total memory check passed

Check: Available memory
  Node Name     Available                 Required                  Status
  ------------  ------------------------  ------------------------  ----------
  rhel6m2       1.8421GB (1931556.0KB)    50MB (51200.0KB)          passed
  rhel6m1       1.7056GB (1788460.0KB)    50MB (51200.0KB)          passed
Result: Available memory check passed

Check: Swap space
  Node Name     Available                 Required                  Status
  ------------  ------------------------  ------------------------  ----------
  rhel6m2       4GB (4194296.0KB)         2.9395GB (3082326.0KB)    passed
  rhel6m1       4GB (4194296.0KB)         2.9395GB (3082326.0KB)    passed
Result: Swap space check passed

Check: Free disk space for "rhel6m2:/tmp"
  Path              Node Name     Mount point   Available     Required      Status
  ----------------  ------------  ------------  ------------  ------------  ------------
  /tmp              rhel6m2       /             22.5312GB     1GB           passed
Result: Free disk space check passed for "rhel6m2:/tmp"

Check: Free disk space for "rhel6m1:/tmp"
  Path              Node Name     Mount point   Available     Required      Status
  ----------------  ------------  ------------  ------------  ------------  ------------
  /tmp              rhel6m1       /             21.4365GB     1GB           passed
Result: Free disk space check passed for "rhel6m1:/tmp"

Check: User existence for "grid"
  Node Name     Status                    Comment
  ------------  ------------------------  ------------------------
  rhel6m2       passed                    exists(502)
  rhel6m1       passed                    exists(502)

Checking for multiple users with UID value 502
Result: Check for multiple users with UID value 502 passed
Result: User existence check passed for "grid"

Check: Group existence for "oinstall"
  Node Name     Status                    Comment
  ------------  ------------------------  ------------------------
  rhel6m2       passed                    exists
  rhel6m1       passed                    exists
Result: Group existence check passed for "oinstall"

Check: Group existence for "dba"
  Node Name     Status                    Comment
  ------------  ------------------------  ------------------------
  rhel6m2       passed                    exists
  rhel6m1       passed                    exists
Result: Group existence check passed for "dba"

Check: Membership of user "grid" in group "oinstall" [as Primary]
  Node Name         User Exists   Group Exists  User in Group  Primary       Status
  ----------------  ------------  ------------  ------------  ------------  ------------
  rhel6m2           yes           yes           yes           yes           passed
  rhel6m1           yes           yes           yes           yes           passed
Result: Membership check for user "grid" in group "oinstall" [as Primary] passed

Check: Membership of user "grid" in group "dba"
  Node Name         User Exists   Group Exists  User in Group  Status
  ----------------  ------------  ------------  ------------  ----------------
  rhel6m2           yes           yes           yes           passed
  rhel6m1           yes           yes           yes           passed
Result: Membership check for user "grid" in group "dba" passed

Check: Run level
  Node Name     run level                 Required                  Status
  ------------  ------------------------  ------------------------  ----------
  rhel6m2       3                         3,5                       passed
  rhel6m1       3                         3,5                       passed
Result: Run level check passed

Check: Hard limits for "maximum open file descriptors"
  Node Name         Type          Available     Required      Status
  ----------------  ------------  ------------  ------------  ----------------
  rhel6m2           hard          65536         65536         passed
  rhel6m1           hard          65536         65536         passed
Result: Hard limits check passed for "maximum open file descriptors"

Check: Soft limits for "maximum open file descriptors"
  Node Name         Type          Available     Required      Status
  ----------------  ------------  ------------  ------------  ----------------
  rhel6m2           soft          1024          1024          passed
  rhel6m1           soft          1024          1024          passed
Result: Soft limits check passed for "maximum open file descriptors"

Check: Hard limits for "maximum user processes"
  Node Name         Type          Available     Required      Status
  ----------------  ------------  ------------  ------------  ----------------
  rhel6m2           hard          16384         16384         passed
  rhel6m1           hard          16384         16384         passed
Result: Hard limits check passed for "maximum user processes"

Check: Soft limits for "maximum user processes"
  Node Name         Type          Available     Required      Status
  ----------------  ------------  ------------  ------------  ----------------
  rhel6m2           soft          2047          2047          passed
  rhel6m1           soft          2047          2047          passed
Result: Soft limits check passed for "maximum user processes"

Check: System architecture
  Node Name     Available                 Required                  Status
  ------------  ------------------------  ------------------------  ----------
  rhel6m2       x86_64                    x86_64                    passed
  rhel6m1       x86_64                    x86_64                    passed
Result: System architecture check passed

Check: Kernel version
  Node Name     Available                 Required                  Status
  ------------  ------------------------  ------------------------  ----------
  rhel6m2       2.6.32-220.el6.x86_64     2.6.9                     passed
  rhel6m1       2.6.32-220.el6.x86_64     2.6.9                     passed
Result: Kernel version check passed

Check: Kernel parameter for "semmsl"
  Node Name         Current       Configured    Required      Status        Comment
  ----------------  ------------  ------------  ------------  ------------  ------------
  rhel6m2           3010          3010          250           passed
  rhel6m1           3010          3010          250           passed
Result: Kernel parameter check passed for "semmsl"

Check: Kernel parameter for "semmns"
  Node Name         Current       Configured    Required      Status        Comment
  ----------------  ------------  ------------  ------------  ------------  ------------
  rhel6m2           385280        385280        32000         passed
  rhel6m1           385280        385280        32000         passed
Result: Kernel parameter check passed for "semmns"

Check: Kernel parameter for "semopm"
  Node Name         Current       Configured    Required      Status        Comment
  ----------------  ------------  ------------  ------------  ------------  ------------
  rhel6m2           3010          3010          100           passed
  rhel6m1           3010          3010          100           passed
Result: Kernel parameter check passed for "semopm"

Check: Kernel parameter for "semmni"
  Node Name         Current       Configured    Required      Status        Comment
  ----------------  ------------  ------------  ------------  ------------  ------------
  rhel6m2           128           128           128           passed
  rhel6m1           128           128           128           passed
Result: Kernel parameter check passed for "semmni"

Check: Kernel parameter for "shmmax"
  Node Name         Current       Configured    Required      Status        Comment
  ----------------  ------------  ------------  ------------  ------------  ------------
  rhel6m2           68719476736   68719476736   1052100608    passed
  rhel6m1           68719476736   68719476736   1052100608    passed
Result: Kernel parameter check passed for "shmmax"

Check: Kernel parameter for "shmmni"
  Node Name         Current       Configured    Required      Status        Comment
  ----------------  ------------  ------------  ------------  ------------  ------------
  rhel6m2           4096          4096          4096          passed
  rhel6m1           4096          4096          4096          passed
Result: Kernel parameter check passed for "shmmni"

Check: Kernel parameter for "shmall"
  Node Name         Current       Configured    Required      Status        Comment
  ----------------  ------------  ------------  ------------  ------------  ------------
  rhel6m2           4294967296    4294967296    2097152       passed
  rhel6m1           4294967296    4294967296    2097152       passed
Result: Kernel parameter check passed for "shmall"

Check: Kernel parameter for "file-max"
  Node Name         Current       Configured    Required      Status        Comment
  ----------------  ------------  ------------  ------------  ------------  ------------
  rhel6m2           6815744       6815744       6815744       passed
  rhel6m1           6815744       6815744       6815744       passed
Result: Kernel parameter check passed for "file-max"

Check: Kernel parameter for "ip_local_port_range"
  Node Name         Current       Configured    Required      Status        Comment
  ----------------  ------------  ------------  ------------  ------------  ------------
  rhel6m2           between 9000.0 & 65500.0  between 9000.0 & 65500.0  between 9000.0 & 65500.0  passed
  rhel6m1           between 9000.0 & 65500.0  between 9000.0 & 65500.0  between 9000.0 & 65500.0  passed
Result: Kernel parameter check passed for "ip_local_port_range"

Check: Kernel parameter for "rmem_default"
  Node Name         Current       Configured    Required      Status        Comment
  ----------------  ------------  ------------  ------------  ------------  ------------
  rhel6m2           4194304       4194304       262144        passed
  rhel6m1           4194304       4194304       262144        passed
Result: Kernel parameter check passed for "rmem_default"

Check: Kernel parameter for "rmem_max"
  Node Name         Current       Configured    Required      Status        Comment
  ----------------  ------------  ------------  ------------  ------------  ------------
  rhel6m2           4194304       4194304       4194304       passed
  rhel6m1           4194304       4194304       4194304       passed
Result: Kernel parameter check passed for "rmem_max"

Check: Kernel parameter for "wmem_default"
  Node Name         Current       Configured    Required      Status        Comment
  ----------------  ------------  ------------  ------------  ------------  ------------
  rhel6m2           1048576       1048576       262144        passed
  rhel6m1           1048576       1048576       262144        passed
Result: Kernel parameter check passed for "wmem_default"

Check: Kernel parameter for "wmem_max"
  Node Name         Current       Configured    Required      Status        Comment
  ----------------  ------------  ------------  ------------  ------------  ------------
  rhel6m2           2097152       2097152       1048576       passed
  rhel6m1           2097152       2097152       1048576       passed
Result: Kernel parameter check passed for "wmem_max"

Check: Kernel parameter for "aio-max-nr"
  Node Name         Current       Configured    Required      Status        Comment
  ----------------  ------------  ------------  ------------  ------------  ------------
  rhel6m2           3145728       3145728       1048576       passed
  rhel6m1           3145728       3145728       1048576       passed
Result: Kernel parameter check passed for "aio-max-nr"

Check: Package existence for "make"
  Node Name     Available                 Required                  Status
  ------------  ------------------------  ------------------------  ----------
  rhel6m2       make-3.81-19.el6          make-3.80                 passed
  rhel6m1       make-3.81-19.el6          make-3.80                 passed
Result: Package existence check passed for "make"

Check: Package existence for "binutils"
  Node Name     Available                 Required                  Status
  ------------  ------------------------  ------------------------  ----------
  rhel6m2       binutils-2.20.51.0.2-5.28.el6  binutils-2.15.92.0.2      passed
  rhel6m1       binutils-2.20.51.0.2-5.28.el6  binutils-2.15.92.0.2      passed
Result: Package existence check passed for "binutils"

Check: Package existence for "gcc(x86_64)"
  Node Name     Available                 Required                  Status
  ------------  ------------------------  ------------------------  ----------
  rhel6m2       gcc(x86_64)-4.4.6-3.el6   gcc(x86_64)-3.4.6         passed
  rhel6m1       gcc(x86_64)-4.4.6-3.el6   gcc(x86_64)-3.4.6         passed
Result: Package existence check passed for "gcc(x86_64)"

Check: Package existence for "libaio(x86_64)"
  Node Name     Available                 Required                  Status
  ------------  ------------------------  ------------------------  ----------
  rhel6m2       libaio(x86_64)-0.3.107-10.el6  libaio(x86_64)-0.3.105    passed
  rhel6m1       libaio(x86_64)-0.3.107-10.el6  libaio(x86_64)-0.3.105    passed
Result: Package existence check passed for "libaio(x86_64)"

Check: Package existence for "glibc(x86_64)"
  Node Name     Available                 Required                  Status
  ------------  ------------------------  ------------------------  ----------
  rhel6m2       glibc(x86_64)-2.12-1.47.el6  glibc(x86_64)-2.3.4-2.41  passed
  rhel6m1       glibc(x86_64)-2.12-1.47.el6  glibc(x86_64)-2.3.4-2.41  passed
Result: Package existence check passed for "glibc(x86_64)"

Check: Package existence for "compat-libstdc++-33(x86_64)"
  Node Name     Available                 Required                  Status
  ------------  ------------------------  ------------------------  ----------
  rhel6m2       compat-libstdc++-33(x86_64)-3.2.3-69.el6  compat-libstdc++-33(x86_64)-3.2.3  passed
  rhel6m1       compat-libstdc++-33(x86_64)-3.2.3-69.el6  compat-libstdc++-33(x86_64)-3.2.3  passed
Result: Package existence check passed for "compat-libstdc++-33(x86_64)"

Check: Package existence for "elfutils-libelf(x86_64)"
  Node Name     Available                 Required                  Status
  ------------  ------------------------  ------------------------  ----------
  rhel6m2       elfutils-libelf(x86_64)-0.152-1.el6  elfutils-libelf(x86_64)-0.97  passed
  rhel6m1       elfutils-libelf(x86_64)-0.152-1.el6  elfutils-libelf(x86_64)-0.97  passed
Result: Package existence check passed for "elfutils-libelf(x86_64)"

Check: Package existence for "elfutils-libelf-devel"
  Node Name     Available                 Required                  Status
  ------------  ------------------------  ------------------------  ----------
  rhel6m2       elfutils-libelf-devel-0.152-1.el6  elfutils-libelf-devel-0.97  passed
  rhel6m1       elfutils-libelf-devel-0.152-1.el6  elfutils-libelf-devel-0.97  passed
Result: Package existence check passed for "elfutils-libelf-devel"

Check: Package existence for "glibc-common"
  Node Name     Available                 Required                  Status
  ------------  ------------------------  ------------------------  ----------
  rhel6m2       glibc-common-2.12-1.47.el6  glibc-common-2.3.4        passed
  rhel6m1       glibc-common-2.12-1.47.el6  glibc-common-2.3.4        passed
Result: Package existence check passed for "glibc-common"

Check: Package existence for "glibc-devel(x86_64)"
  Node Name     Available                 Required                  Status
  ------------  ------------------------  ------------------------  ----------
  rhel6m2       glibc-devel(x86_64)-2.12-1.47.el6  glibc-devel(x86_64)-2.3.4  passed
  rhel6m1       glibc-devel(x86_64)-2.12-1.47.el6  glibc-devel(x86_64)-2.3.4  passed
Result: Package existence check passed for "glibc-devel(x86_64)"

Check: Package existence for "glibc-headers"
  Node Name     Available                 Required                  Status
  ------------  ------------------------  ------------------------  ----------
  rhel6m2       glibc-headers-2.12-1.47.el6  glibc-headers-2.3.4       passed
  rhel6m1       glibc-headers-2.12-1.47.el6  glibc-headers-2.3.4       passed
Result: Package existence check passed for "glibc-headers"

Check: Package existence for "gcc-c++(x86_64)"
  Node Name     Available                 Required                  Status
  ------------  ------------------------  ------------------------  ----------
  rhel6m2       gcc-c++(x86_64)-4.4.6-3.el6  gcc-c++(x86_64)-3.4.6     passed
  rhel6m1       gcc-c++(x86_64)-4.4.6-3.el6  gcc-c++(x86_64)-3.4.6     passed
Result: Package existence check passed for "gcc-c++(x86_64)"

Check: Package existence for "libaio-devel(x86_64)"
  Node Name     Available                 Required                  Status
  ------------  ------------------------  ------------------------  ----------
  rhel6m2       libaio-devel(x86_64)-0.3.107-10.el6  libaio-devel(x86_64)-0.3.105  passed
  rhel6m1       libaio-devel(x86_64)-0.3.107-10.el6  libaio-devel(x86_64)-0.3.105  passed
Result: Package existence check passed for "libaio-devel(x86_64)"

Check: Package existence for "libgcc(x86_64)"
  Node Name     Available                 Required                  Status
  ------------  ------------------------  ------------------------  ----------
  rhel6m2       libgcc(x86_64)-4.4.6-3.el6  libgcc(x86_64)-3.4.6      passed
  rhel6m1       libgcc(x86_64)-4.4.6-3.el6  libgcc(x86_64)-3.4.6      passed
Result: Package existence check passed for "libgcc(x86_64)"

Check: Package existence for "libstdc++(x86_64)"
  Node Name     Available                 Required                  Status
  ------------  ------------------------  ------------------------  ----------
  rhel6m2       libstdc++(x86_64)-4.4.6-3.el6  libstdc++(x86_64)-3.4.6   passed
  rhel6m1       libstdc++(x86_64)-4.4.6-3.el6  libstdc++(x86_64)-3.4.6   passed
Result: Package existence check passed for "libstdc++(x86_64)"

Check: Package existence for "libstdc++-devel(x86_64)"
  Node Name     Available                 Required                  Status
  ------------  ------------------------  ------------------------  ----------
  rhel6m2       libstdc++-devel(x86_64)-4.4.6-3.el6  libstdc++-devel(x86_64)-3.4.6  passed
  rhel6m1       libstdc++-devel(x86_64)-4.4.6-3.el6  libstdc++-devel(x86_64)-3.4.6  passed
Result: Package existence check passed for "libstdc++-devel(x86_64)"

Check: Package existence for "sysstat"
  Node Name     Available                 Required                  Status
  ------------  ------------------------  ------------------------  ----------
  rhel6m2       sysstat-9.0.4-18.el6      sysstat-5.0.5             passed
  rhel6m1       sysstat-9.0.4-18.el6      sysstat-5.0.5             passed
Result: Package existence check passed for "sysstat"

Check: Package existence for "pdksh"
  Node Name     Available                 Required                  Status
  ------------  ------------------------  ------------------------  ----------
  rhel6m2       missing                   pdksh-5.2.14              failed
  rhel6m1       missing                   pdksh-5.2.14              failed
Result: Package existence check failed for "pdksh"

Check: Package existence for "expat(x86_64)"
  Node Name     Available                 Required                  Status
  ------------  ------------------------  ------------------------  ----------
  rhel6m2       expat(x86_64)-2.0.1-9.1.el6  expat(x86_64)-1.95.7      passed
  rhel6m1       expat(x86_64)-2.0.1-9.1.el6  expat(x86_64)-1.95.7      passed
Result: Package existence check passed for "expat(x86_64)"

Checking for multiple users with UID value 0
Result: Check for multiple users with UID value 0 passed

Check: Current group ID
Result: Current group ID check passed

Starting check for consistency of primary group of root user
  Node Name                             Status
  ------------------------------------  ------------------------
  rhel6m2                               passed
  rhel6m1                               passed

Check for consistency of root user's primary group passed

Starting Clock synchronization checks using Network Time Protocol(NTP)...

NTP Configuration file check started...
Network Time Protocol(NTP) configuration file not found on any of the nodes. Oracle Cluster Time Synchronization Service(CTSS) can be used instead of NTP for time synchronization on the cluster nodes
No NTP Daemons or Services were found to be running

Result: Clock synchronization check using Network Time Protocol(NTP) passed

Checking Core file name pattern consistency...
Core file name pattern consistency check passed.

Checking to make sure user "grid" is not in "root" group
  Node Name     Status                    Comment
  ------------  ------------------------  ------------------------
  rhel6m2       passed                    does not exist
  rhel6m1       passed                    does not exist
Result: User "grid" is not part of "root" group. Check passed

Check default user file creation mask
  Node Name     Available                 Required                  Comment
  ------------  ------------------------  ------------------------  ----------
  rhel6m2       0022                      0022                      passed
  rhel6m1       0022                      0022                      passed
Result: Default user file creation mask check passed
Checking consistency of file "/etc/resolv.conf" across nodes

Checking the file "/etc/resolv.conf" to make sure only one of domain and search entries is defined
File "/etc/resolv.conf" does not have both domain and search entries defined
Checking if domain entry in file "/etc/resolv.conf" is consistent across the nodes...
domain entry in file "/etc/resolv.conf" is consistent across nodes
Checking if search entry in file "/etc/resolv.conf" is consistent across the nodes...
search entry in file "/etc/resolv.conf" is consistent across nodes
Checking file "/etc/resolv.conf" to make sure that only one search entry is defined
All nodes have one search entry defined in file "/etc/resolv.conf"
Checking all nodes to make sure that search entry is "domain.net" as found on node "rhel6m2"
All nodes of the cluster have same value for 'search'
Checking DNS response time for an unreachable node
  Node Name                             Status
  ------------------------------------  ------------------------
  rhel6m2                               passed
  rhel6m1                               passed
The DNS response time for an unreachable node is within acceptable limit on all nodes

File "/etc/resolv.conf" is consistent across nodes

Check: Time zone consistency
Result: Time zone consistency check passed

Pre-check for cluster services setup was unsuccessful on all the nodes.

If only the pdksh has the status as failed this could be ignored and consider the pre-check is successful.

11. Start the clusterware installation.

Since block devices are used for ASM disks set the disk discovery path to block devices.

Set OS group chosen for ASM administration

When CV_ASSUME_DISTID=OEL6 is set the OUI doesn't flag the pdksh but correctly identify the ksh pre-check requirement.

Summary

Begin the installation

12. Run root scripts when prompted

Running root.sh on first node (rhel6m1)

# /opt/app/11.2.0/grid/root.sh
Performing root user operation for Oracle 11g

The following environment variables are set as:
    ORACLE_OWNER= grid
    ORACLE_HOME=  /opt/app/11.2.0/grid

Enter the full pathname of the local bin directory: [/usr/local/bin]:
   Copying dbhome to /usr/local/bin ...
   Copying oraenv to /usr/local/bin ...
   Copying coraenv to /usr/local/bin ...


Creating /etc/oratab file...
Entries will be added to the /etc/oratab file as needed by
Database Configuration Assistant when a database is created
Finished running generic part of root script.
Now product-specific root actions will be performed.
Using configuration parameter file: /opt/app/11.2.0/grid/crs/install/crsconfig_params
Creating trace directory
OLR initialization - successful
  root wallet
  root wallet cert
  root cert export
  peer wallet
  profile reader wallet
  pa wallet
  peer wallet keys
  pa wallet keys
  peer cert request
  pa cert request
  peer cert
  pa cert
  peer root cert TP
  profile reader root cert TP
  pa root cert TP
  peer pa cert TP
  pa peer cert TP
  profile reader pa cert TP
  profile reader peer cert TP
  peer user cert
  pa user cert
Adding Clusterware entries to upstart
CRS-2672: Attempting to start 'ora.mdnsd' on 'rhel6m1'
CRS-2676: Start of 'ora.mdnsd' on 'rhel6m1' succeeded
CRS-2672: Attempting to start 'ora.gpnpd' on 'rhel6m1'
CRS-2676: Start of 'ora.gpnpd' on 'rhel6m1' succeeded
CRS-2672: Attempting to start 'ora.cssdmonitor' on 'rhel6m1'
CRS-2672: Attempting to start 'ora.gipcd' on 'rhel6m1'
CRS-2676: Start of 'ora.cssdmonitor' on 'rhel6m1' succeeded
CRS-2676: Start of 'ora.gipcd' on 'rhel6m1' succeeded
CRS-2672: Attempting to start 'ora.cssd' on 'rhel6m1'
CRS-2672: Attempting to start 'ora.diskmon' on 'rhel6m1'
CRS-2676: Start of 'ora.diskmon' on 'rhel6m1' succeeded
CRS-2676: Start of 'ora.cssd' on 'rhel6m1' succeeded

ASM created and started successfully.

Disk Group CLUSTER_DG created successfully.

clscfg: -install mode specified
Successfully accumulated necessary OCR keys.
Creating OCR keys for user 'root', privgrp 'root'..
Operation successful.
CRS-4256: Updating the profile
Successful addition of voting disk 2c90c4f4a0fb4faebfb264244a836b96.
Successful addition of voting disk 1d7890a346284fa4bf0e278b2f3b1d3d.
Successful addition of voting disk 433f41571bf44fcebfe636a97d94777a.
Successfully replaced voting disk group with +CLUSTER_DG.
CRS-4256: Updating the profile
CRS-4266: Voting file(s) successfully replaced
##  STATE    File Universal Id                File Name Disk group
--  -----    -----------------                --------- ---------
 1. ONLINE   2c90c4f4a0fb4faebfb264244a836b96 (/dev/sdb1) [CLUSTER_DG]
 2. ONLINE   1d7890a346284fa4bf0e278b2f3b1d3d (/dev/sdc1) [CLUSTER_DG]
 3. ONLINE   433f41571bf44fcebfe636a97d94777a (/dev/sdd1) [CLUSTER_DG]
Located 3 voting disk(s).
CRS-2672: Attempting to start 'ora.asm' on 'rhel6m1'
CRS-2676: Start of 'ora.asm' on 'rhel6m1' succeeded
CRS-2672: Attempting to start 'ora.CLUSTER_DG.dg' on 'rhel6m1'
CRS-2676: Start of 'ora.CLUSTER_DG.dg' on 'rhel6m1' succeeded
Configure Oracle Grid Infrastructure for a Cluster ... succeeded

Running root.sh on the second node (rhel6m2)

# /opt/app/11.2.0/grid/root.sh
Performing root user operation for Oracle 11g

The following environment variables are set as:
    ORACLE_OWNER= grid
    ORACLE_HOME=  /opt/app/11.2.0/grid

Enter the full pathname of the local bin directory: [/usr/local/bin]:
   Copying dbhome to /usr/local/bin ...
   Copying oraenv to /usr/local/bin ...
   Copying coraenv to /usr/local/bin ...


Creating /etc/oratab file...
Entries will be added to the /etc/oratab file as needed by
Database Configuration Assistant when a database is created
Finished running generic part of root script.
Now product-specific root actions will be performed.
Using configuration parameter file: /opt/app/11.2.0/grid/crs/install/crsconfig_params
Creating trace directory
OLR initialization - successful
Adding Clusterware entries to upstart
CRS-4402: The CSS daemon was started in exclusive mode but found an active CSS daemon on node rhel6m1, number 1, and is terminating
An active cluster was found during exclusive startup, restarting to join the cluster
Configure Oracle Grid Infrastructure for a Cluster ... succeeded

CRS-4402 has been explained on (1212703.1), this is not an error on 11.2.0.3

13. Once the root.sh scripts have run on all nodes the clusterware installation will continue with running configuration assistants.

This will complete the clusterware installation as grid user. Check post installation configuration with

 cluvfy stage -post crsinst -n rhel6m1,rhel6m2

and using asmca created additional diskgroup that will be used when RAC DB is created.

14. Edit the cvu_config file in database software installation. Similar to clusterware check the cluvfy will flag pdksh as failed

cluvfy stage -pre dbinst -n rhel6m1,rhel6m2 -fixup -verbose
...
...
Check: Package existence for "pdksh"
  Node Name     Available                 Required                  Status
  ------------  ------------------------  ------------------------  ----------
  rhel6m2       missing                   pdksh-5.2.14              failed
  rhel6m1       missing                   pdksh-5.2.14              failed
Result: Package existence check failed for "pdksh"
...
...

But OUI will correctly identify the ksh package requirement.

Apart from this rest of the installation steps are same as in RHEL 5.
Select software only install. Database will be created after the installation using DBCA

Select all nodes part of the cluster

Installation path under Oracle base

OS group for database software management

Summary

Begin installation and execute the root.sh script when prompted.

15. Before beginning database created with dbca set read,write and execute permission for cfgtoollogs for Oracle user.

cd $ORACLE_BASE
#chmod 770 cfgtoollogs

Select all nodes the instance will run and complete DB creation.

Continue wizard to complete the remaining steps.

Related Posts
Installing 11gR2 (11.2.0.3) GI with Role Separation on OEL 6
Installing 11gR2 Standalone Server with ASM and Role Separation on RHEL 6
Installing 11.2.0.3 on RHEL 6