Saturday, August 24, 2013

Installing 12c (12.1.0.1) RAC on RHEL 6 with Role Separation - Clusterware

Installing 12cR1 RAC without the use of the new features such as the Oracle Flex ASM is pretty much same as installing 11gR2 RAC with role separation. This post is not a step by step guide to installing 12cR1 RAC but list the main differences in the installation process compared to previous versions and pitfalls to look out for.
Among the noticeable changes and new features 12c doesn't support storing clusterware files in block devices as such provisions must be made either to store in the shared file systems or in ASM. Even if upgrading ocr files must be moved to ASM before the upgrade.
12c supports IPv6 for public IP and VIP. However this post uses IPv4 for public and VIP.
12c has also introduced several new administrative privileges for job role separation for database related tasks in data guard, key management and backup.
1. The RHEL 6 version used for the setup is RHEL 6 - U4
# uname -r
2.6.32-358.el6.x86_64
2. Create OS groups and users needed for role separation. Oracle user will own database software and grid user will own the clusterware.
groupadd dba
groupadd oinstall
groupadd oper
groupadd asmoper
groupadd asmdba
groupadd asmadmin
groupadd backupdba
groupadd dgdba
groupadd kmdba

useradd -g oinstall -G dba,oper,asmdba,asmoper,asmadmin,backupdba,dgdba,kmdba oracle
useradd -g oinstall -G asmadmin,asmdba,asmoper,dba grid
grid user was made part of the dba group as per 1084186.1. (If this issue is still there on 12c was not tested).
3. Create user equivalence manually and do not wait for OUI to do this during the installation even though Oracle documentation says "You can configure SSH from the OUI interface during installation for the user account running the installation. The automatic configuration creates passwordless SSH connectivity between all cluster member nodes. Oracle recommends that you use the automatic procedure if possible.". Without this manual user equivalence none of the cluvfy commands will complete successfully.
4. Create directory structure for clusterware and Oracle base and set with relevant ownership and permission. For GI_HOME
mkdir -p /opt/app/12.1.0/grid 
chown -R grid:oinstall /opt/app
chown grid:oinstall /opt
chmod -R 775 /opt/app
chmod 775 /opt
For ORACLE_BASE
mkdir -p /opt/app/oracle
chown oracle:oinstall /opt/app/oracle
chmod 775 /opt/app/oracle
After running the root.sh the ownership of these directories (for GI_HOME) will be changed to root. But before the installation it must be owned by grid with oinstall as group and both user and group must have read,write and execute permissions.
5. Verify pre-req package list is installed. Follow GI installation guide and metalink note Requirements for Installing Oracle Database 12.1 on RHEL6 or OL6 64-bit (x86-64) [ID 1529864.1] for list of rpms required. Starting with 11.2.0.2 and including 12c "the 32-bit packages are required only for 32-bit client installs". In this setup servers used had both 32-bit and 64-bit version were installed.
6.Set resource limit for both oracle and grid user in /etc/security/limits.conf file
oracle soft nproc 2047
oracle hard nproc 16384
oracle soft nofile 1024
oracle hard nofile 65536
oracle soft stack 10240
oracle hard stack 32768

grid soft nproc 2047
grid hard nproc 16384
grid soft nofile 1024
grid hard nofile 65536
grid soft stack 10240
grid hard stack 32768
7. Disable x11 forward for grid and oracle user by adding the following to each user's .ssh/config file
Host *
 ForwardX11 no
8. Disable tty on .bashrc for both grid and oracle user
if [ -t 0 ]; then
 stty intr ^C
fi
and also set umask value in .bashrc for grid and oracle user
umask 022
9. ASMLib is not used for this RAC setup. Instead block devices are used as ASM disk. Setup udev rules for block devices with following ownerships and permissions.
KERNEL=="sdb[1]", OWNER="grid", GROUP="asmadmin", MODE="660"
If scsi_id command is used to find the UUID then the command listed on the GI install documentation fails on RHEL 6.
# /sbin/scsi_id -g -s /block/sdb/sdb1
/sbin/scsi_id: invalid option -- 's'
-s is not a supported option on RHEL 6. Instead use the following
/sbin/scsi_id -g -u -d /dev/sdb
Further more Oracle documentation mentions with external redundancy a minimum of 1 disk of 700MB size is sufficient to store both OCR and vote disk when not configuring cluster management repository. However it must be stated 700MB is not sufficient the disk space should be at least 1069MB.
In practice it is best to use normal or high redundancy on the disk group used to store OCR and vote disk.
10. Running cluvfy fails with the following message
ERROR:
Reference data is not available for verifying prerequisites on this operating system distribution
Verification cannot proceed
As mentioned in upgrade to 12c post install redhat-release-6Server-1.noarch.rpm found in the pack available with 1514012.1. (Also refer 1567127.1). After the aforementioned rpm is installed cluvfy can continue without any issue. If the cluvfy or OUI cannot find the UUID for a disk during the disk share check following message will be shown. From cluvfy
Checking Devices for ASM...

ERROR:  /dev/sdb1
rhel12c1:Cannot verify the shared state for device /dev/sdb1 due to Universally Unique Identifiers (UUIDs) not being found, or different values being found, for this device across nodes:
        rhel12c2,rhel12c1

Checking for shared devices...

PRVF-5149 : WARNING: Storage "/dev/sdb1" is not shared on all nodes
From OUI

Error is due to test system using VirtualBox to setup the RAC and partitions not returning an UUID. Installation could be continued ignoring this error. In a proper system where UUID is available the cluvfy would have the following messages when these check succeed.
Checking Devices for ASM...

Checking for shared devices...

  Device                                Device Type
  ------------------------------------  ------------------------
  /dev/sdb1                              Disk


Checking consistency of device owner across all nodes...
Consistency check of device owner for "/dev/sdb1" PASSED

Checking consistency of device group across all nodes...
Consistency check of device group for "/dev/sdb1" PASSED

Checking consistency of device permissions across all nodes...
Consistency check of device permissions for "/dev/sdb1" PASSED

Checking consistency of device size across all nodes...
Consistency check of device size for "/dev/sdb1" PASSED
Below is the output from running pre crsinst.
[grid@rhel12c1 grid]$ ./runcluvfy.sh stage -pre crsinst -n rhel12c1,rhel12c2 -asm -asmgrp asmadmin -presence local -asmdev /dev/sdb1 -crshome /opt/app/12.1.0/grid -fixup  -verbose

Performing pre-checks for cluster services setup

Checking node reachability...

Check: Node reachability from node "rhel12c1"
  Destination Node                      Reachable?
  ------------------------------------  ------------------------
  rhel12c1                              yes
  rhel12c2                              yes
Result: Node reachability check passed from node "rhel12c1"


Checking user equivalence...

Check: User equivalence for user "grid"
  Node Name                             Status
  ------------------------------------  ------------------------
  rhel12c2                              passed
  rhel12c1                              passed
Result: User equivalence check passed for user "grid"

Checking node connectivity...

Checking hosts config file...
  Node Name                             Status
  ------------------------------------  ------------------------
  rhel12c1                              passed
  rhel12c2                              passed

Verification of the hosts config file successful


Interface information for node "rhel12c1"
 Name   IP Address      Subnet          Gateway         Def. Gateway    HW Address        MTU
 ------ --------------- --------------- --------------- --------------- ----------------- ------
 eth0   192.168.0.93    192.168.0.0     0.0.0.0         192.168.0.100   08:00:27:CB:D8:AE 1500
 eth1   192.168.1.87    192.168.1.0     0.0.0.0         192.168.0.100   08:00:27:92:0B:69 1500


Interface information for node "rhel12c2"
 Name   IP Address      Subnet          Gateway         Def. Gateway    HW Address        MTU
 ------ --------------- --------------- --------------- --------------- ----------------- ------
 eth0   192.168.0.94    192.168.0.0     0.0.0.0         192.168.0.100   08:00:27:9C:66:81 1500
 eth1   192.168.1.88    192.168.1.0     0.0.0.0         192.168.0.100   08:00:27:1E:0E:A8 1500


Check: Node connectivity of subnet "192.168.0.0"
  Source                          Destination                     Connected?
  ------------------------------  ------------------------------  ----------------
  rhel12c1[192.168.0.93]          rhel12c2[192.168.0.94]          yes
Result: Node connectivity passed for subnet "192.168.0.0" with node(s) rhel12c1,rhel12c2


Check: TCP connectivity of subnet "192.168.0.0"
  Source                          Destination                     Connected?
  ------------------------------  ------------------------------  ----------------
  rhel12c1:192.168.0.93           rhel12c2:192.168.0.94           passed
Result: TCP connectivity check passed for subnet "192.168.0.0"


Check: Node connectivity of subnet "192.168.1.0"
  Source                          Destination                     Connected?
  ------------------------------  ------------------------------  ----------------
  rhel12c1[192.168.1.87]          rhel12c2[192.168.1.88]          yes
Result: Node connectivity passed for subnet "192.168.1.0" with node(s) rhel12c1,rhel12c2


Check: TCP connectivity of subnet "192.168.1.0"
  Source                          Destination                     Connected?
  ------------------------------  ------------------------------  ----------------
  rhel12c1:192.168.1.87           rhel12c2:192.168.1.88           passed
Result: TCP connectivity check passed for subnet "192.168.1.0"


Interfaces found on subnet "192.168.0.0" that are likely candidates for VIP are:
rhel12c1 eth0:192.168.0.93
rhel12c2 eth0:192.168.0.94

Interfaces found on subnet "192.168.1.0" that are likely candidates for a private interconnect are:
rhel12c1 eth1:192.168.1.87
rhel12c2 eth1:192.168.1.88
Checking subnet mask consistency...
Subnet mask consistency check passed for subnet "192.168.0.0".
Subnet mask consistency check passed for subnet "192.168.1.0".
Subnet mask consistency check passed.

Result: Node connectivity check passed

Checking multicast communication...

Checking subnet "192.168.0.0" for multicast communication with multicast group "224.0.0.251"...
Check of subnet "192.168.0.0" for multicast communication with multicast group "224.0.0.251" passed.

Check of multicast communication passed.

Checking ASMLib configuration.
  Node Name                             Status
  ------------------------------------  ------------------------
  rhel12c1                              passed
  rhel12c2                              passed
Result: Check for ASMLib configuration passed.

Check: Total memory
  Node Name     Available                 Required                  Status
  ------------  ------------------------  ------------------------  ----------
  rhel12c2      4.4199GB (4634568.0KB)    4GB (4194304.0KB)         passed
  rhel12c1      4.4199GB (4634568.0KB)    4GB (4194304.0KB)         passed
Result: Total memory check passed

Check: Available memory
  Node Name     Available                 Required                  Status
  ------------  ------------------------  ------------------------  ----------
  rhel12c2      4.2905GB (4498884.0KB)    50MB (51200.0KB)          passed
  rhel12c1      4.1755GB (4378320.0KB)    50MB (51200.0KB)          passed
Result: Available memory check passed

Check: Swap space
  Node Name     Available                 Required                  Status
  ------------  ------------------------  ------------------------  ----------
  rhel12c2      5GB (5242872.0KB)         4.4199GB (4634568.0KB)    passed
  rhel12c1      5GB (5242872.0KB)         4.4199GB (4634568.0KB)    passed
Result: Swap space check passed

Check: Free disk space for "rhel12c2:/usr,rhel12c2:/var,rhel12c2:/etc,rhel12c2:/opt/app/12.1.0/grid,rhel12c2:/sbin,rhel12c2:/tmp"
  Path              Node Name     Mount point   Available     Required      Status
  ----------------  ------------  ------------  ------------  ------------  ------------
  /usr              rhel12c2      /             21.3525GB     7.9635GB      passed
  /var              rhel12c2      /             21.3525GB     7.9635GB      passed
  /etc              rhel12c2      /             21.3525GB     7.9635GB      passed
  /opt/app/12.1.0/grid  rhel12c2      /             21.3525GB     7.9635GB      passed
  /sbin             rhel12c2      /             21.3525GB     7.9635GB      passed
  /tmp              rhel12c2      /             21.3525GB     7.9635GB      passed
Result: Free disk space check passed for "rhel12c2:/usr,rhel12c2:/var,rhel12c2:/etc,rhel12c2:/opt/app/12.1.0/grid,rhel12c2:/sbin,rhel12c2:/tmp"

Check: Free disk space for "rhel12c1:/usr,rhel12c1:/var,rhel12c1:/etc,rhel12c1:/opt/app/12.1.0/grid,rhel12c1:/sbin,rhel12c1:/tmp"
  Path              Node Name     Mount point   Available     Required      Status
  ----------------  ------------  ------------  ------------  ------------  ------------
  /usr              rhel12c1      /             18.3523GB     7.9635GB      passed
  /var              rhel12c1      /             18.3523GB     7.9635GB      passed
  /etc              rhel12c1      /             18.3523GB     7.9635GB      passed
  /opt/app/12.1.0/grid  rhel12c1      /             18.3523GB     7.9635GB      passed
  /sbin             rhel12c1      /             18.3523GB     7.9635GB      passed
  /tmp              rhel12c1      /             18.3523GB     7.9635GB      passed
Result: Free disk space check passed for "rhel12c1:/usr,rhel12c1:/var,rhel12c1:/etc,rhel12c1:/opt/app/12.1.0/grid,rhel12c1:/sbin,rhel12c1:/tmp"

Check: User existence for "grid"
  Node Name     Status                    Comment
  ------------  ------------------------  ------------------------
  rhel12c2      passed                    exists(501)
  rhel12c1      passed                    exists(501)

Checking for multiple users with UID value 501
Result: Check for multiple users with UID value 501 passed
Result: User existence check passed for "grid"

Check: Group existence for "oinstall"
  Node Name     Status                    Comment
  ------------  ------------------------  ------------------------
  rhel12c2      passed                    exists
  rhel12c1      passed                    exists
Result: Group existence check passed for "oinstall"

Check: Group existence for "dba"
  Node Name     Status                    Comment
  ------------  ------------------------  ------------------------
  rhel12c2      passed                    exists
  rhel12c1      passed                    exists
Result: Group existence check passed for "dba"

Check: Group existence for "asmadmin"
  Node Name     Status                    Comment
  ------------  ------------------------  ------------------------
  rhel12c2      passed                    exists
  rhel12c1      passed                    exists
Result: Group existence check passed for "asmadmin"

Check: Membership of user "grid" in group "oinstall" [as Primary]
  Node Name         User Exists   Group Exists  User in Group  Primary       Status
  ----------------  ------------  ------------  ------------  ------------  ------------
  rhel12c2          yes           yes           yes           yes           passed
  rhel12c1          yes           yes           yes           yes           passed
Result: Membership check for user "grid" in group "oinstall" [as Primary] passed

Check: Membership of user "grid" in group "dba"
  Node Name         User Exists   Group Exists  User in Group  Status
  ----------------  ------------  ------------  ------------  ----------------
  rhel12c2          yes           yes           yes           passed
  rhel12c1          yes           yes           yes           passed
Result: Membership check for user "grid" in group "dba" passed

Check: Membership of user "grid" in group "asmadmin"
  Node Name         User Exists   Group Exists  User in Group  Status
  ----------------  ------------  ------------  ------------  ----------------
  rhel12c2          yes           yes           yes           passed
  rhel12c1          yes           yes           yes           passed
Result: Membership check for user "grid" in group "asmadmin" passed

Check: Run level
  Node Name     run level                 Required                  Status
  ------------  ------------------------  ------------------------  ----------
  rhel12c2      3                         3,5                       passed
  rhel12c1      3                         3,5                       passed
Result: Run level check passed

Check: Hard limits for "maximum open file descriptors"
  Node Name         Type          Available     Required      Status
  ----------------  ------------  ------------  ------------  ----------------
  rhel12c2          hard          65536         65536         passed
  rhel12c1          hard          65536         65536         passed
Result: Hard limits check passed for "maximum open file descriptors"

Check: Soft limits for "maximum open file descriptors"
  Node Name         Type          Available     Required      Status
  ----------------  ------------  ------------  ------------  ----------------
  rhel12c2          soft          1024          1024          passed
  rhel12c1          soft          1024          1024          passed
Result: Soft limits check passed for "maximum open file descriptors"

Check: Hard limits for "maximum user processes"
  Node Name         Type          Available     Required      Status
  ----------------  ------------  ------------  ------------  ----------------
  rhel12c2          hard          16384         16384         passed
  rhel12c1          hard          16384         16384         passed
Result: Hard limits check passed for "maximum user processes"

Check: Soft limits for "maximum user processes"
  Node Name         Type          Available     Required      Status
  ----------------  ------------  ------------  ------------  ----------------
  rhel12c2          soft          2047          2047          passed
  rhel12c1          soft          2047          2047          passed
Result: Soft limits check passed for "maximum user processes"

Check: System architecture
  Node Name     Available                 Required                  Status
  ------------  ------------------------  ------------------------  ----------
  rhel12c2      x86_64                    x86_64                    passed
  rhel12c1      x86_64                    x86_64                    passed
Result: System architecture check passed

Check: Kernel version
  Node Name     Available                 Required                  Status
  ------------  ------------------------  ------------------------  ----------
  rhel12c2      2.6.32-358.el6.x86_64     2.6.32                    passed
  rhel12c1      2.6.32-358.el6.x86_64     2.6.32                    passed
Result: Kernel version check passed

Check: Kernel parameter for "semmsl"
  Node Name         Current       Configured    Required      Status        Comment
  ----------------  ------------  ------------  ------------  ------------  ------------
  rhel12c1          3010          3010          250           passed
  rhel12c2          3010          3010          250           passed
Result: Kernel parameter check passed for "semmsl"

Check: Kernel parameter for "semmns"
  Node Name         Current       Configured    Required      Status        Comment
  ----------------  ------------  ------------  ------------  ------------  ------------
  rhel12c1          385280        385280        32000         passed
  rhel12c2          385280        385280        32000         passed
Result: Kernel parameter check passed for "semmns"

Check: Kernel parameter for "semopm"
  Node Name         Current       Configured    Required      Status        Comment
  ----------------  ------------  ------------  ------------  ------------  ------------
  rhel12c1          3010          3010          100           passed
  rhel12c2          3010          3010          100           passed
Result: Kernel parameter check passed for "semopm"

Check: Kernel parameter for "semmni"
  Node Name         Current       Configured    Required      Status        Comment
  ----------------  ------------  ------------  ------------  ------------  ------------
  rhel12c1          128           128           128           passed
  rhel12c2          128           128           128           passed
Result: Kernel parameter check passed for "semmni"

Check: Kernel parameter for "shmmax"
  Node Name         Current       Configured    Required      Status        Comment
  ----------------  ------------  ------------  ------------  ------------  ------------
  rhel12c1          68719476736   68719476736   2372898816    passed
  rhel12c2          68719476736   68719476736   2372898816    passed
Result: Kernel parameter check passed for "shmmax"

Check: Kernel parameter for "shmmni"
  Node Name         Current       Configured    Required      Status        Comment
  ----------------  ------------  ------------  ------------  ------------  ------------
  rhel12c1          4096          4096          4096          passed
  rhel12c2          4096          4096          4096          passed
Result: Kernel parameter check passed for "shmmni"

Check: Kernel parameter for "shmall"
  Node Name         Current       Configured    Required      Status        Comment
  ----------------  ------------  ------------  ------------  ------------  ------------
  rhel12c1          4294967296    4294967296    463456        passed
  rhel12c2          4294967296    4294967296    463456        passed
Result: Kernel parameter check passed for "shmall"

Check: Kernel parameter for "file-max"
  Node Name         Current       Configured    Required      Status        Comment
  ----------------  ------------  ------------  ------------  ------------  ------------
  rhel12c1          6815744       6815744       6815744       passed
  rhel12c2          6815744       6815744       6815744       passed
Result: Kernel parameter check passed for "file-max"

Check: Kernel parameter for "ip_local_port_range"
  Node Name         Current       Configured    Required      Status        Comment
  ----------------  ------------  ------------  ------------  ------------  ------------
  rhel12c1          between 9000 & 65500  between 9000 & 65500  between 9000 & 65535  passed
  rhel12c2          between 9000 & 65500  between 9000 & 65500  between 9000 & 65535  passed
Result: Kernel parameter check passed for "ip_local_port_range"

Check: Kernel parameter for "rmem_default"
  Node Name         Current       Configured    Required      Status        Comment
  ----------------  ------------  ------------  ------------  ------------  ------------
  rhel12c1          4194304       4194304       262144        passed
  rhel12c2          4194304       4194304       262144        passed
Result: Kernel parameter check passed for "rmem_default"

Check: Kernel parameter for "rmem_max"
  Node Name         Current       Configured    Required      Status        Comment
  ----------------  ------------  ------------  ------------  ------------  ------------
  rhel12c1          4194304       4194304       4194304       passed
  rhel12c2          4194304       4194304       4194304       passed
Result: Kernel parameter check passed for "rmem_max"

Check: Kernel parameter for "wmem_default"
  Node Name         Current       Configured    Required      Status        Comment
  ----------------  ------------  ------------  ------------  ------------  ------------
  rhel12c1          1048576       1048576       262144        passed
  rhel12c2          1048576       1048576       262144        passed
Result: Kernel parameter check passed for "wmem_default"

Check: Kernel parameter for "wmem_max"
  Node Name         Current       Configured    Required      Status        Comment
  ----------------  ------------  ------------  ------------  ------------  ------------
  rhel12c1          2097152       2097152       1048576       passed
  rhel12c2          2097152       2097152       1048576       passed
Result: Kernel parameter check passed for "wmem_max"

Check: Kernel parameter for "aio-max-nr"
  Node Name         Current       Configured    Required      Status        Comment
  ----------------  ------------  ------------  ------------  ------------  ------------
  rhel12c1          3145728       3145728       1048576       passed
  rhel12c2          3145728       3145728       1048576       passed
Result: Kernel parameter check passed for "aio-max-nr"

Check: Package existence for "binutils"
  Node Name     Available                 Required                  Status
  ------------  ------------------------  ------------------------  ----------
  rhel12c2      binutils-2.20.51.0.2-5.36.el6  binutils-2.20.51.0.2      passed
  rhel12c1      binutils-2.20.51.0.2-5.36.el6  binutils-2.20.51.0.2      passed
Result: Package existence check passed for "binutils"

Check: Package existence for "compat-libcap1"
  Node Name     Available                 Required                  Status
  ------------  ------------------------  ------------------------  ----------
  rhel12c2      compat-libcap1-1.10-1     compat-libcap1-1.10       passed
  rhel12c1      compat-libcap1-1.10-1     compat-libcap1-1.10       passed
Result: Package existence check passed for "compat-libcap1"

Check: Package existence for "compat-libstdc++-33(x86_64)"
  Node Name     Available                 Required                  Status
  ------------  ------------------------  ------------------------  ----------
  rhel12c2      compat-libstdc++-33(x86_64)-3.2.3-69.el6  compat-libstdc++-33(x86_64)-3.2.3  passed
  rhel12c1      compat-libstdc++-33(x86_64)-3.2.3-69.el6  compat-libstdc++-33(x86_64)-3.2.3  passed
Result: Package existence check passed for "compat-libstdc++-33(x86_64)"

Check: Package existence for "libgcc(x86_64)"
  Node Name     Available                 Required                  Status
  ------------  ------------------------  ------------------------  ----------
  rhel12c2      libgcc(x86_64)-4.4.7-3.el6  libgcc(x86_64)-4.4.4      passed
  rhel12c1      libgcc(x86_64)-4.4.7-3.el6  libgcc(x86_64)-4.4.4      passed
Result: Package existence check passed for "libgcc(x86_64)"

Check: Package existence for "libstdc++(x86_64)"
  Node Name     Available                 Required                  Status
  ------------  ------------------------  ------------------------  ----------
  rhel12c2      libstdc++(x86_64)-4.4.7-3.el6  libstdc++(x86_64)-4.4.4   passed
  rhel12c1      libstdc++(x86_64)-4.4.7-3.el6  libstdc++(x86_64)-4.4.4   passed
Result: Package existence check passed for "libstdc++(x86_64)"

Check: Package existence for "libstdc++-devel(x86_64)"
  Node Name     Available                 Required                  Status
  ------------  ------------------------  ------------------------  ----------
  rhel12c2      libstdc++-devel(x86_64)-4.4.7-3.el6  libstdc++-devel(x86_64)-4.4.4  passed
  rhel12c1      libstdc++-devel(x86_64)-4.4.7-3.el6  libstdc++-devel(x86_64)-4.4.4  passed
Result: Package existence check passed for "libstdc++-devel(x86_64)"

Check: Package existence for "sysstat"
  Node Name     Available                 Required                  Status
  ------------  ------------------------  ------------------------  ----------
  rhel12c2      sysstat-9.0.4-20.el6      sysstat-9.0.4             passed
  rhel12c1      sysstat-9.0.4-20.el6      sysstat-9.0.4             passed
Result: Package existence check passed for "sysstat"

Check: Package existence for "gcc"
  Node Name     Available                 Required                  Status
  ------------  ------------------------  ------------------------  ----------
  rhel12c2      gcc-4.4.7-3.el6           gcc-4.4.4                 passed
  rhel12c1      gcc-4.4.7-3.el6           gcc-4.4.4                 passed
Result: Package existence check passed for "gcc"

Check: Package existence for "gcc-c++"
  Node Name     Available                 Required                  Status
  ------------  ------------------------  ------------------------  ----------
  rhel12c2      gcc-c++-4.4.7-3.el6       gcc-c++-4.4.4             passed
  rhel12c1      gcc-c++-4.4.7-3.el6       gcc-c++-4.4.4             passed
Result: Package existence check passed for "gcc-c++"

Check: Package existence for "ksh"
  Node Name     Available                 Required                  Status
  ------------  ------------------------  ------------------------  ----------
  rhel12c2      ksh-20100621-19.el6       ksh-...                   passed
  rhel12c1      ksh-20100621-19.el6       ksh-...                   passed
Result: Package existence check passed for "ksh"

Check: Package existence for "make"
  Node Name     Available                 Required                  Status
  ------------  ------------------------  ------------------------  ----------
  rhel12c2      make-3.81-20.el6          make-3.81                 passed
  rhel12c1      make-3.81-20.el6          make-3.81                 passed
Result: Package existence check passed for "make"

Check: Package existence for "glibc(x86_64)"
  Node Name     Available                 Required                  Status
  ------------  ------------------------  ------------------------  ----------
  rhel12c2      glibc(x86_64)-2.12-1.107.el6  glibc(x86_64)-2.12        passed
  rhel12c1      glibc(x86_64)-2.12-1.107.el6  glibc(x86_64)-2.12        passed
Result: Package existence check passed for "glibc(x86_64)"

Check: Package existence for "glibc-devel(x86_64)"
  Node Name     Available                 Required                  Status
  ------------  ------------------------  ------------------------  ----------
  rhel12c2      glibc-devel(x86_64)-2.12-1.107.el6  glibc-devel(x86_64)-2.12  passed
  rhel12c1      glibc-devel(x86_64)-2.12-1.107.el6  glibc-devel(x86_64)-2.12  passed
Result: Package existence check passed for "glibc-devel(x86_64)"

Check: Package existence for "libaio(x86_64)"
  Node Name     Available                 Required                  Status
  ------------  ------------------------  ------------------------  ----------
  rhel12c2      libaio(x86_64)-0.3.107-10.el6  libaio(x86_64)-0.3.107    passed
  rhel12c1      libaio(x86_64)-0.3.107-10.el6  libaio(x86_64)-0.3.107    passed
Result: Package existence check passed for "libaio(x86_64)"

Check: Package existence for "libaio-devel(x86_64)"
  Node Name     Available                 Required                  Status
  ------------  ------------------------  ------------------------  ----------
  rhel12c2      libaio-devel(x86_64)-0.3.107-10.el6  libaio-devel(x86_64)-0.3.107  passed
  rhel12c1      libaio-devel(x86_64)-0.3.107-10.el6  libaio-devel(x86_64)-0.3.107  passed
Result: Package existence check passed for "libaio-devel(x86_64)"

Check: Package existence for "nfs-utils"
  Node Name     Available                 Required                  Status
  ------------  ------------------------  ------------------------  ----------
  rhel12c2      nfs-utils-1.2.3-36.el6    nfs-utils-1.2.3-15        passed
  rhel12c1      nfs-utils-1.2.3-36.el6    nfs-utils-1.2.3-15        passed
Result: Package existence check passed for "nfs-utils"

Checking availability of ports "6200,6100" required for component "Oracle Notification Service (ONS)"
  Node Name         Port Number   Protocol      Available     Status
  ----------------  ------------  ------------  ------------  ----------------
  rhel12c2          6200          TCP           yes           successful
  rhel12c1          6200          TCP           yes           successful
  rhel12c2          6100          TCP           yes           successful
  rhel12c1          6100          TCP           yes           successful
Result: Port availability check passed for ports "6200,6100"

Checking for multiple users with UID value 0
Result: Check for multiple users with UID value 0 passed

Check: Current group ID
Result: Current group ID check passed

Starting check for consistency of primary group of root user
  Node Name                             Status
  ------------------------------------  ------------------------
  rhel12c2                              passed
  rhel12c1                              passed

Check for consistency of root user's primary group passed

Check: Package existence for "cvuqdisk"
  Node Name     Available                 Required                  Status
  ------------  ------------------------  ------------------------  ----------
  rhel12c2      cvuqdisk-1.0.9-1          cvuqdisk-1.0.9-1          passed
  rhel12c1      cvuqdisk-1.0.9-1          cvuqdisk-1.0.9-1          passed
Result: Package existence check passed for "cvuqdisk"


Checking Devices for ASM...

ERROR:  /dev/sdb1
rhel12c1:Cannot verify the shared state for device /dev/sdb1 due to Universally Unique Identifiers (UUIDs) not being found, or different values being found, for this device across nodes:
        rhel12c2,rhel12c1

Checking for shared devices...

PRVF-5149 : WARNING: Storage "/dev/sdb1" is not shared on all nodes

UDev attributes check for ASM Disks started...
Checking udev settings for device "/dev/sdb1"
  Device            Owner         Group         Permissions   Result
  ----------------  ------------  ------------  ------------  ----------------
  sdb1              grid          asmadmin      0660          passed

  sdb1              grid          asmadmin      0660          passed

Result: UDev attributes check passed for ASM Disks

Result: Devices check for ASM passed


Starting Clock synchronization checks using Network Time Protocol(NTP)...

NTP Configuration file check started...
Network Time Protocol(NTP) configuration file not found on any of the nodes. Oracle Cluster Time Synchronization Service(CTSS) can be used instead of NTP for time synchronization on the cluster nodes
No NTP Daemons or Services were found to be running

Result: Clock synchronization check using Network Time Protocol(NTP) passed

Checking Core file name pattern consistency...
Core file name pattern consistency check passed.

Checking to make sure user "grid" is not in "root" group
  Node Name     Status                    Comment
  ------------  ------------------------  ------------------------
  rhel12c2      passed                    does not exist
  rhel12c1      passed                    does not exist
Result: User "grid" is not part of "root" group. Check passed

Check default user file creation mask
  Node Name     Available                 Required                  Comment
  ------------  ------------------------  ------------------------  ----------
  rhel12c2      0022                      0022                      passed
  rhel12c1      0022                      0022                      passed
Result: Default user file creation mask check passed
Checking integrity of file "/etc/resolv.conf" across nodes

Checking the file "/etc/resolv.conf" to make sure only one of domain and search entries is defined
"domain" and "search" entries do not coexist in any  "/etc/resolv.conf" file
Checking if domain entry in file "/etc/resolv.conf" is consistent across the nodes...
"domain" entry does not exist in any "/etc/resolv.conf" file
Checking if search entry in file "/etc/resolv.conf" is consistent across the nodes...
Checking file "/etc/resolv.conf" to make sure that only one search entry is defined
More than one "search" entry does not exist in any "/etc/resolv.conf" file
All nodes have same "search" order defined in file "/etc/resolv.conf"
Checking DNS response time for an unreachable node
  Node Name                             Status
  ------------------------------------  ------------------------
  rhel12c1                              passed
  rhel12c2                              passed
The DNS response time for an unreachable node is within acceptable limit on all nodes

Check for integrity of file "/etc/resolv.conf" passed

Check: Time zone consistency
Result: Time zone consistency check passed

Checking integrity of name service switch configuration file "/etc/nsswitch.conf" ...
Checking if "hosts" entry in file "/etc/nsswitch.conf" is consistent across nodes...
Checking file "/etc/nsswitch.conf" to make sure that only one "hosts" entry is defined
More than one "hosts" entry does not exist in any "/etc/nsswitch.conf" file
All nodes have same "hosts" entry defined in file "/etc/nsswitch.conf"
Check for integrity of name service switch configuration file "/etc/nsswitch.conf" passed


Checking daemon "avahi-daemon" is not configured and running

Check: Daemon "avahi-daemon" not configured
  Node Name     Configured                Status
  ------------  ------------------------  ------------------------
  rhel12c2      no                        passed
  rhel12c1      no                        passed
Daemon not configured check passed for process "avahi-daemon"

Check: Daemon "avahi-daemon" not running
  Node Name     Running?                  Status
  ------------  ------------------------  ------------------------
  rhel12c2      no                        passed
  rhel12c1      no                        passed
Daemon not running check passed for process "avahi-daemon"

Starting check for /dev/shm mounted as temporary file system ...

Check for /dev/shm mounted as temporary file system passed


NOTE:
No fixable verification failures to fix

Pre-check for cluster services setup was successful.
Despite the issues on the ASM disk share cluvfy reports pre-reqs to be successful.



11. Start the clusterware installation by executing runInstaller.
12c provides many "use for" options for NICs. Since this is not setting up Flex ASM for eth1 select private. Oracle documentation states "Oracle Flex ASM can use either the same private networks as Oracle Clusterware, or use its own dedicated private networks. Each network can be classified PUBLIC, ASM & PRIVATE, PRIVATE, or ASM."
Selecting no prompts the below shown message. If GI management repository is installed then ASM space requirement would be higher. Refer Oracle documentation and 12c upgrade post for space requirements for management repository

12. When prompted execute the root.sh scripts. Below is the output from running the root.sh on first node.
[root@rhel12c1 ~]# /opt/app/12.1.0/grid/root.sh
Performing root user operation for Oracle 12c

The following environment variables are set as:
    ORACLE_OWNER= grid
    ORACLE_HOME=  /opt/app/12.1.0/grid

Enter the full pathname of the local bin directory: [/usr/local/bin]:
   Copying dbhome to /usr/local/bin ...
   Copying oraenv to /usr/local/bin ...
   Copying coraenv to /usr/local/bin ...


Creating /etc/oratab file...
Entries will be added to the /etc/oratab file as needed by
Database Configuration Assistant when a database is created
Finished running generic part of root script.
Now product-specific root actions will be performed.
Using configuration parameter file: /opt/app/12.1.0/grid/crs/install/crsconfig_params
2013/08/20 17:35:28 CLSRSC-363: User ignored prerequisites during installation

OLR initialization - successful
  root wallet
  root wallet cert
  root cert export
  peer wallet
  profile reader wallet
  pa wallet
  peer wallet keys
  pa wallet keys
  peer cert request
  pa cert request
  peer cert
  pa cert
  peer root cert TP
  profile reader root cert TP
  pa root cert TP
  peer pa cert TP
  pa peer cert TP
  profile reader pa cert TP
  profile reader peer cert TP
  peer user cert
  pa user cert
2013/08/20 17:36:29 CLSRSC-330: Adding Clusterware entries to file 'oracle-ohasd.conf'

CRS-4133: Oracle High Availability Services has been stopped.
CRS-4123: Oracle High Availability Services has been started.
CRS-4133: Oracle High Availability Services has been stopped.
CRS-4123: Oracle High Availability Services has been started.
CRS-2673: Attempting to stop 'ora.drivers.acfs' on 'rhel12c1'
CRS-2677: Stop of 'ora.drivers.acfs' on 'rhel12c1' succeeded
CRS-2672: Attempting to start 'ora.evmd' on 'rhel12c1'
CRS-2672: Attempting to start 'ora.mdnsd' on 'rhel12c1'
CRS-2676: Start of 'ora.evmd' on 'rhel12c1' succeeded
CRS-2676: Start of 'ora.mdnsd' on 'rhel12c1' succeeded
CRS-2672: Attempting to start 'ora.gpnpd' on 'rhel12c1'
CRS-2676: Start of 'ora.gpnpd' on 'rhel12c1' succeeded
CRS-2672: Attempting to start 'ora.cssdmonitor' on 'rhel12c1'
CRS-2672: Attempting to start 'ora.gipcd' on 'rhel12c1'
CRS-2676: Start of 'ora.cssdmonitor' on 'rhel12c1' succeeded
CRS-2676: Start of 'ora.gipcd' on 'rhel12c1' succeeded
CRS-2672: Attempting to start 'ora.cssd' on 'rhel12c1'
CRS-2672: Attempting to start 'ora.diskmon' on 'rhel12c1'
CRS-2676: Start of 'ora.diskmon' on 'rhel12c1' succeeded
CRS-2676: Start of 'ora.cssd' on 'rhel12c1' succeeded

ASM created and started successfully.

Disk Group clusterdg created successfully.

CRS-2672: Attempting to start 'ora.storage' on 'rhel12c1'
CRS-2676: Start of 'ora.storage' on 'rhel12c1' succeeded
CRS-2672: Attempting to start 'ora.crsd' on 'rhel12c1'
CRS-2676: Start of 'ora.crsd' on 'rhel12c1' succeeded
CRS-4256: Updating the profile
Successful addition of voting disk c09b02f830b64f49bf06b75f49b272ee.
Successfully replaced voting disk group with +clusterdg.
CRS-4256: Updating the profile
CRS-4266: Voting file(s) successfully replaced
##  STATE    File Universal Id                File Name Disk group
--  -----    -----------------                --------- ---------
 1. ONLINE   c09b02f830b64f49bf06b75f49b272ee (/dev/sdb1) [CLUSTERDG]
Located 1 voting disk(s).
CRS-2791: Starting shutdown of Oracle High Availability Services-managed resources on 'rhel12c1'
CRS-2673: Attempting to stop 'ora.crsd' on 'rhel12c1'
CRS-2677: Stop of 'ora.crsd' on 'rhel12c1' succeeded
CRS-2673: Attempting to stop 'ora.drivers.acfs' on 'rhel12c1'
CRS-2673: Attempting to stop 'ora.evmd' on 'rhel12c1'
CRS-2673: Attempting to stop 'ora.ctssd' on 'rhel12c1'
CRS-2673: Attempting to stop 'ora.storage' on 'rhel12c1'
CRS-2673: Attempting to stop 'ora.mdnsd' on 'rhel12c1'
CRS-2673: Attempting to stop 'ora.gpnpd' on 'rhel12c1'
CRS-2677: Stop of 'ora.drivers.acfs' on 'rhel12c1' succeeded
CRS-2677: Stop of 'ora.storage' on 'rhel12c1' succeeded
CRS-2673: Attempting to stop 'ora.asm' on 'rhel12c1'
CRS-2677: Stop of 'ora.mdnsd' on 'rhel12c1' succeeded
CRS-2677: Stop of 'ora.gpnpd' on 'rhel12c1' succeeded
CRS-2677: Stop of 'ora.evmd' on 'rhel12c1' succeeded
CRS-2677: Stop of 'ora.ctssd' on 'rhel12c1' succeeded
CRS-2677: Stop of 'ora.asm' on 'rhel12c1' succeeded
CRS-2673: Attempting to stop 'ora.cluster_interconnect.haip' on 'rhel12c1'
CRS-2677: Stop of 'ora.cluster_interconnect.haip' on 'rhel12c1' succeeded
CRS-2673: Attempting to stop 'ora.cssd' on 'rhel12c1'
CRS-2677: Stop of 'ora.cssd' on 'rhel12c1' succeeded
CRS-2673: Attempting to stop 'ora.gipcd' on 'rhel12c1'
CRS-2677: Stop of 'ora.gipcd' on 'rhel12c1' succeeded
CRS-2793: Shutdown of Oracle High Availability Services-managed resources on 'rhel12c1' has completed
CRS-4133: Oracle High Availability Services has been stopped.
CRS-4123: Starting Oracle High Availability Services-managed resources
CRS-2672: Attempting to start 'ora.mdnsd' on 'rhel12c1'
CRS-2672: Attempting to start 'ora.evmd' on 'rhel12c1'
CRS-2676: Start of 'ora.mdnsd' on 'rhel12c1' succeeded
CRS-2676: Start of 'ora.evmd' on 'rhel12c1' succeeded
CRS-2672: Attempting to start 'ora.gpnpd' on 'rhel12c1'
CRS-2676: Start of 'ora.gpnpd' on 'rhel12c1' succeeded
CRS-2672: Attempting to start 'ora.gipcd' on 'rhel12c1'
CRS-2676: Start of 'ora.gipcd' on 'rhel12c1' succeeded
CRS-2672: Attempting to start 'ora.cssdmonitor' on 'rhel12c1'
CRS-2676: Start of 'ora.cssdmonitor' on 'rhel12c1' succeeded
CRS-2672: Attempting to start 'ora.cssd' on 'rhel12c1'
CRS-2672: Attempting to start 'ora.diskmon' on 'rhel12c1'
CRS-2676: Start of 'ora.diskmon' on 'rhel12c1' succeeded
CRS-2789: Cannot stop resource 'ora.diskmon' as it is not running on server 'rhel12c1'
CRS-2676: Start of 'ora.cssd' on 'rhel12c1' succeeded
CRS-2672: Attempting to start 'ora.cluster_interconnect.haip' on 'rhel12c1'
CRS-2672: Attempting to start 'ora.ctssd' on 'rhel12c1'
CRS-2676: Start of 'ora.ctssd' on 'rhel12c1' succeeded
CRS-2676: Start of 'ora.cluster_interconnect.haip' on 'rhel12c1' succeeded
CRS-2672: Attempting to start 'ora.asm' on 'rhel12c1'
CRS-2676: Start of 'ora.asm' on 'rhel12c1' succeeded
CRS-2672: Attempting to start 'ora.storage' on 'rhel12c1'
CRS-2676: Start of 'ora.storage' on 'rhel12c1' succeeded
CRS-2672: Attempting to start 'ora.crsd' on 'rhel12c1'
CRS-2676: Start of 'ora.crsd' on 'rhel12c1' succeeded
CRS-6023: Starting Oracle Cluster Ready Services-managed resources
CRS-6017: Processing resource auto-start for servers: rhel12c1
CRS-6016: Resource auto-start has completed for server rhel12c1
CRS-6024: Completed start of Oracle Cluster Ready Services-managed resources
CRS-4123: Oracle High Availability Services has been started.
2013/08/20 17:44:10 CLSRSC-343: Successfully started Oracle clusterware stack

CRS-2672: Attempting to start 'ora.asm' on 'rhel12c1'
CRS-2676: Start of 'ora.asm' on 'rhel12c1' succeeded
CRS-2672: Attempting to start 'ora.CLUSTERDG.dg' on 'rhel12c1'
CRS-2676: Start of 'ora.CLUSTERDG.dg' on 'rhel12c1' succeeded
2013/08/20 17:45:19 CLSRSC-325: Configure Oracle Grid Infrastructure for a Cluster ... succeeded
Running root.sh on second node.
[root@rhel12c2 ~]# /opt/app/12.1.0/grid/root.sh
Performing root user operation for Oracle 12c

The following environment variables are set as:
    ORACLE_OWNER= grid
    ORACLE_HOME=  /opt/app/12.1.0/grid

Enter the full pathname of the local bin directory: [/usr/local/bin]:
   Copying dbhome to /usr/local/bin ...
   Copying oraenv to /usr/local/bin ...
   Copying coraenv to /usr/local/bin ...


Creating /etc/oratab file...
Entries will be added to the /etc/oratab file as needed by
Database Configuration Assistant when a database is created
Finished running generic part of root script.
Now product-specific root actions will be performed.
Using configuration parameter file: /opt/app/12.1.0/grid/crs/install/crsconfig_params
2013/08/20 17:45:54 CLSRSC-363: User ignored prerequisites during installation

OLR initialization - successful
2013/08/20 17:46:24 CLSRSC-330: Adding Clusterware entries to file 'oracle-ohasd.conf'

CRS-4133: Oracle High Availability Services has been stopped.
CRS-4123: Oracle High Availability Services has been started.
CRS-4133: Oracle High Availability Services has been stopped.
CRS-4123: Oracle High Availability Services has been started.
CRS-2791: Starting shutdown of Oracle High Availability Services-managed resources on 'rhel12c2'
CRS-2673: Attempting to stop 'ora.drivers.acfs' on 'rhel12c2'
CRS-2677: Stop of 'ora.drivers.acfs' on 'rhel12c2' succeeded
CRS-2793: Shutdown of Oracle High Availability Services-managed resources on 'rhel12c2' has completed
CRS-4133: Oracle High Availability Services has been stopped.
CRS-4123: Starting Oracle High Availability Services-managed resources
CRS-2672: Attempting to start 'ora.mdnsd' on 'rhel12c2'
CRS-2672: Attempting to start 'ora.evmd' on 'rhel12c2'
CRS-2676: Start of 'ora.mdnsd' on 'rhel12c2' succeeded
CRS-2676: Start of 'ora.evmd' on 'rhel12c2' succeeded
CRS-2672: Attempting to start 'ora.gpnpd' on 'rhel12c2'
CRS-2676: Start of 'ora.gpnpd' on 'rhel12c2' succeeded
CRS-2672: Attempting to start 'ora.gipcd' on 'rhel12c2'
CRS-2676: Start of 'ora.gipcd' on 'rhel12c2' succeeded
CRS-2672: Attempting to start 'ora.cssdmonitor' on 'rhel12c2'
CRS-2676: Start of 'ora.cssdmonitor' on 'rhel12c2' succeeded
CRS-2672: Attempting to start 'ora.cssd' on 'rhel12c2'
CRS-2672: Attempting to start 'ora.diskmon' on 'rhel12c2'
CRS-2676: Start of 'ora.diskmon' on 'rhel12c2' succeeded
CRS-2789: Cannot stop resource 'ora.diskmon' as it is not running on server 'rhel12c2'
CRS-2676: Start of 'ora.cssd' on 'rhel12c2' succeeded
CRS-2672: Attempting to start 'ora.cluster_interconnect.haip' on 'rhel12c2'
CRS-2672: Attempting to start 'ora.ctssd' on 'rhel12c2'
CRS-2676: Start of 'ora.ctssd' on 'rhel12c2' succeeded
CRS-2676: Start of 'ora.cluster_interconnect.haip' on 'rhel12c2' succeeded
CRS-2672: Attempting to start 'ora.asm' on 'rhel12c2'
CRS-2676: Start of 'ora.asm' on 'rhel12c2' succeeded
CRS-2672: Attempting to start 'ora.storage' on 'rhel12c2'
CRS-2676: Start of 'ora.storage' on 'rhel12c2' succeeded
CRS-2672: Attempting to start 'ora.crsd' on 'rhel12c2'
CRS-2676: Start of 'ora.crsd' on 'rhel12c2' succeeded
CRS-6017: Processing resource auto-start for servers: rhel12c2
CRS-2672: Attempting to start 'ora.ons' on 'rhel12c2'
CRS-2676: Start of 'ora.ons' on 'rhel12c2' succeeded
CRS-6016: Resource auto-start has completed for server rhel12c2
CRS-6024: Completed start of Oracle Cluster Ready Services-managed resources
CRS-4123: Oracle High Availability Services has been started.
2013/08/20 17:52:08 CLSRSC-343: Successfully started Oracle clusterware stack

2013/08/20 17:52:32 CLSRSC-325: Configure Oracle Grid Infrastructure for a Cluster ... succeeded
13. Once root.sh is completed click OK on the execute script dialog to continue with the remaining configurations after which clusterware installation is complete.
14. Verify the cluster installation with cluvfy post crsinst
cluvfy stage -post crsinst -n rhel12c1,rhel12c2 -verbose
15. Use ASMCA to create additional ASM disk groups that will be used for data and flash recovery when database is setup. Cluvfy could be used to verify if the disk are shared
cluvfy comp ssa -n rhel12c1,rhel12c2 -s /dev/sdc1,/dev/sdd1 -t data -r 12.1 -verbose

Next step is to install the Oracle Database software.

Related Posts
Installing 11gR2 (11.2.0.3) GI with Role Separation on RHEL 6
Installing 11gR2 (11.2.0.3) GI with Role Separation on OEL 6
Installing 11gR2 Standalone Server with ASM and Role Separation on RHEL 6
11gR2 Standalone Data Guard (with ASM and Role Separation)