Saturday, August 24, 2013

Installing 12c (12.1.0.1) RAC on RHEL 6 with Role Separation - Creating CDB & PDB

After installing clusterware and database software remaining step is to create the database. In this post a container database (CDB) and a pluggable database (PDB) will be created using DBCA. CDB and PDB are part of 12c new multitenant feature.
1. Before starting DBCA change the permission on cfgtoollogs and admin folders. cfgtoolslogs permission was an issue in 11.2 when installed with role separation as well(see step 15). In 12c admin directory in ORACLE_BASE also causes issue as it doesn't have write permission for the oinstall group thus making oracle user unable to create necessary directories inside it.

To fix it change to ORACLE_BASE as grid user (or root) and execute the following on all nodes
cd $ORACLE_BASE
chmod 770 cfgtoollogs
chmod 770 admin
2. Verify the database installation pre-req check with cluvfy -pre dbcfg
[oracle@rhel12c1 ~]$ cluvfy stage -pre dbcfg -n rhel12c1,rhel12c2 -d /opt/app/oracle/product/12.1.0/dbhome_1

Performing pre-checks for database configuration

Checking node reachability...
Node reachability check passed from node "rhel12c1"


Checking user equivalence...
User equivalence check passed for user "oracle"

Checking node connectivity...

Checking hosts config file...

Verification of the hosts config file successful

Check: Node connectivity using interfaces on subnet "192.168.0.0"
Node connectivity passed for subnet "192.168.0.0" with node(s) rhel12c2,rhel12c1
TCP connectivity check passed for subnet "192.168.0.0"


Check: Node connectivity using interfaces on subnet "192.168.1.0"
Node connectivity passed for subnet "192.168.1.0" with node(s) rhel12c2,rhel12c1
TCP connectivity check passed for subnet "192.168.1.0"

Checking subnet mask consistency...
Subnet mask consistency check passed for subnet "192.168.0.0".
Subnet mask consistency check passed for subnet "192.168.1.0".
Subnet mask consistency check passed.

Node connectivity check passed

Checking multicast communication...

Checking subnet "192.168.1.0" for multicast communication with multicast group "224.0.0.251"...
Check of subnet "192.168.1.0" for multicast communication with multicast group "224.0.0.251" passed.

Check of multicast communication passed.
Group existence check passed for "oper"
Group existence check passed for "dba"
Group existence check passed for "backupdba"
Group existence check passed for "dgdba"
Group existence check passed for "kmdba"
Total memory check passed
Available memory check passed
Swap space check passed
Free disk space check passed for "rhel12c2:/tmp"
Free disk space check passed for "rhel12c1:/tmp"
Check for multiple users with UID value 500 passed
User existence check passed for "oracle"
Group existence check passed for "oinstall"
Group existence check passed for "dba"
Membership check for user "oracle" in group "oinstall" [as Primary] passed
Membership check for user "oracle" in group "dba" passed
Run level check passed
Hard limits check passed for "maximum open file descriptors"
Soft limits check passed for "maximum open file descriptors"
Hard limits check passed for "maximum user processes"
Soft limits check passed for "maximum user processes"
System architecture check passed
Kernel version check passed
Kernel parameter check passed for "semmsl"
Kernel parameter check passed for "semmns"
Kernel parameter check passed for "semopm"
Kernel parameter check passed for "semmni"
Kernel parameter check passed for "shmmax"
Kernel parameter check passed for "shmmni"
Kernel parameter check passed for "shmall"
Kernel parameter check passed for "file-max"
Kernel parameter check passed for "ip_local_port_range"
Kernel parameter check passed for "rmem_default"
Kernel parameter check passed for "rmem_max"
Kernel parameter check passed for "wmem_default"
Kernel parameter check passed for "wmem_max"
Kernel parameter check passed for "aio-max-nr"
Package existence check passed for "binutils"
Package existence check passed for "compat-libcap1"
Package existence check passed for "compat-libstdc++-33(x86_64)"
Package existence check passed for "libgcc(x86_64)"
Package existence check passed for "libstdc++(x86_64)"
Package existence check passed for "libstdc++-devel(x86_64)"
Package existence check passed for "sysstat"
Package existence check passed for "gcc"
Package existence check passed for "gcc-c++"
Package existence check passed for "ksh"
Package existence check passed for "make"
Package existence check passed for "glibc(x86_64)"
Package existence check passed for "glibc-devel(x86_64)"
Package existence check passed for "libaio(x86_64)"
Package existence check passed for "libaio-devel(x86_64)"
Check for multiple users with UID value 0 passed
Current group ID check passed

Starting check for consistency of primary group of root user

Check for consistency of root user's primary group passed

Checking CRS integrity...

Clusterware version consistency passed.

CRS integrity check passed

Checking node application existence...

Checking existence of VIP node application (required)
VIP node application check passed

Checking existence of NETWORK node application (required)
NETWORK node application check passed

Checking existence of ONS node application (optional)
ONS node application check passed

Time zone consistency check passed

Pre-check for database configuration was successful.



3.Start DBCA and create the CDB first.
It was noted that the time to create database using custom option without the use of OLTP and warehouse templates takes longer compared to 11.2
It is possible to create both CDB and PDB in one step but here only the CDB is created initially.
With 12c database vault and label security could be configured with DBCA at the time of database creation.

4. PDBs could be created with DBCA as well as using command "create pluggable database". Time it takes to provision a PDB is much shorter compared to creating a database using DBCA in previous releases. This is one of the reasons why oracle is "marketing/advertising/touting" that 12c is the database for the cloud.
Start DBCA to create the PDB.
Each PDB needs a CDB and earlier created CDB is specified as the CDB for this PDB being created.

SRVCTL could be used to manage PDBs in a RAC environment. Dynamic services must be created for each PDBs. In this for the admin managed CDB and PDBs service is created with following command
srvctl add service -d cdb12c -pdb pdb12c -s pdbsvc -preferred "cdb12c1,cdb12c2"

srvctl config service -d cdb12c -s pdbsvc
Service name: pdbsvc
Service is enabled
Server pool: cdb12c_pdbsvc
Cardinality: 2
Disconnect: false
Service role: PRIMARY
Management policy: AUTOMATIC
DTP transaction: false
AQ HA notifications: false
Global: false
Commit Outcome: false
Failover type:
Failover method:
TAF failover retries:
TAF failover delay:
Connection Load Balancing Goal: LONG
Runtime Load Balancing Goal: NONE
TAF policy specification: NONE
Edition:
Pluggable database name: pdb12c
Maximum lag time: ANY
SQL Translation Profile:
Retention: 86400 seconds
Replay Initiation Time: 300 seconds
Session State Consistency:
Preferred instances: cdb12c1,cdb12c2
Available instances:
Starting the service will open the PDBs in read,write mode.
SQL>  select inst_id,name,open_mode from gv$pdbs;

   INST_ID NAME                           OPEN_MODE
---------- ------------------------------ ----------
         2 PDB12C                         MOUNTED
         1 PDB12C                         MOUNTED

srvctl start service -d cdb12c -s pdbsvc

SQL> select inst_id,name,open_mode from gv$pdbs;

   INST_ID NAME                           OPEN_MODE
---------- ------------------------------ ----------
         2 PDB12C                         READ WRITE
         1 PDB12C                         READ WRITE
on alert log following could be seen
Mon Oct 14 14:14:09 2013
ALTER PLUGGABLE DATABASE pdb12c OPEN  /* svc agent *//* {1:23451:950} */
Mon Oct 14 14:14:09 2013
This instance was first to open pluggable database PDB12C (container=3)
Opening pdb PDB12C (3) with no Resource Manager plan active
Pluggable database PDB12C opened read write
Completed: ALTER PLUGGABLE DATABASE pdb12c OPEN  /* svc agent *//* {1:23451:950} */
Modify the service's auto start status and PDBs will be open in read write mode each time the service is started
# crsctl modify resource ora.cdb12c.pdbsvc.svc -attr "AUTO_START=always"
However stopping the service using srvctl has no effect on the open mode of the PDB. It will stop the service and prevent any new connection coming in but the PDB will remain in read write mode. Even though on Oracle documentation it says stopping the service will stop the PDB.
This text could lead to confusion if one would expect the PDB will be closed when it is stopped. The real behavior of the stop is mentioned in another document which reads (from Admin Guide)"When your database is being managed by Oracle Restart or Oracle Clusterware, and you use the SRVCTL utility to start a service with a PDB property for a PDB that is closed, the PDB is opened in read/write mode on the nodes where the service is started. However, stopping a PDB service does not change the open mode of the PDB."
To stop the PDBs in RAC uses alter pluggable database command with "instances=all".

Creation of the PDB concludes the installing of 12c RAC.

Related Posts
Installing 11gR2 (11.2.0.3) GI with Role Separation on RHEL 6
Installing 11gR2 (11.2.0.3) GI with Role Separation on OEL 6
Installing 11gR2 Standalone Server with ASM and Role Separation on RHEL 6
11gR2 Standalone Data Guard (with ASM and Role Separation)

Installing 12c (12.1.0.1) RAC on RHEL 6 with Role Separation - Database Software

Once the clusterware is installed as mentioned in the previous post next step is to install the database software. Database software is installed as a non-shared oracle homes. In this role separation setup the database software will be owned by the oracle user.
1.Check the database software installation readiness with cluvfy. Below is the output from running pre dbinst.
[oracle@rhel12c1 bin]$ ./cluvfy stage -pre dbinst -n rhel12c1,rhel12c2 -fixup -verbose

Performing pre-checks for database installation

Checking node reachability...

Check: Node reachability from node "rhel12c1"
  Destination Node                      Reachable?
  ------------------------------------  ------------------------
  rhel12c1                              yes
  rhel12c2                              yes
Result: Node reachability check passed from node "rhel12c1"


Checking user equivalence...

Check: User equivalence for user "oracle"
  Node Name                             Status
  ------------------------------------  ------------------------
  rhel12c2                              passed
  rhel12c1                              passed
Result: User equivalence check passed for user "oracle"

Checking node connectivity...

Checking hosts config file...
  Node Name                             Status
  ------------------------------------  ------------------------
  rhel12c1                              passed
  rhel12c2                              passed

Verification of the hosts config file successful


Interface information for node "rhel12c1"
 Name   IP Address      Subnet          Gateway         Def. Gateway    HW Address        MTU
 ------ --------------- --------------- --------------- --------------- ----------------- ------
 eth0   192.168.0.93    192.168.0.0     0.0.0.0         192.168.0.100   08:00:27:CB:D8:AE 1500
 eth0   192.168.0.91    192.168.0.0     0.0.0.0         192.168.0.100   08:00:27:CB:D8:AE 1500
 eth0   192.168.0.89    192.168.0.0     0.0.0.0         192.168.0.100   08:00:27:CB:D8:AE 1500
 eth1   192.168.1.87    192.168.1.0     0.0.0.0         192.168.0.100   08:00:27:92:0B:69 1500
 eth1   169.254.111.199 169.254.0.0     0.0.0.0         192.168.0.100   08:00:27:92:0B:69 1500


Interface information for node "rhel12c2"
 Name   IP Address      Subnet          Gateway         Def. Gateway    HW Address        MTU
 ------ --------------- --------------- --------------- --------------- ----------------- ------
 eth0   192.168.0.94    192.168.0.0     0.0.0.0         192.168.0.100   08:00:27:9C:66:81 1500
 eth0   192.168.0.90    192.168.0.0     0.0.0.0         192.168.0.100   08:00:27:9C:66:81 1500
 eth1   192.168.1.88    192.168.1.0     0.0.0.0         192.168.0.100   08:00:27:1E:0E:A8 1500
 eth1   169.254.59.84   169.254.0.0     0.0.0.0         192.168.0.100   08:00:27:1E:0E:A8 1500


Check: Node connectivity using interfaces on subnet "192.168.1.0"

Check: Node connectivity of subnet "192.168.1.0"
  Source                          Destination                     Connected?
  ------------------------------  ------------------------------  ----------------
  rhel12c1[192.168.1.87]          rhel12c2[192.168.1.88]          yes
Result: Node connectivity passed for subnet "192.168.1.0" with node(s) rhel12c1,rhel12c2


Check: TCP connectivity of subnet "192.168.1.0"
  Source                          Destination                     Connected?
  ------------------------------  ------------------------------  ----------------
  rhel12c1:192.168.1.87           rhel12c2:192.168.1.88           passed
Result: TCP connectivity check passed for subnet "192.168.1.0"


Check: Node connectivity using interfaces on subnet "192.168.0.0"

Check: Node connectivity of subnet "192.168.0.0"
  Source                          Destination                     Connected?
  ------------------------------  ------------------------------  ----------------
  rhel12c2[192.168.0.90]          rhel12c1[192.168.0.91]          yes
  rhel12c2[192.168.0.90]          rhel12c1[192.168.0.89]          yes
  rhel12c2[192.168.0.90]          rhel12c2[192.168.0.94]          yes
  rhel12c2[192.168.0.90]          rhel12c1[192.168.0.93]          yes
  rhel12c1[192.168.0.91]          rhel12c1[192.168.0.89]          yes
  rhel12c1[192.168.0.91]          rhel12c2[192.168.0.94]          yes
  rhel12c1[192.168.0.91]          rhel12c1[192.168.0.93]          yes
  rhel12c1[192.168.0.89]          rhel12c2[192.168.0.94]          yes
  rhel12c1[192.168.0.89]          rhel12c1[192.168.0.93]          yes
  rhel12c2[192.168.0.94]          rhel12c1[192.168.0.93]          yes
Result: Node connectivity passed for subnet "192.168.0.0" with node(s) rhel12c2,rhel12c1


Check: TCP connectivity of subnet "192.168.0.0"
  Source                          Destination                     Connected?
  ------------------------------  ------------------------------  ----------------
  rhel12c2:192.168.0.90           rhel12c1:192.168.0.91           passed
  rhel12c2:192.168.0.90           rhel12c1:192.168.0.89           passed
  rhel12c2:192.168.0.90           rhel12c2:192.168.0.94           passed
  rhel12c2:192.168.0.90           rhel12c1:192.168.0.93           passed
Result: TCP connectivity check passed for subnet "192.168.0.0"

Checking subnet mask consistency...
Subnet mask consistency check passed for subnet "192.168.0.0".
Subnet mask consistency check passed for subnet "192.168.1.0".
Subnet mask consistency check passed.

Result: Node connectivity check passed

Checking multicast communication...

Checking subnet "192.168.1.0" for multicast communication with multicast group "224.0.0.251"...
Check of subnet "192.168.1.0" for multicast communication with multicast group "224.0.0.251" passed.

Check of multicast communication passed.

Check: Total memory
  Node Name     Available                 Required                  Status
  ------------  ------------------------  ------------------------  ----------
  rhel12c2      4.4199GB (4634568.0KB)    1GB (1048576.0KB)         passed
  rhel12c1      4.4199GB (4634568.0KB)    1GB (1048576.0KB)         passed
Result: Total memory check passed

Check: Available memory
  Node Name     Available                 Required                  Status
  ------------  ------------------------  ------------------------  ----------
  rhel12c2      3.0389GB (3186556.0KB)    50MB (51200.0KB)          passed
  rhel12c1      2.8197GB (2956660.0KB)    50MB (51200.0KB)          passed
Result: Available memory check passed

Check: Swap space
  Node Name     Available                 Required                  Status
  ------------  ------------------------  ------------------------  ----------
  rhel12c2      5GB (5242872.0KB)         4.4199GB (4634568.0KB)    passed
  rhel12c1      5GB (5242872.0KB)         4.4199GB (4634568.0KB)    passed
Result: Swap space check passed

Check: Free disk space for "rhel12c2:/tmp"
  Path              Node Name     Mount point   Available     Required      Status
  ----------------  ------------  ------------  ------------  ------------  ------------
  /tmp              rhel12c2      /             9.165GB       1GB           passed
Result: Free disk space check passed for "rhel12c2:/tmp"

Check: Free disk space for "rhel12c1:/tmp"
  Path              Node Name     Mount point   Available     Required      Status
  ----------------  ------------  ------------  ------------  ------------  ------------
  /tmp              rhel12c1      /             9.7314GB      1GB           passed
Result: Free disk space check passed for "rhel12c1:/tmp"

Check: User existence for "oracle"
  Node Name     Status                    Comment
  ------------  ------------------------  ------------------------
  rhel12c2      passed                    exists(500)
  rhel12c1      passed                    exists(500)

Checking for multiple users with UID value 500
Result: Check for multiple users with UID value 500 passed
Result: User existence check passed for "oracle"

Check: Group existence for "oinstall"
  Node Name     Status                    Comment
  ------------  ------------------------  ------------------------
  rhel12c2      passed                    exists
  rhel12c1      passed                    exists
Result: Group existence check passed for "oinstall"

Check: Group existence for "dba"
  Node Name     Status                    Comment
  ------------  ------------------------  ------------------------
  rhel12c2      passed                    exists
  rhel12c1      passed                    exists
Result: Group existence check passed for "dba"

Check: Group existence for "asmdba"
  Node Name     Status                    Comment
  ------------  ------------------------  ------------------------
  rhel12c2      passed                    exists
  rhel12c1      passed                    exists
Result: Group existence check passed for "asmdba"

Check: Membership of user "oracle" in group "oinstall" [as Primary]
  Node Name         User Exists   Group Exists  User in Group  Primary       Status
  ----------------  ------------  ------------  ------------  ------------  ------------
  rhel12c2          yes           yes           yes           yes           passed
  rhel12c1          yes           yes           yes           yes           passed
Result: Membership check for user "oracle" in group "oinstall" [as Primary] passed

Check: Membership of user "oracle" in group "dba"
  Node Name         User Exists   Group Exists  User in Group  Status
  ----------------  ------------  ------------  ------------  ----------------
  rhel12c2          yes           yes           yes           passed
  rhel12c1          yes           yes           yes           passed
Result: Membership check for user "oracle" in group "dba" passed

Check: Membership of user "oracle" in group "asmdba"
  Node Name         User Exists   Group Exists  User in Group  Status
  ----------------  ------------  ------------  ------------  ----------------
  rhel12c2          yes           yes           yes           passed
  rhel12c1          yes           yes           yes           passed
Result: Membership check for user "oracle" in group "asmdba" passed

Check: Run level
  Node Name     run level                 Required                  Status
  ------------  ------------------------  ------------------------  ----------
  rhel12c2      3                         3,5                       passed
  rhel12c1      3                         3,5                       passed
Result: Run level check passed

Check: Hard limits for "maximum open file descriptors"
  Node Name         Type          Available     Required      Status
  ----------------  ------------  ------------  ------------  ----------------
  rhel12c2          hard          65536         65536         passed
  rhel12c1          hard          65536         65536         passed
Result: Hard limits check passed for "maximum open file descriptors"

Check: Soft limits for "maximum open file descriptors"
  Node Name         Type          Available     Required      Status
  ----------------  ------------  ------------  ------------  ----------------
  rhel12c2          soft          1024          1024          passed
  rhel12c1          soft          1024          1024          passed
Result: Soft limits check passed for "maximum open file descriptors"

Check: Hard limits for "maximum user processes"
  Node Name         Type          Available     Required      Status
  ----------------  ------------  ------------  ------------  ----------------
  rhel12c2          hard          16384         16384         passed
  rhel12c1          hard          16384         16384         passed
Result: Hard limits check passed for "maximum user processes"

Check: Soft limits for "maximum user processes"
  Node Name         Type          Available     Required      Status
  ----------------  ------------  ------------  ------------  ----------------
  rhel12c2          soft          2047          2047          passed
  rhel12c1          soft          2047          2047          passed
Result: Soft limits check passed for "maximum user processes"

Check: System architecture
  Node Name     Available                 Required                  Status
  ------------  ------------------------  ------------------------  ----------
  rhel12c2      x86_64                    x86_64                    passed
  rhel12c1      x86_64                    x86_64                    passed
Result: System architecture check passed

Check: Kernel version
  Node Name     Available                 Required                  Status
  ------------  ------------------------  ------------------------  ----------
  rhel12c2      2.6.32-358.el6.x86_64     2.6.32                    passed
  rhel12c1      2.6.32-358.el6.x86_64     2.6.32                    passed
Result: Kernel version check passed

Check: Kernel parameter for "semmsl"
  Node Name         Current       Configured    Required      Status        Comment
  ----------------  ------------  ------------  ------------  ------------  ------------
  rhel12c1          3010          3010          250           passed
  rhel12c2          3010          3010          250           passed
Result: Kernel parameter check passed for "semmsl"

Check: Kernel parameter for "semmns"
  Node Name         Current       Configured    Required      Status        Comment
  ----------------  ------------  ------------  ------------  ------------  ------------
  rhel12c1          385280        385280        32000         passed
  rhel12c2          385280        385280        32000         passed
Result: Kernel parameter check passed for "semmns"

Check: Kernel parameter for "semopm"
  Node Name         Current       Configured    Required      Status        Comment
  ----------------  ------------  ------------  ------------  ------------  ------------
  rhel12c1          3010          3010          100           passed
  rhel12c2          3010          3010          100           passed
Result: Kernel parameter check passed for "semopm"

Check: Kernel parameter for "semmni"
  Node Name         Current       Configured    Required      Status        Comment
  ----------------  ------------  ------------  ------------  ------------  ------------
  rhel12c1          128           128           128           passed
  rhel12c2          128           128           128           passed
Result: Kernel parameter check passed for "semmni"

Check: Kernel parameter for "shmmax"
  Node Name         Current       Configured    Required      Status        Comment
  ----------------  ------------  ------------  ------------  ------------  ------------
  rhel12c1          68719476736   68719476736   2372898816    passed
  rhel12c2          68719476736   68719476736   2372898816    passed
Result: Kernel parameter check passed for "shmmax"

Check: Kernel parameter for "shmmni"
  Node Name         Current       Configured    Required      Status        Comment
  ----------------  ------------  ------------  ------------  ------------  ------------
  rhel12c1          4096          4096          4096          passed
  rhel12c2          4096          4096          4096          passed
Result: Kernel parameter check passed for "shmmni"

Check: Kernel parameter for "shmall"
  Node Name         Current       Configured    Required      Status        Comment
  ----------------  ------------  ------------  ------------  ------------  ------------
  rhel12c1          4294967296    4294967296    463456        passed
  rhel12c2          4294967296    4294967296    463456        passed
Result: Kernel parameter check passed for "shmall"

Check: Kernel parameter for "file-max"
  Node Name         Current       Configured    Required      Status        Comment
  ----------------  ------------  ------------  ------------  ------------  ------------
  rhel12c1          6815744       6815744       6815744       passed
  rhel12c2          6815744       6815744       6815744       passed
Result: Kernel parameter check passed for "file-max"

Check: Kernel parameter for "ip_local_port_range"
  Node Name         Current       Configured    Required      Status        Comment
  ----------------  ------------  ------------  ------------  ------------  ------------
  rhel12c1          between 9000 & 65500  between 9000 & 65500  between 9000 & 65535  passed
  rhel12c2          between 9000 & 65500  between 9000 & 65500  between 9000 & 65535  passed
Result: Kernel parameter check passed for "ip_local_port_range"

Check: Kernel parameter for "rmem_default"
  Node Name         Current       Configured    Required      Status        Comment
  ----------------  ------------  ------------  ------------  ------------  ------------
  rhel12c1          4194304       4194304       262144        passed
  rhel12c2          4194304       4194304       262144        passed
Result: Kernel parameter check passed for "rmem_default"

Check: Kernel parameter for "rmem_max"
  Node Name         Current       Configured    Required      Status        Comment
  ----------------  ------------  ------------  ------------  ------------  ------------
  rhel12c1          4194304       4194304       4194304       passed
  rhel12c2          4194304       4194304       4194304       passed
Result: Kernel parameter check passed for "rmem_max"

Check: Kernel parameter for "wmem_default"
  Node Name         Current       Configured    Required      Status        Comment
  ----------------  ------------  ------------  ------------  ------------  ------------
  rhel12c1          1048576       1048576       262144        passed
  rhel12c2          1048576       1048576       262144        passed
Result: Kernel parameter check passed for "wmem_default"

Check: Kernel parameter for "wmem_max"
  Node Name         Current       Configured    Required      Status        Comment
  ----------------  ------------  ------------  ------------  ------------  ------------
  rhel12c1          2097152       2097152       1048576       passed
  rhel12c2          2097152       2097152       1048576       passed
Result: Kernel parameter check passed for "wmem_max"

Check: Kernel parameter for "aio-max-nr"
  Node Name         Current       Configured    Required      Status        Comment
  ----------------  ------------  ------------  ------------  ------------  ------------
  rhel12c1          3145728       3145728       1048576       passed
  rhel12c2          3145728       3145728       1048576       passed
Result: Kernel parameter check passed for "aio-max-nr"

Check: Package existence for "binutils"
  Node Name     Available                 Required                  Status
  ------------  ------------------------  ------------------------  ----------
  rhel12c2      binutils-2.20.51.0.2-5.36.el6  binutils-2.20.51.0.2      passed
  rhel12c1      binutils-2.20.51.0.2-5.36.el6  binutils-2.20.51.0.2      passed
Result: Package existence check passed for "binutils"

Check: Package existence for "compat-libcap1"
  Node Name     Available                 Required                  Status
  ------------  ------------------------  ------------------------  ----------
  rhel12c2      compat-libcap1-1.10-1     compat-libcap1-1.10       passed
  rhel12c1      compat-libcap1-1.10-1     compat-libcap1-1.10       passed
Result: Package existence check passed for "compat-libcap1"

Check: Package existence for "compat-libstdc++-33(x86_64)"
  Node Name     Available                 Required                  Status
  ------------  ------------------------  ------------------------  ----------
  rhel12c2      compat-libstdc++-33(x86_64)-3.2.3-69.el6  compat-libstdc++-33(x86_64)-3.2.3  passed
  rhel12c1      compat-libstdc++-33(x86_64)-3.2.3-69.el6  compat-libstdc++-33(x86_64)-3.2.3  passed
Result: Package existence check passed for "compat-libstdc++-33(x86_64)"

Check: Package existence for "libgcc(x86_64)"
  Node Name     Available                 Required                  Status
  ------------  ------------------------  ------------------------  ----------
  rhel12c2      libgcc(x86_64)-4.4.7-3.el6  libgcc(x86_64)-4.4.4      passed
  rhel12c1      libgcc(x86_64)-4.4.7-3.el6  libgcc(x86_64)-4.4.4      passed
Result: Package existence check passed for "libgcc(x86_64)"

Check: Package existence for "libstdc++(x86_64)"
  Node Name     Available                 Required                  Status
  ------------  ------------------------  ------------------------  ----------
  rhel12c2      libstdc++(x86_64)-4.4.7-3.el6  libstdc++(x86_64)-4.4.4   passed
  rhel12c1      libstdc++(x86_64)-4.4.7-3.el6  libstdc++(x86_64)-4.4.4   passed
Result: Package existence check passed for "libstdc++(x86_64)"

Check: Package existence for "libstdc++-devel(x86_64)"
  Node Name     Available                 Required                  Status
  ------------  ------------------------  ------------------------  ----------
  rhel12c2      libstdc++-devel(x86_64)-4.4.7-3.el6  libstdc++-devel(x86_64)-4.4.4  passed
  rhel12c1      libstdc++-devel(x86_64)-4.4.7-3.el6  libstdc++-devel(x86_64)-4.4.4  passed
Result: Package existence check passed for "libstdc++-devel(x86_64)"

Check: Package existence for "sysstat"
  Node Name     Available                 Required                  Status
  ------------  ------------------------  ------------------------  ----------
  rhel12c2      sysstat-9.0.4-20.el6      sysstat-9.0.4             passed
  rhel12c1      sysstat-9.0.4-20.el6      sysstat-9.0.4             passed
Result: Package existence check passed for "sysstat"

Check: Package existence for "gcc"
  Node Name     Available                 Required                  Status
  ------------  ------------------------  ------------------------  ----------
  rhel12c2      gcc-4.4.7-3.el6           gcc-4.4.4                 passed
  rhel12c1      gcc-4.4.7-3.el6           gcc-4.4.4                 passed
Result: Package existence check passed for "gcc"

Check: Package existence for "gcc-c++"
  Node Name     Available                 Required                  Status
  ------------  ------------------------  ------------------------  ----------
  rhel12c2      gcc-c++-4.4.7-3.el6       gcc-c++-4.4.4             passed
  rhel12c1      gcc-c++-4.4.7-3.el6       gcc-c++-4.4.4             passed
Result: Package existence check passed for "gcc-c++"

Check: Package existence for "ksh"
  Node Name     Available                 Required                  Status
  ------------  ------------------------  ------------------------  ----------
  rhel12c2      ksh-20100621-19.el6       ksh-...                   passed
  rhel12c1      ksh-20100621-19.el6       ksh-...                   passed
Result: Package existence check passed for "ksh"

Check: Package existence for "make"
  Node Name     Available                 Required                  Status
  ------------  ------------------------  ------------------------  ----------
  rhel12c2      make-3.81-20.el6          make-3.81                 passed
  rhel12c1      make-3.81-20.el6          make-3.81                 passed
Result: Package existence check passed for "make"

Check: Package existence for "glibc(x86_64)"
  Node Name     Available                 Required                  Status
  ------------  ------------------------  ------------------------  ----------
  rhel12c2      glibc(x86_64)-2.12-1.107.el6  glibc(x86_64)-2.12        passed
  rhel12c1      glibc(x86_64)-2.12-1.107.el6  glibc(x86_64)-2.12        passed
Result: Package existence check passed for "glibc(x86_64)"

Check: Package existence for "glibc-devel(x86_64)"
  Node Name     Available                 Required                  Status
  ------------  ------------------------  ------------------------  ----------
  rhel12c2      glibc-devel(x86_64)-2.12-1.107.el6  glibc-devel(x86_64)-2.12  passed
  rhel12c1      glibc-devel(x86_64)-2.12-1.107.el6  glibc-devel(x86_64)-2.12  passed
Result: Package existence check passed for "glibc-devel(x86_64)"

Check: Package existence for "libaio(x86_64)"
  Node Name     Available                 Required                  Status
  ------------  ------------------------  ------------------------  ----------
  rhel12c2      libaio(x86_64)-0.3.107-10.el6  libaio(x86_64)-0.3.107    passed
  rhel12c1      libaio(x86_64)-0.3.107-10.el6  libaio(x86_64)-0.3.107    passed
Result: Package existence check passed for "libaio(x86_64)"

Check: Package existence for "libaio-devel(x86_64)"
  Node Name     Available                 Required                  Status
  ------------  ------------------------  ------------------------  ----------
  rhel12c2      libaio-devel(x86_64)-0.3.107-10.el6  libaio-devel(x86_64)-0.3.107  passed
  rhel12c1      libaio-devel(x86_64)-0.3.107-10.el6  libaio-devel(x86_64)-0.3.107  passed
Result: Package existence check passed for "libaio-devel(x86_64)"

Checking for multiple users with UID value 0
Result: Check for multiple users with UID value 0 passed

Check: Current group ID
Result: Current group ID check passed

Starting check for consistency of primary group of root user
  Node Name                             Status
  ------------------------------------  ------------------------
  rhel12c2                              passed
  rhel12c1                              passed

Check for consistency of root user's primary group passed

Check default user file creation mask
  Node Name     Available                 Required                  Comment
  ------------  ------------------------  ------------------------  ----------
  rhel12c2      0022                      0022                      passed
  rhel12c1      0022                      0022                      passed
Result: Default user file creation mask check passed

Checking CRS integrity...

Clusterware version consistency passed.
The Oracle Clusterware is healthy on node "rhel12c1"
The Oracle Clusterware is healthy on node "rhel12c2"

CRS integrity check passed

Checking Cluster manager integrity...


Checking CSS daemon...

  Node Name                             Status
  ------------------------------------  ------------------------
  rhel12c1                              running
  rhel12c2                              running

Oracle Cluster Synchronization Services appear to be online.

Cluster manager integrity check passed


Checking node application existence...

Checking existence of VIP node application (required)
  Node Name     Required                  Running?                  Comment
  ------------  ------------------------  ------------------------  ----------
  rhel12c1      yes                       yes                       passed
  rhel12c2      yes                       yes                       passed
VIP node application check passed

Checking existence of NETWORK node application (required)
  Node Name     Required                  Running?                  Comment
  ------------  ------------------------  ------------------------  ----------
  rhel12c1      yes                       yes                       passed
  rhel12c2      yes                       yes                       passed
NETWORK node application check passed

Checking existence of ONS node application (optional)
  Node Name     Required                  Running?                  Comment
  ------------  ------------------------  ------------------------  ----------
  rhel12c1      no                        yes                       passed
  rhel12c2      no                        yes                       passed
ONS node application check passed


Checking if Clusterware is installed on all nodes...
Check of Clusterware install passed

Checking if CTSS Resource is running on all nodes...
Check: CTSS Resource running on all nodes
  Node Name                             Status
  ------------------------------------  ------------------------
  rhel12c1                              passed
  rhel12c2                              passed
Result: CTSS resource check passed


Querying CTSS for time offset on all nodes...
Result: Query of CTSS for time offset passed

Check CTSS state started...
Check: CTSS state
  Node Name                             State
  ------------------------------------  ------------------------
  rhel12c2                              Active
  rhel12c1                              Active
CTSS is in Active state. Proceeding with check of clock time offsets on all nodes...
Reference Time Offset Limit: 1000.0 msecs
Check: Reference Time Offset
  Node Name     Time Offset               Status
  ------------  ------------------------  ------------------------
  rhel12c2      0.0                       passed
  rhel12c1      0.0                       passed

Time offset is within the specified limits on the following set of nodes:
"[rhel12c2, rhel12c1]"
Result: Check of clock time offsets passed


Oracle Cluster Time Synchronization Services check passed
Checking integrity of file "/etc/resolv.conf" across nodes

Checking the file "/etc/resolv.conf" to make sure only one of domain and search entries is defined
"domain" and "search" entries do not coexist in any  "/etc/resolv.conf" file
Checking if domain entry in file "/etc/resolv.conf" is consistent across the nodes...
"domain" entry does not exist in any "/etc/resolv.conf" file
Checking if search entry in file "/etc/resolv.conf" is consistent across the nodes...
Checking file "/etc/resolv.conf" to make sure that only one search entry is defined
More than one "search" entry does not exist in any "/etc/resolv.conf" file
All nodes have same "search" order defined in file "/etc/resolv.conf"
Checking DNS response time for an unreachable node
  Node Name                             Status
  ------------------------------------  ------------------------
  rhel12c1                              passed
  rhel12c2                              passed
The DNS response time for an unreachable node is within acceptable limit on all nodes

Check for integrity of file "/etc/resolv.conf" passed

Check: Time zone consistency
Result: Time zone consistency check passed

Checking Single Client Access Name (SCAN)...
  SCAN Name         Node          Running?      ListenerName  Port          Running?
  ----------------  ------------  ------------  ------------  ------------  ------------
  rhel12c-scan      rhel12c1      true          LISTENER_SCAN1  1521          true

Checking TCP connectivity to SCAN Listeners...
  Node          ListenerName              TCP connectivity?
  ------------  ------------------------  ------------------------
  rhel12c1      LISTENER_SCAN1            yes
TCP connectivity to SCAN Listeners exists on all cluster nodes

Checking name resolution setup for "rhel12c-scan"...

Checking integrity of name service switch configuration file "/etc/nsswitch.conf" ...
Checking if "hosts" entry in file "/etc/nsswitch.conf" is consistent across nodes...
Checking file "/etc/nsswitch.conf" to make sure that only one "hosts" entry is defined
More than one "hosts" entry does not exist in any "/etc/nsswitch.conf" file
All nodes have same "hosts" entry defined in file "/etc/nsswitch.conf"
Check for integrity of name service switch configuration file "/etc/nsswitch.conf" passed

  SCAN Name     IP Address                Status                    Comment
  ------------  ------------------------  ------------------------  ----------
  rhel12c-scan  192.168.0.91            passed

Checking SCAN IP addresses...
Check of SCAN IP addresses passed

Verification of SCAN VIP and Listener setup passed
Checking VIP configuration.
Checking VIP Subnet configuration.
Check for VIP Subnet configuration passed.
Checking VIP reachability
Check for VIP reachability passed.

Checking Database and Clusterware version compatibility


Checking ASM and CRS version compatibility
ASM and CRS versions are compatible
Database version "12.1" is compatible with the Clusterware version "12.1.0.1.0".
Database Clusterware version compatibility passed.

Starting check for /dev/shm mounted as temporary file system ...

Check for /dev/shm mounted as temporary file system passed


NOTE:
No fixable verification failures to fix

Pre-check for database installation was successful.



2. set ORACLE_BASE environment variable and start the database software installation by executing runInstaller as oracle user.

The new administrative privileges introduced in 12c are used with the database software installation.

Once the database software is installed next step is to create the database.

Related Posts
Installing 11gR2 (11.2.0.3) GI with Role Separation on RHEL 6
Installing 11gR2 (11.2.0.3) GI with Role Separation on OEL 6
Installing 11gR2 Standalone Server with ASM and Role Separation on RHEL 6
11gR2 Standalone Data Guard (with ASM and Role Separation)

Installing 12c (12.1.0.1) RAC on RHEL 6 with Role Separation - Clusterware

Installing 12cR1 RAC without the use of the new features such as the Oracle Flex ASM is pretty much same as installing 11gR2 RAC with role separation. This post is not a step by step guide to installing 12cR1 RAC but list the main differences in the installation process compared to previous versions and pitfalls to look out for.
Among the noticeable changes and new features 12c doesn't support storing clusterware files in block devices as such provisions must be made either to store in the shared file systems or in ASM. Even if upgrading ocr files must be moved to ASM before the upgrade.
12c supports IPv6 for public IP and VIP. However this post uses IPv4 for public and VIP.
12c has also introduced several new administrative privileges for job role separation for database related tasks in data guard, key management and backup.
1. The RHEL 6 version used for the setup is RHEL 6 - U4
# uname -r
2.6.32-358.el6.x86_64
2. Create OS groups and users needed for role separation. Oracle user will own database software and grid user will own the clusterware.
groupadd dba
groupadd oinstall
groupadd oper
groupadd asmoper
groupadd asmdba
groupadd asmadmin
groupadd backupdba
groupadd dgdba
groupadd kmdba

useradd -g oinstall -G dba,oper,asmdba,backupdba,dgdba,kmdba oracle
useradd -g oinstall -G asmadmin,asmdba,asmoper,dba grid
grid user was made part of the dba group as per 1084186.1. (If this issue is still there on 12c was not tested).
3. Create user equivalence manually and do not wait for OUI to do this during the installation even though Oracle documentation says "You can configure SSH from the OUI interface during installation for the user account running the installation. The automatic configuration creates passwordless SSH connectivity between all cluster member nodes. Oracle recommends that you use the automatic procedure if possible.". Without this manual user equivalence none of the cluvfy commands will complete successfully.
4. Create directory structure for clusterware and Oracle base and set with relevant ownership and permission. For GI_HOME
mkdir -p /opt/app/12.1.0/grid 
chown -R grid:oinstall /opt/app
chown grid:oinstall /opt
chmod -R 775 /opt/app
chmod 775 /opt
For ORACLE_BASE
mkdir -p /opt/app/oracle
chown oracle:oinstall /opt/app/oracle
chmod 775 /opt/app/oracle
After running the root.sh the ownership of these directories (for GI_HOME) will be changed to root. But before the installation it must be owned by grid with oinstall as group and both user and group must have read,write and execute permissions.
5. Verify pre-req package list is installed. Follow GI installation guide and metalink note Requirements for Installing Oracle Database 12.1 on RHEL6 or OL6 64-bit (x86-64) [ID 1529864.1] for list of rpms required. Starting with 11.2.0.2 and including 12c "the 32-bit packages are required only for 32-bit client installs". In this setup servers used had both 32-bit and 64-bit version were installed.
6.Set resource limit for both oracle and grid user in /etc/security/limits.conf file
oracle soft nproc 2047
oracle hard nproc 16384
oracle soft nofile 1024
oracle hard nofile 65536
oracle soft stack 10240
oracle hard stack 32768

grid soft nproc 2047
grid hard nproc 16384
grid soft nofile 1024
grid hard nofile 65536
grid soft stack 10240
grid hard stack 32768
7. Disable x11 forward for grid and oracle user by adding the following to each user's .ssh/config file
Host *
 ForwardX11 no
8. Disable tty on .bashrc for both grid and oracle user
if [ -t 0 ]; then
 stty intr ^C
fi
and also set umask value in .bashrc for grid and oracle user
umask 022
9. ASMLib is not used for this RAC setup. Instead block devices are used as ASM disk. Setup udev rules for block devices with following ownerships and permissions.
KERNEL=="sdb[1]", OWNER="grid", GROUP="asmadmin", MODE="660"
If scsi_id command is used to find the UUID then the command listed on the GI install documentation fails on RHEL 6.
# /sbin/scsi_id -g -s /block/sdb/sdb1
/sbin/scsi_id: invalid option -- 's'
-s is not a supported option on RHEL 6. Instead use the following
/sbin/scsi_id -g -u -d /dev/sdb
Further more Oracle documentation mentions with external redundancy a minimum of 1 disk of 700MB size is sufficient to store both OCR and vote disk when not configuring cluster management repository. However it must be stated 700MB is not sufficient the disk space should be at least 1069MB.
In practice it is best to use normal or high redundancy on the disk group used to store OCR and vote disk.
10. Running cluvfy fails with the following message
ERROR:
Reference data is not available for verifying prerequisites on this operating system distribution
Verification cannot proceed
As mentioned in upgrade to 12c post install redhat-release-6Server-1.noarch.rpm found in the pack available with 1514012.1. (Also refer 1567127.1). After the aforementioned rpm is installed cluvfy can continue without any issue. If the cluvfy or OUI cannot find the UUID for a disk during the disk share check following message will be shown. From cluvfy
Checking Devices for ASM...

ERROR:  /dev/sdb1
rhel12c1:Cannot verify the shared state for device /dev/sdb1 due to Universally Unique Identifiers (UUIDs) not being found, or different values being found, for this device across nodes:
        rhel12c2,rhel12c1

Checking for shared devices...

PRVF-5149 : WARNING: Storage "/dev/sdb1" is not shared on all nodes
From OUI

Error is due to test system using VirtualBox to setup the RAC and partitions not returning an UUID. Installation could be continued ignoring this error. In a proper system where UUID is available the cluvfy would have the following messages when these check succeed.
Checking Devices for ASM...

Checking for shared devices...

  Device                                Device Type
  ------------------------------------  ------------------------
  /dev/sdb1                              Disk


Checking consistency of device owner across all nodes...
Consistency check of device owner for "/dev/sdb1" PASSED

Checking consistency of device group across all nodes...
Consistency check of device group for "/dev/sdb1" PASSED

Checking consistency of device permissions across all nodes...
Consistency check of device permissions for "/dev/sdb1" PASSED

Checking consistency of device size across all nodes...
Consistency check of device size for "/dev/sdb1" PASSED
Below is the output from running pre crsinst.
[grid@rhel12c1 grid]$ ./runcluvfy.sh stage -pre crsinst -n rhel12c1,rhel12c2 -asm -asmgrp asmadmin -presence local -asmdev /dev/sdb1 -crshome /opt/app/12.1.0/grid -fixup  -verbose

Performing pre-checks for cluster services setup

Checking node reachability...

Check: Node reachability from node "rhel12c1"
  Destination Node                      Reachable?
  ------------------------------------  ------------------------
  rhel12c1                              yes
  rhel12c2                              yes
Result: Node reachability check passed from node "rhel12c1"


Checking user equivalence...

Check: User equivalence for user "grid"
  Node Name                             Status
  ------------------------------------  ------------------------
  rhel12c2                              passed
  rhel12c1                              passed
Result: User equivalence check passed for user "grid"

Checking node connectivity...

Checking hosts config file...
  Node Name                             Status
  ------------------------------------  ------------------------
  rhel12c1                              passed
  rhel12c2                              passed

Verification of the hosts config file successful


Interface information for node "rhel12c1"
 Name   IP Address      Subnet          Gateway         Def. Gateway    HW Address        MTU
 ------ --------------- --------------- --------------- --------------- ----------------- ------
 eth0   192.168.0.93    192.168.0.0     0.0.0.0         192.168.0.100   08:00:27:CB:D8:AE 1500
 eth1   192.168.1.87    192.168.1.0     0.0.0.0         192.168.0.100   08:00:27:92:0B:69 1500


Interface information for node "rhel12c2"
 Name   IP Address      Subnet          Gateway         Def. Gateway    HW Address        MTU
 ------ --------------- --------------- --------------- --------------- ----------------- ------
 eth0   192.168.0.94    192.168.0.0     0.0.0.0         192.168.0.100   08:00:27:9C:66:81 1500
 eth1   192.168.1.88    192.168.1.0     0.0.0.0         192.168.0.100   08:00:27:1E:0E:A8 1500


Check: Node connectivity of subnet "192.168.0.0"
  Source                          Destination                     Connected?
  ------------------------------  ------------------------------  ----------------
  rhel12c1[192.168.0.93]          rhel12c2[192.168.0.94]          yes
Result: Node connectivity passed for subnet "192.168.0.0" with node(s) rhel12c1,rhel12c2


Check: TCP connectivity of subnet "192.168.0.0"
  Source                          Destination                     Connected?
  ------------------------------  ------------------------------  ----------------
  rhel12c1:192.168.0.93           rhel12c2:192.168.0.94           passed
Result: TCP connectivity check passed for subnet "192.168.0.0"


Check: Node connectivity of subnet "192.168.1.0"
  Source                          Destination                     Connected?
  ------------------------------  ------------------------------  ----------------
  rhel12c1[192.168.1.87]          rhel12c2[192.168.1.88]          yes
Result: Node connectivity passed for subnet "192.168.1.0" with node(s) rhel12c1,rhel12c2


Check: TCP connectivity of subnet "192.168.1.0"
  Source                          Destination                     Connected?
  ------------------------------  ------------------------------  ----------------
  rhel12c1:192.168.1.87           rhel12c2:192.168.1.88           passed
Result: TCP connectivity check passed for subnet "192.168.1.0"


Interfaces found on subnet "192.168.0.0" that are likely candidates for VIP are:
rhel12c1 eth0:192.168.0.93
rhel12c2 eth0:192.168.0.94

Interfaces found on subnet "192.168.1.0" that are likely candidates for a private interconnect are:
rhel12c1 eth1:192.168.1.87
rhel12c2 eth1:192.168.1.88
Checking subnet mask consistency...
Subnet mask consistency check passed for subnet "192.168.0.0".
Subnet mask consistency check passed for subnet "192.168.1.0".
Subnet mask consistency check passed.

Result: Node connectivity check passed

Checking multicast communication...

Checking subnet "192.168.0.0" for multicast communication with multicast group "224.0.0.251"...
Check of subnet "192.168.0.0" for multicast communication with multicast group "224.0.0.251" passed.

Check of multicast communication passed.

Checking ASMLib configuration.
  Node Name                             Status
  ------------------------------------  ------------------------
  rhel12c1                              passed
  rhel12c2                              passed
Result: Check for ASMLib configuration passed.

Check: Total memory
  Node Name     Available                 Required                  Status
  ------------  ------------------------  ------------------------  ----------
  rhel12c2      4.4199GB (4634568.0KB)    4GB (4194304.0KB)         passed
  rhel12c1      4.4199GB (4634568.0KB)    4GB (4194304.0KB)         passed
Result: Total memory check passed

Check: Available memory
  Node Name     Available                 Required                  Status
  ------------  ------------------------  ------------------------  ----------
  rhel12c2      4.2905GB (4498884.0KB)    50MB (51200.0KB)          passed
  rhel12c1      4.1755GB (4378320.0KB)    50MB (51200.0KB)          passed
Result: Available memory check passed

Check: Swap space
  Node Name     Available                 Required                  Status
  ------------  ------------------------  ------------------------  ----------
  rhel12c2      5GB (5242872.0KB)         4.4199GB (4634568.0KB)    passed
  rhel12c1      5GB (5242872.0KB)         4.4199GB (4634568.0KB)    passed
Result: Swap space check passed

Check: Free disk space for "rhel12c2:/usr,rhel12c2:/var,rhel12c2:/etc,rhel12c2:/opt/app/12.1.0/grid,rhel12c2:/sbin,rhel12c2:/tmp"
  Path              Node Name     Mount point   Available     Required      Status
  ----------------  ------------  ------------  ------------  ------------  ------------
  /usr              rhel12c2      /             21.3525GB     7.9635GB      passed
  /var              rhel12c2      /             21.3525GB     7.9635GB      passed
  /etc              rhel12c2      /             21.3525GB     7.9635GB      passed
  /opt/app/12.1.0/grid  rhel12c2      /             21.3525GB     7.9635GB      passed
  /sbin             rhel12c2      /             21.3525GB     7.9635GB      passed
  /tmp              rhel12c2      /             21.3525GB     7.9635GB      passed
Result: Free disk space check passed for "rhel12c2:/usr,rhel12c2:/var,rhel12c2:/etc,rhel12c2:/opt/app/12.1.0/grid,rhel12c2:/sbin,rhel12c2:/tmp"

Check: Free disk space for "rhel12c1:/usr,rhel12c1:/var,rhel12c1:/etc,rhel12c1:/opt/app/12.1.0/grid,rhel12c1:/sbin,rhel12c1:/tmp"
  Path              Node Name     Mount point   Available     Required      Status
  ----------------  ------------  ------------  ------------  ------------  ------------
  /usr              rhel12c1      /             18.3523GB     7.9635GB      passed
  /var              rhel12c1      /             18.3523GB     7.9635GB      passed
  /etc              rhel12c1      /             18.3523GB     7.9635GB      passed
  /opt/app/12.1.0/grid  rhel12c1      /             18.3523GB     7.9635GB      passed
  /sbin             rhel12c1      /             18.3523GB     7.9635GB      passed
  /tmp              rhel12c1      /             18.3523GB     7.9635GB      passed
Result: Free disk space check passed for "rhel12c1:/usr,rhel12c1:/var,rhel12c1:/etc,rhel12c1:/opt/app/12.1.0/grid,rhel12c1:/sbin,rhel12c1:/tmp"

Check: User existence for "grid"
  Node Name     Status                    Comment
  ------------  ------------------------  ------------------------
  rhel12c2      passed                    exists(501)
  rhel12c1      passed                    exists(501)

Checking for multiple users with UID value 501
Result: Check for multiple users with UID value 501 passed
Result: User existence check passed for "grid"

Check: Group existence for "oinstall"
  Node Name     Status                    Comment
  ------------  ------------------------  ------------------------
  rhel12c2      passed                    exists
  rhel12c1      passed                    exists
Result: Group existence check passed for "oinstall"

Check: Group existence for "dba"
  Node Name     Status                    Comment
  ------------  ------------------------  ------------------------
  rhel12c2      passed                    exists
  rhel12c1      passed                    exists
Result: Group existence check passed for "dba"

Check: Group existence for "asmadmin"
  Node Name     Status                    Comment
  ------------  ------------------------  ------------------------
  rhel12c2      passed                    exists
  rhel12c1      passed                    exists
Result: Group existence check passed for "asmadmin"

Check: Membership of user "grid" in group "oinstall" [as Primary]
  Node Name         User Exists   Group Exists  User in Group  Primary       Status
  ----------------  ------------  ------------  ------------  ------------  ------------
  rhel12c2          yes           yes           yes           yes           passed
  rhel12c1          yes           yes           yes           yes           passed
Result: Membership check for user "grid" in group "oinstall" [as Primary] passed

Check: Membership of user "grid" in group "dba"
  Node Name         User Exists   Group Exists  User in Group  Status
  ----------------  ------------  ------------  ------------  ----------------
  rhel12c2          yes           yes           yes           passed
  rhel12c1          yes           yes           yes           passed
Result: Membership check for user "grid" in group "dba" passed

Check: Membership of user "grid" in group "asmadmin"
  Node Name         User Exists   Group Exists  User in Group  Status
  ----------------  ------------  ------------  ------------  ----------------
  rhel12c2          yes           yes           yes           passed
  rhel12c1          yes           yes           yes           passed
Result: Membership check for user "grid" in group "asmadmin" passed

Check: Run level
  Node Name     run level                 Required                  Status
  ------------  ------------------------  ------------------------  ----------
  rhel12c2      3                         3,5                       passed
  rhel12c1      3                         3,5                       passed
Result: Run level check passed

Check: Hard limits for "maximum open file descriptors"
  Node Name         Type          Available     Required      Status
  ----------------  ------------  ------------  ------------  ----------------
  rhel12c2          hard          65536         65536         passed
  rhel12c1          hard          65536         65536         passed
Result: Hard limits check passed for "maximum open file descriptors"

Check: Soft limits for "maximum open file descriptors"
  Node Name         Type          Available     Required      Status
  ----------------  ------------  ------------  ------------  ----------------
  rhel12c2          soft          1024          1024          passed
  rhel12c1          soft          1024          1024          passed
Result: Soft limits check passed for "maximum open file descriptors"

Check: Hard limits for "maximum user processes"
  Node Name         Type          Available     Required      Status
  ----------------  ------------  ------------  ------------  ----------------
  rhel12c2          hard          16384         16384         passed
  rhel12c1          hard          16384         16384         passed
Result: Hard limits check passed for "maximum user processes"

Check: Soft limits for "maximum user processes"
  Node Name         Type          Available     Required      Status
  ----------------  ------------  ------------  ------------  ----------------
  rhel12c2          soft          2047          2047          passed
  rhel12c1          soft          2047          2047          passed
Result: Soft limits check passed for "maximum user processes"

Check: System architecture
  Node Name     Available                 Required                  Status
  ------------  ------------------------  ------------------------  ----------
  rhel12c2      x86_64                    x86_64                    passed
  rhel12c1      x86_64                    x86_64                    passed
Result: System architecture check passed

Check: Kernel version
  Node Name     Available                 Required                  Status
  ------------  ------------------------  ------------------------  ----------
  rhel12c2      2.6.32-358.el6.x86_64     2.6.32                    passed
  rhel12c1      2.6.32-358.el6.x86_64     2.6.32                    passed
Result: Kernel version check passed

Check: Kernel parameter for "semmsl"
  Node Name         Current       Configured    Required      Status        Comment
  ----------------  ------------  ------------  ------------  ------------  ------------
  rhel12c1          3010          3010          250           passed
  rhel12c2          3010          3010          250           passed
Result: Kernel parameter check passed for "semmsl"

Check: Kernel parameter for "semmns"
  Node Name         Current       Configured    Required      Status        Comment
  ----------------  ------------  ------------  ------------  ------------  ------------
  rhel12c1          385280        385280        32000         passed
  rhel12c2          385280        385280        32000         passed
Result: Kernel parameter check passed for "semmns"

Check: Kernel parameter for "semopm"
  Node Name         Current       Configured    Required      Status        Comment
  ----------------  ------------  ------------  ------------  ------------  ------------
  rhel12c1          3010          3010          100           passed
  rhel12c2          3010          3010          100           passed
Result: Kernel parameter check passed for "semopm"

Check: Kernel parameter for "semmni"
  Node Name         Current       Configured    Required      Status        Comment
  ----------------  ------------  ------------  ------------  ------------  ------------
  rhel12c1          128           128           128           passed
  rhel12c2          128           128           128           passed
Result: Kernel parameter check passed for "semmni"

Check: Kernel parameter for "shmmax"
  Node Name         Current       Configured    Required      Status        Comment
  ----------------  ------------  ------------  ------------  ------------  ------------
  rhel12c1          68719476736   68719476736   2372898816    passed
  rhel12c2          68719476736   68719476736   2372898816    passed
Result: Kernel parameter check passed for "shmmax"

Check: Kernel parameter for "shmmni"
  Node Name         Current       Configured    Required      Status        Comment
  ----------------  ------------  ------------  ------------  ------------  ------------
  rhel12c1          4096          4096          4096          passed
  rhel12c2          4096          4096          4096          passed
Result: Kernel parameter check passed for "shmmni"

Check: Kernel parameter for "shmall"
  Node Name         Current       Configured    Required      Status        Comment
  ----------------  ------------  ------------  ------------  ------------  ------------
  rhel12c1          4294967296    4294967296    463456        passed
  rhel12c2          4294967296    4294967296    463456        passed
Result: Kernel parameter check passed for "shmall"

Check: Kernel parameter for "file-max"
  Node Name         Current       Configured    Required      Status        Comment
  ----------------  ------------  ------------  ------------  ------------  ------------
  rhel12c1          6815744       6815744       6815744       passed
  rhel12c2          6815744       6815744       6815744       passed
Result: Kernel parameter check passed for "file-max"

Check: Kernel parameter for "ip_local_port_range"
  Node Name         Current       Configured    Required      Status        Comment
  ----------------  ------------  ------------  ------------  ------------  ------------
  rhel12c1          between 9000 & 65500  between 9000 & 65500  between 9000 & 65535  passed
  rhel12c2          between 9000 & 65500  between 9000 & 65500  between 9000 & 65535  passed
Result: Kernel parameter check passed for "ip_local_port_range"

Check: Kernel parameter for "rmem_default"
  Node Name         Current       Configured    Required      Status        Comment
  ----------------  ------------  ------------  ------------  ------------  ------------
  rhel12c1          4194304       4194304       262144        passed
  rhel12c2          4194304       4194304       262144        passed
Result: Kernel parameter check passed for "rmem_default"

Check: Kernel parameter for "rmem_max"
  Node Name         Current       Configured    Required      Status        Comment
  ----------------  ------------  ------------  ------------  ------------  ------------
  rhel12c1          4194304       4194304       4194304       passed
  rhel12c2          4194304       4194304       4194304       passed
Result: Kernel parameter check passed for "rmem_max"

Check: Kernel parameter for "wmem_default"
  Node Name         Current       Configured    Required      Status        Comment
  ----------------  ------------  ------------  ------------  ------------  ------------
  rhel12c1          1048576       1048576       262144        passed
  rhel12c2          1048576       1048576       262144        passed
Result: Kernel parameter check passed for "wmem_default"

Check: Kernel parameter for "wmem_max"
  Node Name         Current       Configured    Required      Status        Comment
  ----------------  ------------  ------------  ------------  ------------  ------------
  rhel12c1          2097152       2097152       1048576       passed
  rhel12c2          2097152       2097152       1048576       passed
Result: Kernel parameter check passed for "wmem_max"

Check: Kernel parameter for "aio-max-nr"
  Node Name         Current       Configured    Required      Status        Comment
  ----------------  ------------  ------------  ------------  ------------  ------------
  rhel12c1          3145728       3145728       1048576       passed
  rhel12c2          3145728       3145728       1048576       passed
Result: Kernel parameter check passed for "aio-max-nr"

Check: Package existence for "binutils"
  Node Name     Available                 Required                  Status
  ------------  ------------------------  ------------------------  ----------
  rhel12c2      binutils-2.20.51.0.2-5.36.el6  binutils-2.20.51.0.2      passed
  rhel12c1      binutils-2.20.51.0.2-5.36.el6  binutils-2.20.51.0.2      passed
Result: Package existence check passed for "binutils"

Check: Package existence for "compat-libcap1"
  Node Name     Available                 Required                  Status
  ------------  ------------------------  ------------------------  ----------
  rhel12c2      compat-libcap1-1.10-1     compat-libcap1-1.10       passed
  rhel12c1      compat-libcap1-1.10-1     compat-libcap1-1.10       passed
Result: Package existence check passed for "compat-libcap1"

Check: Package existence for "compat-libstdc++-33(x86_64)"
  Node Name     Available                 Required                  Status
  ------------  ------------------------  ------------------------  ----------
  rhel12c2      compat-libstdc++-33(x86_64)-3.2.3-69.el6  compat-libstdc++-33(x86_64)-3.2.3  passed
  rhel12c1      compat-libstdc++-33(x86_64)-3.2.3-69.el6  compat-libstdc++-33(x86_64)-3.2.3  passed
Result: Package existence check passed for "compat-libstdc++-33(x86_64)"

Check: Package existence for "libgcc(x86_64)"
  Node Name     Available                 Required                  Status
  ------------  ------------------------  ------------------------  ----------
  rhel12c2      libgcc(x86_64)-4.4.7-3.el6  libgcc(x86_64)-4.4.4      passed
  rhel12c1      libgcc(x86_64)-4.4.7-3.el6  libgcc(x86_64)-4.4.4      passed
Result: Package existence check passed for "libgcc(x86_64)"

Check: Package existence for "libstdc++(x86_64)"
  Node Name     Available                 Required                  Status
  ------------  ------------------------  ------------------------  ----------
  rhel12c2      libstdc++(x86_64)-4.4.7-3.el6  libstdc++(x86_64)-4.4.4   passed
  rhel12c1      libstdc++(x86_64)-4.4.7-3.el6  libstdc++(x86_64)-4.4.4   passed
Result: Package existence check passed for "libstdc++(x86_64)"

Check: Package existence for "libstdc++-devel(x86_64)"
  Node Name     Available                 Required                  Status
  ------------  ------------------------  ------------------------  ----------
  rhel12c2      libstdc++-devel(x86_64)-4.4.7-3.el6  libstdc++-devel(x86_64)-4.4.4  passed
  rhel12c1      libstdc++-devel(x86_64)-4.4.7-3.el6  libstdc++-devel(x86_64)-4.4.4  passed
Result: Package existence check passed for "libstdc++-devel(x86_64)"

Check: Package existence for "sysstat"
  Node Name     Available                 Required                  Status
  ------------  ------------------------  ------------------------  ----------
  rhel12c2      sysstat-9.0.4-20.el6      sysstat-9.0.4             passed
  rhel12c1      sysstat-9.0.4-20.el6      sysstat-9.0.4             passed
Result: Package existence check passed for "sysstat"

Check: Package existence for "gcc"
  Node Name     Available                 Required                  Status
  ------------  ------------------------  ------------------------  ----------
  rhel12c2      gcc-4.4.7-3.el6           gcc-4.4.4                 passed
  rhel12c1      gcc-4.4.7-3.el6           gcc-4.4.4                 passed
Result: Package existence check passed for "gcc"

Check: Package existence for "gcc-c++"
  Node Name     Available                 Required                  Status
  ------------  ------------------------  ------------------------  ----------
  rhel12c2      gcc-c++-4.4.7-3.el6       gcc-c++-4.4.4             passed
  rhel12c1      gcc-c++-4.4.7-3.el6       gcc-c++-4.4.4             passed
Result: Package existence check passed for "gcc-c++"

Check: Package existence for "ksh"
  Node Name     Available                 Required                  Status
  ------------  ------------------------  ------------------------  ----------
  rhel12c2      ksh-20100621-19.el6       ksh-...                   passed
  rhel12c1      ksh-20100621-19.el6       ksh-...                   passed
Result: Package existence check passed for "ksh"

Check: Package existence for "make"
  Node Name     Available                 Required                  Status
  ------------  ------------------------  ------------------------  ----------
  rhel12c2      make-3.81-20.el6          make-3.81                 passed
  rhel12c1      make-3.81-20.el6          make-3.81                 passed
Result: Package existence check passed for "make"

Check: Package existence for "glibc(x86_64)"
  Node Name     Available                 Required                  Status
  ------------  ------------------------  ------------------------  ----------
  rhel12c2      glibc(x86_64)-2.12-1.107.el6  glibc(x86_64)-2.12        passed
  rhel12c1      glibc(x86_64)-2.12-1.107.el6  glibc(x86_64)-2.12        passed
Result: Package existence check passed for "glibc(x86_64)"

Check: Package existence for "glibc-devel(x86_64)"
  Node Name     Available                 Required                  Status
  ------------  ------------------------  ------------------------  ----------
  rhel12c2      glibc-devel(x86_64)-2.12-1.107.el6  glibc-devel(x86_64)-2.12  passed
  rhel12c1      glibc-devel(x86_64)-2.12-1.107.el6  glibc-devel(x86_64)-2.12  passed
Result: Package existence check passed for "glibc-devel(x86_64)"

Check: Package existence for "libaio(x86_64)"
  Node Name     Available                 Required                  Status
  ------------  ------------------------  ------------------------  ----------
  rhel12c2      libaio(x86_64)-0.3.107-10.el6  libaio(x86_64)-0.3.107    passed
  rhel12c1      libaio(x86_64)-0.3.107-10.el6  libaio(x86_64)-0.3.107    passed
Result: Package existence check passed for "libaio(x86_64)"

Check: Package existence for "libaio-devel(x86_64)"
  Node Name     Available                 Required                  Status
  ------------  ------------------------  ------------------------  ----------
  rhel12c2      libaio-devel(x86_64)-0.3.107-10.el6  libaio-devel(x86_64)-0.3.107  passed
  rhel12c1      libaio-devel(x86_64)-0.3.107-10.el6  libaio-devel(x86_64)-0.3.107  passed
Result: Package existence check passed for "libaio-devel(x86_64)"

Check: Package existence for "nfs-utils"
  Node Name     Available                 Required                  Status
  ------------  ------------------------  ------------------------  ----------
  rhel12c2      nfs-utils-1.2.3-36.el6    nfs-utils-1.2.3-15        passed
  rhel12c1      nfs-utils-1.2.3-36.el6    nfs-utils-1.2.3-15        passed
Result: Package existence check passed for "nfs-utils"

Checking availability of ports "6200,6100" required for component "Oracle Notification Service (ONS)"
  Node Name         Port Number   Protocol      Available     Status
  ----------------  ------------  ------------  ------------  ----------------
  rhel12c2          6200          TCP           yes           successful
  rhel12c1          6200          TCP           yes           successful
  rhel12c2          6100          TCP           yes           successful
  rhel12c1          6100          TCP           yes           successful
Result: Port availability check passed for ports "6200,6100"

Checking for multiple users with UID value 0
Result: Check for multiple users with UID value 0 passed

Check: Current group ID
Result: Current group ID check passed

Starting check for consistency of primary group of root user
  Node Name                             Status
  ------------------------------------  ------------------------
  rhel12c2                              passed
  rhel12c1                              passed

Check for consistency of root user's primary group passed

Check: Package existence for "cvuqdisk"
  Node Name     Available                 Required                  Status
  ------------  ------------------------  ------------------------  ----------
  rhel12c2      cvuqdisk-1.0.9-1          cvuqdisk-1.0.9-1          passed
  rhel12c1      cvuqdisk-1.0.9-1          cvuqdisk-1.0.9-1          passed
Result: Package existence check passed for "cvuqdisk"


Checking Devices for ASM...

ERROR:  /dev/sdb1
rhel12c1:Cannot verify the shared state for device /dev/sdb1 due to Universally Unique Identifiers (UUIDs) not being found, or different values being found, for this device across nodes:
        rhel12c2,rhel12c1

Checking for shared devices...

PRVF-5149 : WARNING: Storage "/dev/sdb1" is not shared on all nodes

UDev attributes check for ASM Disks started...
Checking udev settings for device "/dev/sdb1"
  Device            Owner         Group         Permissions   Result
  ----------------  ------------  ------------  ------------  ----------------
  sdb1              grid          asmadmin      0660          passed

  sdb1              grid          asmadmin      0660          passed

Result: UDev attributes check passed for ASM Disks

Result: Devices check for ASM passed


Starting Clock synchronization checks using Network Time Protocol(NTP)...

NTP Configuration file check started...
Network Time Protocol(NTP) configuration file not found on any of the nodes. Oracle Cluster Time Synchronization Service(CTSS) can be used instead of NTP for time synchronization on the cluster nodes
No NTP Daemons or Services were found to be running

Result: Clock synchronization check using Network Time Protocol(NTP) passed

Checking Core file name pattern consistency...
Core file name pattern consistency check passed.

Checking to make sure user "grid" is not in "root" group
  Node Name     Status                    Comment
  ------------  ------------------------  ------------------------
  rhel12c2      passed                    does not exist
  rhel12c1      passed                    does not exist
Result: User "grid" is not part of "root" group. Check passed

Check default user file creation mask
  Node Name     Available                 Required                  Comment
  ------------  ------------------------  ------------------------  ----------
  rhel12c2      0022                      0022                      passed
  rhel12c1      0022                      0022                      passed
Result: Default user file creation mask check passed
Checking integrity of file "/etc/resolv.conf" across nodes

Checking the file "/etc/resolv.conf" to make sure only one of domain and search entries is defined
"domain" and "search" entries do not coexist in any  "/etc/resolv.conf" file
Checking if domain entry in file "/etc/resolv.conf" is consistent across the nodes...
"domain" entry does not exist in any "/etc/resolv.conf" file
Checking if search entry in file "/etc/resolv.conf" is consistent across the nodes...
Checking file "/etc/resolv.conf" to make sure that only one search entry is defined
More than one "search" entry does not exist in any "/etc/resolv.conf" file
All nodes have same "search" order defined in file "/etc/resolv.conf"
Checking DNS response time for an unreachable node
  Node Name                             Status
  ------------------------------------  ------------------------
  rhel12c1                              passed
  rhel12c2                              passed
The DNS response time for an unreachable node is within acceptable limit on all nodes

Check for integrity of file "/etc/resolv.conf" passed

Check: Time zone consistency
Result: Time zone consistency check passed

Checking integrity of name service switch configuration file "/etc/nsswitch.conf" ...
Checking if "hosts" entry in file "/etc/nsswitch.conf" is consistent across nodes...
Checking file "/etc/nsswitch.conf" to make sure that only one "hosts" entry is defined
More than one "hosts" entry does not exist in any "/etc/nsswitch.conf" file
All nodes have same "hosts" entry defined in file "/etc/nsswitch.conf"
Check for integrity of name service switch configuration file "/etc/nsswitch.conf" passed


Checking daemon "avahi-daemon" is not configured and running

Check: Daemon "avahi-daemon" not configured
  Node Name     Configured                Status
  ------------  ------------------------  ------------------------
  rhel12c2      no                        passed
  rhel12c1      no                        passed
Daemon not configured check passed for process "avahi-daemon"

Check: Daemon "avahi-daemon" not running
  Node Name     Running?                  Status
  ------------  ------------------------  ------------------------
  rhel12c2      no                        passed
  rhel12c1      no                        passed
Daemon not running check passed for process "avahi-daemon"

Starting check for /dev/shm mounted as temporary file system ...

Check for /dev/shm mounted as temporary file system passed


NOTE:
No fixable verification failures to fix

Pre-check for cluster services setup was successful.
Despite the issues on the ASM disk share cluvfy reports pre-reqs to be successful.



11. Start the clusterware installation by executing runInstaller.
12c provides many "use for" options for NICs. Since this is not setting up Flex ASM for eth1 select private. Oracle documentation states "Oracle Flex ASM can use either the same private networks as Oracle Clusterware, or use its own dedicated private networks. Each network can be classified PUBLIC, ASM & PRIVATE, PRIVATE, or ASM."
Selecting no prompts the below shown message. If GI management repository is installed then ASM space requirement would be higher. Refer Oracle documentation and 12c upgrade post for space requirements for management repository

12. When prompted execute the root.sh scripts. Below is the output from running the root.sh on first node.
[root@rhel12c1 ~]# /opt/app/12.1.0/grid/root.sh
Performing root user operation for Oracle 12c

The following environment variables are set as:
    ORACLE_OWNER= grid
    ORACLE_HOME=  /opt/app/12.1.0/grid

Enter the full pathname of the local bin directory: [/usr/local/bin]:
   Copying dbhome to /usr/local/bin ...
   Copying oraenv to /usr/local/bin ...
   Copying coraenv to /usr/local/bin ...


Creating /etc/oratab file...
Entries will be added to the /etc/oratab file as needed by
Database Configuration Assistant when a database is created
Finished running generic part of root script.
Now product-specific root actions will be performed.
Using configuration parameter file: /opt/app/12.1.0/grid/crs/install/crsconfig_params
2013/08/20 17:35:28 CLSRSC-363: User ignored prerequisites during installation

OLR initialization - successful
  root wallet
  root wallet cert
  root cert export
  peer wallet
  profile reader wallet
  pa wallet
  peer wallet keys
  pa wallet keys
  peer cert request
  pa cert request
  peer cert
  pa cert
  peer root cert TP
  profile reader root cert TP
  pa root cert TP
  peer pa cert TP
  pa peer cert TP
  profile reader pa cert TP
  profile reader peer cert TP
  peer user cert
  pa user cert
2013/08/20 17:36:29 CLSRSC-330: Adding Clusterware entries to file 'oracle-ohasd.conf'

CRS-4133: Oracle High Availability Services has been stopped.
CRS-4123: Oracle High Availability Services has been started.
CRS-4133: Oracle High Availability Services has been stopped.
CRS-4123: Oracle High Availability Services has been started.
CRS-2673: Attempting to stop 'ora.drivers.acfs' on 'rhel12c1'
CRS-2677: Stop of 'ora.drivers.acfs' on 'rhel12c1' succeeded
CRS-2672: Attempting to start 'ora.evmd' on 'rhel12c1'
CRS-2672: Attempting to start 'ora.mdnsd' on 'rhel12c1'
CRS-2676: Start of 'ora.evmd' on 'rhel12c1' succeeded
CRS-2676: Start of 'ora.mdnsd' on 'rhel12c1' succeeded
CRS-2672: Attempting to start 'ora.gpnpd' on 'rhel12c1'
CRS-2676: Start of 'ora.gpnpd' on 'rhel12c1' succeeded
CRS-2672: Attempting to start 'ora.cssdmonitor' on 'rhel12c1'
CRS-2672: Attempting to start 'ora.gipcd' on 'rhel12c1'
CRS-2676: Start of 'ora.cssdmonitor' on 'rhel12c1' succeeded
CRS-2676: Start of 'ora.gipcd' on 'rhel12c1' succeeded
CRS-2672: Attempting to start 'ora.cssd' on 'rhel12c1'
CRS-2672: Attempting to start 'ora.diskmon' on 'rhel12c1'
CRS-2676: Start of 'ora.diskmon' on 'rhel12c1' succeeded
CRS-2676: Start of 'ora.cssd' on 'rhel12c1' succeeded

ASM created and started successfully.

Disk Group clusterdg created successfully.

CRS-2672: Attempting to start 'ora.storage' on 'rhel12c1'
CRS-2676: Start of 'ora.storage' on 'rhel12c1' succeeded
CRS-2672: Attempting to start 'ora.crsd' on 'rhel12c1'
CRS-2676: Start of 'ora.crsd' on 'rhel12c1' succeeded
CRS-4256: Updating the profile
Successful addition of voting disk c09b02f830b64f49bf06b75f49b272ee.
Successfully replaced voting disk group with +clusterdg.
CRS-4256: Updating the profile
CRS-4266: Voting file(s) successfully replaced
##  STATE    File Universal Id                File Name Disk group
--  -----    -----------------                --------- ---------
 1. ONLINE   c09b02f830b64f49bf06b75f49b272ee (/dev/sdb1) [CLUSTERDG]
Located 1 voting disk(s).
CRS-2791: Starting shutdown of Oracle High Availability Services-managed resources on 'rhel12c1'
CRS-2673: Attempting to stop 'ora.crsd' on 'rhel12c1'
CRS-2677: Stop of 'ora.crsd' on 'rhel12c1' succeeded
CRS-2673: Attempting to stop 'ora.drivers.acfs' on 'rhel12c1'
CRS-2673: Attempting to stop 'ora.evmd' on 'rhel12c1'
CRS-2673: Attempting to stop 'ora.ctssd' on 'rhel12c1'
CRS-2673: Attempting to stop 'ora.storage' on 'rhel12c1'
CRS-2673: Attempting to stop 'ora.mdnsd' on 'rhel12c1'
CRS-2673: Attempting to stop 'ora.gpnpd' on 'rhel12c1'
CRS-2677: Stop of 'ora.drivers.acfs' on 'rhel12c1' succeeded
CRS-2677: Stop of 'ora.storage' on 'rhel12c1' succeeded
CRS-2673: Attempting to stop 'ora.asm' on 'rhel12c1'
CRS-2677: Stop of 'ora.mdnsd' on 'rhel12c1' succeeded
CRS-2677: Stop of 'ora.gpnpd' on 'rhel12c1' succeeded
CRS-2677: Stop of 'ora.evmd' on 'rhel12c1' succeeded
CRS-2677: Stop of 'ora.ctssd' on 'rhel12c1' succeeded
CRS-2677: Stop of 'ora.asm' on 'rhel12c1' succeeded
CRS-2673: Attempting to stop 'ora.cluster_interconnect.haip' on 'rhel12c1'
CRS-2677: Stop of 'ora.cluster_interconnect.haip' on 'rhel12c1' succeeded
CRS-2673: Attempting to stop 'ora.cssd' on 'rhel12c1'
CRS-2677: Stop of 'ora.cssd' on 'rhel12c1' succeeded
CRS-2673: Attempting to stop 'ora.gipcd' on 'rhel12c1'
CRS-2677: Stop of 'ora.gipcd' on 'rhel12c1' succeeded
CRS-2793: Shutdown of Oracle High Availability Services-managed resources on 'rhel12c1' has completed
CRS-4133: Oracle High Availability Services has been stopped.
CRS-4123: Starting Oracle High Availability Services-managed resources
CRS-2672: Attempting to start 'ora.mdnsd' on 'rhel12c1'
CRS-2672: Attempting to start 'ora.evmd' on 'rhel12c1'
CRS-2676: Start of 'ora.mdnsd' on 'rhel12c1' succeeded
CRS-2676: Start of 'ora.evmd' on 'rhel12c1' succeeded
CRS-2672: Attempting to start 'ora.gpnpd' on 'rhel12c1'
CRS-2676: Start of 'ora.gpnpd' on 'rhel12c1' succeeded
CRS-2672: Attempting to start 'ora.gipcd' on 'rhel12c1'
CRS-2676: Start of 'ora.gipcd' on 'rhel12c1' succeeded
CRS-2672: Attempting to start 'ora.cssdmonitor' on 'rhel12c1'
CRS-2676: Start of 'ora.cssdmonitor' on 'rhel12c1' succeeded
CRS-2672: Attempting to start 'ora.cssd' on 'rhel12c1'
CRS-2672: Attempting to start 'ora.diskmon' on 'rhel12c1'
CRS-2676: Start of 'ora.diskmon' on 'rhel12c1' succeeded
CRS-2789: Cannot stop resource 'ora.diskmon' as it is not running on server 'rhel12c1'
CRS-2676: Start of 'ora.cssd' on 'rhel12c1' succeeded
CRS-2672: Attempting to start 'ora.cluster_interconnect.haip' on 'rhel12c1'
CRS-2672: Attempting to start 'ora.ctssd' on 'rhel12c1'
CRS-2676: Start of 'ora.ctssd' on 'rhel12c1' succeeded
CRS-2676: Start of 'ora.cluster_interconnect.haip' on 'rhel12c1' succeeded
CRS-2672: Attempting to start 'ora.asm' on 'rhel12c1'
CRS-2676: Start of 'ora.asm' on 'rhel12c1' succeeded
CRS-2672: Attempting to start 'ora.storage' on 'rhel12c1'
CRS-2676: Start of 'ora.storage' on 'rhel12c1' succeeded
CRS-2672: Attempting to start 'ora.crsd' on 'rhel12c1'
CRS-2676: Start of 'ora.crsd' on 'rhel12c1' succeeded
CRS-6023: Starting Oracle Cluster Ready Services-managed resources
CRS-6017: Processing resource auto-start for servers: rhel12c1
CRS-6016: Resource auto-start has completed for server rhel12c1
CRS-6024: Completed start of Oracle Cluster Ready Services-managed resources
CRS-4123: Oracle High Availability Services has been started.
2013/08/20 17:44:10 CLSRSC-343: Successfully started Oracle clusterware stack

CRS-2672: Attempting to start 'ora.asm' on 'rhel12c1'
CRS-2676: Start of 'ora.asm' on 'rhel12c1' succeeded
CRS-2672: Attempting to start 'ora.CLUSTERDG.dg' on 'rhel12c1'
CRS-2676: Start of 'ora.CLUSTERDG.dg' on 'rhel12c1' succeeded
2013/08/20 17:45:19 CLSRSC-325: Configure Oracle Grid Infrastructure for a Cluster ... succeeded
Running root.sh on second node.
[root@rhel12c2 ~]# /opt/app/12.1.0/grid/root.sh
Performing root user operation for Oracle 12c

The following environment variables are set as:
    ORACLE_OWNER= grid
    ORACLE_HOME=  /opt/app/12.1.0/grid

Enter the full pathname of the local bin directory: [/usr/local/bin]:
   Copying dbhome to /usr/local/bin ...
   Copying oraenv to /usr/local/bin ...
   Copying coraenv to /usr/local/bin ...


Creating /etc/oratab file...
Entries will be added to the /etc/oratab file as needed by
Database Configuration Assistant when a database is created
Finished running generic part of root script.
Now product-specific root actions will be performed.
Using configuration parameter file: /opt/app/12.1.0/grid/crs/install/crsconfig_params
2013/08/20 17:45:54 CLSRSC-363: User ignored prerequisites during installation

OLR initialization - successful
2013/08/20 17:46:24 CLSRSC-330: Adding Clusterware entries to file 'oracle-ohasd.conf'

CRS-4133: Oracle High Availability Services has been stopped.
CRS-4123: Oracle High Availability Services has been started.
CRS-4133: Oracle High Availability Services has been stopped.
CRS-4123: Oracle High Availability Services has been started.
CRS-2791: Starting shutdown of Oracle High Availability Services-managed resources on 'rhel12c2'
CRS-2673: Attempting to stop 'ora.drivers.acfs' on 'rhel12c2'
CRS-2677: Stop of 'ora.drivers.acfs' on 'rhel12c2' succeeded
CRS-2793: Shutdown of Oracle High Availability Services-managed resources on 'rhel12c2' has completed
CRS-4133: Oracle High Availability Services has been stopped.
CRS-4123: Starting Oracle High Availability Services-managed resources
CRS-2672: Attempting to start 'ora.mdnsd' on 'rhel12c2'
CRS-2672: Attempting to start 'ora.evmd' on 'rhel12c2'
CRS-2676: Start of 'ora.mdnsd' on 'rhel12c2' succeeded
CRS-2676: Start of 'ora.evmd' on 'rhel12c2' succeeded
CRS-2672: Attempting to start 'ora.gpnpd' on 'rhel12c2'
CRS-2676: Start of 'ora.gpnpd' on 'rhel12c2' succeeded
CRS-2672: Attempting to start 'ora.gipcd' on 'rhel12c2'
CRS-2676: Start of 'ora.gipcd' on 'rhel12c2' succeeded
CRS-2672: Attempting to start 'ora.cssdmonitor' on 'rhel12c2'
CRS-2676: Start of 'ora.cssdmonitor' on 'rhel12c2' succeeded
CRS-2672: Attempting to start 'ora.cssd' on 'rhel12c2'
CRS-2672: Attempting to start 'ora.diskmon' on 'rhel12c2'
CRS-2676: Start of 'ora.diskmon' on 'rhel12c2' succeeded
CRS-2789: Cannot stop resource 'ora.diskmon' as it is not running on server 'rhel12c2'
CRS-2676: Start of 'ora.cssd' on 'rhel12c2' succeeded
CRS-2672: Attempting to start 'ora.cluster_interconnect.haip' on 'rhel12c2'
CRS-2672: Attempting to start 'ora.ctssd' on 'rhel12c2'
CRS-2676: Start of 'ora.ctssd' on 'rhel12c2' succeeded
CRS-2676: Start of 'ora.cluster_interconnect.haip' on 'rhel12c2' succeeded
CRS-2672: Attempting to start 'ora.asm' on 'rhel12c2'
CRS-2676: Start of 'ora.asm' on 'rhel12c2' succeeded
CRS-2672: Attempting to start 'ora.storage' on 'rhel12c2'
CRS-2676: Start of 'ora.storage' on 'rhel12c2' succeeded
CRS-2672: Attempting to start 'ora.crsd' on 'rhel12c2'
CRS-2676: Start of 'ora.crsd' on 'rhel12c2' succeeded
CRS-6017: Processing resource auto-start for servers: rhel12c2
CRS-2672: Attempting to start 'ora.ons' on 'rhel12c2'
CRS-2676: Start of 'ora.ons' on 'rhel12c2' succeeded
CRS-6016: Resource auto-start has completed for server rhel12c2
CRS-6024: Completed start of Oracle Cluster Ready Services-managed resources
CRS-4123: Oracle High Availability Services has been started.
2013/08/20 17:52:08 CLSRSC-343: Successfully started Oracle clusterware stack

2013/08/20 17:52:32 CLSRSC-325: Configure Oracle Grid Infrastructure for a Cluster ... succeeded
13. Once root.sh is completed click OK on the execute script dialog to continue with the remaining configurations after which clusterware installation is complete.
14. Verify the cluster installation with cluvfy post crsinst
cluvfy stage -post crsinst -n rhel12c1,rhel12c2 -verbose
15. Use ASMCA to create additional ASM disk groups that will be used for data and flash recovery when database is setup. Cluvfy could be used to verify if the disk are shared
cluvfy comp ssa -n rhel12c1,rhel12c2 -s /dev/sdc1,/dev/sdd1 -t data -r 12.1 -verbose

Next step is to install the Oracle Database software.

Related Posts
Installing 11gR2 (11.2.0.3) GI with Role Separation on RHEL 6
Installing 11gR2 (11.2.0.3) GI with Role Separation on OEL 6
Installing 11gR2 Standalone Server with ASM and Role Separation on RHEL 6
11gR2 Standalone Data Guard (with ASM and Role Separation)