Installing 12c (12.1.0.2) Flex Cluster on RHEL 6 with Role Separation

Oracle 12c introduced a new type of cluster configuration called Flex Cluster. This post highlights the steps of installing a Flex cluster. It's not a complete "how to guide". Most of the steps are similar to 12 standard cluster installation. But for flex cluster, GNS setup is a must.
Also it seems that the ocr and vote diskgroup size requirements are also different when installing a flex cluster. 12.1.0.2 made it mandatory to have the cluster management DB as such the space requirement for diskgroup containing OCR and vote disks is higher compared to 12.1.0.1

The new space requirement is 5528MB (for external redundancy diskgroup) but earlier for upgrading from 12.1.0.1 to 12.1.0.2 it was 4424MB (for external redundancy diskgroup).
If the disk redundancy is normal then the size of the disks must be same, dissimilar size disks are not allowed (this restriction doesn't apply to external redundancy diskgroups).

Other per-installation requirements are similar to previous version of 12.1.0.1.
Start the installation specify it's a flex cluster configuration.

Specify the GNS configuration

Specify the node list and also the role of the node. In this two node cluster setup both nodes are set as hub nodes.

With flex cluster an additional network could be defined for Flex ASM or as in this case allow the private interconnect network to be used for for flex ASM as well.

The installer only prompt to run orainstRoot.sh only on one of the nodes. However root.sh is to be run on all the nodes.

Output from the root.sh

[root@rhel12c1 app]# /opt/app/12.1.0/grid/root.sh
Performing root user operation.

The following environment variables are set as:
    ORACLE_OWNER= grid
    ORACLE_HOME=  /opt/app/12.1.0/grid

Enter the full pathname of the local bin directory: [/usr/local/bin]:
The contents of "dbhome" have not changed. No need to overwrite.
The contents of "oraenv" have not changed. No need to overwrite.
The contents of "coraenv" have not changed. No need to overwrite.


Creating /etc/oratab file...
Entries will be added to the /etc/oratab file as needed by
Database Configuration Assistant when a database is created
Finished running generic part of root script.
Now product-specific root actions will be performed.
Using configuration parameter file: /opt/app/12.1.0/grid/crs/install/crsconfig_params
2015/08/14 13:03:13 CLSRSC-4001: Installing Oracle Trace File Analyzer (TFA) Collector.

2015/08/14 13:03:47 CLSRSC-4002: Successfully installed Oracle Trace File Analyzer (TFA) Collector.

2015/08/14 13:03:50 CLSRSC-363: User ignored prerequisites during installation

OLR initialization - successful
  root wallet
  root wallet cert
  root cert export
  peer wallet
  profile reader wallet
  pa wallet
  peer wallet keys
  pa wallet keys
  peer cert request
  pa cert request
  peer cert
  pa cert
  peer root cert TP
  profile reader root cert TP
  pa root cert TP
  peer pa cert TP
  pa peer cert TP
  profile reader pa cert TP
  profile reader peer cert TP
  peer user cert
  pa user cert
2015/08/14 13:05:01 CLSRSC-330: Adding Clusterware entries to file 'oracle-ohasd.conf'

CRS-4133: Oracle High Availability Services has been stopped.
CRS-4123: Oracle High Availability Services has been started.
CRS-4133: Oracle High Availability Services has been stopped.
CRS-4123: Oracle High Availability Services has been started.
CRS-2672: Attempting to start 'ora.evmd' on 'rhel12c1'
CRS-2672: Attempting to start 'ora.mdnsd' on 'rhel12c1'
CRS-2676: Start of 'ora.mdnsd' on 'rhel12c1' succeeded
CRS-2676: Start of 'ora.evmd' on 'rhel12c1' succeeded
CRS-2672: Attempting to start 'ora.gpnpd' on 'rhel12c1'
CRS-2676: Start of 'ora.gpnpd' on 'rhel12c1' succeeded
CRS-2672: Attempting to start 'ora.cssdmonitor' on 'rhel12c1'
CRS-2672: Attempting to start 'ora.gipcd' on 'rhel12c1'
CRS-2676: Start of 'ora.cssdmonitor' on 'rhel12c1' succeeded
CRS-2676: Start of 'ora.gipcd' on 'rhel12c1' succeeded
CRS-2672: Attempting to start 'ora.cssd' on 'rhel12c1'
CRS-2672: Attempting to start 'ora.diskmon' on 'rhel12c1'
CRS-2676: Start of 'ora.diskmon' on 'rhel12c1' succeeded
CRS-2676: Start of 'ora.cssd' on 'rhel12c1' succeeded

ASM created and started successfully.

Disk Group clusterdg created successfully.

CRS-2672: Attempting to start 'ora.crf' on 'rhel12c1'
CRS-2672: Attempting to start 'ora.storage' on 'rhel12c1'
CRS-2676: Start of 'ora.storage' on 'rhel12c1' succeeded
CRS-2676: Start of 'ora.crf' on 'rhel12c1' succeeded
CRS-2672: Attempting to start 'ora.crsd' on 'rhel12c1'
CRS-2676: Start of 'ora.crsd' on 'rhel12c1' succeeded
CRS-4256: Updating the profile
Successful addition of voting disk 27c22c794c2a4f7fbfca7979664be132.
Successfully replaced voting disk group with +clusterdg.
CRS-4256: Updating the profile
CRS-4266: Voting file(s) successfully replaced
##  STATE    File Universal Id                File Name Disk group
--  -----    -----------------                --------- ---------
 1. ONLINE   27c22c794c2a4f7fbfca7979664be132 (/dev/sde1) [CLUSTERDG]
Located 1 voting disk(s).
CRS-2791: Starting shutdown of Oracle High Availability Services-managed resources on 'rhel12c1'
CRS-2673: Attempting to stop 'ora.crsd' on 'rhel12c1'
CRS-2677: Stop of 'ora.crsd' on 'rhel12c1' succeeded
CRS-2673: Attempting to stop 'ora.storage' on 'rhel12c1'
CRS-2673: Attempting to stop 'ora.mdnsd' on 'rhel12c1'
CRS-2673: Attempting to stop 'ora.gpnpd' on 'rhel12c1'
CRS-2673: Attempting to stop 'ora.drivers.acfs' on 'rhel12c1'
CRS-2677: Stop of 'ora.storage' on 'rhel12c1' succeeded
CRS-2677: Stop of 'ora.drivers.acfs' on 'rhel12c1' succeeded
CRS-2673: Attempting to stop 'ora.crf' on 'rhel12c1'
CRS-2673: Attempting to stop 'ora.ctssd' on 'rhel12c1'
CRS-2673: Attempting to stop 'ora.evmd' on 'rhel12c1'
CRS-2673: Attempting to stop 'ora.asm' on 'rhel12c1'
CRS-2677: Stop of 'ora.gpnpd' on 'rhel12c1' succeeded
CRS-2677: Stop of 'ora.mdnsd' on 'rhel12c1' succeeded
CRS-2677: Stop of 'ora.ctssd' on 'rhel12c1' succeeded
CRS-2677: Stop of 'ora.crf' on 'rhel12c1' succeeded
CRS-2677: Stop of 'ora.evmd' on 'rhel12c1' succeeded
CRS-2677: Stop of 'ora.asm' on 'rhel12c1' succeeded
CRS-2673: Attempting to stop 'ora.cluster_interconnect.haip' on 'rhel12c1'
CRS-2677: Stop of 'ora.cluster_interconnect.haip' on 'rhel12c1' succeeded
CRS-2673: Attempting to stop 'ora.cssd' on 'rhel12c1'
CRS-2677: Stop of 'ora.cssd' on 'rhel12c1' succeeded
CRS-2673: Attempting to stop 'ora.gipcd' on 'rhel12c1'
CRS-2677: Stop of 'ora.gipcd' on 'rhel12c1' succeeded
CRS-2793: Shutdown of Oracle High Availability Services-managed resources on 'rhel12c1' has completed
CRS-4133: Oracle High Availability Services has been stopped.
CRS-4123: Starting Oracle High Availability Services-managed resources
CRS-2672: Attempting to start 'ora.mdnsd' on 'rhel12c1'
CRS-2672: Attempting to start 'ora.evmd' on 'rhel12c1'
CRS-2676: Start of 'ora.evmd' on 'rhel12c1' succeeded
CRS-2676: Start of 'ora.mdnsd' on 'rhel12c1' succeeded
CRS-2672: Attempting to start 'ora.gpnpd' on 'rhel12c1'
CRS-2676: Start of 'ora.gpnpd' on 'rhel12c1' succeeded
CRS-2672: Attempting to start 'ora.gipcd' on 'rhel12c1'
CRS-2676: Start of 'ora.gipcd' on 'rhel12c1' succeeded
CRS-2672: Attempting to start 'ora.cssdmonitor' on 'rhel12c1'
CRS-2676: Start of 'ora.cssdmonitor' on 'rhel12c1' succeeded
CRS-2672: Attempting to start 'ora.cssd' on 'rhel12c1'
CRS-2672: Attempting to start 'ora.diskmon' on 'rhel12c1'
CRS-2676: Start of 'ora.diskmon' on 'rhel12c1' succeeded
CRS-2676: Start of 'ora.cssd' on 'rhel12c1' succeeded
CRS-2672: Attempting to start 'ora.cluster_interconnect.haip' on 'rhel12c1'
CRS-2672: Attempting to start 'ora.ctssd' on 'rhel12c1'
CRS-2676: Start of 'ora.ctssd' on 'rhel12c1' succeeded
CRS-2676: Start of 'ora.cluster_interconnect.haip' on 'rhel12c1' succeeded
CRS-2672: Attempting to start 'ora.asm' on 'rhel12c1'
CRS-2676: Start of 'ora.asm' on 'rhel12c1' succeeded
CRS-2672: Attempting to start 'ora.storage' on 'rhel12c1'
CRS-2676: Start of 'ora.storage' on 'rhel12c1' succeeded
CRS-2672: Attempting to start 'ora.crf' on 'rhel12c1'
CRS-2676: Start of 'ora.crf' on 'rhel12c1' succeeded
CRS-2672: Attempting to start 'ora.crsd' on 'rhel12c1'
CRS-2676: Start of 'ora.crsd' on 'rhel12c1' succeeded
CRS-6023: Starting Oracle Cluster Ready Services-managed resources
CRS-6017: Processing resource auto-start for servers: rhel12c1
CRS-6016: Resource auto-start has completed for server rhel12c1
CRS-6024: Completed start of Oracle Cluster Ready Services-managed resources
CRS-4123: Oracle High Availability Services has been started.
2015/08/14 13:11:52 CLSRSC-343: Successfully started Oracle Clusterware stack

CRS-2672: Attempting to start 'ora.net1.network' on 'rhel12c1'
CRS-2676: Start of 'ora.net1.network' on 'rhel12c1' succeeded
CRS-2672: Attempting to start 'ora.gns.vip' on 'rhel12c1'
CRS-2676: Start of 'ora.gns.vip' on 'rhel12c1' succeeded
CRS-2672: Attempting to start 'ora.gns' on 'rhel12c1'
CRS-2676: Start of 'ora.gns' on 'rhel12c1' succeeded
CRS-2672: Attempting to start 'ora.ASMNET1LSNR_ASM.lsnr' on 'rhel12c1'
CRS-2676: Start of 'ora.ASMNET1LSNR_ASM.lsnr' on 'rhel12c1' succeeded
CRS-2672: Attempting to start 'ora.asm' on 'rhel12c1'
CRS-2676: Start of 'ora.asm' on 'rhel12c1' succeeded
CRS-2672: Attempting to start 'ora.CLUSTERDG.dg' on 'rhel12c1'
CRS-2676: Start of 'ora.CLUSTERDG.dg' on 'rhel12c1' succeeded
CRS-2791: Starting shutdown of Oracle High Availability Services-managed resources on 'rhel12c1'
CRS-2673: Attempting to stop 'ora.crsd' on 'rhel12c1'
CRS-2790: Starting shutdown of Cluster Ready Services-managed resources on 'rhel12c1'
CRS-2673: Attempting to stop 'ora.rhel12c1.vip' on 'rhel12c1'
CRS-2673: Attempting to stop 'ora.CLUSTERDG.dg' on 'rhel12c1'
CRS-2673: Attempting to stop 'ora.cvu' on 'rhel12c1'
CRS-2673: Attempting to stop 'ora.oc4j' on 'rhel12c1'
CRS-2673: Attempting to stop 'ora.LISTENER_SCAN3.lsnr' on 'rhel12c1'
CRS-2673: Attempting to stop 'ora.LISTENER_SCAN2.lsnr' on 'rhel12c1'
CRS-2673: Attempting to stop 'ora.LISTENER_SCAN1.lsnr' on 'rhel12c1'
CRS-2673: Attempting to stop 'ora.gns' on 'rhel12c1'
CRS-2677: Stop of 'ora.cvu' on 'rhel12c1' succeeded
CRS-2677: Stop of 'ora.CLUSTERDG.dg' on 'rhel12c1' succeeded
CRS-2673: Attempting to stop 'ora.asm' on 'rhel12c1'
CRS-2677: Stop of 'ora.asm' on 'rhel12c1' succeeded
CRS-2673: Attempting to stop 'ora.ASMNET1LSNR_ASM.lsnr' on 'rhel12c1'
CRS-2677: Stop of 'ora.LISTENER_SCAN3.lsnr' on 'rhel12c1' succeeded
CRS-2673: Attempting to stop 'ora.scan3.vip' on 'rhel12c1'
CRS-2677: Stop of 'ora.LISTENER_SCAN2.lsnr' on 'rhel12c1' succeeded
CRS-2673: Attempting to stop 'ora.scan2.vip' on 'rhel12c1'
CRS-2677: Stop of 'ora.LISTENER_SCAN1.lsnr' on 'rhel12c1' succeeded
CRS-2673: Attempting to stop 'ora.scan1.vip' on 'rhel12c1'
CRS-2677: Stop of 'ora.ASMNET1LSNR_ASM.lsnr' on 'rhel12c1' succeeded
CRS-2677: Stop of 'ora.rhel12c1.vip' on 'rhel12c1' succeeded
CRS-2677: Stop of 'ora.scan2.vip' on 'rhel12c1' succeeded
CRS-2677: Stop of 'ora.scan3.vip' on 'rhel12c1' succeeded
CRS-2677: Stop of 'ora.scan1.vip' on 'rhel12c1' succeeded
CRS-2677: Stop of 'ora.gns' on 'rhel12c1' succeeded
CRS-2673: Attempting to stop 'ora.gns.vip' on 'rhel12c1'
CRS-2677: Stop of 'ora.gns.vip' on 'rhel12c1' succeeded
CRS-2677: Stop of 'ora.oc4j' on 'rhel12c1' succeeded
CRS-2673: Attempting to stop 'ora.ons' on 'rhel12c1'
CRS-2677: Stop of 'ora.ons' on 'rhel12c1' succeeded
CRS-2673: Attempting to stop 'ora.net1.network' on 'rhel12c1'
CRS-2677: Stop of 'ora.net1.network' on 'rhel12c1' succeeded
CRS-2792: Shutdown of Cluster Ready Services-managed resources on 'rhel12c1' has completed
CRS-2677: Stop of 'ora.crsd' on 'rhel12c1' succeeded
CRS-2673: Attempting to stop 'ora.evmd' on 'rhel12c1'
CRS-2673: Attempting to stop 'ora.ctssd' on 'rhel12c1'
CRS-2673: Attempting to stop 'ora.storage' on 'rhel12c1'
CRS-2673: Attempting to stop 'ora.crf' on 'rhel12c1'
CRS-2673: Attempting to stop 'ora.drivers.acfs' on 'rhel12c1'
CRS-2673: Attempting to stop 'ora.mdnsd' on 'rhel12c1'
CRS-2673: Attempting to stop 'ora.gpnpd' on 'rhel12c1'
CRS-2677: Stop of 'ora.storage' on 'rhel12c1' succeeded
CRS-2677: Stop of 'ora.drivers.acfs' on 'rhel12c1' succeeded
CRS-2677: Stop of 'ora.evmd' on 'rhel12c1' succeeded
CRS-2677: Stop of 'ora.ctssd' on 'rhel12c1' succeeded
CRS-2673: Attempting to stop 'ora.asm' on 'rhel12c1'
CRS-2677: Stop of 'ora.crf' on 'rhel12c1' succeeded
CRS-2677: Stop of 'ora.gpnpd' on 'rhel12c1' succeeded
CRS-2677: Stop of 'ora.mdnsd' on 'rhel12c1' succeeded
CRS-2677: Stop of 'ora.asm' on 'rhel12c1' succeeded
CRS-2673: Attempting to stop 'ora.cluster_interconnect.haip' on 'rhel12c1'
CRS-2677: Stop of 'ora.cluster_interconnect.haip' on 'rhel12c1' succeeded
CRS-2673: Attempting to stop 'ora.cssd' on 'rhel12c1'
CRS-2677: Stop of 'ora.cssd' on 'rhel12c1' succeeded
CRS-2673: Attempting to stop 'ora.gipcd' on 'rhel12c1'
CRS-2677: Stop of 'ora.gipcd' on 'rhel12c1' succeeded
CRS-2793: Shutdown of Oracle High Availability Services-managed resources on 'rhel12c1' has completed
CRS-4133: Oracle High Availability Services has been stopped.
CRS-4123: Starting Oracle High Availability Services-managed resources
CRS-2672: Attempting to start 'ora.mdnsd' on 'rhel12c1'
CRS-2672: Attempting to start 'ora.evmd' on 'rhel12c1'
CRS-2676: Start of 'ora.mdnsd' on 'rhel12c1' succeeded
CRS-2676: Start of 'ora.evmd' on 'rhel12c1' succeeded
CRS-2672: Attempting to start 'ora.gpnpd' on 'rhel12c1'
CRS-2676: Start of 'ora.gpnpd' on 'rhel12c1' succeeded
CRS-2672: Attempting to start 'ora.gipcd' on 'rhel12c1'
CRS-2676: Start of 'ora.gipcd' on 'rhel12c1' succeeded
CRS-2672: Attempting to start 'ora.cssdmonitor' on 'rhel12c1'
CRS-2676: Start of 'ora.cssdmonitor' on 'rhel12c1' succeeded
CRS-2672: Attempting to start 'ora.cssd' on 'rhel12c1'
CRS-2672: Attempting to start 'ora.diskmon' on 'rhel12c1'
CRS-2676: Start of 'ora.diskmon' on 'rhel12c1' succeeded
CRS-2676: Start of 'ora.cssd' on 'rhel12c1' succeeded
CRS-2672: Attempting to start 'ora.cluster_interconnect.haip' on 'rhel12c1'
CRS-2672: Attempting to start 'ora.ctssd' on 'rhel12c1'
CRS-2676: Start of 'ora.ctssd' on 'rhel12c1' succeeded
CRS-2676: Start of 'ora.cluster_interconnect.haip' on 'rhel12c1' succeeded
CRS-2672: Attempting to start 'ora.asm' on 'rhel12c1'
CRS-2676: Start of 'ora.asm' on 'rhel12c1' succeeded
CRS-2672: Attempting to start 'ora.storage' on 'rhel12c1'
CRS-2676: Start of 'ora.storage' on 'rhel12c1' succeeded
CRS-2672: Attempting to start 'ora.crf' on 'rhel12c1'
CRS-2676: Start of 'ora.crf' on 'rhel12c1' succeeded
CRS-2672: Attempting to start 'ora.crsd' on 'rhel12c1'
CRS-2676: Start of 'ora.crsd' on 'rhel12c1' succeeded
CRS-6023: Starting Oracle Cluster Ready Services-managed resources
CRS-2664: Resource 'ora.CLUSTERDG.dg' is already running on 'rhel12c1'
CRS-6017: Processing resource auto-start for servers: rhel12c1
CRS-2672: Attempting to start 'ora.oc4j' on 'rhel12c1'
CRS-2672: Attempting to start 'ora.LISTENER_SCAN1.lsnr' on 'rhel12c1'
CRS-2676: Start of 'ora.LISTENER_SCAN1.lsnr' on 'rhel12c1' succeeded
CRS-2676: Start of 'ora.oc4j' on 'rhel12c1' succeeded
CRS-6016: Resource auto-start has completed for server rhel12c1
CRS-6024: Completed start of Oracle Cluster Ready Services-managed resources
CRS-4123: Oracle High Availability Services has been started.
2015/08/14 13:16:58 CLSRSC-325: Configure Oracle Grid Infrastructure for a Cluster ... succeeded

Root.sh output on node 2

[root@rhel12c2 app]# /opt/app/12.1.0/grid/root.sh
Performing root user operation.

The following environment variables are set as:
    ORACLE_OWNER= grid
    ORACLE_HOME=  /opt/app/12.1.0/grid

Enter the full pathname of the local bin directory: [/usr/local/bin]:
The contents of "dbhome" have not changed. No need to overwrite.
The contents of "oraenv" have not changed. No need to overwrite.
The contents of "coraenv" have not changed. No need to overwrite.


Creating /etc/oratab file...
Entries will be added to the /etc/oratab file as needed by
Database Configuration Assistant when a database is created
Finished running generic part of root script.
Now product-specific root actions will be performed.
Using configuration parameter file: /opt/app/12.1.0/grid/crs/install/crsconfig_params
2015/08/14 13:17:41 CLSRSC-4001: Installing Oracle Trace File Analyzer (TFA) Collector.

2015/08/14 13:18:14 CLSRSC-4002: Successfully installed Oracle Trace File Analyzer (TFA) Collector.

2015/08/14 13:18:16 CLSRSC-363: User ignored prerequisites during installation

OLR initialization - successful
2015/08/14 13:19:47 CLSRSC-330: Adding Clusterware entries to file 'oracle-ohasd.conf'

CRS-4133: Oracle High Availability Services has been stopped.
CRS-4123: Oracle High Availability Services has been started.
CRS-4133: Oracle High Availability Services has been stopped.
CRS-4123: Oracle High Availability Services has been started.
CRS-2791: Starting shutdown of Oracle High Availability Services-managed resources on 'rhel12c2'
CRS-2673: Attempting to stop 'ora.drivers.acfs' on 'rhel12c2'
CRS-2677: Stop of 'ora.drivers.acfs' on 'rhel12c2' succeeded
CRS-2793: Shutdown of Oracle High Availability Services-managed resources on 'rhel12c2' has completed
CRS-4133: Oracle High Availability Services has been stopped.
CRS-4123: Starting Oracle High Availability Services-managed resources
CRS-2672: Attempting to start 'ora.mdnsd' on 'rhel12c2'
CRS-2672: Attempting to start 'ora.evmd' on 'rhel12c2'
CRS-2676: Start of 'ora.mdnsd' on 'rhel12c2' succeeded
CRS-2676: Start of 'ora.evmd' on 'rhel12c2' succeeded
CRS-2672: Attempting to start 'ora.gpnpd' on 'rhel12c2'
CRS-2676: Start of 'ora.gpnpd' on 'rhel12c2' succeeded
CRS-2672: Attempting to start 'ora.gipcd' on 'rhel12c2'
CRS-2676: Start of 'ora.gipcd' on 'rhel12c2' succeeded
CRS-2672: Attempting to start 'ora.cssdmonitor' on 'rhel12c2'
CRS-2676: Start of 'ora.cssdmonitor' on 'rhel12c2' succeeded
CRS-2672: Attempting to start 'ora.cssd' on 'rhel12c2'
CRS-2672: Attempting to start 'ora.diskmon' on 'rhel12c2'
CRS-2676: Start of 'ora.diskmon' on 'rhel12c2' succeeded
CRS-2676: Start of 'ora.cssd' on 'rhel12c2' succeeded
CRS-2672: Attempting to start 'ora.cluster_interconnect.haip' on 'rhel12c2'
CRS-2672: Attempting to start 'ora.ctssd' on 'rhel12c2'
CRS-2676: Start of 'ora.ctssd' on 'rhel12c2' succeeded
CRS-2676: Start of 'ora.cluster_interconnect.haip' on 'rhel12c2' succeeded
CRS-2672: Attempting to start 'ora.asm' on 'rhel12c2'
CRS-2676: Start of 'ora.asm' on 'rhel12c2' succeeded
CRS-2672: Attempting to start 'ora.storage' on 'rhel12c2'
CRS-2676: Start of 'ora.storage' on 'rhel12c2' succeeded
CRS-2672: Attempting to start 'ora.crf' on 'rhel12c2'
CRS-2676: Start of 'ora.crf' on 'rhel12c2' succeeded
CRS-2672: Attempting to start 'ora.crsd' on 'rhel12c2'
CRS-2676: Start of 'ora.crsd' on 'rhel12c2' succeeded
CRS-6017: Processing resource auto-start for servers: rhel12c2
CRS-2672: Attempting to start 'ora.net1.network' on 'rhel12c2'
CRS-2672: Attempting to start 'ora.ASMNET1LSNR_ASM.lsnr' on 'rhel12c2'
CRS-2676: Start of 'ora.net1.network' on 'rhel12c2' succeeded
CRS-2672: Attempting to start 'ora.ons' on 'rhel12c2'
CRS-2676: Start of 'ora.ons' on 'rhel12c2' succeeded
CRS-2673: Attempting to stop 'ora.LISTENER_SCAN1.lsnr' on 'rhel12c1'
CRS-2676: Start of 'ora.ASMNET1LSNR_ASM.lsnr' on 'rhel12c2' succeeded
CRS-2672: Attempting to start 'ora.asm' on 'rhel12c2'
CRS-2677: Stop of 'ora.LISTENER_SCAN1.lsnr' on 'rhel12c1' succeeded
CRS-2673: Attempting to stop 'ora.scan1.vip' on 'rhel12c1'
CRS-2677: Stop of 'ora.scan1.vip' on 'rhel12c1' succeeded
CRS-2672: Attempting to start 'ora.scan1.vip' on 'rhel12c2'
CRS-2676: Start of 'ora.scan1.vip' on 'rhel12c2' succeeded
CRS-2672: Attempting to start 'ora.LISTENER_SCAN1.lsnr' on 'rhel12c2'
CRS-2676: Start of 'ora.LISTENER_SCAN1.lsnr' on 'rhel12c2' succeeded
CRS-2676: Start of 'ora.asm' on 'rhel12c2' succeeded
CRS-6016: Resource auto-start has completed for server rhel12c2
CRS-6024: Completed start of Oracle Cluster Ready Services-managed resources
CRS-4123: Oracle High Availability Services has been started.
2015/08/14 13:24:22 CLSRSC-343: Successfully started Oracle Clusterware stack

2015/08/14 13:24:42 CLSRSC-325: Configure Oracle Grid Infrastructure for a Cluster ... succeeded

Once the root.sh completes the confiugration assistants are run and cluster installation complete. The type of the cluster and nodes could be checked as follows

[root@rhel12c1 oracle]# crsctl get cluster mode status
Cluster is running in "flex" mode

[root@rhel12c1 oracle]# crsctl get node role config
Node 'rhel12c1' configured role is 'hub'

[grid@rhel12c2 ~]$ crsctl get node role config
Node 'rhel12c2' configured role is 'hub'

ASM will have the cluster ASM listener and default ASM instance count of 3

srvctl config asm -detail
ASM home: 
Password file: +clusterdg/orapwASM
ASM listener: LISTENER
ASM is enabled.
ASM is individually enabled on nodes:
ASM is individually disabled on nodes:
ASM instance count: 3
Cluster ASM listener: ASMNET1LSNR_ASM

But on this two node cluster only two ASM instances would be active

ora.asm
      1        ONLINE  ONLINE       rhel12c1                 Started,STABLE
      2        ONLINE  ONLINE       rhel12c2                 Started,STABLE
      3        OFFLINE OFFLINE                               STABLE

Also check the ASM cluster mode

ASMCMD> showclustermode
ASM cluster : Flex mode enabled

Once the cluster is setup install the database software. There's no additional steps to this due to flex cluster, installation is same as 12.1.0.1. Once the database software is installed create the database either CDB or non-CDB.
With version 12.1.0.2 the cluster log files were moved inside diagnostic dest as such the ownership of diag folder is grid:oinstall with permission 755. Due to this fact creation of database will fail as Oracle user doesn't have the write permission inside the diag directory. This is a change from previous 12.1.0.1 version. Before running the DBCA change the permission of following directories so oracle user has write permission

chmod 770 cfgtoollogs
chmod 770 admin
chmod 770 cfgtoollogs/dbca
chmod 770 audit
chmod 770 diag

There are no other additional steps to create a database in flex cluster compared a standard cluster.

Related Posts
Installing 12c (12.1.0.1) RAC on RHEL 6 with Role Separation - Clusterware
Installing 11gR2 (11.2.0.3) GI with Role Separation on RHEL 6
Installing 11gR2 (11.2.0.3) GI with Role Separation on OEL 6
Installing 11gR2 Standalone Server with ASM and Role Separation on RHEL 6
11gR2 Standalone Data Guard (with ASM and Role Separation)

Moving non-RAC Database and ASM Between Servers

There are many strategies to moving a database from one server to another. Backup/duplication, exports base solutions are most common. This post list steps for moving a non-RAC database and ASM (standlone) by way of attaching the underlying storage to the new server. The steps here has been tested using Amazon EC2/ESB combination and using Oracle VirtualBox. As stated in title the current setup includes a standalone non-RAC database using ASM.
1. To able to use ASM in the new sever, install grid infrastructure with software only option. This allow installation of GI without having to create ASM disk groups.

2. When prompted run the root scripts.

# /opt/app/oracle/product/11.2.0/grid_1/root.sh
Performing root user operation for Oracle 11g

The following environment variables are set as:
    ORACLE_OWNER= grid
    ORACLE_HOME=  /opt/app/oracle/product/11.2.0/grid_1

Enter the full pathname of the local bin directory: [/usr/local/bin]:
The contents of "dbhome" have not changed. No need to overwrite.
The contents of "oraenv" have not changed. No need to overwrite.
The contents of "coraenv" have not changed. No need to overwrite.


Creating /etc/oratab file...
Entries will be added to the /etc/oratab file as needed by
Database Configuration Assistant when a database is created
Finished running generic part of root script.
Now product-specific root actions will be performed.

To configure Grid Infrastructure for a Stand-Alone Server run the following command as the root user:
/opt/app/oracle/product/11.2.0/grid_1/perl/bin/perl -I/opt/app/oracle/product/11.2.0/grid_1/perl/lib -I/opt/app/oracle/product/11.2.0/grid_1/crs/install /opt/app/oracle/product/11.2.0/grid_1/crs/install/roothas.pl


To configure Grid Infrastructure for a Cluster execute the following command:
/opt/app/oracle/product/11.2.0/grid_1/crs/config/config.sh
This command launches the Grid Infrastructure Configuration Wizard. The wizard also supports silent operation, and the parameters can be passed through the response file that is available in the installation media.


# /opt/app/oracle/product/11.2.0/grid_1/perl/bin/perl -I/opt/app/oracle/product/11.2.0/grid_1/perl/lib -I/opt/app/oracle/product/11.2.0/grid_1/crs/install /opt/app/oracle/product/11.2.0/grid_1/crs/install/roothas.pl
Using configuration parameter file: /opt/app/oracle/product/11.2.0/grid_1/crs/install/crsconfig_params
Creating trace directory
User ignored Prerequisites during installation
LOCAL ADD MODE
Creating OCR keys for user 'grid', privgrp 'oinstall'..
Operation successful.
LOCAL ONLY MODE
Successfully accumulated necessary OCR keys.
Creating OCR keys for user 'root', privgrp 'root'..
Operation successful.
CRS-4664: Node rhel6m1 successfully pinned.
Adding Clusterware entries to upstart

rhel6m1     2015/03/10 16:40:10     /opt/app/oracle/product/11.2.0/grid_1/cdata/rhel6m1/backup_20150310_164010.olr
Successfully configured Oracle Grid Infrastructure for a Standalone Server

3. GI will have following resources created at this stage

$ crsctl stat res
NAME=ora.cssd
TYPE=ora.cssd.type
TARGET=OFFLINE
STATE=OFFLINE

NAME=ora.diskmon
TYPE=ora.diskmon.type
TARGET=OFFLINE
STATE=OFFLINE

NAME=ora.evmd
TYPE=ora.evm.type
TARGET=ONLINE
STATE=ONLINE on rhel6m1

NAME=ora.ons
TYPE=ora.ons.type
TARGET=OFFLINE
STATE=OFFLINE

4. Install database software with software only option. Patch both GI and database software to the same patch level as the current database setup where database being moved from.
5. If role separation being used them some directory locations ($ORACLE_BASE/admin, $ORACLE_BASE/cfgtoollogs) may not have write permissions for Oracle user, permissions might be set as 755 grid : oinstall. Change permissions as 775 to allow oracle user the write access to these location. Also manually create the audit file location

mkdir -p /opt/app/oracle/admin/ent11g2/adump

6. Copy the listener file from current setup to new server's $GI_HOME/network/admin. Change all reference to the hostname to reflect the new hostname.



7. Attach the storage device(s) to the new server. Depending on the storage type this may require server to shutdown. Once the server is restarted verify the disks that will be used by ASM are visible.Also HA services are running

# crsctl stat res

NAME=ora.cssd
TYPE=ora.cssd.type
TARGET=ONLINE
STATE=ONLINE on rhel6m1

8. With storage attached its only a matter of configuring listener,ASM and database so HAS manage them. First add the listener as grid user.

$ srvctl add listener -l listener -o /opt/app/oracle/product/11.2.0/grid_1 -p 1521
$ srvctl start listener -l listener
$ srvctl status listener -l listener
Listener LISTENER is enabled
Listener LISTENER is running on node(s): rhel6m1

9. Secondly add the ASM as grid user. The ASM pfile is located in the local server as opposed to ASM itself.

$ srvctl add asm -l listener -p /opt/app/oracle/product/11.2.0/grid_1/dbs/spfile+ASM.ora -d "/dev/sd*"
$ srvctl start asm
$ srvctl config asm
ASM home: /opt/app/oracle/product/11.2.0/grid_1
ASM listener: LISTENER
Spfile: /opt/app/oracle/product/11.2.0/grid_1/dbs/spfile+ASM.ora
ASM diskgroup discovery string: /dev/sd*

10. Finally as oracle user add the database and modify the auto start options of the resources

srvctl add database -d ent11g2 -o /opt/app/oracle/product/11.2.0/dbhome_1 -p +DATA/ent11g2/spfileent11g2.ora -a "data,flash,apps"
srvctl start database -d ent11g2

crsctl modify resource ora.DATA.dg -attr "AUTO_START"="always"
crsctl modify resource ora.FLASH.dg -attr "AUTO_START"="always"  
crsctl modify resource ora.LISTENER.lsnr -attr "AUTO_START"="always"  
crsctl modify resource ora.ent11g2.db -attr "AUTO_START"="always"

This concludes the steps for moving non-RAC database and ASM by way of attaching storage from one server to another.

Applying of GI PSU 11.2.0.4.5 Fails Due to -customLogDir Opton Used by OPatch 11.2.0.3.10

Use of opatch auto option has become the default for applying PSU for both GI and Oracle home in an automated fashion. Applying 11.2.0.4.5 GI PSU (July 2015) for both GI and Oracle Home in a single instance setup failed with following message.

$ /opt/app/oracle/product/11.2.0/grid_4/OPatch/opatch version
OPatch Version: 11.2.0.3.10

# /opt/app/oracle/product/11.2.0/grid_4/OPatch/opatch auto ./20996923 -ocmrf ../ocm.rsp
Executing /opt/app/oracle/product/11.2.0/grid_4/perl/bin/perl /opt/app/oracle/product/11.2.0/grid_4/OPatch/crs/patch11203.pl -patchdir . -patchn 20996923 -ocmrf ../ocm.rsp -paramfile /opt/app/oracle/product/11.2.0/grid_4/crs/install/crsconfig_params
This is the main log file: /opt/app/oracle/product/11.2.0/grid_4/cfgtoollogs/opatchauto2015-08-10_14-06-35.log
This file will show your detected configuration and all the steps that opatchauto attempted to do on your system:
/opt/app/oracle/product/11.2.0/grid_4/cfgtoollogs/opatchauto2015-08-10_14-06-35.report.log
2015-08-10 14:06:35: Starting Oracle Restart Patch Setup
Using configuration parameter file: /opt/app/oracle/product/11.2.0/grid_4/crs/install/crsconfig_params
The opatch Component check failed. This patch is not applicable for /opt/app/oracle/product/11.2.0/dbhome_4
The opatch Component check failed. This patch is not applicable for /opt/app/oracle/product/11.2.0/dbhome_4
Conflict check failed for oracle home  /opt/app/oracle/product/11.2.0/dbhome_4
Conflict check failed
ERROR: Conflict-Check has failed . Please refer to /opt/app/oracle/product/11.2.0/grid_4/cfgtoollogs/opatchauto2015-08-10_14-06-35.log for details

Though the error message says conflict check failed, it is not the real reason in this case. Examining the log file revealed following

2015-08-10 14:07:13: Status of opatch version check  for /opt/app/oracle/product/11.2.0/grid_4 is 1
2015-08-10 14:07:13: Opatch version check passed for oracle home  /opt/app/oracle/product/11.2.0/grid_4
2015-08-10 14:07:13: Opatch version check passed  for all oracle homes
2015-08-10 14:07:13: Processing oracle home /opt/app/oracle/product/11.2.0/dbhome_4
2015-08-10 14:07:13: Opening file /etc/oracle/ocr.loc
2015-08-10 14:07:13: Value (TRUE) is set for key=local_only
2015-08-10 14:07:13: Home type of /opt/app/oracle/product/11.2.0/dbhome_4 is DB
2015-08-10 14:07:13: Oracle user for /opt/app/oracle/product/11.2.0/dbhome_4 is oracle
2015-08-10 14:07:13: Running as user oracle: /opt/app/oracle/product/11.2.0/dbhome_4/OPatch/opatch prereq CheckComponents -ph ./20996923/20760982 -oh /opt/app/oracle/product/11.2.0/dbhome_4 
-customLogDir /opt/app/oracle/product/11.2.0/grid_4/cfgtoollogs/opatchauto/core
2015-08-10 14:07:13: s_run_as_user2: Running /bin/su oracle -c ' /opt/app/oracle/product/11.2.0/dbhome_4/OPatch/opatch prereq CheckComponents -ph ./20996923/20760982 -oh /opt/app/oracle/product/11.2.0/dbhome_4 
-customLogDir /opt/app/oracle/product/11.2.0/grid_4/cfgtoollogs/opatchauto/core '
2015-08-10 14:07:14: Removing file /tmp/fileWJrEf9
2015-08-10 14:07:14: Successfully removed file: /tmp/fileWJrEf9
2015-08-10 14:07:14: /bin/su exited with rc=1

2015-08-10 14:07:14: The opatch Component check failed. This patch is not applicable for /opt/app/oracle/product/11.2.0/dbhome_4
2015-08-10 14:07:14: The component check failed with following error
2015-08-10 14:07:14: Input custom log directory "/opt/app/oracle/product/11.2.0/grid_4/cfgtoollogs/opatchauto/core" does not exists or not a valid directory.

 OPatch failed with error code 1

From the output it could be seen that when opatch is checking components for oracle home patching it is setting a custom log directory which is inside the grid home. This wouldn't be a problem if both GI home and Oracle home are owned by the same user. But in this case role separation is used and the directories cfgtoollogs,opatchauto,core all have permission set to either 700 or 755 and ownership grid:oinstall. This makes oracle user unable to write to these directories and the failure of the patch apply.
Metalink notes 2022797.1 and 1964569.1 (These MOS notes are related to 12c) suggest to use -customLogDir option to specify a directory but this is not applicable for "opatch auto" option.



Two ways to overcome this problem. One is to grant the oinstall group the write permission to cfgtoollogs/opatchauto/core before the patch apply and revoke once the the patch is successfully applied.
Second option is to use a lower version OPatch, in this case using OPatch version 11.2.0.3.6 (minimum version required according to PSU's readme.html) enabled to successfully apply the patch.
As patch apply went ahead without a problem using a lower version of opatch mean that customLogDir option was introduced in higher version OPatch (11.2.0.3.10) without the consideration to possible failures that could arise due to use of role separation.

Useful Metalink Notes
opatchauto wasn't able to run ... because 'bash' command could not be found [ID 2022797.1]
"opatch lsinventory" on CRS/GI home fails with error code 15 [ID 1964569.1]

ASM Diks Resizing After Underlying Storage Disk Expansion

There are two ways to add more space to an ASM diskgroup. One is to add more disks to the group and second is to expand the disk already in the group. This post shows the step for the latter case. The current setup contains two ASM diskgroup each with exactly one disk.

GROUP_NAME DISK_NAME       GROUP_SIZE_MB DISK_SIZE_MB PATH
---------- --------------- ------------- ------------ ----------
DATA       DATA_0000               10236        10236 /dev/sdb1
FLASH      FLASH_0000              10236        10236 /dev/sdc1

Size of the disk added to the flash diskgroup is roughly 10GB and this will be expanded to 12GB. ASM is used for a standalone database (11.2.0.4) created on VirtulBox VM. The expansion of the underlying storage disk is achieved using VBoxManagment. If this is a SAN LUN then vendor provided tools/commands could be used to achieve the same.

# VBoxManage modifyhd rhel5flash.vdi --resize 12288

Shutdown the GI stack on the server before the disk expansion. Once the expansion is complete GI stack could be started but the additional space won't be visible to Oracle and trying to resize the ASM disk would fail.

SQL> alter diskgroup flash resize disk FLASH_0000 size 12000m;
alter diskgroup flash resize disk FLASH_0000 size 12000m
*
ERROR at line 1:
ORA-15032: not all alterations performed
ORA-15057: specified size of 12000 MB is larger than actual size of 10236 MB

This is because even though the disk was expanded the storage disk partition created for the ASM disk is still at the original size.

fdisk -l /dev/sdc

Disk /dev/sdc: 12.9 GB, 12884901888 bytes
255 heads, 63 sectors/track, 1566 cylinders
Units = cylinders of 16065 * 512 = 8225280 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disk identifier: 0x00000000

   Device Boot      Start         End      Blocks   Id  System
/dev/sdc1               1        1305    10482381   83  Linux

Only way to resize the partition is to re-create. This would entail deleting the current partition as such stop the GI stack before partition recreation.

$ crsctl stop has

# fdisk /dev/sdc

WARNING: DOS-compatible mode is deprecated. It's strongly recommended to
         switch off the mode (command 'c') and change display units to
         sectors (command 'u').

Command (m for help): d
Selected partition 1

Command (m for help): n
Command action
   e   extended
   p   primary partition (1-4)
p
Partition number (1-4): 1
First cylinder (1-1566, default 1):
Using default value 1
Last cylinder, +cylinders or +size{K,M,G} (1-1566, default 1566):
Using default value 1566

Command (m for help): w
The partition table has been altered!
Calling ioctl() to re-read partition table.
Syncing disks.

Once the partition is recreated verify size has increased

# fdisk -l /dev/sdc

Disk /dev/sdc: 12.9 GB, 12884901888 bytes
255 heads, 63 sectors/track, 1566 cylinders
Units = cylinders of 16065 * 512 = 8225280 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disk identifier: 0x00000000

   Device Boot      Start         End      Blocks   Id  System
/dev/sdc1               1        1566    12578863+  83  Linux

Start the GI stack and resize ASM disk

crsctl start has
CRS-4123: Oracle High Availability Services has been started.

SQL> alter diskgroup flash resize disk FLASH_0000 size 12000m;

If multiple disks are resized then it is possible to add the re-balance option to the alter diskgroup. ASM alert log the resize operation

SQL> alter diskgroup flash resize disk FLASH_0000 size 12000m
NOTE: requesting all-instance membership refresh for group=2
NOTE: requesting all-instance disk validation for group=2
Thu Jul 30 15:42:42 2015
NOTE: disk validation pending for group 2/0x2c5db20f (FLASH)
SUCCESS: validated disks for 2/0x2c5db20f (FLASH)
NOTE: increased size in header on grp 2 disk FLASH_0000
NOTE: membership refresh pending for group 2/0x2c5db20f (FLASH)
Thu Jul 30 15:42:49 2015
GMON querying group 2 at 9 for pid 13, osid 5803
SUCCESS: refreshed membership for 2/0x2c5db20f (FLASH)
NOTE: starting rebalance of group 2/0x2c5db20f (FLASH) at power 1
SUCCESS: alter diskgroup flash resize disk FLASH_0000 size 12000m
Starting background process ARB0
Thu Jul 30 15:42:49 2015
ARB0 started with pid=28, OS id=6019
NOTE: assigning ARB0 to group 2/0x2c5db20f (FLASH) with 1 parallel I/O
cellip.ora not found.
NOTE: stopping process ARB0
SUCCESS: rebalance completed for group 2/0x2c5db20f (FLASH)
NOTE: Attempting voting file refresh on diskgroup FLASH
Thu Jul 30 15:42:52 2015
NOTE: requesting all-instance membership refresh for group=2
Thu Jul 30 15:42:55 2015
NOTE: membership refresh pending for group 2/0x2c5db20f (FLASH)
GMON querying group 2 at 10 for pid 13, osid 5803
SUCCESS: refreshed membership for 2/0x2c5db20f (FLASH)
NOTE: Attempting voting file refresh on diskgroup FLASH

Query the diskgroup and disk for the increase in size.

GROUP_NAME DISK_NAME       GROUP_SIZE_MB DISK_SIZE_MB PATH
---------- --------------- ------------- ------------ ----------
DATA       DATA_0000               10236        10236 /dev/sdb1
FLASH      FLASH_0000              12000        10236 /dev/sdc1

This concludes resizing of ASM disk after underlying storage expansion.

Useful Metalink Notes
How To Resize an ASM Disk? [ID 373426.1]
Master Note for Automatic Storage Management (ASM) [ID 1187723.1]
Resize of ASM Disk Fails with ORA-600 [KFDATSHRINK_NOTFREE] [ID 2015973.1]
Versions 11.2.0.4 and later: When a LUN is expanded, unable to resize ASM disks online [ID 2000490.1]
Master Note For Automatic Storage Management (ASM) For Operating System (OS) Administrators [ID 1345562.1]