Sunday, August 23, 2015

Installing 12c (12.1.0.2) Flex Cluster on RHEL 6 with Role Seperation

Oracle 12c introduced a new type of cluster configuration called Flex Cluster. This post highlights the steps of installing a Flex cluster. It's not a complete "how to guide". Most of the steps are similar to 12 standard cluster installation. But for flex cluster, GNS setup is a must.
Also it seems that the ocr and vote diskgroup size requirements are also different when installing a flex cluster. 12.1.0.2 made it mandatory to have the cluster management DB as such the space requirement for diskgroup containing OCR and vote disks is higher compared to 12.1.0.1
The new space requirement is 5528MB (for external redundancy diskgroup) but earlier for upgrading from 12.1.0.1 to 12.1.0.2 it was 4424MB (for external redundancy diskgroup).
If the disk redundancy is normal then the size of the disks must be same, dissimilar size disks are not allowed (this restriction doesn't apply to external redundancy diskgroups).

Other per-installation requirements are similar to previous version of 12.1.0.1.
Start the installation specify it's a flex cluster configuration.
Specify the GNS configuration
Specify the node list and also the role of the node. In this two node cluster setup both nodes are set as hub nodes.
With flex cluster an additional network could be defined for Flex ASM or as in this case allow the private interconnect network to be used for for flex ASM as well.
The installer only prompt to run orainstRoot.sh only on one of the nodes. However root.sh is to be run on all the nodes.




Output from the root.sh
[root@rhel12c1 app]# /opt/app/12.1.0/grid/root.sh
Performing root user operation.

The following environment variables are set as:
    ORACLE_OWNER= grid
    ORACLE_HOME=  /opt/app/12.1.0/grid

Enter the full pathname of the local bin directory: [/usr/local/bin]:
The contents of "dbhome" have not changed. No need to overwrite.
The contents of "oraenv" have not changed. No need to overwrite.
The contents of "coraenv" have not changed. No need to overwrite.


Creating /etc/oratab file...
Entries will be added to the /etc/oratab file as needed by
Database Configuration Assistant when a database is created
Finished running generic part of root script.
Now product-specific root actions will be performed.
Using configuration parameter file: /opt/app/12.1.0/grid/crs/install/crsconfig_params
2015/08/14 13:03:13 CLSRSC-4001: Installing Oracle Trace File Analyzer (TFA) Collector.

2015/08/14 13:03:47 CLSRSC-4002: Successfully installed Oracle Trace File Analyzer (TFA) Collector.

2015/08/14 13:03:50 CLSRSC-363: User ignored prerequisites during installation

OLR initialization - successful
  root wallet
  root wallet cert
  root cert export
  peer wallet
  profile reader wallet
  pa wallet
  peer wallet keys
  pa wallet keys
  peer cert request
  pa cert request
  peer cert
  pa cert
  peer root cert TP
  profile reader root cert TP
  pa root cert TP
  peer pa cert TP
  pa peer cert TP
  profile reader pa cert TP
  profile reader peer cert TP
  peer user cert
  pa user cert
2015/08/14 13:05:01 CLSRSC-330: Adding Clusterware entries to file 'oracle-ohasd.conf'

CRS-4133: Oracle High Availability Services has been stopped.
CRS-4123: Oracle High Availability Services has been started.
CRS-4133: Oracle High Availability Services has been stopped.
CRS-4123: Oracle High Availability Services has been started.
CRS-2672: Attempting to start 'ora.evmd' on 'rhel12c1'
CRS-2672: Attempting to start 'ora.mdnsd' on 'rhel12c1'
CRS-2676: Start of 'ora.mdnsd' on 'rhel12c1' succeeded
CRS-2676: Start of 'ora.evmd' on 'rhel12c1' succeeded
CRS-2672: Attempting to start 'ora.gpnpd' on 'rhel12c1'
CRS-2676: Start of 'ora.gpnpd' on 'rhel12c1' succeeded
CRS-2672: Attempting to start 'ora.cssdmonitor' on 'rhel12c1'
CRS-2672: Attempting to start 'ora.gipcd' on 'rhel12c1'
CRS-2676: Start of 'ora.cssdmonitor' on 'rhel12c1' succeeded
CRS-2676: Start of 'ora.gipcd' on 'rhel12c1' succeeded
CRS-2672: Attempting to start 'ora.cssd' on 'rhel12c1'
CRS-2672: Attempting to start 'ora.diskmon' on 'rhel12c1'
CRS-2676: Start of 'ora.diskmon' on 'rhel12c1' succeeded
CRS-2676: Start of 'ora.cssd' on 'rhel12c1' succeeded

ASM created and started successfully.

Disk Group clusterdg created successfully.

CRS-2672: Attempting to start 'ora.crf' on 'rhel12c1'
CRS-2672: Attempting to start 'ora.storage' on 'rhel12c1'
CRS-2676: Start of 'ora.storage' on 'rhel12c1' succeeded
CRS-2676: Start of 'ora.crf' on 'rhel12c1' succeeded
CRS-2672: Attempting to start 'ora.crsd' on 'rhel12c1'
CRS-2676: Start of 'ora.crsd' on 'rhel12c1' succeeded
CRS-4256: Updating the profile
Successful addition of voting disk 27c22c794c2a4f7fbfca7979664be132.
Successfully replaced voting disk group with +clusterdg.
CRS-4256: Updating the profile
CRS-4266: Voting file(s) successfully replaced
##  STATE    File Universal Id                File Name Disk group
--  -----    -----------------                --------- ---------
 1. ONLINE   27c22c794c2a4f7fbfca7979664be132 (/dev/sde1) [CLUSTERDG]
Located 1 voting disk(s).
CRS-2791: Starting shutdown of Oracle High Availability Services-managed resources on 'rhel12c1'
CRS-2673: Attempting to stop 'ora.crsd' on 'rhel12c1'
CRS-2677: Stop of 'ora.crsd' on 'rhel12c1' succeeded
CRS-2673: Attempting to stop 'ora.storage' on 'rhel12c1'
CRS-2673: Attempting to stop 'ora.mdnsd' on 'rhel12c1'
CRS-2673: Attempting to stop 'ora.gpnpd' on 'rhel12c1'
CRS-2673: Attempting to stop 'ora.drivers.acfs' on 'rhel12c1'
CRS-2677: Stop of 'ora.storage' on 'rhel12c1' succeeded
CRS-2677: Stop of 'ora.drivers.acfs' on 'rhel12c1' succeeded
CRS-2673: Attempting to stop 'ora.crf' on 'rhel12c1'
CRS-2673: Attempting to stop 'ora.ctssd' on 'rhel12c1'
CRS-2673: Attempting to stop 'ora.evmd' on 'rhel12c1'
CRS-2673: Attempting to stop 'ora.asm' on 'rhel12c1'
CRS-2677: Stop of 'ora.gpnpd' on 'rhel12c1' succeeded
CRS-2677: Stop of 'ora.mdnsd' on 'rhel12c1' succeeded
CRS-2677: Stop of 'ora.ctssd' on 'rhel12c1' succeeded
CRS-2677: Stop of 'ora.crf' on 'rhel12c1' succeeded
CRS-2677: Stop of 'ora.evmd' on 'rhel12c1' succeeded
CRS-2677: Stop of 'ora.asm' on 'rhel12c1' succeeded
CRS-2673: Attempting to stop 'ora.cluster_interconnect.haip' on 'rhel12c1'
CRS-2677: Stop of 'ora.cluster_interconnect.haip' on 'rhel12c1' succeeded
CRS-2673: Attempting to stop 'ora.cssd' on 'rhel12c1'
CRS-2677: Stop of 'ora.cssd' on 'rhel12c1' succeeded
CRS-2673: Attempting to stop 'ora.gipcd' on 'rhel12c1'
CRS-2677: Stop of 'ora.gipcd' on 'rhel12c1' succeeded
CRS-2793: Shutdown of Oracle High Availability Services-managed resources on 'rhel12c1' has completed
CRS-4133: Oracle High Availability Services has been stopped.
CRS-4123: Starting Oracle High Availability Services-managed resources
CRS-2672: Attempting to start 'ora.mdnsd' on 'rhel12c1'
CRS-2672: Attempting to start 'ora.evmd' on 'rhel12c1'
CRS-2676: Start of 'ora.evmd' on 'rhel12c1' succeeded
CRS-2676: Start of 'ora.mdnsd' on 'rhel12c1' succeeded
CRS-2672: Attempting to start 'ora.gpnpd' on 'rhel12c1'
CRS-2676: Start of 'ora.gpnpd' on 'rhel12c1' succeeded
CRS-2672: Attempting to start 'ora.gipcd' on 'rhel12c1'
CRS-2676: Start of 'ora.gipcd' on 'rhel12c1' succeeded
CRS-2672: Attempting to start 'ora.cssdmonitor' on 'rhel12c1'
CRS-2676: Start of 'ora.cssdmonitor' on 'rhel12c1' succeeded
CRS-2672: Attempting to start 'ora.cssd' on 'rhel12c1'
CRS-2672: Attempting to start 'ora.diskmon' on 'rhel12c1'
CRS-2676: Start of 'ora.diskmon' on 'rhel12c1' succeeded
CRS-2676: Start of 'ora.cssd' on 'rhel12c1' succeeded
CRS-2672: Attempting to start 'ora.cluster_interconnect.haip' on 'rhel12c1'
CRS-2672: Attempting to start 'ora.ctssd' on 'rhel12c1'
CRS-2676: Start of 'ora.ctssd' on 'rhel12c1' succeeded
CRS-2676: Start of 'ora.cluster_interconnect.haip' on 'rhel12c1' succeeded
CRS-2672: Attempting to start 'ora.asm' on 'rhel12c1'
CRS-2676: Start of 'ora.asm' on 'rhel12c1' succeeded
CRS-2672: Attempting to start 'ora.storage' on 'rhel12c1'
CRS-2676: Start of 'ora.storage' on 'rhel12c1' succeeded
CRS-2672: Attempting to start 'ora.crf' on 'rhel12c1'
CRS-2676: Start of 'ora.crf' on 'rhel12c1' succeeded
CRS-2672: Attempting to start 'ora.crsd' on 'rhel12c1'
CRS-2676: Start of 'ora.crsd' on 'rhel12c1' succeeded
CRS-6023: Starting Oracle Cluster Ready Services-managed resources
CRS-6017: Processing resource auto-start for servers: rhel12c1
CRS-6016: Resource auto-start has completed for server rhel12c1
CRS-6024: Completed start of Oracle Cluster Ready Services-managed resources
CRS-4123: Oracle High Availability Services has been started.
2015/08/14 13:11:52 CLSRSC-343: Successfully started Oracle Clusterware stack

CRS-2672: Attempting to start 'ora.net1.network' on 'rhel12c1'
CRS-2676: Start of 'ora.net1.network' on 'rhel12c1' succeeded
CRS-2672: Attempting to start 'ora.gns.vip' on 'rhel12c1'
CRS-2676: Start of 'ora.gns.vip' on 'rhel12c1' succeeded
CRS-2672: Attempting to start 'ora.gns' on 'rhel12c1'
CRS-2676: Start of 'ora.gns' on 'rhel12c1' succeeded
CRS-2672: Attempting to start 'ora.ASMNET1LSNR_ASM.lsnr' on 'rhel12c1'
CRS-2676: Start of 'ora.ASMNET1LSNR_ASM.lsnr' on 'rhel12c1' succeeded
CRS-2672: Attempting to start 'ora.asm' on 'rhel12c1'
CRS-2676: Start of 'ora.asm' on 'rhel12c1' succeeded
CRS-2672: Attempting to start 'ora.CLUSTERDG.dg' on 'rhel12c1'
CRS-2676: Start of 'ora.CLUSTERDG.dg' on 'rhel12c1' succeeded
CRS-2791: Starting shutdown of Oracle High Availability Services-managed resources on 'rhel12c1'
CRS-2673: Attempting to stop 'ora.crsd' on 'rhel12c1'
CRS-2790: Starting shutdown of Cluster Ready Services-managed resources on 'rhel12c1'
CRS-2673: Attempting to stop 'ora.rhel12c1.vip' on 'rhel12c1'
CRS-2673: Attempting to stop 'ora.CLUSTERDG.dg' on 'rhel12c1'
CRS-2673: Attempting to stop 'ora.cvu' on 'rhel12c1'
CRS-2673: Attempting to stop 'ora.oc4j' on 'rhel12c1'
CRS-2673: Attempting to stop 'ora.LISTENER_SCAN3.lsnr' on 'rhel12c1'
CRS-2673: Attempting to stop 'ora.LISTENER_SCAN2.lsnr' on 'rhel12c1'
CRS-2673: Attempting to stop 'ora.LISTENER_SCAN1.lsnr' on 'rhel12c1'
CRS-2673: Attempting to stop 'ora.gns' on 'rhel12c1'
CRS-2677: Stop of 'ora.cvu' on 'rhel12c1' succeeded
CRS-2677: Stop of 'ora.CLUSTERDG.dg' on 'rhel12c1' succeeded
CRS-2673: Attempting to stop 'ora.asm' on 'rhel12c1'
CRS-2677: Stop of 'ora.asm' on 'rhel12c1' succeeded
CRS-2673: Attempting to stop 'ora.ASMNET1LSNR_ASM.lsnr' on 'rhel12c1'
CRS-2677: Stop of 'ora.LISTENER_SCAN3.lsnr' on 'rhel12c1' succeeded
CRS-2673: Attempting to stop 'ora.scan3.vip' on 'rhel12c1'
CRS-2677: Stop of 'ora.LISTENER_SCAN2.lsnr' on 'rhel12c1' succeeded
CRS-2673: Attempting to stop 'ora.scan2.vip' on 'rhel12c1'
CRS-2677: Stop of 'ora.LISTENER_SCAN1.lsnr' on 'rhel12c1' succeeded
CRS-2673: Attempting to stop 'ora.scan1.vip' on 'rhel12c1'
CRS-2677: Stop of 'ora.ASMNET1LSNR_ASM.lsnr' on 'rhel12c1' succeeded
CRS-2677: Stop of 'ora.rhel12c1.vip' on 'rhel12c1' succeeded
CRS-2677: Stop of 'ora.scan2.vip' on 'rhel12c1' succeeded
CRS-2677: Stop of 'ora.scan3.vip' on 'rhel12c1' succeeded
CRS-2677: Stop of 'ora.scan1.vip' on 'rhel12c1' succeeded
CRS-2677: Stop of 'ora.gns' on 'rhel12c1' succeeded
CRS-2673: Attempting to stop 'ora.gns.vip' on 'rhel12c1'
CRS-2677: Stop of 'ora.gns.vip' on 'rhel12c1' succeeded
CRS-2677: Stop of 'ora.oc4j' on 'rhel12c1' succeeded
CRS-2673: Attempting to stop 'ora.ons' on 'rhel12c1'
CRS-2677: Stop of 'ora.ons' on 'rhel12c1' succeeded
CRS-2673: Attempting to stop 'ora.net1.network' on 'rhel12c1'
CRS-2677: Stop of 'ora.net1.network' on 'rhel12c1' succeeded
CRS-2792: Shutdown of Cluster Ready Services-managed resources on 'rhel12c1' has completed
CRS-2677: Stop of 'ora.crsd' on 'rhel12c1' succeeded
CRS-2673: Attempting to stop 'ora.evmd' on 'rhel12c1'
CRS-2673: Attempting to stop 'ora.ctssd' on 'rhel12c1'
CRS-2673: Attempting to stop 'ora.storage' on 'rhel12c1'
CRS-2673: Attempting to stop 'ora.crf' on 'rhel12c1'
CRS-2673: Attempting to stop 'ora.drivers.acfs' on 'rhel12c1'
CRS-2673: Attempting to stop 'ora.mdnsd' on 'rhel12c1'
CRS-2673: Attempting to stop 'ora.gpnpd' on 'rhel12c1'
CRS-2677: Stop of 'ora.storage' on 'rhel12c1' succeeded
CRS-2677: Stop of 'ora.drivers.acfs' on 'rhel12c1' succeeded
CRS-2677: Stop of 'ora.evmd' on 'rhel12c1' succeeded
CRS-2677: Stop of 'ora.ctssd' on 'rhel12c1' succeeded
CRS-2673: Attempting to stop 'ora.asm' on 'rhel12c1'
CRS-2677: Stop of 'ora.crf' on 'rhel12c1' succeeded
CRS-2677: Stop of 'ora.gpnpd' on 'rhel12c1' succeeded
CRS-2677: Stop of 'ora.mdnsd' on 'rhel12c1' succeeded
CRS-2677: Stop of 'ora.asm' on 'rhel12c1' succeeded
CRS-2673: Attempting to stop 'ora.cluster_interconnect.haip' on 'rhel12c1'
CRS-2677: Stop of 'ora.cluster_interconnect.haip' on 'rhel12c1' succeeded
CRS-2673: Attempting to stop 'ora.cssd' on 'rhel12c1'
CRS-2677: Stop of 'ora.cssd' on 'rhel12c1' succeeded
CRS-2673: Attempting to stop 'ora.gipcd' on 'rhel12c1'
CRS-2677: Stop of 'ora.gipcd' on 'rhel12c1' succeeded
CRS-2793: Shutdown of Oracle High Availability Services-managed resources on 'rhel12c1' has completed
CRS-4133: Oracle High Availability Services has been stopped.
CRS-4123: Starting Oracle High Availability Services-managed resources
CRS-2672: Attempting to start 'ora.mdnsd' on 'rhel12c1'
CRS-2672: Attempting to start 'ora.evmd' on 'rhel12c1'
CRS-2676: Start of 'ora.mdnsd' on 'rhel12c1' succeeded
CRS-2676: Start of 'ora.evmd' on 'rhel12c1' succeeded
CRS-2672: Attempting to start 'ora.gpnpd' on 'rhel12c1'
CRS-2676: Start of 'ora.gpnpd' on 'rhel12c1' succeeded
CRS-2672: Attempting to start 'ora.gipcd' on 'rhel12c1'
CRS-2676: Start of 'ora.gipcd' on 'rhel12c1' succeeded
CRS-2672: Attempting to start 'ora.cssdmonitor' on 'rhel12c1'
CRS-2676: Start of 'ora.cssdmonitor' on 'rhel12c1' succeeded
CRS-2672: Attempting to start 'ora.cssd' on 'rhel12c1'
CRS-2672: Attempting to start 'ora.diskmon' on 'rhel12c1'
CRS-2676: Start of 'ora.diskmon' on 'rhel12c1' succeeded
CRS-2676: Start of 'ora.cssd' on 'rhel12c1' succeeded
CRS-2672: Attempting to start 'ora.cluster_interconnect.haip' on 'rhel12c1'
CRS-2672: Attempting to start 'ora.ctssd' on 'rhel12c1'
CRS-2676: Start of 'ora.ctssd' on 'rhel12c1' succeeded
CRS-2676: Start of 'ora.cluster_interconnect.haip' on 'rhel12c1' succeeded
CRS-2672: Attempting to start 'ora.asm' on 'rhel12c1'
CRS-2676: Start of 'ora.asm' on 'rhel12c1' succeeded
CRS-2672: Attempting to start 'ora.storage' on 'rhel12c1'
CRS-2676: Start of 'ora.storage' on 'rhel12c1' succeeded
CRS-2672: Attempting to start 'ora.crf' on 'rhel12c1'
CRS-2676: Start of 'ora.crf' on 'rhel12c1' succeeded
CRS-2672: Attempting to start 'ora.crsd' on 'rhel12c1'
CRS-2676: Start of 'ora.crsd' on 'rhel12c1' succeeded
CRS-6023: Starting Oracle Cluster Ready Services-managed resources
CRS-2664: Resource 'ora.CLUSTERDG.dg' is already running on 'rhel12c1'
CRS-6017: Processing resource auto-start for servers: rhel12c1
CRS-2672: Attempting to start 'ora.oc4j' on 'rhel12c1'
CRS-2672: Attempting to start 'ora.LISTENER_SCAN1.lsnr' on 'rhel12c1'
CRS-2676: Start of 'ora.LISTENER_SCAN1.lsnr' on 'rhel12c1' succeeded
CRS-2676: Start of 'ora.oc4j' on 'rhel12c1' succeeded
CRS-6016: Resource auto-start has completed for server rhel12c1
CRS-6024: Completed start of Oracle Cluster Ready Services-managed resources
CRS-4123: Oracle High Availability Services has been started.
2015/08/14 13:16:58 CLSRSC-325: Configure Oracle Grid Infrastructure for a Cluster ... succeeded
Root.sh output on node 2
[root@rhel12c2 app]# /opt/app/12.1.0/grid/root.sh
Performing root user operation.

The following environment variables are set as:
    ORACLE_OWNER= grid
    ORACLE_HOME=  /opt/app/12.1.0/grid

Enter the full pathname of the local bin directory: [/usr/local/bin]:
The contents of "dbhome" have not changed. No need to overwrite.
The contents of "oraenv" have not changed. No need to overwrite.
The contents of "coraenv" have not changed. No need to overwrite.


Creating /etc/oratab file...
Entries will be added to the /etc/oratab file as needed by
Database Configuration Assistant when a database is created
Finished running generic part of root script.
Now product-specific root actions will be performed.
Using configuration parameter file: /opt/app/12.1.0/grid/crs/install/crsconfig_params
2015/08/14 13:17:41 CLSRSC-4001: Installing Oracle Trace File Analyzer (TFA) Collector.

2015/08/14 13:18:14 CLSRSC-4002: Successfully installed Oracle Trace File Analyzer (TFA) Collector.

2015/08/14 13:18:16 CLSRSC-363: User ignored prerequisites during installation

OLR initialization - successful
2015/08/14 13:19:47 CLSRSC-330: Adding Clusterware entries to file 'oracle-ohasd.conf'

CRS-4133: Oracle High Availability Services has been stopped.
CRS-4123: Oracle High Availability Services has been started.
CRS-4133: Oracle High Availability Services has been stopped.
CRS-4123: Oracle High Availability Services has been started.
CRS-2791: Starting shutdown of Oracle High Availability Services-managed resources on 'rhel12c2'
CRS-2673: Attempting to stop 'ora.drivers.acfs' on 'rhel12c2'
CRS-2677: Stop of 'ora.drivers.acfs' on 'rhel12c2' succeeded
CRS-2793: Shutdown of Oracle High Availability Services-managed resources on 'rhel12c2' has completed
CRS-4133: Oracle High Availability Services has been stopped.
CRS-4123: Starting Oracle High Availability Services-managed resources
CRS-2672: Attempting to start 'ora.mdnsd' on 'rhel12c2'
CRS-2672: Attempting to start 'ora.evmd' on 'rhel12c2'
CRS-2676: Start of 'ora.mdnsd' on 'rhel12c2' succeeded
CRS-2676: Start of 'ora.evmd' on 'rhel12c2' succeeded
CRS-2672: Attempting to start 'ora.gpnpd' on 'rhel12c2'
CRS-2676: Start of 'ora.gpnpd' on 'rhel12c2' succeeded
CRS-2672: Attempting to start 'ora.gipcd' on 'rhel12c2'
CRS-2676: Start of 'ora.gipcd' on 'rhel12c2' succeeded
CRS-2672: Attempting to start 'ora.cssdmonitor' on 'rhel12c2'
CRS-2676: Start of 'ora.cssdmonitor' on 'rhel12c2' succeeded
CRS-2672: Attempting to start 'ora.cssd' on 'rhel12c2'
CRS-2672: Attempting to start 'ora.diskmon' on 'rhel12c2'
CRS-2676: Start of 'ora.diskmon' on 'rhel12c2' succeeded
CRS-2676: Start of 'ora.cssd' on 'rhel12c2' succeeded
CRS-2672: Attempting to start 'ora.cluster_interconnect.haip' on 'rhel12c2'
CRS-2672: Attempting to start 'ora.ctssd' on 'rhel12c2'
CRS-2676: Start of 'ora.ctssd' on 'rhel12c2' succeeded
CRS-2676: Start of 'ora.cluster_interconnect.haip' on 'rhel12c2' succeeded
CRS-2672: Attempting to start 'ora.asm' on 'rhel12c2'
CRS-2676: Start of 'ora.asm' on 'rhel12c2' succeeded
CRS-2672: Attempting to start 'ora.storage' on 'rhel12c2'
CRS-2676: Start of 'ora.storage' on 'rhel12c2' succeeded
CRS-2672: Attempting to start 'ora.crf' on 'rhel12c2'
CRS-2676: Start of 'ora.crf' on 'rhel12c2' succeeded
CRS-2672: Attempting to start 'ora.crsd' on 'rhel12c2'
CRS-2676: Start of 'ora.crsd' on 'rhel12c2' succeeded
CRS-6017: Processing resource auto-start for servers: rhel12c2
CRS-2672: Attempting to start 'ora.net1.network' on 'rhel12c2'
CRS-2672: Attempting to start 'ora.ASMNET1LSNR_ASM.lsnr' on 'rhel12c2'
CRS-2676: Start of 'ora.net1.network' on 'rhel12c2' succeeded
CRS-2672: Attempting to start 'ora.ons' on 'rhel12c2'
CRS-2676: Start of 'ora.ons' on 'rhel12c2' succeeded
CRS-2673: Attempting to stop 'ora.LISTENER_SCAN1.lsnr' on 'rhel12c1'
CRS-2676: Start of 'ora.ASMNET1LSNR_ASM.lsnr' on 'rhel12c2' succeeded
CRS-2672: Attempting to start 'ora.asm' on 'rhel12c2'
CRS-2677: Stop of 'ora.LISTENER_SCAN1.lsnr' on 'rhel12c1' succeeded
CRS-2673: Attempting to stop 'ora.scan1.vip' on 'rhel12c1'
CRS-2677: Stop of 'ora.scan1.vip' on 'rhel12c1' succeeded
CRS-2672: Attempting to start 'ora.scan1.vip' on 'rhel12c2'
CRS-2676: Start of 'ora.scan1.vip' on 'rhel12c2' succeeded
CRS-2672: Attempting to start 'ora.LISTENER_SCAN1.lsnr' on 'rhel12c2'
CRS-2676: Start of 'ora.LISTENER_SCAN1.lsnr' on 'rhel12c2' succeeded
CRS-2676: Start of 'ora.asm' on 'rhel12c2' succeeded
CRS-6016: Resource auto-start has completed for server rhel12c2
CRS-6024: Completed start of Oracle Cluster Ready Services-managed resources
CRS-4123: Oracle High Availability Services has been started.
2015/08/14 13:24:22 CLSRSC-343: Successfully started Oracle Clusterware stack

2015/08/14 13:24:42 CLSRSC-325: Configure Oracle Grid Infrastructure for a Cluster ... succeeded
Once the root.sh completes the confiugration assistants are run and cluster installation complete. The type of the cluster and nodes could be checked as follows
[root@rhel12c1 oracle]# crsctl get cluster mode status
Cluster is running in "flex" mode

[root@rhel12c1 oracle]# crsctl get node role config
Node 'rhel12c1' configured role is 'hub'

[grid@rhel12c2 ~]$ crsctl get node role config
Node 'rhel12c2' configured role is 'hub'
ASM will have the cluster ASM listener and default ASM instance count of 3
srvctl config asm -detail
ASM home: 
Password file: +clusterdg/orapwASM
ASM listener: LISTENER
ASM is enabled.
ASM is individually enabled on nodes:
ASM is individually disabled on nodes:
ASM instance count: 3
Cluster ASM listener: ASMNET1LSNR_ASM
But on this two node cluster only two ASM instances would be active
ora.asm
      1        ONLINE  ONLINE       rhel12c1                 Started,STABLE
      2        ONLINE  ONLINE       rhel12c2                 Started,STABLE
      3        OFFLINE OFFLINE                               STABLE
Also check the ASM cluster mode
ASMCMD> showclustermode
ASM cluster : Flex mode enabled
Once the cluster is setup install the database software. There's no additional steps to this due to flex cluster, installation is same as 12.1.0.1. Once the database software is installed create the database either CDB or non-CDB.
With version 12.1.0.2 the cluster log files were moved inside diagnostic dest as such the ownership of diag folder is grid:oinstall with permission 755. Due to this fact creation of database will fail as Oracle user doesn't have the write permission inside the diag directory. This is a change from previous 12.1.0.1 version. Before running the DBCA change the permission of following directories so oracle user has write permission
chmod 770 cfgtoollogs
chmod 770 admin
chmod 770 cfgtoollogs/dbca
chmod 770 audit
chmod 770 diag
There are no other additional steps to create a database in flex cluster compared a standard cluster.

Related Posts
Installing 12c (12.1.0.1) RAC on RHEL 6 with Role Seperation - Clusterware
Installing 11gR2 (11.2.0.3) GI with Role Separation on RHEL 6
Installing 11gR2 (11.2.0.3) GI with Role Separation on OEL 6
Installing 11gR2 Standalone Server with ASM and Role Separation on RHEL 6
11gR2 Standalone Data Guard (with ASM and Role Separation)

Friday, August 14, 2015

Moving non-RAC Database and ASM Between Servers

There are many strategies to moving a database from one server to another. Backup/duplication, exports base solutions are most common. This post list steps for moving a non-RAC database and ASM (standlone) by way of attaching the underlying storage to the new server. The steps here has been tested using Amazon EC2/ESB combination and using Oracle VirtualBox. As stated in title the current setup includes a standalone non-RAC database using ASM.
1. To able to use ASM in the new sever, install grid infrastructure with software only option. This allow installation of GI without having to create ASM disk groups.

2. When prompted run the root scripts.
# /opt/app/oracle/product/11.2.0/grid_1/root.sh
Performing root user operation for Oracle 11g

The following environment variables are set as:
    ORACLE_OWNER= grid
    ORACLE_HOME=  /opt/app/oracle/product/11.2.0/grid_1

Enter the full pathname of the local bin directory: [/usr/local/bin]:
The contents of "dbhome" have not changed. No need to overwrite.
The contents of "oraenv" have not changed. No need to overwrite.
The contents of "coraenv" have not changed. No need to overwrite.


Creating /etc/oratab file...
Entries will be added to the /etc/oratab file as needed by
Database Configuration Assistant when a database is created
Finished running generic part of root script.
Now product-specific root actions will be performed.

To configure Grid Infrastructure for a Stand-Alone Server run the following command as the root user:
/opt/app/oracle/product/11.2.0/grid_1/perl/bin/perl -I/opt/app/oracle/product/11.2.0/grid_1/perl/lib -I/opt/app/oracle/product/11.2.0/grid_1/crs/install /opt/app/oracle/product/11.2.0/grid_1/crs/install/roothas.pl


To configure Grid Infrastructure for a Cluster execute the following command:
/opt/app/oracle/product/11.2.0/grid_1/crs/config/config.sh
This command launches the Grid Infrastructure Configuration Wizard. The wizard also supports silent operation, and the parameters can be passed through the response file that is available in the installation media.


# /opt/app/oracle/product/11.2.0/grid_1/perl/bin/perl -I/opt/app/oracle/product/11.2.0/grid_1/perl/lib -I/opt/app/oracle/product/11.2.0/grid_1/crs/install /opt/app/oracle/product/11.2.0/grid_1/crs/install/roothas.pl
Using configuration parameter file: /opt/app/oracle/product/11.2.0/grid_1/crs/install/crsconfig_params
Creating trace directory
User ignored Prerequisites during installation
LOCAL ADD MODE
Creating OCR keys for user 'grid', privgrp 'oinstall'..
Operation successful.
LOCAL ONLY MODE
Successfully accumulated necessary OCR keys.
Creating OCR keys for user 'root', privgrp 'root'..
Operation successful.
CRS-4664: Node rhel6m1 successfully pinned.
Adding Clusterware entries to upstart

rhel6m1     2015/03/10 16:40:10     /opt/app/oracle/product/11.2.0/grid_1/cdata/rhel6m1/backup_20150310_164010.olr
Successfully configured Oracle Grid Infrastructure for a Standalone Server
3. GI will have following resources created at this stage
$ crsctl stat res
NAME=ora.cssd
TYPE=ora.cssd.type
TARGET=OFFLINE
STATE=OFFLINE

NAME=ora.diskmon
TYPE=ora.diskmon.type
TARGET=OFFLINE
STATE=OFFLINE

NAME=ora.evmd
TYPE=ora.evm.type
TARGET=ONLINE
STATE=ONLINE on rhel6m1

NAME=ora.ons
TYPE=ora.ons.type
TARGET=OFFLINE
STATE=OFFLINE
4. Install database software with software only option. Patch both GI and database software to the same patch level as the current database setup where database being moved from.
5. If role separation being used them some directory locations ($ORACLE_BASE/admin, $ORACLE_BASE/cfgtoollogs) may not have write permissions for Oracle user, permissions might be set as 755 grid : oinstall. Change permissions as 775 to allow oracle user the write access to these location. Also manually create the audit file location
mkdir -p /opt/app/oracle/admin/ent11g2/adump
6. Copy the listener file from current setup to new server's $GI_HOME/network/admin. Change all reference to the hostname to reflect the new hostname.



7. Attach the storage device(s) to the new server. Depending on the storage type this may require server to shutdown. Once the server is restarted verify the disks that will be used by ASM are visible.Also HA services are running
# crsctl stat res

NAME=ora.cssd
TYPE=ora.cssd.type
TARGET=ONLINE
STATE=ONLINE on rhel6m1
8. With storage attached its only a matter of configuring listener,ASM and database so HAS manage them. First add the listener as grid user.
$ srvctl add listener -l listener -o /opt/app/oracle/product/11.2.0/grid_1 -p 1521
$ srvctl start listener -l listener
$ srvctl status listener -l listener
Listener LISTENER is enabled
Listener LISTENER is running on node(s): rhel6m1
9. Secondly add the ASM as grid user. The ASM pfile is located in the local server as opposed to ASM itself.
$ srvctl add asm -l listener -p /opt/app/oracle/product/11.2.0/grid_1/dbs/spfile+ASM.ora -d "/dev/sd*"
$ srvctl start asm
$ srvctl config asm
ASM home: /opt/app/oracle/product/11.2.0/grid_1
ASM listener: LISTENER
Spfile: /opt/app/oracle/product/11.2.0/grid_1/dbs/spfile+ASM.ora
ASM diskgroup discovery string: /dev/sd*
10. Finally as oracle user add the database and modify the auto start options of the resources
srvctl add database -d ent11g2 -o /opt/app/oracle/product/11.2.0/dbhome_1 -p +DATA/ent11g2/spfileent11g2.ora -a "data,flash,apps"
srvctl start database -d ent11g2

crsctl modify resource ora.DATA.dg -attr "AUTO_START"="always"
crsctl modify resource ora.FLASH.dg -attr "AUTO_START"="always"  
crsctl modify resource ora.LISTENER.lsnr -attr "AUTO_START"="always"  
crsctl modify resource ora.ent11g2.db -attr "AUTO_START"="always"
This concludes the steps for moving non-RAC database and ASM by way of attaching storage from one server to another.

Monday, August 10, 2015

Applying of GI PSU 11.2.0.4.5 Fails Due to -customLogDir Opton Used by OPatch 11.2.0.3.10

Use of opatch auto option has become the default for applying PSU for both GI and Oracle home in an automated fashion. Applying 11.2.0.4.5 GI PSU (July 2015) for both GI and Oracle Home in a single instance setup failed with following message.
$ /opt/app/oracle/product/11.2.0/grid_4/OPatch/opatch version
OPatch Version: 11.2.0.3.10

# /opt/app/oracle/product/11.2.0/grid_4/OPatch/opatch auto ./20996923 -ocmrf ../ocm.rsp
Executing /opt/app/oracle/product/11.2.0/grid_4/perl/bin/perl /opt/app/oracle/product/11.2.0/grid_4/OPatch/crs/patch11203.pl -patchdir . -patchn 20996923 -ocmrf ../ocm.rsp -paramfile /opt/app/oracle/product/11.2.0/grid_4/crs/install/crsconfig_params
This is the main log file: /opt/app/oracle/product/11.2.0/grid_4/cfgtoollogs/opatchauto2015-08-10_14-06-35.log
This file will show your detected configuration and all the steps that opatchauto attempted to do on your system:
/opt/app/oracle/product/11.2.0/grid_4/cfgtoollogs/opatchauto2015-08-10_14-06-35.report.log
2015-08-10 14:06:35: Starting Oracle Restart Patch Setup
Using configuration parameter file: /opt/app/oracle/product/11.2.0/grid_4/crs/install/crsconfig_params
The opatch Component check failed. This patch is not applicable for /opt/app/oracle/product/11.2.0/dbhome_4
The opatch Component check failed. This patch is not applicable for /opt/app/oracle/product/11.2.0/dbhome_4
Conflict check failed for oracle home  /opt/app/oracle/product/11.2.0/dbhome_4
Conflict check failed
ERROR: Conflict-Check has failed . Please refer to /opt/app/oracle/product/11.2.0/grid_4/cfgtoollogs/opatchauto2015-08-10_14-06-35.log for details
Though the error message says conflict check failed, it is not the real reason in this case. Examining the log file revealed following
2015-08-10 14:07:13: Status of opatch version check  for /opt/app/oracle/product/11.2.0/grid_4 is 1
2015-08-10 14:07:13: Opatch version check passed for oracle home  /opt/app/oracle/product/11.2.0/grid_4
2015-08-10 14:07:13: Opatch version check passed  for all oracle homes
2015-08-10 14:07:13: Processing oracle home /opt/app/oracle/product/11.2.0/dbhome_4
2015-08-10 14:07:13: Opening file /etc/oracle/ocr.loc
2015-08-10 14:07:13: Value (TRUE) is set for key=local_only
2015-08-10 14:07:13: Home type of /opt/app/oracle/product/11.2.0/dbhome_4 is DB
2015-08-10 14:07:13: Oracle user for /opt/app/oracle/product/11.2.0/dbhome_4 is oracle
2015-08-10 14:07:13: Running as user oracle: /opt/app/oracle/product/11.2.0/dbhome_4/OPatch/opatch prereq CheckComponents -ph ./20996923/20760982 -oh /opt/app/oracle/product/11.2.0/dbhome_4 
-customLogDir /opt/app/oracle/product/11.2.0/grid_4/cfgtoollogs/opatchauto/core
2015-08-10 14:07:13: s_run_as_user2: Running /bin/su oracle -c ' /opt/app/oracle/product/11.2.0/dbhome_4/OPatch/opatch prereq CheckComponents -ph ./20996923/20760982 -oh /opt/app/oracle/product/11.2.0/dbhome_4 
-customLogDir /opt/app/oracle/product/11.2.0/grid_4/cfgtoollogs/opatchauto/core '
2015-08-10 14:07:14: Removing file /tmp/fileWJrEf9
2015-08-10 14:07:14: Successfully removed file: /tmp/fileWJrEf9
2015-08-10 14:07:14: /bin/su exited with rc=1

2015-08-10 14:07:14: The opatch Component check failed. This patch is not applicable for /opt/app/oracle/product/11.2.0/dbhome_4
2015-08-10 14:07:14: The component check failed with following error
2015-08-10 14:07:14: Input custom log directory "/opt/app/oracle/product/11.2.0/grid_4/cfgtoollogs/opatchauto/core" does not exists or not a valid directory.

 OPatch failed with error code 1
From the output it could be seen that when opatch is checking components for oracle home patching it is setting a custom log directory which is inside the grid home. This wouldn't be a problem if both GI home and Oracle home are owned by the same user. But in this case role separation is used and the directories cfgtoollogs,opatchauto,core all have permission set to either 700 or 755 and ownership grid:oinstall. This makes oracle user unable to write to these directories and the failure of the patch apply.
Metalink notes 2022797.1 and 1964569.1 (These MOS notes are related to 12c) suggest to use -customLogDir option to specify a directory but this is not applicable for "opatch auto" option.



Two ways to overcome this problem. One is to grant the oinstall group the write permission to cfgtoollogs/opatchauto/core before the patch apply and revoke once the the patch is successfully applied.
Second option is to use a lower version OPatch, in this case using OPatch version 11.2.0.3.6 (minimum version required according to PSU's readme.html) enabled to successfully apply the patch.
As patch apply went ahead without a problem using a lower version of opatch mean that customLogDir option was introduced in higher version OPatch (11.2.0.3.10) without the consideration to possible failures that could arise due to use of role separation.

Useful Metalink Notes
opatchauto wasn't able to run ... because 'bash' command could not be found [ID 2022797.1]
"opatch lsinventory" on CRS/GI home fails with error code 15 [ID 1964569.1]