Changing Hub Node to Leaf Node and Vice Versa

The flex cluster setup used here consists of two hub nodes. One of the nodes will be changed from hub node to leaf node and back to a hub node. The nodes are named rhel12c1 and rhel12c2.

Changing Hub Node to a Leaf Node
Node rhel12c1 will be changed from a hub node to a leaf node. The current role could listed with

[root@rhel12c1 oracle]# crsctl get node role config
Node 'rhel12c1' configured role is 'hub'

To change the role of the node run the following command. This must be run on the node that's undergoing the role change as root

[root@rhel12c1 grid]# crsctl set node role leaf
CRS-4408: Node 'rhel12c1' configured role successfully changed; restart Oracle High Availability Services for new role to take effect.

Restart the node after the role change command has been executed

# crsctl stop crs
# crsctl start crs -wait

Verify the node started as a leaf node.

[root@rhel12c1 grid]# crsctl get node role config
Node 'rhel12c1' configured role is 'leaf'

The cluster now consists of a hub node and a leaf node

[root@rhel12c2 grid]# crsctl get node role config -all
Node 'rhel12c1' configured role is 'leaf'
Node 'rhel12c2' configured role is 'hub'

The database instance that was running while the rhel12c1 was a hub node will no longer be active and is in a shutdown state

ora.rac12c1.db
      1        ONLINE  OFFLINE                               Instance Shutdown,STABLE
      2        ONLINE  ONLINE       rhel12c2                 Open,STABLE

There's no resources running on the leaf node.

[root@rhel12c1 grid]# crsctl stat res -t -c rhel12c1

Finally update inventory as follows. Run the update node list command on hub nodes listing all remaining hub nodes in the cluster_nodes option. In this case only rhel12c2 is the hub node.

[grid@rhel12c2 ~]$ $GI_HOME/oui/bin/runInstaller -updateNodeList ORACLE_HOME=/opt/app/12.1.0/grid2 "CLUSTER_NODES={rhel12c2}" -silent -local CRS=TRUE
Starting Oracle Universal Installer...

Checking swap space: must be greater than 500 MB.   Actual 4506 MB    Passed
The inventory pointer is located at /etc/oraInst.loc
'UpdateNodeList' was successful.

On the leaf node run the update node list containing only the leaf node in the cluster_nodes option.

[grid@rhel12c1 ~]$ $GI_HOME/oui/bin/runInstaller -updateNodeList ORACLE_HOME=/opt/app/12.1.0/grid2 "CLUSTER_NODES={rhel12c1}" -silent -local CRS=TRUE
Starting Oracle Universal Installer...

Checking swap space: must be greater than 500 MB.   Actual 5112 MB    Passed
The inventory pointer is located at /etc/oraInst.loc
'UpdateNodeList' was successful.

This conclude converting from hub node to leaf node.



Changing Leaf Node to a Hub Node
Leaf node created in the previous step will be changed back to a hub node in this place. The current node role is

[grid@rhel12c1 ~]$ crsctl get node role config
Node 'rhel12c1' configured role is 'leaf'

As root change the node role to hub node with

[root@rhel12c1 grid]# crsctl set node role hub
CRS-4408: Node 'rhel12c1' configured role successfully changed; restart Oracle High Availability Services for new role to take effect.

Stop the cluster stack on the node undergoing the role change

# crsctl stop crs

Configure the Oracle ASM Filter Driver by running the following as root

# crsctl stop crs

[root@rhel12c1 grid]# /opt/app/12.1.0/grid2/bin/asmcmd afd_configure
Connected to an idle instance.
AFD-627: AFD distribution files found.
AFD-636: Installing requested AFD software.
AFD-637: Loading installed AFD drivers.
AFD-9321: Creating udev for AFD.
AFD-9323: Creating module dependencies - this may take some time.
AFD-9154: Loading 'oracleafd.ko' driver.
AFD-649: Verifying AFD devices.
AFD-9156: Detecting control device '/dev/oracleafd/admin'.
AFD-638: AFD installation correctness verified.
Modifying resource dependencies - this may take some time.

If the node started as a leaf node then it may also require VIP to be configured on the node before changing to hub node. As this started off as a leaf node the VIP already exists and this step is not needed. Once the ASM driver is configured start cluster stack on the node and verify the node role

[root@rhel12c1 grid]# crsctl start crs -wait
[grid@rhel12c1 ~]$ crsctl get node role config
Node 'rhel12c1' configured role is 'hub'

Resources there were down when the role was in leaf node role will be up and running again now.

[root@rhel12c1 grid]# crsctl stat res -t -c rhel12c1
--------------------------------------------------------------------------------
Name           Target  State        Server                   State details
--------------------------------------------------------------------------------
Local Resources
--------------------------------------------------------------------------------
ora.ASMNET1LSNR_ASM.lsnr
               ONLINE  ONLINE       rhel12c1                 STABLE
ora.DATA.dg
               ONLINE  ONLINE       rhel12c1                 STABLE
ora.FRA.dg
               ONLINE  ONLINE       rhel12c1                 STABLE
ora.LISTENER.lsnr
               ONLINE  ONLINE       rhel12c1                 STABLE
ora.NEWCLUSTERDG.dg
               ONLINE  ONLINE       rhel12c1                 STABLE
ora.net1.network
               ONLINE  ONLINE       rhel12c1                 STABLE
ora.ons
               ONLINE  ONLINE       rhel12c1                 STABLE
--------------------------------------------------------------------------------
Cluster Resources
--------------------------------------------------------------------------------
ora.LISTENER_SCAN1.lsnr
      1        ONLINE  ONLINE       rhel12c1                 STABLE
ora.asm
      2        ONLINE  ONLINE       rhel12c1                 Started,STABLE
ora.rac12c1.db
      1        ONLINE  ONLINE       rhel12c1                 Open,STABLE
ora.rhel12c1.vip
      1        ONLINE  ONLINE       rhel12c1                 STABLE
ora.scan1.vip
      1        ONLINE  ONLINE       rhel12c1                 STABLE
--------------------------------------------------------------------------------

Run the inventory update specifying all hub nodes in the cluster_node option

$GI_HOME/oui/bin/runInstaller -updateNodeList ORACLE_HOME=/opt/app/12.1.0/grid2 "CLUSTER_NODES={rhel12c1,rhel12c2}" -silent -local CRS=TRUE

This conclude the changing leaf node to hub node.

Converting 12c Standard Cluster to Flex Cluster

An earlier post showed steps for installing a 12c flex cluster. This post shows the steps for converting a standard cluster to a flex cluster. There are two phases to converting a standard cluster to flex cluster. First phase involves converting Oracle ASM to Flex ASM and second phase is to change the cluster mode to flex.
The current setup is a two node standard cluster.

$  crsctl get cluster mode status
Cluster is running in "standard" mode

$ asmcmd
ASMCMD> showclustermode
ASM cluster : Flex mode disabled

There are few per-requisites to converting ASM to flex ASM, these include that OCR, SPFile and password file all stored in a disk group and this group have ASM compatibility (COMPATIBLE.ASM) set to 12.1 or higher. If this is not the case then make the necessary changes to move these files to disk group. As the current setup is a newly installed 12c cluster (as oppose to upgraded to 12c) these files all reside in disk group.
A point to note is that the diskgroup with above mentioned files has the size of 5114MB (below output has been modified by omitting some columns).

ASMCMD> lsdg
State    Type    Rebal  Sector  Block       AU  Total_MB  Free_MB  Voting_files  Name
MOUNTED  EXTERN  N         512   4096  1048576     10236     7609             N  DATA/
MOUNTED  EXTERN  N         512   4096  1048576     10236     8609             N  FRA/
MOUNTED  EXTERN  N         512   4096  1048576      5114      475             Y  NEWCLUSTERDG/

This is smaller than the required diskgroup size for new flex cluster being installed. Despite the smaller size diskgroup the conversion to flex cluster succeeded without any issue. The current ASM configuration is

$  srvctl config asm
ASM home: 
Password file: +NEWCLUSTERDG/orapwasm
ASM listener: LISTENER

Another key requirement is that flex cluster requires GNS. If a GNS doesn't exists setup GNS and add the GNS VIP to cluster. The standard cluster was already setup with GNS.

$  srvctl config gns
GNS is enabled.
GNS VIP addresses: 192.168.0.87
Domain served by GNS: rac.domain.net

To start the first phase of converting to flex ASM run asmca and click on ASM instances tab. If role separation is used then make sure grid user has write permission for directory $ORACLE_BASE/cfgtoollogs as the conversion script is generated inside it.

Click convert to Flex ASM button. This would prompt to select a network interface for ASM communication and also to specify a port. The default port is 1521.

However there is a problem in using the port 1521.

The port 1521 is used on the private network (192.168.1.0 in this case) by the management listener.

$ netstat -ntl | grep 1521
tcp        0      0 192.168.0.89:1521           0.0.0.0:*                   LISTEN      3323/tnslsnr
tcp        0      0 192.168.0.96:1521           0.0.0.0:*                   LISTEN      3323/tnslsnr
tcp        0      0 192.168.0.97:1521           0.0.0.0:*                   LISTEN      3323/tnslsnr
tcp        0      0 192.168.0.93:1521           0.0.0.0:*                   LISTEN      3323/tnslsnr
tcp        0      0 192.168.0.95:1521           0.0.0.0:*                   LISTEN      3323/tnslsnr
tcp        0      0 169.254.11.195:1521         0.0.0.0:*                   LISTEN      3323/tnslsnr
tcp        0      0 192.168.1.87:1521           0.0.0.0:*                   LISTEN      3323/tnslsnr

# ps ax | grep 3323
 3323 ?        Ssl    0:00 /opt/app/12.1.0/grid2/bin/tnslsnr MGMTLSNR -no_crs_notify -inherit

Even after stopping the MGMTLSNR

$ srvctl stop listener -l MGMTLSNR
PRCR-1065 : Failed to stop resource ora.MGMTLSNR
CRS-2529: Unable to act on 'ora.MGMTLSNR' because that would require stopping or relocating 'ora.mgmtdb', but the force option was not specified
$ srvctl stop listener -l MGMTLSNR -f

ora.MGMTLSNR
      1        OFFLINE OFFLINE                               169.254.11.195 192.1
                                                             68.1.87,STABLE

and all other listeners (scan_listener and listener) that were using port 1521 (even though they were on a different network) the error did not get resolved. It could be port usage is looked up statically from a config file. There's no problem to use 1521 for both database listener and ASM listener when a new flex cluster is installed. Finally had to proceed using a different port for cluster ASM listener to continue forward.

Conversion to flex ASM will continue restarting one node at time.

When prompted execute the script on the same node ASMCA was run.

Output from converttoFlexASM.sh execution

# /opt/app/oracle/cfgtoollogs/asmca/scripts/converttoFlexASM.sh
CRS-2673: Attempting to stop 'ora.crsd' on 'rhel12c1'
CRS-2790: Starting shutdown of Cluster Ready Services-managed resources on 'rhel12c1'
CRS-2673: Attempting to stop 'ora.mgmtdb' on 'rhel12c1'
CRS-2673: Attempting to stop 'ora.gns' on 'rhel12c1'
CRS-2673: Attempting to stop 'ora.LISTENER_SCAN2.lsnr' on 'rhel12c1'
CRS-2673: Attempting to stop 'ora.NEWCLUSTERDG.dg' on 'rhel12c1'
CRS-2673: Attempting to stop 'ora.LISTENER.lsnr' on 'rhel12c1'
CRS-2673: Attempting to stop 'ora.rac12c1.db' on 'rhel12c1'
CRS-2673: Attempting to stop 'ora.LISTENER_SCAN3.lsnr' on 'rhel12c1'
CRS-2673: Attempting to stop 'ora.DATA.dg' on 'rhel12c1'
CRS-2673: Attempting to stop 'ora.oc4j' on 'rhel12c1'
CRS-2673: Attempting to stop 'ora.cvu' on 'rhel12c1'
CRS-2673: Attempting to stop 'ora.FRA.dg' on 'rhel12c1'
CRS-2677: Stop of 'ora.cvu' on 'rhel12c1' succeeded
CRS-2672: Attempting to start 'ora.cvu' on 'rhel12c2'
CRS-2677: Stop of 'ora.LISTENER_SCAN2.lsnr' on 'rhel12c1' succeeded
CRS-2673: Attempting to stop 'ora.scan2.vip' on 'rhel12c1'
CRS-2677: Stop of 'ora.LISTENER.lsnr' on 'rhel12c1' succeeded
CRS-2677: Stop of 'ora.DATA.dg' on 'rhel12c1' succeeded
CRS-2677: Stop of 'ora.NEWCLUSTERDG.dg' on 'rhel12c1' succeeded
CRS-2677: Stop of 'ora.FRA.dg' on 'rhel12c1' succeeded
CRS-2677: Stop of 'ora.gns' on 'rhel12c1' succeeded
CRS-2673: Attempting to stop 'ora.gns.vip' on 'rhel12c1'
CRS-2677: Stop of 'ora.LISTENER_SCAN3.lsnr' on 'rhel12c1' succeeded
CRS-2673: Attempting to stop 'ora.scan3.vip' on 'rhel12c1'
CRS-2673: Attempting to stop 'ora.rhel12c1.vip' on 'rhel12c1'
CRS-2676: Start of 'ora.cvu' on 'rhel12c2' succeeded
CRS-2677: Stop of 'ora.scan2.vip' on 'rhel12c1' succeeded
CRS-2672: Attempting to start 'ora.scan2.vip' on 'rhel12c2'
CRS-2677: Stop of 'ora.rac12c1.db' on 'rhel12c1' succeeded
CRS-2677: Stop of 'ora.mgmtdb' on 'rhel12c1' succeeded
CRS-2673: Attempting to stop 'ora.MGMTLSNR' on 'rhel12c1'
CRS-2677: Stop of 'ora.MGMTLSNR' on 'rhel12c1' succeeded
CRS-2672: Attempting to start 'ora.MGMTLSNR' on 'rhel12c2'
CRS-2677: Stop of 'ora.gns.vip' on 'rhel12c1' succeeded
CRS-2672: Attempting to start 'ora.gns.vip' on 'rhel12c2'
CRS-2677: Stop of 'ora.scan3.vip' on 'rhel12c1' succeeded
CRS-2672: Attempting to start 'ora.scan3.vip' on 'rhel12c2'
CRS-2677: Stop of 'ora.rhel12c1.vip' on 'rhel12c1' succeeded
CRS-2672: Attempting to start 'ora.rhel12c1.vip' on 'rhel12c2'
CRS-2676: Start of 'ora.scan2.vip' on 'rhel12c2' succeeded
CRS-2672: Attempting to start 'ora.LISTENER_SCAN2.lsnr' on 'rhel12c2'
CRS-2676: Start of 'ora.MGMTLSNR' on 'rhel12c2' succeeded
CRS-2672: Attempting to start 'ora.mgmtdb' on 'rhel12c2'
CRS-2676: Start of 'ora.gns.vip' on 'rhel12c2' succeeded
CRS-2672: Attempting to start 'ora.gns' on 'rhel12c2'
CRS-2677: Stop of 'ora.oc4j' on 'rhel12c1' succeeded
CRS-2672: Attempting to start 'ora.oc4j' on 'rhel12c2'
CRS-2676: Start of 'ora.scan3.vip' on 'rhel12c2' succeeded
CRS-2672: Attempting to start 'ora.LISTENER_SCAN3.lsnr' on 'rhel12c2'
CRS-2676: Start of 'ora.rhel12c1.vip' on 'rhel12c2' succeeded
CRS-2676: Start of 'ora.gns' on 'rhel12c2' succeeded
CRS-2676: Start of 'ora.LISTENER_SCAN2.lsnr' on 'rhel12c2' succeeded
CRS-2676: Start of 'ora.LISTENER_SCAN3.lsnr' on 'rhel12c2' succeeded
CRS-2676: Start of 'ora.oc4j' on 'rhel12c2' succeeded
CRS-2676: Start of 'ora.mgmtdb' on 'rhel12c2' succeeded
CRS-2673: Attempting to stop 'ora.ons' on 'rhel12c1'
CRS-2677: Stop of 'ora.ons' on 'rhel12c1' succeeded
CRS-2673: Attempting to stop 'ora.net1.network' on 'rhel12c1'
CRS-2677: Stop of 'ora.net1.network' on 'rhel12c1' succeeded
CRS-2792: Shutdown of Cluster Ready Services-managed resources on 'rhel12c1' has completed
CRS-2677: Stop of 'ora.crsd' on 'rhel12c1' succeeded
CRS-2673: Attempting to stop 'ora.ctssd' on 'rhel12c1'
CRS-2673: Attempting to stop 'ora.evmd' on 'rhel12c1'
CRS-2673: Attempting to stop 'ora.storage' on 'rhel12c1'
CRS-2677: Stop of 'ora.storage' on 'rhel12c1' succeeded
CRS-2673: Attempting to stop 'ora.asm' on 'rhel12c1'
CRS-2677: Stop of 'ora.ctssd' on 'rhel12c1' succeeded
CRS-2677: Stop of 'ora.evmd' on 'rhel12c1' succeeded
CRS-2677: Stop of 'ora.asm' on 'rhel12c1' succeeded
CRS-2673: Attempting to stop 'ora.cluster_interconnect.haip' on 'rhel12c1'
CRS-2677: Stop of 'ora.cluster_interconnect.haip' on 'rhel12c1' succeeded
CRS-2673: Attempting to stop 'ora.cssd' on 'rhel12c1'
CRS-2677: Stop of 'ora.cssd' on 'rhel12c1' succeeded
CRS-2672: Attempting to start 'ora.evmd' on 'rhel12c1'
CRS-2672: Attempting to start 'ora.cssdmonitor' on 'rhel12c1'
CRS-2676: Start of 'ora.cssdmonitor' on 'rhel12c1' succeeded
CRS-2672: Attempting to start 'ora.cssd' on 'rhel12c1'
CRS-2672: Attempting to start 'ora.diskmon' on 'rhel12c1'
CRS-2676: Start of 'ora.diskmon' on 'rhel12c1' succeeded
CRS-2676: Start of 'ora.evmd' on 'rhel12c1' succeeded
CRS-2676: Start of 'ora.cssd' on 'rhel12c1' succeeded
CRS-2672: Attempting to start 'ora.ctssd' on 'rhel12c1'
CRS-2672: Attempting to start 'ora.cluster_interconnect.haip' on 'rhel12c1'
CRS-2676: Start of 'ora.ctssd' on 'rhel12c1' succeeded
CRS-2676: Start of 'ora.cluster_interconnect.haip' on 'rhel12c1' succeeded
CRS-2672: Attempting to start 'ora.asm' on 'rhel12c1'
CRS-2676: Start of 'ora.asm' on 'rhel12c1' succeeded
CRS-2672: Attempting to start 'ora.storage' on 'rhel12c1'
CRS-2676: Start of 'ora.storage' on 'rhel12c1' succeeded
CRS-2672: Attempting to start 'ora.crsd' on 'rhel12c1'
CRS-2676: Start of 'ora.crsd' on 'rhel12c1' succeeded
Oracle Grid Infrastructure restarted in node rhel12c1
PRCC-1014 : ASMNET1LSNR_ASM was already running
PRCR-1004 : Resource ora.ASMNET1LSNR_ASM.lsnr is already running
PRCR-1079 : Failed to start resource ora.ASMNET1LSNR_ASM.lsnr
CRS-5702: Resource 'ora.ASMNET1LSNR_ASM.lsnr' is already running on 'rhel12c2'
CRS-5702: Resource 'ora.ASMNET1LSNR_ASM.lsnr' is already running on 'rhel12c1'
ASM listener ASMNET1LSNR_ASM running already
CRS-2673: Attempting to stop 'ora.crsd' on 'rhel12c2'
CRS-2790: Starting shutdown of Cluster Ready Services-managed resources on 'rhel12c2'
CRS-2673: Attempting to stop 'ora.rac12c1.cbx.svc' on 'rhel12c2'
CRS-2673: Attempting to stop 'ora.LISTENER_SCAN3.lsnr' on 'rhel12c2'
CRS-2673: Attempting to stop 'ora.LISTENER_SCAN2.lsnr' on 'rhel12c2'
CRS-2673: Attempting to stop 'ora.mgmtdb' on 'rhel12c2'
CRS-2673: Attempting to stop 'ora.LISTENER.lsnr' on 'rhel12c2'
CRS-2673: Attempting to stop 'ora.oc4j' on 'rhel12c2'
CRS-2673: Attempting to stop 'ora.cvu' on 'rhel12c2'
CRS-2673: Attempting to stop 'ora.NEWCLUSTERDG.dg' on 'rhel12c2'
CRS-2673: Attempting to stop 'ora.ASMNET1LSNR_ASM.lsnr' on 'rhel12c2'
CRS-2677: Stop of 'ora.cvu' on 'rhel12c2' succeeded
CRS-2672: Attempting to start 'ora.cvu' on 'rhel12c1'
CRS-2677: Stop of 'ora.NEWCLUSTERDG.dg' on 'rhel12c2' succeeded
CRS-2677: Stop of 'ora.LISTENER_SCAN2.lsnr' on 'rhel12c2' succeeded
CRS-2673: Attempting to stop 'ora.scan2.vip' on 'rhel12c2'
CRS-2677: Stop of 'ora.LISTENER_SCAN3.lsnr' on 'rhel12c2' succeeded
CRS-2673: Attempting to stop 'ora.scan3.vip' on 'rhel12c2'
CRS-2677: Stop of 'ora.LISTENER.lsnr' on 'rhel12c2' succeeded
CRS-2677: Stop of 'ora.rac12c1.cbx.svc' on 'rhel12c2' succeeded
CRS-2673: Attempting to stop 'ora.rac12c1.db' on 'rhel12c2'
CRS-2677: Stop of 'ora.ASMNET1LSNR_ASM.lsnr' on 'rhel12c2' succeeded
CRS-2676: Start of 'ora.cvu' on 'rhel12c1' succeeded
CRS-2677: Stop of 'ora.scan2.vip' on 'rhel12c2' succeeded
CRS-2672: Attempting to start 'ora.scan2.vip' on 'rhel12c1'
CRS-2677: Stop of 'ora.scan3.vip' on 'rhel12c2' succeeded
CRS-2672: Attempting to start 'ora.scan3.vip' on 'rhel12c1'
CRS-2677: Stop of 'ora.rac12c1.db' on 'rhel12c2' succeeded
CRS-2677: Stop of 'ora.mgmtdb' on 'rhel12c2' succeeded
CRS-2673: Attempting to stop 'ora.MGMTLSNR' on 'rhel12c2'
CRS-2677: Stop of 'ora.MGMTLSNR' on 'rhel12c2' succeeded
CRS-2672: Attempting to start 'ora.MGMTLSNR' on 'rhel12c1'
CRS-2673: Attempting to stop 'ora.rhel12c2.vip' on 'rhel12c2'
CRS-2676: Start of 'ora.scan2.vip' on 'rhel12c1' succeeded
CRS-2672: Attempting to start 'ora.LISTENER_SCAN2.lsnr' on 'rhel12c1'
CRS-2677: Stop of 'ora.rhel12c2.vip' on 'rhel12c2' succeeded
CRS-2672: Attempting to start 'ora.rhel12c2.vip' on 'rhel12c1'
CRS-2676: Start of 'ora.scan3.vip' on 'rhel12c1' succeeded
CRS-2672: Attempting to start 'ora.LISTENER_SCAN3.lsnr' on 'rhel12c1'
CRS-2676: Start of 'ora.MGMTLSNR' on 'rhel12c1' succeeded
CRS-2672: Attempting to start 'ora.mgmtdb' on 'rhel12c1'
CRS-2677: Stop of 'ora.oc4j' on 'rhel12c2' succeeded
CRS-2672: Attempting to start 'ora.oc4j' on 'rhel12c1'
CRS-2676: Start of 'ora.rhel12c2.vip' on 'rhel12c1' succeeded
CRS-2676: Start of 'ora.LISTENER_SCAN2.lsnr' on 'rhel12c1' succeeded
CRS-2676: Start of 'ora.LISTENER_SCAN3.lsnr' on 'rhel12c1' succeeded
CRS-2673: Attempting to stop 'ora.DATA.dg' on 'rhel12c2'
CRS-2673: Attempting to stop 'ora.FRA.dg' on 'rhel12c2'
CRS-2673: Attempting to stop 'ora.gns' on 'rhel12c2'
CRS-2677: Stop of 'ora.FRA.dg' on 'rhel12c2' succeeded
CRS-2677: Stop of 'ora.DATA.dg' on 'rhel12c2' succeeded
CRS-2677: Stop of 'ora.gns' on 'rhel12c2' succeeded
CRS-2673: Attempting to stop 'ora.gns.vip' on 'rhel12c2'
CRS-2677: Stop of 'ora.gns.vip' on 'rhel12c2' succeeded
CRS-2672: Attempting to start 'ora.gns.vip' on 'rhel12c1'
CRS-2676: Start of 'ora.gns.vip' on 'rhel12c1' succeeded
CRS-2672: Attempting to start 'ora.gns' on 'rhel12c1'
CRS-2676: Start of 'ora.gns' on 'rhel12c1' succeeded
CRS-2676: Start of 'ora.oc4j' on 'rhel12c1' succeeded
CRS-2676: Start of 'ora.mgmtdb' on 'rhel12c1' succeeded
CRS-2673: Attempting to stop 'ora.ons' on 'rhel12c2'
CRS-2677: Stop of 'ora.ons' on 'rhel12c2' succeeded
CRS-2673: Attempting to stop 'ora.net1.network' on 'rhel12c2'
CRS-2677: Stop of 'ora.net1.network' on 'rhel12c2' succeeded
CRS-2792: Shutdown of Cluster Ready Services-managed resources on 'rhel12c2' has completed
CRS-2677: Stop of 'ora.crsd' on 'rhel12c2' succeeded
CRS-2673: Attempting to stop 'ora.ctssd' on 'rhel12c2'
CRS-2673: Attempting to stop 'ora.evmd' on 'rhel12c2'
CRS-2673: Attempting to stop 'ora.storage' on 'rhel12c2'
CRS-2677: Stop of 'ora.storage' on 'rhel12c2' succeeded
CRS-2673: Attempting to stop 'ora.asm' on 'rhel12c2'
CRS-2677: Stop of 'ora.ctssd' on 'rhel12c2' succeeded
CRS-2677: Stop of 'ora.evmd' on 'rhel12c2' succeeded
CRS-2677: Stop of 'ora.asm' on 'rhel12c2' succeeded
CRS-2673: Attempting to stop 'ora.cluster_interconnect.haip' on 'rhel12c2'
CRS-2677: Stop of 'ora.cluster_interconnect.haip' on 'rhel12c2' succeeded
CRS-2673: Attempting to stop 'ora.cssd' on 'rhel12c2'
CRS-2677: Stop of 'ora.cssd' on 'rhel12c2' succeeded
CRS-2672: Attempting to start 'ora.cssdmonitor' on 'rhel12c2'
CRS-2672: Attempting to start 'ora.evmd' on 'rhel12c2'
CRS-2676: Start of 'ora.cssdmonitor' on 'rhel12c2' succeeded
CRS-2672: Attempting to start 'ora.cssd' on 'rhel12c2'
CRS-2672: Attempting to start 'ora.diskmon' on 'rhel12c2'
CRS-2676: Start of 'ora.diskmon' on 'rhel12c2' succeeded
CRS-2676: Start of 'ora.evmd' on 'rhel12c2' succeeded
CRS-2676: Start of 'ora.cssd' on 'rhel12c2' succeeded
CRS-2672: Attempting to start 'ora.ctssd' on 'rhel12c2'
CRS-2672: Attempting to start 'ora.cluster_interconnect.haip' on 'rhel12c2'
CRS-2676: Start of 'ora.ctssd' on 'rhel12c2' succeeded
CRS-2676: Start of 'ora.cluster_interconnect.haip' on 'rhel12c2' succeeded
CRS-2672: Attempting to start 'ora.asm' on 'rhel12c2'
CRS-2676: Start of 'ora.asm' on 'rhel12c2' succeeded
CRS-2672: Attempting to start 'ora.storage' on 'rhel12c2'
CRS-2676: Start of 'ora.storage' on 'rhel12c2' succeeded
CRS-2672: Attempting to start 'ora.crsd' on 'rhel12c2'
CRS-2676: Start of 'ora.crsd' on 'rhel12c2' succeeded
Oracle Grid Infrastructure restarted in node rhel12c2

End of the conversion the ASMCA will exit and console will have following output

Unable to establish connection to ASM instance.
 ORA-01034: ORACLE not available
 ASMCA will exit now.

This due to the restart of ASM and ASMCA unable to establish the connection. Rerun ASMCA and verify ASM instances are up an running. Notice the convert to flex ASM button is not there anymore.

List the new ASM configuration

$  srvctl config asm
ASM home: 
Password file: +NEWCLUSTERDG/orapwasm
ASM listener: LISTENER
ASM instance count: ALL
Cluster ASM listener: ASMNET1LSNR_ASM

The instance count is ALL because the option of running database version lower than 12.1 was selected. If this isn't the case the instance count would be

ASM instance count: 3

But only two ASM instances would be running.
The cluster ASM listener is available across the cluster

--------------------------------------------------------------------------------
Local Resources
--------------------------------------------------------------------------------
ora.ASMNET1LSNR_ASM.lsnr
               ONLINE  ONLINE       rhel12c1                 STABLE
               ONLINE  ONLINE       rhel12c2                 STABLE

Also verify the ASM flex mode with asmcmd

$ asmcmd
ASMCMD> showclustermode
ASM cluster : Flex mode enabled

The second phase is to convert the standard cluster mode to flex. This requires a cluster downtime.
To change the cluster mode to flex as root run the following(only on one of the nodes)

# crsctl set cluster mode flex
CRS-4933: Cluster mode set to "flex"; restart Oracle High Availability Services on all nodes for cluster to run in "flex" mode.

Shutdown clusterware on all nodes and start one node at a time

[root@rhel12c1 oracle]# crsctl stop crs
[root@rhel12c2 oracle]# crsctl stop crs

[root@rhel12c1 oracle]# crsctl start crs -wait
[root@rhel12c2 oracle]# crsctl start crs -wait

Once the cluster has started check the cluster mode and node roles

# crsctl get cluster mode status
Cluster is running in "flex" mode

[root@rhel12c1 oracle]# crsctl get node role config
Node 'rhel12c1' configured role is 'hub'

[grid@rhel12c2 ~]$ crsctl get node role config
Node 'rhel12c2' configured role is 'hub'

Make a note of ASM and network configurations as well

# srvctl config asm -detail
ASM home: 
Password file: +NEWCLUSTERDG/orapwasm
ASM listener: LISTENER
ASM is enabled.
ASM is individually enabled on nodes:
ASM is individually disabled on nodes:
ASM instance count: ALL
Cluster ASM listener: ASMNET1LSNR_ASM

oifcfg getif
eth0  192.168.0.0  global  public
eth1  192.168.1.0  global  cluster_interconnect,asm

This concludes the converting of a standard cluster to a flex cluster.

One of the key features of flex ASM is there's no one-one relationship between ASM instance and database instance. In a non flex ASM configuration failure of ASM instance would mean that database instance on that node will also fail. However with flex ASM in the event of an ASM instance failure the database instance is free to continue using a remaining ASM instance. This could be tested by stopping an ASM instance in one of the nodes

$ srvctl stop asm -node rhel12c1 -force

$ srvctl status asm -detail
ASM is running on rhel12c2
ASM is enabled.

ASM is running only one node 2. But both database instances would be up and running

ora.rac12c1.db
      1        ONLINE  ONLINE       rhel12c1                 Open,STABLE
      2        ONLINE  ONLINE       rhel12c2                 Open,STABLE

There's no restriction to making application connections to the db instance 1

SQL> select instance_name from v$instance;

INSTANCE_NAME
----------------
rac12c11

Connect to ASM instance on node 2 and check the asm clients which will list among others the db instance 1

SQL> select db_name,CLUSTER_NAME,STATUS,INSTANCE_NAME from v$asm_client;

DB_NAME  CLUSTER_NAME                    STATUS       INSTANCE_NAME
-------- ------------------------------- ------------ -----------------
_mgmtdb  rhel12c                         CONNECTED    -MGMTDB
+ASM     rhel12c                         CONNECTED    +ASM2
+ASM     rhel12c                         CONNECTED    +ASM2
rac12c1  rhel12c                         CONNECTED    rac12c11
rac12c1  rhel12c                         CONNECTED    rac12c11
rac12c1  rhel12c                         CONNECTED    rac12c12
rac12c1  rhel12c                         CONNECTED    rac12c12

Bringing back the stopped asm instance will not automatically relocate the db instance to it.

srvctl start asm -node rhel12c1
SQL>select db_name,CLUSTER_NAME,STATUS,INSTANCE_NAME from v$asm_client;
no rows selected

This has to be done manually with relocate command. Run this on node 2 by specifying instance_name:db_name

SQL> alter system relocate client 'rac12c11:rac12c1';
System altered.

The ASM alert log will capture the following during relocation

NOTE: Flex client id 0x10002 [rac12c11:rac12c1:rhel12c] attempting to reconnect
NOTE: registered owner id 0x10002 for rac12c11:rac12c1:rhel12c (reconnecting)
NOTE: Flex client rac12c11:rac12c1:rhel12c re-registered, osid 14767, mbr 0x0, asmb 6115 (reg:1427811453)
NOTE: client [rac12c11:rac12c1:rhel12c] declared 0 additional pending writes
Fri Aug 14 10:43:07 2015
NOTE: cleaning stale remote ownerid 0x10002 for client rac12c11:rac12c1:rhel12c (reg:1447305939)

Useful metalink notes
FAQ: Oracle Flex ASM 12c / 12.1 [ID 1573137.1]
Steps to Convert STANDARD ASM to FLEX ASM in 12C [ID 1915814.1]

Installing 12c (12.1.0.2) Flex Cluster on RHEL 6 with Role Separation

Oracle 12c introduced a new type of cluster configuration called Flex Cluster. This post highlights the steps of installing a Flex cluster. It's not a complete "how to guide". Most of the steps are similar to 12 standard cluster installation. But for flex cluster, GNS setup is a must.
Also it seems that the ocr and vote diskgroup size requirements are also different when installing a flex cluster. 12.1.0.2 made it mandatory to have the cluster management DB as such the space requirement for diskgroup containing OCR and vote disks is higher compared to 12.1.0.1

The new space requirement is 5528MB (for external redundancy diskgroup) but earlier for upgrading from 12.1.0.1 to 12.1.0.2 it was 4424MB (for external redundancy diskgroup).
If the disk redundancy is normal then the size of the disks must be same, dissimilar size disks are not allowed (this restriction doesn't apply to external redundancy diskgroups).

Other per-installation requirements are similar to previous version of 12.1.0.1.
Start the installation specify it's a flex cluster configuration.

Specify the GNS configuration

Specify the node list and also the role of the node. In this two node cluster setup both nodes are set as hub nodes.

With flex cluster an additional network could be defined for Flex ASM or as in this case allow the private interconnect network to be used for for flex ASM as well.

The installer only prompt to run orainstRoot.sh only on one of the nodes. However root.sh is to be run on all the nodes.

Output from the root.sh

[root@rhel12c1 app]# /opt/app/12.1.0/grid/root.sh
Performing root user operation.

The following environment variables are set as:
    ORACLE_OWNER= grid
    ORACLE_HOME=  /opt/app/12.1.0/grid

Enter the full pathname of the local bin directory: [/usr/local/bin]:
The contents of "dbhome" have not changed. No need to overwrite.
The contents of "oraenv" have not changed. No need to overwrite.
The contents of "coraenv" have not changed. No need to overwrite.


Creating /etc/oratab file...
Entries will be added to the /etc/oratab file as needed by
Database Configuration Assistant when a database is created
Finished running generic part of root script.
Now product-specific root actions will be performed.
Using configuration parameter file: /opt/app/12.1.0/grid/crs/install/crsconfig_params
2015/08/14 13:03:13 CLSRSC-4001: Installing Oracle Trace File Analyzer (TFA) Collector.

2015/08/14 13:03:47 CLSRSC-4002: Successfully installed Oracle Trace File Analyzer (TFA) Collector.

2015/08/14 13:03:50 CLSRSC-363: User ignored prerequisites during installation

OLR initialization - successful
  root wallet
  root wallet cert
  root cert export
  peer wallet
  profile reader wallet
  pa wallet
  peer wallet keys
  pa wallet keys
  peer cert request
  pa cert request
  peer cert
  pa cert
  peer root cert TP
  profile reader root cert TP
  pa root cert TP
  peer pa cert TP
  pa peer cert TP
  profile reader pa cert TP
  profile reader peer cert TP
  peer user cert
  pa user cert
2015/08/14 13:05:01 CLSRSC-330: Adding Clusterware entries to file 'oracle-ohasd.conf'

CRS-4133: Oracle High Availability Services has been stopped.
CRS-4123: Oracle High Availability Services has been started.
CRS-4133: Oracle High Availability Services has been stopped.
CRS-4123: Oracle High Availability Services has been started.
CRS-2672: Attempting to start 'ora.evmd' on 'rhel12c1'
CRS-2672: Attempting to start 'ora.mdnsd' on 'rhel12c1'
CRS-2676: Start of 'ora.mdnsd' on 'rhel12c1' succeeded
CRS-2676: Start of 'ora.evmd' on 'rhel12c1' succeeded
CRS-2672: Attempting to start 'ora.gpnpd' on 'rhel12c1'
CRS-2676: Start of 'ora.gpnpd' on 'rhel12c1' succeeded
CRS-2672: Attempting to start 'ora.cssdmonitor' on 'rhel12c1'
CRS-2672: Attempting to start 'ora.gipcd' on 'rhel12c1'
CRS-2676: Start of 'ora.cssdmonitor' on 'rhel12c1' succeeded
CRS-2676: Start of 'ora.gipcd' on 'rhel12c1' succeeded
CRS-2672: Attempting to start 'ora.cssd' on 'rhel12c1'
CRS-2672: Attempting to start 'ora.diskmon' on 'rhel12c1'
CRS-2676: Start of 'ora.diskmon' on 'rhel12c1' succeeded
CRS-2676: Start of 'ora.cssd' on 'rhel12c1' succeeded

ASM created and started successfully.

Disk Group clusterdg created successfully.

CRS-2672: Attempting to start 'ora.crf' on 'rhel12c1'
CRS-2672: Attempting to start 'ora.storage' on 'rhel12c1'
CRS-2676: Start of 'ora.storage' on 'rhel12c1' succeeded
CRS-2676: Start of 'ora.crf' on 'rhel12c1' succeeded
CRS-2672: Attempting to start 'ora.crsd' on 'rhel12c1'
CRS-2676: Start of 'ora.crsd' on 'rhel12c1' succeeded
CRS-4256: Updating the profile
Successful addition of voting disk 27c22c794c2a4f7fbfca7979664be132.
Successfully replaced voting disk group with +clusterdg.
CRS-4256: Updating the profile
CRS-4266: Voting file(s) successfully replaced
##  STATE    File Universal Id                File Name Disk group
--  -----    -----------------                --------- ---------
 1. ONLINE   27c22c794c2a4f7fbfca7979664be132 (/dev/sde1) [CLUSTERDG]
Located 1 voting disk(s).
CRS-2791: Starting shutdown of Oracle High Availability Services-managed resources on 'rhel12c1'
CRS-2673: Attempting to stop 'ora.crsd' on 'rhel12c1'
CRS-2677: Stop of 'ora.crsd' on 'rhel12c1' succeeded
CRS-2673: Attempting to stop 'ora.storage' on 'rhel12c1'
CRS-2673: Attempting to stop 'ora.mdnsd' on 'rhel12c1'
CRS-2673: Attempting to stop 'ora.gpnpd' on 'rhel12c1'
CRS-2673: Attempting to stop 'ora.drivers.acfs' on 'rhel12c1'
CRS-2677: Stop of 'ora.storage' on 'rhel12c1' succeeded
CRS-2677: Stop of 'ora.drivers.acfs' on 'rhel12c1' succeeded
CRS-2673: Attempting to stop 'ora.crf' on 'rhel12c1'
CRS-2673: Attempting to stop 'ora.ctssd' on 'rhel12c1'
CRS-2673: Attempting to stop 'ora.evmd' on 'rhel12c1'
CRS-2673: Attempting to stop 'ora.asm' on 'rhel12c1'
CRS-2677: Stop of 'ora.gpnpd' on 'rhel12c1' succeeded
CRS-2677: Stop of 'ora.mdnsd' on 'rhel12c1' succeeded
CRS-2677: Stop of 'ora.ctssd' on 'rhel12c1' succeeded
CRS-2677: Stop of 'ora.crf' on 'rhel12c1' succeeded
CRS-2677: Stop of 'ora.evmd' on 'rhel12c1' succeeded
CRS-2677: Stop of 'ora.asm' on 'rhel12c1' succeeded
CRS-2673: Attempting to stop 'ora.cluster_interconnect.haip' on 'rhel12c1'
CRS-2677: Stop of 'ora.cluster_interconnect.haip' on 'rhel12c1' succeeded
CRS-2673: Attempting to stop 'ora.cssd' on 'rhel12c1'
CRS-2677: Stop of 'ora.cssd' on 'rhel12c1' succeeded
CRS-2673: Attempting to stop 'ora.gipcd' on 'rhel12c1'
CRS-2677: Stop of 'ora.gipcd' on 'rhel12c1' succeeded
CRS-2793: Shutdown of Oracle High Availability Services-managed resources on 'rhel12c1' has completed
CRS-4133: Oracle High Availability Services has been stopped.
CRS-4123: Starting Oracle High Availability Services-managed resources
CRS-2672: Attempting to start 'ora.mdnsd' on 'rhel12c1'
CRS-2672: Attempting to start 'ora.evmd' on 'rhel12c1'
CRS-2676: Start of 'ora.evmd' on 'rhel12c1' succeeded
CRS-2676: Start of 'ora.mdnsd' on 'rhel12c1' succeeded
CRS-2672: Attempting to start 'ora.gpnpd' on 'rhel12c1'
CRS-2676: Start of 'ora.gpnpd' on 'rhel12c1' succeeded
CRS-2672: Attempting to start 'ora.gipcd' on 'rhel12c1'
CRS-2676: Start of 'ora.gipcd' on 'rhel12c1' succeeded
CRS-2672: Attempting to start 'ora.cssdmonitor' on 'rhel12c1'
CRS-2676: Start of 'ora.cssdmonitor' on 'rhel12c1' succeeded
CRS-2672: Attempting to start 'ora.cssd' on 'rhel12c1'
CRS-2672: Attempting to start 'ora.diskmon' on 'rhel12c1'
CRS-2676: Start of 'ora.diskmon' on 'rhel12c1' succeeded
CRS-2676: Start of 'ora.cssd' on 'rhel12c1' succeeded
CRS-2672: Attempting to start 'ora.cluster_interconnect.haip' on 'rhel12c1'
CRS-2672: Attempting to start 'ora.ctssd' on 'rhel12c1'
CRS-2676: Start of 'ora.ctssd' on 'rhel12c1' succeeded
CRS-2676: Start of 'ora.cluster_interconnect.haip' on 'rhel12c1' succeeded
CRS-2672: Attempting to start 'ora.asm' on 'rhel12c1'
CRS-2676: Start of 'ora.asm' on 'rhel12c1' succeeded
CRS-2672: Attempting to start 'ora.storage' on 'rhel12c1'
CRS-2676: Start of 'ora.storage' on 'rhel12c1' succeeded
CRS-2672: Attempting to start 'ora.crf' on 'rhel12c1'
CRS-2676: Start of 'ora.crf' on 'rhel12c1' succeeded
CRS-2672: Attempting to start 'ora.crsd' on 'rhel12c1'
CRS-2676: Start of 'ora.crsd' on 'rhel12c1' succeeded
CRS-6023: Starting Oracle Cluster Ready Services-managed resources
CRS-6017: Processing resource auto-start for servers: rhel12c1
CRS-6016: Resource auto-start has completed for server rhel12c1
CRS-6024: Completed start of Oracle Cluster Ready Services-managed resources
CRS-4123: Oracle High Availability Services has been started.
2015/08/14 13:11:52 CLSRSC-343: Successfully started Oracle Clusterware stack

CRS-2672: Attempting to start 'ora.net1.network' on 'rhel12c1'
CRS-2676: Start of 'ora.net1.network' on 'rhel12c1' succeeded
CRS-2672: Attempting to start 'ora.gns.vip' on 'rhel12c1'
CRS-2676: Start of 'ora.gns.vip' on 'rhel12c1' succeeded
CRS-2672: Attempting to start 'ora.gns' on 'rhel12c1'
CRS-2676: Start of 'ora.gns' on 'rhel12c1' succeeded
CRS-2672: Attempting to start 'ora.ASMNET1LSNR_ASM.lsnr' on 'rhel12c1'
CRS-2676: Start of 'ora.ASMNET1LSNR_ASM.lsnr' on 'rhel12c1' succeeded
CRS-2672: Attempting to start 'ora.asm' on 'rhel12c1'
CRS-2676: Start of 'ora.asm' on 'rhel12c1' succeeded
CRS-2672: Attempting to start 'ora.CLUSTERDG.dg' on 'rhel12c1'
CRS-2676: Start of 'ora.CLUSTERDG.dg' on 'rhel12c1' succeeded
CRS-2791: Starting shutdown of Oracle High Availability Services-managed resources on 'rhel12c1'
CRS-2673: Attempting to stop 'ora.crsd' on 'rhel12c1'
CRS-2790: Starting shutdown of Cluster Ready Services-managed resources on 'rhel12c1'
CRS-2673: Attempting to stop 'ora.rhel12c1.vip' on 'rhel12c1'
CRS-2673: Attempting to stop 'ora.CLUSTERDG.dg' on 'rhel12c1'
CRS-2673: Attempting to stop 'ora.cvu' on 'rhel12c1'
CRS-2673: Attempting to stop 'ora.oc4j' on 'rhel12c1'
CRS-2673: Attempting to stop 'ora.LISTENER_SCAN3.lsnr' on 'rhel12c1'
CRS-2673: Attempting to stop 'ora.LISTENER_SCAN2.lsnr' on 'rhel12c1'
CRS-2673: Attempting to stop 'ora.LISTENER_SCAN1.lsnr' on 'rhel12c1'
CRS-2673: Attempting to stop 'ora.gns' on 'rhel12c1'
CRS-2677: Stop of 'ora.cvu' on 'rhel12c1' succeeded
CRS-2677: Stop of 'ora.CLUSTERDG.dg' on 'rhel12c1' succeeded
CRS-2673: Attempting to stop 'ora.asm' on 'rhel12c1'
CRS-2677: Stop of 'ora.asm' on 'rhel12c1' succeeded
CRS-2673: Attempting to stop 'ora.ASMNET1LSNR_ASM.lsnr' on 'rhel12c1'
CRS-2677: Stop of 'ora.LISTENER_SCAN3.lsnr' on 'rhel12c1' succeeded
CRS-2673: Attempting to stop 'ora.scan3.vip' on 'rhel12c1'
CRS-2677: Stop of 'ora.LISTENER_SCAN2.lsnr' on 'rhel12c1' succeeded
CRS-2673: Attempting to stop 'ora.scan2.vip' on 'rhel12c1'
CRS-2677: Stop of 'ora.LISTENER_SCAN1.lsnr' on 'rhel12c1' succeeded
CRS-2673: Attempting to stop 'ora.scan1.vip' on 'rhel12c1'
CRS-2677: Stop of 'ora.ASMNET1LSNR_ASM.lsnr' on 'rhel12c1' succeeded
CRS-2677: Stop of 'ora.rhel12c1.vip' on 'rhel12c1' succeeded
CRS-2677: Stop of 'ora.scan2.vip' on 'rhel12c1' succeeded
CRS-2677: Stop of 'ora.scan3.vip' on 'rhel12c1' succeeded
CRS-2677: Stop of 'ora.scan1.vip' on 'rhel12c1' succeeded
CRS-2677: Stop of 'ora.gns' on 'rhel12c1' succeeded
CRS-2673: Attempting to stop 'ora.gns.vip' on 'rhel12c1'
CRS-2677: Stop of 'ora.gns.vip' on 'rhel12c1' succeeded
CRS-2677: Stop of 'ora.oc4j' on 'rhel12c1' succeeded
CRS-2673: Attempting to stop 'ora.ons' on 'rhel12c1'
CRS-2677: Stop of 'ora.ons' on 'rhel12c1' succeeded
CRS-2673: Attempting to stop 'ora.net1.network' on 'rhel12c1'
CRS-2677: Stop of 'ora.net1.network' on 'rhel12c1' succeeded
CRS-2792: Shutdown of Cluster Ready Services-managed resources on 'rhel12c1' has completed
CRS-2677: Stop of 'ora.crsd' on 'rhel12c1' succeeded
CRS-2673: Attempting to stop 'ora.evmd' on 'rhel12c1'
CRS-2673: Attempting to stop 'ora.ctssd' on 'rhel12c1'
CRS-2673: Attempting to stop 'ora.storage' on 'rhel12c1'
CRS-2673: Attempting to stop 'ora.crf' on 'rhel12c1'
CRS-2673: Attempting to stop 'ora.drivers.acfs' on 'rhel12c1'
CRS-2673: Attempting to stop 'ora.mdnsd' on 'rhel12c1'
CRS-2673: Attempting to stop 'ora.gpnpd' on 'rhel12c1'
CRS-2677: Stop of 'ora.storage' on 'rhel12c1' succeeded
CRS-2677: Stop of 'ora.drivers.acfs' on 'rhel12c1' succeeded
CRS-2677: Stop of 'ora.evmd' on 'rhel12c1' succeeded
CRS-2677: Stop of 'ora.ctssd' on 'rhel12c1' succeeded
CRS-2673: Attempting to stop 'ora.asm' on 'rhel12c1'
CRS-2677: Stop of 'ora.crf' on 'rhel12c1' succeeded
CRS-2677: Stop of 'ora.gpnpd' on 'rhel12c1' succeeded
CRS-2677: Stop of 'ora.mdnsd' on 'rhel12c1' succeeded
CRS-2677: Stop of 'ora.asm' on 'rhel12c1' succeeded
CRS-2673: Attempting to stop 'ora.cluster_interconnect.haip' on 'rhel12c1'
CRS-2677: Stop of 'ora.cluster_interconnect.haip' on 'rhel12c1' succeeded
CRS-2673: Attempting to stop 'ora.cssd' on 'rhel12c1'
CRS-2677: Stop of 'ora.cssd' on 'rhel12c1' succeeded
CRS-2673: Attempting to stop 'ora.gipcd' on 'rhel12c1'
CRS-2677: Stop of 'ora.gipcd' on 'rhel12c1' succeeded
CRS-2793: Shutdown of Oracle High Availability Services-managed resources on 'rhel12c1' has completed
CRS-4133: Oracle High Availability Services has been stopped.
CRS-4123: Starting Oracle High Availability Services-managed resources
CRS-2672: Attempting to start 'ora.mdnsd' on 'rhel12c1'
CRS-2672: Attempting to start 'ora.evmd' on 'rhel12c1'
CRS-2676: Start of 'ora.mdnsd' on 'rhel12c1' succeeded
CRS-2676: Start of 'ora.evmd' on 'rhel12c1' succeeded
CRS-2672: Attempting to start 'ora.gpnpd' on 'rhel12c1'
CRS-2676: Start of 'ora.gpnpd' on 'rhel12c1' succeeded
CRS-2672: Attempting to start 'ora.gipcd' on 'rhel12c1'
CRS-2676: Start of 'ora.gipcd' on 'rhel12c1' succeeded
CRS-2672: Attempting to start 'ora.cssdmonitor' on 'rhel12c1'
CRS-2676: Start of 'ora.cssdmonitor' on 'rhel12c1' succeeded
CRS-2672: Attempting to start 'ora.cssd' on 'rhel12c1'
CRS-2672: Attempting to start 'ora.diskmon' on 'rhel12c1'
CRS-2676: Start of 'ora.diskmon' on 'rhel12c1' succeeded
CRS-2676: Start of 'ora.cssd' on 'rhel12c1' succeeded
CRS-2672: Attempting to start 'ora.cluster_interconnect.haip' on 'rhel12c1'
CRS-2672: Attempting to start 'ora.ctssd' on 'rhel12c1'
CRS-2676: Start of 'ora.ctssd' on 'rhel12c1' succeeded
CRS-2676: Start of 'ora.cluster_interconnect.haip' on 'rhel12c1' succeeded
CRS-2672: Attempting to start 'ora.asm' on 'rhel12c1'
CRS-2676: Start of 'ora.asm' on 'rhel12c1' succeeded
CRS-2672: Attempting to start 'ora.storage' on 'rhel12c1'
CRS-2676: Start of 'ora.storage' on 'rhel12c1' succeeded
CRS-2672: Attempting to start 'ora.crf' on 'rhel12c1'
CRS-2676: Start of 'ora.crf' on 'rhel12c1' succeeded
CRS-2672: Attempting to start 'ora.crsd' on 'rhel12c1'
CRS-2676: Start of 'ora.crsd' on 'rhel12c1' succeeded
CRS-6023: Starting Oracle Cluster Ready Services-managed resources
CRS-2664: Resource 'ora.CLUSTERDG.dg' is already running on 'rhel12c1'
CRS-6017: Processing resource auto-start for servers: rhel12c1
CRS-2672: Attempting to start 'ora.oc4j' on 'rhel12c1'
CRS-2672: Attempting to start 'ora.LISTENER_SCAN1.lsnr' on 'rhel12c1'
CRS-2676: Start of 'ora.LISTENER_SCAN1.lsnr' on 'rhel12c1' succeeded
CRS-2676: Start of 'ora.oc4j' on 'rhel12c1' succeeded
CRS-6016: Resource auto-start has completed for server rhel12c1
CRS-6024: Completed start of Oracle Cluster Ready Services-managed resources
CRS-4123: Oracle High Availability Services has been started.
2015/08/14 13:16:58 CLSRSC-325: Configure Oracle Grid Infrastructure for a Cluster ... succeeded

Root.sh output on node 2

[root@rhel12c2 app]# /opt/app/12.1.0/grid/root.sh
Performing root user operation.

The following environment variables are set as:
    ORACLE_OWNER= grid
    ORACLE_HOME=  /opt/app/12.1.0/grid

Enter the full pathname of the local bin directory: [/usr/local/bin]:
The contents of "dbhome" have not changed. No need to overwrite.
The contents of "oraenv" have not changed. No need to overwrite.
The contents of "coraenv" have not changed. No need to overwrite.


Creating /etc/oratab file...
Entries will be added to the /etc/oratab file as needed by
Database Configuration Assistant when a database is created
Finished running generic part of root script.
Now product-specific root actions will be performed.
Using configuration parameter file: /opt/app/12.1.0/grid/crs/install/crsconfig_params
2015/08/14 13:17:41 CLSRSC-4001: Installing Oracle Trace File Analyzer (TFA) Collector.

2015/08/14 13:18:14 CLSRSC-4002: Successfully installed Oracle Trace File Analyzer (TFA) Collector.

2015/08/14 13:18:16 CLSRSC-363: User ignored prerequisites during installation

OLR initialization - successful
2015/08/14 13:19:47 CLSRSC-330: Adding Clusterware entries to file 'oracle-ohasd.conf'

CRS-4133: Oracle High Availability Services has been stopped.
CRS-4123: Oracle High Availability Services has been started.
CRS-4133: Oracle High Availability Services has been stopped.
CRS-4123: Oracle High Availability Services has been started.
CRS-2791: Starting shutdown of Oracle High Availability Services-managed resources on 'rhel12c2'
CRS-2673: Attempting to stop 'ora.drivers.acfs' on 'rhel12c2'
CRS-2677: Stop of 'ora.drivers.acfs' on 'rhel12c2' succeeded
CRS-2793: Shutdown of Oracle High Availability Services-managed resources on 'rhel12c2' has completed
CRS-4133: Oracle High Availability Services has been stopped.
CRS-4123: Starting Oracle High Availability Services-managed resources
CRS-2672: Attempting to start 'ora.mdnsd' on 'rhel12c2'
CRS-2672: Attempting to start 'ora.evmd' on 'rhel12c2'
CRS-2676: Start of 'ora.mdnsd' on 'rhel12c2' succeeded
CRS-2676: Start of 'ora.evmd' on 'rhel12c2' succeeded
CRS-2672: Attempting to start 'ora.gpnpd' on 'rhel12c2'
CRS-2676: Start of 'ora.gpnpd' on 'rhel12c2' succeeded
CRS-2672: Attempting to start 'ora.gipcd' on 'rhel12c2'
CRS-2676: Start of 'ora.gipcd' on 'rhel12c2' succeeded
CRS-2672: Attempting to start 'ora.cssdmonitor' on 'rhel12c2'
CRS-2676: Start of 'ora.cssdmonitor' on 'rhel12c2' succeeded
CRS-2672: Attempting to start 'ora.cssd' on 'rhel12c2'
CRS-2672: Attempting to start 'ora.diskmon' on 'rhel12c2'
CRS-2676: Start of 'ora.diskmon' on 'rhel12c2' succeeded
CRS-2676: Start of 'ora.cssd' on 'rhel12c2' succeeded
CRS-2672: Attempting to start 'ora.cluster_interconnect.haip' on 'rhel12c2'
CRS-2672: Attempting to start 'ora.ctssd' on 'rhel12c2'
CRS-2676: Start of 'ora.ctssd' on 'rhel12c2' succeeded
CRS-2676: Start of 'ora.cluster_interconnect.haip' on 'rhel12c2' succeeded
CRS-2672: Attempting to start 'ora.asm' on 'rhel12c2'
CRS-2676: Start of 'ora.asm' on 'rhel12c2' succeeded
CRS-2672: Attempting to start 'ora.storage' on 'rhel12c2'
CRS-2676: Start of 'ora.storage' on 'rhel12c2' succeeded
CRS-2672: Attempting to start 'ora.crf' on 'rhel12c2'
CRS-2676: Start of 'ora.crf' on 'rhel12c2' succeeded
CRS-2672: Attempting to start 'ora.crsd' on 'rhel12c2'
CRS-2676: Start of 'ora.crsd' on 'rhel12c2' succeeded
CRS-6017: Processing resource auto-start for servers: rhel12c2
CRS-2672: Attempting to start 'ora.net1.network' on 'rhel12c2'
CRS-2672: Attempting to start 'ora.ASMNET1LSNR_ASM.lsnr' on 'rhel12c2'
CRS-2676: Start of 'ora.net1.network' on 'rhel12c2' succeeded
CRS-2672: Attempting to start 'ora.ons' on 'rhel12c2'
CRS-2676: Start of 'ora.ons' on 'rhel12c2' succeeded
CRS-2673: Attempting to stop 'ora.LISTENER_SCAN1.lsnr' on 'rhel12c1'
CRS-2676: Start of 'ora.ASMNET1LSNR_ASM.lsnr' on 'rhel12c2' succeeded
CRS-2672: Attempting to start 'ora.asm' on 'rhel12c2'
CRS-2677: Stop of 'ora.LISTENER_SCAN1.lsnr' on 'rhel12c1' succeeded
CRS-2673: Attempting to stop 'ora.scan1.vip' on 'rhel12c1'
CRS-2677: Stop of 'ora.scan1.vip' on 'rhel12c1' succeeded
CRS-2672: Attempting to start 'ora.scan1.vip' on 'rhel12c2'
CRS-2676: Start of 'ora.scan1.vip' on 'rhel12c2' succeeded
CRS-2672: Attempting to start 'ora.LISTENER_SCAN1.lsnr' on 'rhel12c2'
CRS-2676: Start of 'ora.LISTENER_SCAN1.lsnr' on 'rhel12c2' succeeded
CRS-2676: Start of 'ora.asm' on 'rhel12c2' succeeded
CRS-6016: Resource auto-start has completed for server rhel12c2
CRS-6024: Completed start of Oracle Cluster Ready Services-managed resources
CRS-4123: Oracle High Availability Services has been started.
2015/08/14 13:24:22 CLSRSC-343: Successfully started Oracle Clusterware stack

2015/08/14 13:24:42 CLSRSC-325: Configure Oracle Grid Infrastructure for a Cluster ... succeeded

Once the root.sh completes the confiugration assistants are run and cluster installation complete. The type of the cluster and nodes could be checked as follows

[root@rhel12c1 oracle]# crsctl get cluster mode status
Cluster is running in "flex" mode

[root@rhel12c1 oracle]# crsctl get node role config
Node 'rhel12c1' configured role is 'hub'

[grid@rhel12c2 ~]$ crsctl get node role config
Node 'rhel12c2' configured role is 'hub'

ASM will have the cluster ASM listener and default ASM instance count of 3

srvctl config asm -detail
ASM home: 
Password file: +clusterdg/orapwASM
ASM listener: LISTENER
ASM is enabled.
ASM is individually enabled on nodes:
ASM is individually disabled on nodes:
ASM instance count: 3
Cluster ASM listener: ASMNET1LSNR_ASM

But on this two node cluster only two ASM instances would be active

ora.asm
      1        ONLINE  ONLINE       rhel12c1                 Started,STABLE
      2        ONLINE  ONLINE       rhel12c2                 Started,STABLE
      3        OFFLINE OFFLINE                               STABLE

Also check the ASM cluster mode

ASMCMD> showclustermode
ASM cluster : Flex mode enabled

Once the cluster is setup install the database software. There's no additional steps to this due to flex cluster, installation is same as 12.1.0.1. Once the database software is installed create the database either CDB or non-CDB.
With version 12.1.0.2 the cluster log files were moved inside diagnostic dest as such the ownership of diag folder is grid:oinstall with permission 755. Due to this fact creation of database will fail as Oracle user doesn't have the write permission inside the diag directory. This is a change from previous 12.1.0.1 version. Before running the DBCA change the permission of following directories so oracle user has write permission

chmod 770 cfgtoollogs
chmod 770 admin
chmod 770 cfgtoollogs/dbca
chmod 770 audit
chmod 770 diag

There are no other additional steps to create a database in flex cluster compared a standard cluster.

Related Posts
Installing 12c (12.1.0.1) RAC on RHEL 6 with Role Separation - Clusterware
Installing 11gR2 (11.2.0.3) GI with Role Separation on RHEL 6
Installing 11gR2 (11.2.0.3) GI with Role Separation on OEL 6
Installing 11gR2 Standalone Server with ASM and Role Separation on RHEL 6
11gR2 Standalone Data Guard (with ASM and Role Separation)