Tuesday, September 1, 2015

Converting 12c Standard Cluster to Flex Cluster

An earlier post showed steps for installing a 12c flex cluster. This post shows the steps for converting a standard cluster to a flex cluster. There are two phases to converting a standard cluster to flex cluster. First phase involves converting Oracle ASM to Flex ASM and second phase is to change the cluster mode to flex.
The current setup is a two node standard cluster.
$  crsctl get cluster mode status
Cluster is running in "standard" mode

$ asmcmd
ASMCMD> showclustermode
ASM cluster : Flex mode disabled
There are few per-requisites to converting ASM to flex ASM, these include that OCR, SPFile and password file all stored in a disk group and this group have ASM compatibility (COMPATIBLE.ASM) set to 12.1 or higher. If this is not the case then make the necessary changes to move these files to disk group. As the current setup is a newly installed 12c cluster (as oppose to upgraded to 12c) these files all reside in disk group.
A point to note is that the diskgroup with above mentioned files has the size of 5114MB (below output has been modified by omitting some columns).
ASMCMD> lsdg
State    Type    Rebal  Sector  Block       AU  Total_MB  Free_MB  Voting_files  Name
MOUNTED  EXTERN  N         512   4096  1048576     10236     7609             N  DATA/
MOUNTED  EXTERN  N         512   4096  1048576     10236     8609             N  FRA/
MOUNTED  EXTERN  N         512   4096  1048576      5114      475             Y  NEWCLUSTERDG/
This is smaller than the required diskgroup size for new flex cluster being installed. Despite the smaller size diskgroup the conversion to flex cluster succeeded without any issue. The current ASM configuration is
$  srvctl config asm
ASM home: 
Password file: +NEWCLUSTERDG/orapwasm
ASM listener: LISTENER
Another key requirement is that flex cluster requires GNS. If a GNS doesn't exists setup GNS and add the GNS VIP to cluster. The standard cluster was already setup with GNS.
$  srvctl config gns
GNS is enabled.
GNS VIP addresses: 192.168.0.87
Domain served by GNS: rac.domain.net
To start the first phase of converting to flex ASM run asmca and click on ASM instances tab. If role separation is used then make sure grid user has write permission for directory $ORACLE_BASE/cfgtoollogs as the conversion script is generated inside it.
Click convert to Flex ASM button. This would prompt to select a network interface for ASM communication and also to specify a port. The default port is 1521.
However there is a problem in using the port 1521.
The port 1521 is used on the private network (192.168.1.0 in this case) by the management listener.
$ netstat -ntl | grep 1521
tcp        0      0 192.168.0.89:1521           0.0.0.0:*                   LISTEN      3323/tnslsnr
tcp        0      0 192.168.0.96:1521           0.0.0.0:*                   LISTEN      3323/tnslsnr
tcp        0      0 192.168.0.97:1521           0.0.0.0:*                   LISTEN      3323/tnslsnr
tcp        0      0 192.168.0.93:1521           0.0.0.0:*                   LISTEN      3323/tnslsnr
tcp        0      0 192.168.0.95:1521           0.0.0.0:*                   LISTEN      3323/tnslsnr
tcp        0      0 169.254.11.195:1521         0.0.0.0:*                   LISTEN      3323/tnslsnr
tcp        0      0 192.168.1.87:1521           0.0.0.0:*                   LISTEN      3323/tnslsnr

# ps ax | grep 3323
 3323 ?        Ssl    0:00 /opt/app/12.1.0/grid2/bin/tnslsnr MGMTLSNR -no_crs_notify -inherit
Even after stopping the MGMTLSNR
$ srvctl stop listener -l MGMTLSNR
PRCR-1065 : Failed to stop resource ora.MGMTLSNR
CRS-2529: Unable to act on 'ora.MGMTLSNR' because that would require stopping or relocating 'ora.mgmtdb', but the force option was not specified
$ srvctl stop listener -l MGMTLSNR -f

ora.MGMTLSNR
      1        OFFLINE OFFLINE                               169.254.11.195 192.1
                                                             68.1.87,STABLE
and all other listeners (scan_listener and listener) that were using port 1521 (even though they were on a different network) the error did not get resolved. It could be port usage is looked up statically from a config file. There's no problem to use 1521 for both database listener and ASM listener when a new flex cluster is installed. Finally had to proceed using a different port for cluster ASM listener to continue forward.
Conversion to flex ASM will continue restarting one node at time.
When prompted execute the script on the same node ASMCA was run.
Output from converttoFlexASM.sh execution
# /opt/app/oracle/cfgtoollogs/asmca/scripts/converttoFlexASM.sh
CRS-2673: Attempting to stop 'ora.crsd' on 'rhel12c1'
CRS-2790: Starting shutdown of Cluster Ready Services-managed resources on 'rhel12c1'
CRS-2673: Attempting to stop 'ora.mgmtdb' on 'rhel12c1'
CRS-2673: Attempting to stop 'ora.gns' on 'rhel12c1'
CRS-2673: Attempting to stop 'ora.LISTENER_SCAN2.lsnr' on 'rhel12c1'
CRS-2673: Attempting to stop 'ora.NEWCLUSTERDG.dg' on 'rhel12c1'
CRS-2673: Attempting to stop 'ora.LISTENER.lsnr' on 'rhel12c1'
CRS-2673: Attempting to stop 'ora.rac12c1.db' on 'rhel12c1'
CRS-2673: Attempting to stop 'ora.LISTENER_SCAN3.lsnr' on 'rhel12c1'
CRS-2673: Attempting to stop 'ora.DATA.dg' on 'rhel12c1'
CRS-2673: Attempting to stop 'ora.oc4j' on 'rhel12c1'
CRS-2673: Attempting to stop 'ora.cvu' on 'rhel12c1'
CRS-2673: Attempting to stop 'ora.FRA.dg' on 'rhel12c1'
CRS-2677: Stop of 'ora.cvu' on 'rhel12c1' succeeded
CRS-2672: Attempting to start 'ora.cvu' on 'rhel12c2'
CRS-2677: Stop of 'ora.LISTENER_SCAN2.lsnr' on 'rhel12c1' succeeded
CRS-2673: Attempting to stop 'ora.scan2.vip' on 'rhel12c1'
CRS-2677: Stop of 'ora.LISTENER.lsnr' on 'rhel12c1' succeeded
CRS-2677: Stop of 'ora.DATA.dg' on 'rhel12c1' succeeded
CRS-2677: Stop of 'ora.NEWCLUSTERDG.dg' on 'rhel12c1' succeeded
CRS-2677: Stop of 'ora.FRA.dg' on 'rhel12c1' succeeded
CRS-2677: Stop of 'ora.gns' on 'rhel12c1' succeeded
CRS-2673: Attempting to stop 'ora.gns.vip' on 'rhel12c1'
CRS-2677: Stop of 'ora.LISTENER_SCAN3.lsnr' on 'rhel12c1' succeeded
CRS-2673: Attempting to stop 'ora.scan3.vip' on 'rhel12c1'
CRS-2673: Attempting to stop 'ora.rhel12c1.vip' on 'rhel12c1'
CRS-2676: Start of 'ora.cvu' on 'rhel12c2' succeeded
CRS-2677: Stop of 'ora.scan2.vip' on 'rhel12c1' succeeded
CRS-2672: Attempting to start 'ora.scan2.vip' on 'rhel12c2'
CRS-2677: Stop of 'ora.rac12c1.db' on 'rhel12c1' succeeded
CRS-2677: Stop of 'ora.mgmtdb' on 'rhel12c1' succeeded
CRS-2673: Attempting to stop 'ora.MGMTLSNR' on 'rhel12c1'
CRS-2677: Stop of 'ora.MGMTLSNR' on 'rhel12c1' succeeded
CRS-2672: Attempting to start 'ora.MGMTLSNR' on 'rhel12c2'
CRS-2677: Stop of 'ora.gns.vip' on 'rhel12c1' succeeded
CRS-2672: Attempting to start 'ora.gns.vip' on 'rhel12c2'
CRS-2677: Stop of 'ora.scan3.vip' on 'rhel12c1' succeeded
CRS-2672: Attempting to start 'ora.scan3.vip' on 'rhel12c2'
CRS-2677: Stop of 'ora.rhel12c1.vip' on 'rhel12c1' succeeded
CRS-2672: Attempting to start 'ora.rhel12c1.vip' on 'rhel12c2'
CRS-2676: Start of 'ora.scan2.vip' on 'rhel12c2' succeeded
CRS-2672: Attempting to start 'ora.LISTENER_SCAN2.lsnr' on 'rhel12c2'
CRS-2676: Start of 'ora.MGMTLSNR' on 'rhel12c2' succeeded
CRS-2672: Attempting to start 'ora.mgmtdb' on 'rhel12c2'
CRS-2676: Start of 'ora.gns.vip' on 'rhel12c2' succeeded
CRS-2672: Attempting to start 'ora.gns' on 'rhel12c2'
CRS-2677: Stop of 'ora.oc4j' on 'rhel12c1' succeeded
CRS-2672: Attempting to start 'ora.oc4j' on 'rhel12c2'
CRS-2676: Start of 'ora.scan3.vip' on 'rhel12c2' succeeded
CRS-2672: Attempting to start 'ora.LISTENER_SCAN3.lsnr' on 'rhel12c2'
CRS-2676: Start of 'ora.rhel12c1.vip' on 'rhel12c2' succeeded
CRS-2676: Start of 'ora.gns' on 'rhel12c2' succeeded
CRS-2676: Start of 'ora.LISTENER_SCAN2.lsnr' on 'rhel12c2' succeeded
CRS-2676: Start of 'ora.LISTENER_SCAN3.lsnr' on 'rhel12c2' succeeded
CRS-2676: Start of 'ora.oc4j' on 'rhel12c2' succeeded
CRS-2676: Start of 'ora.mgmtdb' on 'rhel12c2' succeeded
CRS-2673: Attempting to stop 'ora.ons' on 'rhel12c1'
CRS-2677: Stop of 'ora.ons' on 'rhel12c1' succeeded
CRS-2673: Attempting to stop 'ora.net1.network' on 'rhel12c1'
CRS-2677: Stop of 'ora.net1.network' on 'rhel12c1' succeeded
CRS-2792: Shutdown of Cluster Ready Services-managed resources on 'rhel12c1' has completed
CRS-2677: Stop of 'ora.crsd' on 'rhel12c1' succeeded
CRS-2673: Attempting to stop 'ora.ctssd' on 'rhel12c1'
CRS-2673: Attempting to stop 'ora.evmd' on 'rhel12c1'
CRS-2673: Attempting to stop 'ora.storage' on 'rhel12c1'
CRS-2677: Stop of 'ora.storage' on 'rhel12c1' succeeded
CRS-2673: Attempting to stop 'ora.asm' on 'rhel12c1'
CRS-2677: Stop of 'ora.ctssd' on 'rhel12c1' succeeded
CRS-2677: Stop of 'ora.evmd' on 'rhel12c1' succeeded
CRS-2677: Stop of 'ora.asm' on 'rhel12c1' succeeded
CRS-2673: Attempting to stop 'ora.cluster_interconnect.haip' on 'rhel12c1'
CRS-2677: Stop of 'ora.cluster_interconnect.haip' on 'rhel12c1' succeeded
CRS-2673: Attempting to stop 'ora.cssd' on 'rhel12c1'
CRS-2677: Stop of 'ora.cssd' on 'rhel12c1' succeeded
CRS-2672: Attempting to start 'ora.evmd' on 'rhel12c1'
CRS-2672: Attempting to start 'ora.cssdmonitor' on 'rhel12c1'
CRS-2676: Start of 'ora.cssdmonitor' on 'rhel12c1' succeeded
CRS-2672: Attempting to start 'ora.cssd' on 'rhel12c1'
CRS-2672: Attempting to start 'ora.diskmon' on 'rhel12c1'
CRS-2676: Start of 'ora.diskmon' on 'rhel12c1' succeeded
CRS-2676: Start of 'ora.evmd' on 'rhel12c1' succeeded
CRS-2676: Start of 'ora.cssd' on 'rhel12c1' succeeded
CRS-2672: Attempting to start 'ora.ctssd' on 'rhel12c1'
CRS-2672: Attempting to start 'ora.cluster_interconnect.haip' on 'rhel12c1'
CRS-2676: Start of 'ora.ctssd' on 'rhel12c1' succeeded
CRS-2676: Start of 'ora.cluster_interconnect.haip' on 'rhel12c1' succeeded
CRS-2672: Attempting to start 'ora.asm' on 'rhel12c1'
CRS-2676: Start of 'ora.asm' on 'rhel12c1' succeeded
CRS-2672: Attempting to start 'ora.storage' on 'rhel12c1'
CRS-2676: Start of 'ora.storage' on 'rhel12c1' succeeded
CRS-2672: Attempting to start 'ora.crsd' on 'rhel12c1'
CRS-2676: Start of 'ora.crsd' on 'rhel12c1' succeeded
Oracle Grid Infrastructure restarted in node rhel12c1
PRCC-1014 : ASMNET1LSNR_ASM was already running
PRCR-1004 : Resource ora.ASMNET1LSNR_ASM.lsnr is already running
PRCR-1079 : Failed to start resource ora.ASMNET1LSNR_ASM.lsnr
CRS-5702: Resource 'ora.ASMNET1LSNR_ASM.lsnr' is already running on 'rhel12c2'
CRS-5702: Resource 'ora.ASMNET1LSNR_ASM.lsnr' is already running on 'rhel12c1'
ASM listener ASMNET1LSNR_ASM running already
CRS-2673: Attempting to stop 'ora.crsd' on 'rhel12c2'
CRS-2790: Starting shutdown of Cluster Ready Services-managed resources on 'rhel12c2'
CRS-2673: Attempting to stop 'ora.rac12c1.cbx.svc' on 'rhel12c2'
CRS-2673: Attempting to stop 'ora.LISTENER_SCAN3.lsnr' on 'rhel12c2'
CRS-2673: Attempting to stop 'ora.LISTENER_SCAN2.lsnr' on 'rhel12c2'
CRS-2673: Attempting to stop 'ora.mgmtdb' on 'rhel12c2'
CRS-2673: Attempting to stop 'ora.LISTENER.lsnr' on 'rhel12c2'
CRS-2673: Attempting to stop 'ora.oc4j' on 'rhel12c2'
CRS-2673: Attempting to stop 'ora.cvu' on 'rhel12c2'
CRS-2673: Attempting to stop 'ora.NEWCLUSTERDG.dg' on 'rhel12c2'
CRS-2673: Attempting to stop 'ora.ASMNET1LSNR_ASM.lsnr' on 'rhel12c2'
CRS-2677: Stop of 'ora.cvu' on 'rhel12c2' succeeded
CRS-2672: Attempting to start 'ora.cvu' on 'rhel12c1'
CRS-2677: Stop of 'ora.NEWCLUSTERDG.dg' on 'rhel12c2' succeeded
CRS-2677: Stop of 'ora.LISTENER_SCAN2.lsnr' on 'rhel12c2' succeeded
CRS-2673: Attempting to stop 'ora.scan2.vip' on 'rhel12c2'
CRS-2677: Stop of 'ora.LISTENER_SCAN3.lsnr' on 'rhel12c2' succeeded
CRS-2673: Attempting to stop 'ora.scan3.vip' on 'rhel12c2'
CRS-2677: Stop of 'ora.LISTENER.lsnr' on 'rhel12c2' succeeded
CRS-2677: Stop of 'ora.rac12c1.cbx.svc' on 'rhel12c2' succeeded
CRS-2673: Attempting to stop 'ora.rac12c1.db' on 'rhel12c2'
CRS-2677: Stop of 'ora.ASMNET1LSNR_ASM.lsnr' on 'rhel12c2' succeeded
CRS-2676: Start of 'ora.cvu' on 'rhel12c1' succeeded
CRS-2677: Stop of 'ora.scan2.vip' on 'rhel12c2' succeeded
CRS-2672: Attempting to start 'ora.scan2.vip' on 'rhel12c1'
CRS-2677: Stop of 'ora.scan3.vip' on 'rhel12c2' succeeded
CRS-2672: Attempting to start 'ora.scan3.vip' on 'rhel12c1'
CRS-2677: Stop of 'ora.rac12c1.db' on 'rhel12c2' succeeded
CRS-2677: Stop of 'ora.mgmtdb' on 'rhel12c2' succeeded
CRS-2673: Attempting to stop 'ora.MGMTLSNR' on 'rhel12c2'
CRS-2677: Stop of 'ora.MGMTLSNR' on 'rhel12c2' succeeded
CRS-2672: Attempting to start 'ora.MGMTLSNR' on 'rhel12c1'
CRS-2673: Attempting to stop 'ora.rhel12c2.vip' on 'rhel12c2'
CRS-2676: Start of 'ora.scan2.vip' on 'rhel12c1' succeeded
CRS-2672: Attempting to start 'ora.LISTENER_SCAN2.lsnr' on 'rhel12c1'
CRS-2677: Stop of 'ora.rhel12c2.vip' on 'rhel12c2' succeeded
CRS-2672: Attempting to start 'ora.rhel12c2.vip' on 'rhel12c1'
CRS-2676: Start of 'ora.scan3.vip' on 'rhel12c1' succeeded
CRS-2672: Attempting to start 'ora.LISTENER_SCAN3.lsnr' on 'rhel12c1'
CRS-2676: Start of 'ora.MGMTLSNR' on 'rhel12c1' succeeded
CRS-2672: Attempting to start 'ora.mgmtdb' on 'rhel12c1'
CRS-2677: Stop of 'ora.oc4j' on 'rhel12c2' succeeded
CRS-2672: Attempting to start 'ora.oc4j' on 'rhel12c1'
CRS-2676: Start of 'ora.rhel12c2.vip' on 'rhel12c1' succeeded
CRS-2676: Start of 'ora.LISTENER_SCAN2.lsnr' on 'rhel12c1' succeeded
CRS-2676: Start of 'ora.LISTENER_SCAN3.lsnr' on 'rhel12c1' succeeded
CRS-2673: Attempting to stop 'ora.DATA.dg' on 'rhel12c2'
CRS-2673: Attempting to stop 'ora.FRA.dg' on 'rhel12c2'
CRS-2673: Attempting to stop 'ora.gns' on 'rhel12c2'
CRS-2677: Stop of 'ora.FRA.dg' on 'rhel12c2' succeeded
CRS-2677: Stop of 'ora.DATA.dg' on 'rhel12c2' succeeded
CRS-2677: Stop of 'ora.gns' on 'rhel12c2' succeeded
CRS-2673: Attempting to stop 'ora.gns.vip' on 'rhel12c2'
CRS-2677: Stop of 'ora.gns.vip' on 'rhel12c2' succeeded
CRS-2672: Attempting to start 'ora.gns.vip' on 'rhel12c1'
CRS-2676: Start of 'ora.gns.vip' on 'rhel12c1' succeeded
CRS-2672: Attempting to start 'ora.gns' on 'rhel12c1'
CRS-2676: Start of 'ora.gns' on 'rhel12c1' succeeded
CRS-2676: Start of 'ora.oc4j' on 'rhel12c1' succeeded
CRS-2676: Start of 'ora.mgmtdb' on 'rhel12c1' succeeded
CRS-2673: Attempting to stop 'ora.ons' on 'rhel12c2'
CRS-2677: Stop of 'ora.ons' on 'rhel12c2' succeeded
CRS-2673: Attempting to stop 'ora.net1.network' on 'rhel12c2'
CRS-2677: Stop of 'ora.net1.network' on 'rhel12c2' succeeded
CRS-2792: Shutdown of Cluster Ready Services-managed resources on 'rhel12c2' has completed
CRS-2677: Stop of 'ora.crsd' on 'rhel12c2' succeeded
CRS-2673: Attempting to stop 'ora.ctssd' on 'rhel12c2'
CRS-2673: Attempting to stop 'ora.evmd' on 'rhel12c2'
CRS-2673: Attempting to stop 'ora.storage' on 'rhel12c2'
CRS-2677: Stop of 'ora.storage' on 'rhel12c2' succeeded
CRS-2673: Attempting to stop 'ora.asm' on 'rhel12c2'
CRS-2677: Stop of 'ora.ctssd' on 'rhel12c2' succeeded
CRS-2677: Stop of 'ora.evmd' on 'rhel12c2' succeeded
CRS-2677: Stop of 'ora.asm' on 'rhel12c2' succeeded
CRS-2673: Attempting to stop 'ora.cluster_interconnect.haip' on 'rhel12c2'
CRS-2677: Stop of 'ora.cluster_interconnect.haip' on 'rhel12c2' succeeded
CRS-2673: Attempting to stop 'ora.cssd' on 'rhel12c2'
CRS-2677: Stop of 'ora.cssd' on 'rhel12c2' succeeded
CRS-2672: Attempting to start 'ora.cssdmonitor' on 'rhel12c2'
CRS-2672: Attempting to start 'ora.evmd' on 'rhel12c2'
CRS-2676: Start of 'ora.cssdmonitor' on 'rhel12c2' succeeded
CRS-2672: Attempting to start 'ora.cssd' on 'rhel12c2'
CRS-2672: Attempting to start 'ora.diskmon' on 'rhel12c2'
CRS-2676: Start of 'ora.diskmon' on 'rhel12c2' succeeded
CRS-2676: Start of 'ora.evmd' on 'rhel12c2' succeeded
CRS-2676: Start of 'ora.cssd' on 'rhel12c2' succeeded
CRS-2672: Attempting to start 'ora.ctssd' on 'rhel12c2'
CRS-2672: Attempting to start 'ora.cluster_interconnect.haip' on 'rhel12c2'
CRS-2676: Start of 'ora.ctssd' on 'rhel12c2' succeeded
CRS-2676: Start of 'ora.cluster_interconnect.haip' on 'rhel12c2' succeeded
CRS-2672: Attempting to start 'ora.asm' on 'rhel12c2'
CRS-2676: Start of 'ora.asm' on 'rhel12c2' succeeded
CRS-2672: Attempting to start 'ora.storage' on 'rhel12c2'
CRS-2676: Start of 'ora.storage' on 'rhel12c2' succeeded
CRS-2672: Attempting to start 'ora.crsd' on 'rhel12c2'
CRS-2676: Start of 'ora.crsd' on 'rhel12c2' succeeded
Oracle Grid Infrastructure restarted in node rhel12c2
End of the conversion the ASMCA will exit and console will have following output
Unable to establish connection to ASM instance.
 ORA-01034: ORACLE not available
 ASMCA will exit now.
This due to the restart of ASM and ASMCA unable to establish the connection. Rerun ASMCA and verify ASM instances are up an running. Notice the convert to flex ASM button is not there anymore.
List the new ASM configuration
$  srvctl config asm
ASM home: 
Password file: +NEWCLUSTERDG/orapwasm
ASM listener: LISTENER
ASM instance count: ALL
Cluster ASM listener: ASMNET1LSNR_ASM
The instance count is ALL because the option of running database version lower than 12.1 was selected. If this isn't the case the instance count would be
ASM instance count: 3
But only two ASM instances would be running.
The cluster ASM listener is available across the cluster
--------------------------------------------------------------------------------
Local Resources
--------------------------------------------------------------------------------
ora.ASMNET1LSNR_ASM.lsnr
               ONLINE  ONLINE       rhel12c1                 STABLE
               ONLINE  ONLINE       rhel12c2                 STABLE
Also verify the ASM flex mode with asmcmd
$ asmcmd
ASMCMD> showclustermode
ASM cluster : Flex mode enabled


The second phase is to convert the standard cluster mode to flex. This requires a cluster downtime.
To change the cluster mode to flex as root run the following(only on one of the nodes)
# crsctl set cluster mode flex
CRS-4933: Cluster mode set to "flex"; restart Oracle High Availability Services on all nodes for cluster to run in "flex" mode.
Shutdown clusterware on all nodes and start one node at a time
[root@rhel12c1 oracle]# crsctl stop crs
[root@rhel12c2 oracle]# crsctl stop crs

[root@rhel12c1 oracle]# crsctl start crs -wait
[root@rhel12c2 oracle]# crsctl start crs -wait
Once the cluster has started check the cluster mode and node roles
# crsctl get cluster mode status
Cluster is running in "flex" mode

[root@rhel12c1 oracle]# crsctl get node role config
Node 'rhel12c1' configured role is 'hub'

[grid@rhel12c2 ~]$ crsctl get node role config
Node 'rhel12c2' configured role is 'hub'
Make a note of ASM and network configurations as well
# srvctl config asm -detail
ASM home: 
Password file: +NEWCLUSTERDG/orapwasm
ASM listener: LISTENER
ASM is enabled.
ASM is individually enabled on nodes:
ASM is individually disabled on nodes:
ASM instance count: ALL
Cluster ASM listener: ASMNET1LSNR_ASM

oifcfg getif
eth0  192.168.0.0  global  public
eth1  192.168.1.0  global  cluster_interconnect,asm
This concludes the converting of a standard cluster to a flex cluster.

One of the key features of flex ASM is there's no one-one relationship between ASM instance and database instance. In a non flex ASM configuration failure of ASM instance would mean that database instance on that node will also fail. However with flex ASM in the event of an ASM instance failure the database instance is free to continue using a remaining ASM instance. This could be tested by stopping an ASM instance in one of the nodes
$ srvctl stop asm -node rhel12c1 -force

$ srvctl status asm -detail
ASM is running on rhel12c2
ASM is enabled.
ASM is running only one node 2. But both database instances would be up and running
ora.rac12c1.db
      1        ONLINE  ONLINE       rhel12c1                 Open,STABLE
      2        ONLINE  ONLINE       rhel12c2                 Open,STABLE
There's no restriction to making application connections to the db instance 1
SQL> select instance_name from v$instance;

INSTANCE_NAME
----------------
rac12c11
Connect to ASM instance on node 2 and check the asm clients which will list among others the db instance 1
SQL> select db_name,CLUSTER_NAME,STATUS,INSTANCE_NAME from v$asm_client;

DB_NAME  CLUSTER_NAME                    STATUS       INSTANCE_NAME
-------- ------------------------------- ------------ -----------------
_mgmtdb  rhel12c                         CONNECTED    -MGMTDB
+ASM     rhel12c                         CONNECTED    +ASM2
+ASM     rhel12c                         CONNECTED    +ASM2
rac12c1  rhel12c                         CONNECTED    rac12c11
rac12c1  rhel12c                         CONNECTED    rac12c11
rac12c1  rhel12c                         CONNECTED    rac12c12
rac12c1  rhel12c                         CONNECTED    rac12c12
Bringing back the stopped asm instance will not automatically relocate the db instance to it.
srvctl start asm -node rhel12c1
SQL>select db_name,CLUSTER_NAME,STATUS,INSTANCE_NAME from v$asm_client;
no rows selected
This has to be done manually with relocate command. Run this on node 2 by specifying instance_name:db_name
SQL> alter system relocate client 'rac12c11:rac12c1';
System altered.
The ASM alert log will capture the following during relocation
NOTE: Flex client id 0x10002 [rac12c11:rac12c1:rhel12c] attempting to reconnect
NOTE: registered owner id 0x10002 for rac12c11:rac12c1:rhel12c (reconnecting)
NOTE: Flex client rac12c11:rac12c1:rhel12c re-registered, osid 14767, mbr 0x0, asmb 6115 (reg:1427811453)
NOTE: client [rac12c11:rac12c1:rhel12c] declared 0 additional pending writes
Fri Aug 14 10:43:07 2015
NOTE: cleaning stale remote ownerid 0x10002 for client rac12c11:rac12c1:rhel12c (reg:1447305939)
Useful metalink notes
FAQ: Oracle Flex ASM 12c / 12.1 [ID 1573137.1]
Steps to Convert STANDARD ASM to FLEX ASM in 12C [ID 1915814.1]