Lock file left by a different patch, OPatch will not try re-using the lock file

Retrying a patch apply after a failed attempted resulted in following error.

$ /opt/app/11.2.0/grid4/OPatch/opatch apply -local
Oracle Interim Patch Installer version 11.2.0.3.6
Copyright (c) 2013, Oracle Corporation.  All rights reserved.

Oracle Home       : /opt/app/11.2.0/grid4
Central Inventory : /opt/app/oraInventory
   from           : /opt/app/11.2.0/grid4/oraInst.loc
OPatch version    : 11.2.0.3.6
OUI version       : 11.2.0.4.0
Log file location : /opt/app/11.2.0/grid4/cfgtoollogs/opatch/19852360_Sep_24_2015_13_35_27/apply2015-09-24_13-35-27PM_1.log

Applying interim patch '19852360' to OH '/opt/app/11.2.0/grid4'
Verifying environment and performing prerequisite checks...
[ Error during Oracle Home discovery Phase]. Detail: OPatchSession cannot load inventory for the given Oracle Home /opt/app/11.2.0/grid4. Possible causes are:
   No read or write permission to ORACLE_HOME/.patch_storage
   Central Inventory is locked by another OUI instance
   No read permission to Central Inventory
   The lock file exists in ORACLE_HOME/.patch_storage
   The Oracle Home does not exist in Central Inventory

[ Error during Oracle Home discovery Phase]. Detail: OPatch failed: ApplySession failed to prepare the system. Lock file left by a different patch, OPatch will not try re-using the lock file.
Log file location: /opt/app/11.2.0/grid4/cfgtoollogs/opatch/19852360_Sep_24_2015_13_35_27/apply2015-09-24_13-35-27PM_1.log

Recommended actions: Please make sure no other OPatch or OUI processes is running. Try running $ORACLE_HOME/oui/bin/runInstsaller.

OPatch failed with error code 22

Looking in the oraInventory showed a lock directory which was empty. Removing it didn't resolve the issue

[grid@rhel6m1 oraInventory]$ ls
backup  ContentsXML  locks  logs  oraInst.loc  orainstRoot.sh  oui
[grid@rhel6m1 oraInventory]$ cd locks/
[grid@rhel6m1 oraInventory]$ rmdir locks

Inside the .patch_storage there was another file patch_locked which had the information for which patch the lock is taken for

cat $ORACLE_HOME/.patch_storage/patch_locked
Locked for patch  :  21068539
Locked by class   :  rollback

Removing this file resolved the issue

rm $ORACLE_HOME/.patch_storage/patch_locked

Changing Hub Node to Leaf Node and Vice Versa

The flex cluster setup used here consists of two hub nodes. One of the nodes will be changed from hub node to leaf node and back to a hub node. The nodes are named rhel12c1 and rhel12c2.

Changing Hub Node to a Leaf Node
Node rhel12c1 will be changed from a hub node to a leaf node. The current role could listed with

[root@rhel12c1 oracle]# crsctl get node role config
Node 'rhel12c1' configured role is 'hub'

To change the role of the node run the following command. This must be run on the node that's undergoing the role change as root

[root@rhel12c1 grid]# crsctl set node role leaf
CRS-4408: Node 'rhel12c1' configured role successfully changed; restart Oracle High Availability Services for new role to take effect.

Restart the node after the role change command has been executed

# crsctl stop crs
# crsctl start crs -wait

Verify the node started as a leaf node.

[root@rhel12c1 grid]# crsctl get node role config
Node 'rhel12c1' configured role is 'leaf'

The cluster now consists of a hub node and a leaf node

[root@rhel12c2 grid]# crsctl get node role config -all
Node 'rhel12c1' configured role is 'leaf'
Node 'rhel12c2' configured role is 'hub'

The database instance that was running while the rhel12c1 was a hub node will no longer be active and is in a shutdown state

ora.rac12c1.db
      1        ONLINE  OFFLINE                               Instance Shutdown,STABLE
      2        ONLINE  ONLINE       rhel12c2                 Open,STABLE

There's no resources running on the leaf node.

[root@rhel12c1 grid]# crsctl stat res -t -c rhel12c1

Finally update inventory as follows. Run the update node list command on hub nodes listing all remaining hub nodes in the cluster_nodes option. In this case only rhel12c2 is the hub node.

[grid@rhel12c2 ~]$ $GI_HOME/oui/bin/runInstaller -updateNodeList ORACLE_HOME=/opt/app/12.1.0/grid2 "CLUSTER_NODES={rhel12c2}" -silent -local CRS=TRUE
Starting Oracle Universal Installer...

Checking swap space: must be greater than 500 MB.   Actual 4506 MB    Passed
The inventory pointer is located at /etc/oraInst.loc
'UpdateNodeList' was successful.

On the leaf node run the update node list containing only the leaf node in the cluster_nodes option.

[grid@rhel12c1 ~]$ $GI_HOME/oui/bin/runInstaller -updateNodeList ORACLE_HOME=/opt/app/12.1.0/grid2 "CLUSTER_NODES={rhel12c1}" -silent -local CRS=TRUE
Starting Oracle Universal Installer...

Checking swap space: must be greater than 500 MB.   Actual 5112 MB    Passed
The inventory pointer is located at /etc/oraInst.loc
'UpdateNodeList' was successful.

This conclude converting from hub node to leaf node.



Changing Leaf Node to a Hub Node
Leaf node created in the previous step will be changed back to a hub node in this place. The current node role is

[grid@rhel12c1 ~]$ crsctl get node role config
Node 'rhel12c1' configured role is 'leaf'

As root change the node role to hub node with

[root@rhel12c1 grid]# crsctl set node role hub
CRS-4408: Node 'rhel12c1' configured role successfully changed; restart Oracle High Availability Services for new role to take effect.

Stop the cluster stack on the node undergoing the role change

# crsctl stop crs

Configure the Oracle ASM Filter Driver by running the following as root

# crsctl stop crs

[root@rhel12c1 grid]# /opt/app/12.1.0/grid2/bin/asmcmd afd_configure
Connected to an idle instance.
AFD-627: AFD distribution files found.
AFD-636: Installing requested AFD software.
AFD-637: Loading installed AFD drivers.
AFD-9321: Creating udev for AFD.
AFD-9323: Creating module dependencies - this may take some time.
AFD-9154: Loading 'oracleafd.ko' driver.
AFD-649: Verifying AFD devices.
AFD-9156: Detecting control device '/dev/oracleafd/admin'.
AFD-638: AFD installation correctness verified.
Modifying resource dependencies - this may take some time.

If the node started as a leaf node then it may also require VIP to be configured on the node before changing to hub node. As this started off as a leaf node the VIP already exists and this step is not needed. Once the ASM driver is configured start cluster stack on the node and verify the node role

[root@rhel12c1 grid]# crsctl start crs -wait
[grid@rhel12c1 ~]$ crsctl get node role config
Node 'rhel12c1' configured role is 'hub'

Resources there were down when the role was in leaf node role will be up and running again now.

[root@rhel12c1 grid]# crsctl stat res -t -c rhel12c1
--------------------------------------------------------------------------------
Name           Target  State        Server                   State details
--------------------------------------------------------------------------------
Local Resources
--------------------------------------------------------------------------------
ora.ASMNET1LSNR_ASM.lsnr
               ONLINE  ONLINE       rhel12c1                 STABLE
ora.DATA.dg
               ONLINE  ONLINE       rhel12c1                 STABLE
ora.FRA.dg
               ONLINE  ONLINE       rhel12c1                 STABLE
ora.LISTENER.lsnr
               ONLINE  ONLINE       rhel12c1                 STABLE
ora.NEWCLUSTERDG.dg
               ONLINE  ONLINE       rhel12c1                 STABLE
ora.net1.network
               ONLINE  ONLINE       rhel12c1                 STABLE
ora.ons
               ONLINE  ONLINE       rhel12c1                 STABLE
--------------------------------------------------------------------------------
Cluster Resources
--------------------------------------------------------------------------------
ora.LISTENER_SCAN1.lsnr
      1        ONLINE  ONLINE       rhel12c1                 STABLE
ora.asm
      2        ONLINE  ONLINE       rhel12c1                 Started,STABLE
ora.rac12c1.db
      1        ONLINE  ONLINE       rhel12c1                 Open,STABLE
ora.rhel12c1.vip
      1        ONLINE  ONLINE       rhel12c1                 STABLE
ora.scan1.vip
      1        ONLINE  ONLINE       rhel12c1                 STABLE
--------------------------------------------------------------------------------

Run the inventory update specifying all hub nodes in the cluster_node option

$GI_HOME/oui/bin/runInstaller -updateNodeList ORACLE_HOME=/opt/app/12.1.0/grid2 "CLUSTER_NODES={rhel12c1,rhel12c2}" -silent -local CRS=TRUE

This conclude the changing leaf node to hub node.

Converting 12c Standard Cluster to Flex Cluster

An earlier post showed steps for installing a 12c flex cluster. This post shows the steps for converting a standard cluster to a flex cluster. There are two phases to converting a standard cluster to flex cluster. First phase involves converting Oracle ASM to Flex ASM and second phase is to change the cluster mode to flex.
The current setup is a two node standard cluster.

$  crsctl get cluster mode status
Cluster is running in "standard" mode

$ asmcmd
ASMCMD> showclustermode
ASM cluster : Flex mode disabled

There are few per-requisites to converting ASM to flex ASM, these include that OCR, SPFile and password file all stored in a disk group and this group have ASM compatibility (COMPATIBLE.ASM) set to 12.1 or higher. If this is not the case then make the necessary changes to move these files to disk group. As the current setup is a newly installed 12c cluster (as oppose to upgraded to 12c) these files all reside in disk group.
A point to note is that the diskgroup with above mentioned files has the size of 5114MB (below output has been modified by omitting some columns).

ASMCMD> lsdg
State    Type    Rebal  Sector  Block       AU  Total_MB  Free_MB  Voting_files  Name
MOUNTED  EXTERN  N         512   4096  1048576     10236     7609             N  DATA/
MOUNTED  EXTERN  N         512   4096  1048576     10236     8609             N  FRA/
MOUNTED  EXTERN  N         512   4096  1048576      5114      475             Y  NEWCLUSTERDG/

This is smaller than the required diskgroup size for new flex cluster being installed. Despite the smaller size diskgroup the conversion to flex cluster succeeded without any issue. The current ASM configuration is

$  srvctl config asm
ASM home: 
Password file: +NEWCLUSTERDG/orapwasm
ASM listener: LISTENER

Another key requirement is that flex cluster requires GNS. If a GNS doesn't exists setup GNS and add the GNS VIP to cluster. The standard cluster was already setup with GNS.

$  srvctl config gns
GNS is enabled.
GNS VIP addresses: 192.168.0.87
Domain served by GNS: rac.domain.net

To start the first phase of converting to flex ASM run asmca and click on ASM instances tab. If role separation is used then make sure grid user has write permission for directory $ORACLE_BASE/cfgtoollogs as the conversion script is generated inside it.

Click convert to Flex ASM button. This would prompt to select a network interface for ASM communication and also to specify a port. The default port is 1521.

However there is a problem in using the port 1521.

The port 1521 is used on the private network (192.168.1.0 in this case) by the management listener.

$ netstat -ntl | grep 1521
tcp        0      0 192.168.0.89:1521           0.0.0.0:*                   LISTEN      3323/tnslsnr
tcp        0      0 192.168.0.96:1521           0.0.0.0:*                   LISTEN      3323/tnslsnr
tcp        0      0 192.168.0.97:1521           0.0.0.0:*                   LISTEN      3323/tnslsnr
tcp        0      0 192.168.0.93:1521           0.0.0.0:*                   LISTEN      3323/tnslsnr
tcp        0      0 192.168.0.95:1521           0.0.0.0:*                   LISTEN      3323/tnslsnr
tcp        0      0 169.254.11.195:1521         0.0.0.0:*                   LISTEN      3323/tnslsnr
tcp        0      0 192.168.1.87:1521           0.0.0.0:*                   LISTEN      3323/tnslsnr

# ps ax | grep 3323
 3323 ?        Ssl    0:00 /opt/app/12.1.0/grid2/bin/tnslsnr MGMTLSNR -no_crs_notify -inherit

Even after stopping the MGMTLSNR

$ srvctl stop listener -l MGMTLSNR
PRCR-1065 : Failed to stop resource ora.MGMTLSNR
CRS-2529: Unable to act on 'ora.MGMTLSNR' because that would require stopping or relocating 'ora.mgmtdb', but the force option was not specified
$ srvctl stop listener -l MGMTLSNR -f

ora.MGMTLSNR
      1        OFFLINE OFFLINE                               169.254.11.195 192.1
                                                             68.1.87,STABLE

and all other listeners (scan_listener and listener) that were using port 1521 (even though they were on a different network) the error did not get resolved. It could be port usage is looked up statically from a config file. There's no problem to use 1521 for both database listener and ASM listener when a new flex cluster is installed. Finally had to proceed using a different port for cluster ASM listener to continue forward.

Conversion to flex ASM will continue restarting one node at time.

When prompted execute the script on the same node ASMCA was run.

Output from converttoFlexASM.sh execution

# /opt/app/oracle/cfgtoollogs/asmca/scripts/converttoFlexASM.sh
CRS-2673: Attempting to stop 'ora.crsd' on 'rhel12c1'
CRS-2790: Starting shutdown of Cluster Ready Services-managed resources on 'rhel12c1'
CRS-2673: Attempting to stop 'ora.mgmtdb' on 'rhel12c1'
CRS-2673: Attempting to stop 'ora.gns' on 'rhel12c1'
CRS-2673: Attempting to stop 'ora.LISTENER_SCAN2.lsnr' on 'rhel12c1'
CRS-2673: Attempting to stop 'ora.NEWCLUSTERDG.dg' on 'rhel12c1'
CRS-2673: Attempting to stop 'ora.LISTENER.lsnr' on 'rhel12c1'
CRS-2673: Attempting to stop 'ora.rac12c1.db' on 'rhel12c1'
CRS-2673: Attempting to stop 'ora.LISTENER_SCAN3.lsnr' on 'rhel12c1'
CRS-2673: Attempting to stop 'ora.DATA.dg' on 'rhel12c1'
CRS-2673: Attempting to stop 'ora.oc4j' on 'rhel12c1'
CRS-2673: Attempting to stop 'ora.cvu' on 'rhel12c1'
CRS-2673: Attempting to stop 'ora.FRA.dg' on 'rhel12c1'
CRS-2677: Stop of 'ora.cvu' on 'rhel12c1' succeeded
CRS-2672: Attempting to start 'ora.cvu' on 'rhel12c2'
CRS-2677: Stop of 'ora.LISTENER_SCAN2.lsnr' on 'rhel12c1' succeeded
CRS-2673: Attempting to stop 'ora.scan2.vip' on 'rhel12c1'
CRS-2677: Stop of 'ora.LISTENER.lsnr' on 'rhel12c1' succeeded
CRS-2677: Stop of 'ora.DATA.dg' on 'rhel12c1' succeeded
CRS-2677: Stop of 'ora.NEWCLUSTERDG.dg' on 'rhel12c1' succeeded
CRS-2677: Stop of 'ora.FRA.dg' on 'rhel12c1' succeeded
CRS-2677: Stop of 'ora.gns' on 'rhel12c1' succeeded
CRS-2673: Attempting to stop 'ora.gns.vip' on 'rhel12c1'
CRS-2677: Stop of 'ora.LISTENER_SCAN3.lsnr' on 'rhel12c1' succeeded
CRS-2673: Attempting to stop 'ora.scan3.vip' on 'rhel12c1'
CRS-2673: Attempting to stop 'ora.rhel12c1.vip' on 'rhel12c1'
CRS-2676: Start of 'ora.cvu' on 'rhel12c2' succeeded
CRS-2677: Stop of 'ora.scan2.vip' on 'rhel12c1' succeeded
CRS-2672: Attempting to start 'ora.scan2.vip' on 'rhel12c2'
CRS-2677: Stop of 'ora.rac12c1.db' on 'rhel12c1' succeeded
CRS-2677: Stop of 'ora.mgmtdb' on 'rhel12c1' succeeded
CRS-2673: Attempting to stop 'ora.MGMTLSNR' on 'rhel12c1'
CRS-2677: Stop of 'ora.MGMTLSNR' on 'rhel12c1' succeeded
CRS-2672: Attempting to start 'ora.MGMTLSNR' on 'rhel12c2'
CRS-2677: Stop of 'ora.gns.vip' on 'rhel12c1' succeeded
CRS-2672: Attempting to start 'ora.gns.vip' on 'rhel12c2'
CRS-2677: Stop of 'ora.scan3.vip' on 'rhel12c1' succeeded
CRS-2672: Attempting to start 'ora.scan3.vip' on 'rhel12c2'
CRS-2677: Stop of 'ora.rhel12c1.vip' on 'rhel12c1' succeeded
CRS-2672: Attempting to start 'ora.rhel12c1.vip' on 'rhel12c2'
CRS-2676: Start of 'ora.scan2.vip' on 'rhel12c2' succeeded
CRS-2672: Attempting to start 'ora.LISTENER_SCAN2.lsnr' on 'rhel12c2'
CRS-2676: Start of 'ora.MGMTLSNR' on 'rhel12c2' succeeded
CRS-2672: Attempting to start 'ora.mgmtdb' on 'rhel12c2'
CRS-2676: Start of 'ora.gns.vip' on 'rhel12c2' succeeded
CRS-2672: Attempting to start 'ora.gns' on 'rhel12c2'
CRS-2677: Stop of 'ora.oc4j' on 'rhel12c1' succeeded
CRS-2672: Attempting to start 'ora.oc4j' on 'rhel12c2'
CRS-2676: Start of 'ora.scan3.vip' on 'rhel12c2' succeeded
CRS-2672: Attempting to start 'ora.LISTENER_SCAN3.lsnr' on 'rhel12c2'
CRS-2676: Start of 'ora.rhel12c1.vip' on 'rhel12c2' succeeded
CRS-2676: Start of 'ora.gns' on 'rhel12c2' succeeded
CRS-2676: Start of 'ora.LISTENER_SCAN2.lsnr' on 'rhel12c2' succeeded
CRS-2676: Start of 'ora.LISTENER_SCAN3.lsnr' on 'rhel12c2' succeeded
CRS-2676: Start of 'ora.oc4j' on 'rhel12c2' succeeded
CRS-2676: Start of 'ora.mgmtdb' on 'rhel12c2' succeeded
CRS-2673: Attempting to stop 'ora.ons' on 'rhel12c1'
CRS-2677: Stop of 'ora.ons' on 'rhel12c1' succeeded
CRS-2673: Attempting to stop 'ora.net1.network' on 'rhel12c1'
CRS-2677: Stop of 'ora.net1.network' on 'rhel12c1' succeeded
CRS-2792: Shutdown of Cluster Ready Services-managed resources on 'rhel12c1' has completed
CRS-2677: Stop of 'ora.crsd' on 'rhel12c1' succeeded
CRS-2673: Attempting to stop 'ora.ctssd' on 'rhel12c1'
CRS-2673: Attempting to stop 'ora.evmd' on 'rhel12c1'
CRS-2673: Attempting to stop 'ora.storage' on 'rhel12c1'
CRS-2677: Stop of 'ora.storage' on 'rhel12c1' succeeded
CRS-2673: Attempting to stop 'ora.asm' on 'rhel12c1'
CRS-2677: Stop of 'ora.ctssd' on 'rhel12c1' succeeded
CRS-2677: Stop of 'ora.evmd' on 'rhel12c1' succeeded
CRS-2677: Stop of 'ora.asm' on 'rhel12c1' succeeded
CRS-2673: Attempting to stop 'ora.cluster_interconnect.haip' on 'rhel12c1'
CRS-2677: Stop of 'ora.cluster_interconnect.haip' on 'rhel12c1' succeeded
CRS-2673: Attempting to stop 'ora.cssd' on 'rhel12c1'
CRS-2677: Stop of 'ora.cssd' on 'rhel12c1' succeeded
CRS-2672: Attempting to start 'ora.evmd' on 'rhel12c1'
CRS-2672: Attempting to start 'ora.cssdmonitor' on 'rhel12c1'
CRS-2676: Start of 'ora.cssdmonitor' on 'rhel12c1' succeeded
CRS-2672: Attempting to start 'ora.cssd' on 'rhel12c1'
CRS-2672: Attempting to start 'ora.diskmon' on 'rhel12c1'
CRS-2676: Start of 'ora.diskmon' on 'rhel12c1' succeeded
CRS-2676: Start of 'ora.evmd' on 'rhel12c1' succeeded
CRS-2676: Start of 'ora.cssd' on 'rhel12c1' succeeded
CRS-2672: Attempting to start 'ora.ctssd' on 'rhel12c1'
CRS-2672: Attempting to start 'ora.cluster_interconnect.haip' on 'rhel12c1'
CRS-2676: Start of 'ora.ctssd' on 'rhel12c1' succeeded
CRS-2676: Start of 'ora.cluster_interconnect.haip' on 'rhel12c1' succeeded
CRS-2672: Attempting to start 'ora.asm' on 'rhel12c1'
CRS-2676: Start of 'ora.asm' on 'rhel12c1' succeeded
CRS-2672: Attempting to start 'ora.storage' on 'rhel12c1'
CRS-2676: Start of 'ora.storage' on 'rhel12c1' succeeded
CRS-2672: Attempting to start 'ora.crsd' on 'rhel12c1'
CRS-2676: Start of 'ora.crsd' on 'rhel12c1' succeeded
Oracle Grid Infrastructure restarted in node rhel12c1
PRCC-1014 : ASMNET1LSNR_ASM was already running
PRCR-1004 : Resource ora.ASMNET1LSNR_ASM.lsnr is already running
PRCR-1079 : Failed to start resource ora.ASMNET1LSNR_ASM.lsnr
CRS-5702: Resource 'ora.ASMNET1LSNR_ASM.lsnr' is already running on 'rhel12c2'
CRS-5702: Resource 'ora.ASMNET1LSNR_ASM.lsnr' is already running on 'rhel12c1'
ASM listener ASMNET1LSNR_ASM running already
CRS-2673: Attempting to stop 'ora.crsd' on 'rhel12c2'
CRS-2790: Starting shutdown of Cluster Ready Services-managed resources on 'rhel12c2'
CRS-2673: Attempting to stop 'ora.rac12c1.cbx.svc' on 'rhel12c2'
CRS-2673: Attempting to stop 'ora.LISTENER_SCAN3.lsnr' on 'rhel12c2'
CRS-2673: Attempting to stop 'ora.LISTENER_SCAN2.lsnr' on 'rhel12c2'
CRS-2673: Attempting to stop 'ora.mgmtdb' on 'rhel12c2'
CRS-2673: Attempting to stop 'ora.LISTENER.lsnr' on 'rhel12c2'
CRS-2673: Attempting to stop 'ora.oc4j' on 'rhel12c2'
CRS-2673: Attempting to stop 'ora.cvu' on 'rhel12c2'
CRS-2673: Attempting to stop 'ora.NEWCLUSTERDG.dg' on 'rhel12c2'
CRS-2673: Attempting to stop 'ora.ASMNET1LSNR_ASM.lsnr' on 'rhel12c2'
CRS-2677: Stop of 'ora.cvu' on 'rhel12c2' succeeded
CRS-2672: Attempting to start 'ora.cvu' on 'rhel12c1'
CRS-2677: Stop of 'ora.NEWCLUSTERDG.dg' on 'rhel12c2' succeeded
CRS-2677: Stop of 'ora.LISTENER_SCAN2.lsnr' on 'rhel12c2' succeeded
CRS-2673: Attempting to stop 'ora.scan2.vip' on 'rhel12c2'
CRS-2677: Stop of 'ora.LISTENER_SCAN3.lsnr' on 'rhel12c2' succeeded
CRS-2673: Attempting to stop 'ora.scan3.vip' on 'rhel12c2'
CRS-2677: Stop of 'ora.LISTENER.lsnr' on 'rhel12c2' succeeded
CRS-2677: Stop of 'ora.rac12c1.cbx.svc' on 'rhel12c2' succeeded
CRS-2673: Attempting to stop 'ora.rac12c1.db' on 'rhel12c2'
CRS-2677: Stop of 'ora.ASMNET1LSNR_ASM.lsnr' on 'rhel12c2' succeeded
CRS-2676: Start of 'ora.cvu' on 'rhel12c1' succeeded
CRS-2677: Stop of 'ora.scan2.vip' on 'rhel12c2' succeeded
CRS-2672: Attempting to start 'ora.scan2.vip' on 'rhel12c1'
CRS-2677: Stop of 'ora.scan3.vip' on 'rhel12c2' succeeded
CRS-2672: Attempting to start 'ora.scan3.vip' on 'rhel12c1'
CRS-2677: Stop of 'ora.rac12c1.db' on 'rhel12c2' succeeded
CRS-2677: Stop of 'ora.mgmtdb' on 'rhel12c2' succeeded
CRS-2673: Attempting to stop 'ora.MGMTLSNR' on 'rhel12c2'
CRS-2677: Stop of 'ora.MGMTLSNR' on 'rhel12c2' succeeded
CRS-2672: Attempting to start 'ora.MGMTLSNR' on 'rhel12c1'
CRS-2673: Attempting to stop 'ora.rhel12c2.vip' on 'rhel12c2'
CRS-2676: Start of 'ora.scan2.vip' on 'rhel12c1' succeeded
CRS-2672: Attempting to start 'ora.LISTENER_SCAN2.lsnr' on 'rhel12c1'
CRS-2677: Stop of 'ora.rhel12c2.vip' on 'rhel12c2' succeeded
CRS-2672: Attempting to start 'ora.rhel12c2.vip' on 'rhel12c1'
CRS-2676: Start of 'ora.scan3.vip' on 'rhel12c1' succeeded
CRS-2672: Attempting to start 'ora.LISTENER_SCAN3.lsnr' on 'rhel12c1'
CRS-2676: Start of 'ora.MGMTLSNR' on 'rhel12c1' succeeded
CRS-2672: Attempting to start 'ora.mgmtdb' on 'rhel12c1'
CRS-2677: Stop of 'ora.oc4j' on 'rhel12c2' succeeded
CRS-2672: Attempting to start 'ora.oc4j' on 'rhel12c1'
CRS-2676: Start of 'ora.rhel12c2.vip' on 'rhel12c1' succeeded
CRS-2676: Start of 'ora.LISTENER_SCAN2.lsnr' on 'rhel12c1' succeeded
CRS-2676: Start of 'ora.LISTENER_SCAN3.lsnr' on 'rhel12c1' succeeded
CRS-2673: Attempting to stop 'ora.DATA.dg' on 'rhel12c2'
CRS-2673: Attempting to stop 'ora.FRA.dg' on 'rhel12c2'
CRS-2673: Attempting to stop 'ora.gns' on 'rhel12c2'
CRS-2677: Stop of 'ora.FRA.dg' on 'rhel12c2' succeeded
CRS-2677: Stop of 'ora.DATA.dg' on 'rhel12c2' succeeded
CRS-2677: Stop of 'ora.gns' on 'rhel12c2' succeeded
CRS-2673: Attempting to stop 'ora.gns.vip' on 'rhel12c2'
CRS-2677: Stop of 'ora.gns.vip' on 'rhel12c2' succeeded
CRS-2672: Attempting to start 'ora.gns.vip' on 'rhel12c1'
CRS-2676: Start of 'ora.gns.vip' on 'rhel12c1' succeeded
CRS-2672: Attempting to start 'ora.gns' on 'rhel12c1'
CRS-2676: Start of 'ora.gns' on 'rhel12c1' succeeded
CRS-2676: Start of 'ora.oc4j' on 'rhel12c1' succeeded
CRS-2676: Start of 'ora.mgmtdb' on 'rhel12c1' succeeded
CRS-2673: Attempting to stop 'ora.ons' on 'rhel12c2'
CRS-2677: Stop of 'ora.ons' on 'rhel12c2' succeeded
CRS-2673: Attempting to stop 'ora.net1.network' on 'rhel12c2'
CRS-2677: Stop of 'ora.net1.network' on 'rhel12c2' succeeded
CRS-2792: Shutdown of Cluster Ready Services-managed resources on 'rhel12c2' has completed
CRS-2677: Stop of 'ora.crsd' on 'rhel12c2' succeeded
CRS-2673: Attempting to stop 'ora.ctssd' on 'rhel12c2'
CRS-2673: Attempting to stop 'ora.evmd' on 'rhel12c2'
CRS-2673: Attempting to stop 'ora.storage' on 'rhel12c2'
CRS-2677: Stop of 'ora.storage' on 'rhel12c2' succeeded
CRS-2673: Attempting to stop 'ora.asm' on 'rhel12c2'
CRS-2677: Stop of 'ora.ctssd' on 'rhel12c2' succeeded
CRS-2677: Stop of 'ora.evmd' on 'rhel12c2' succeeded
CRS-2677: Stop of 'ora.asm' on 'rhel12c2' succeeded
CRS-2673: Attempting to stop 'ora.cluster_interconnect.haip' on 'rhel12c2'
CRS-2677: Stop of 'ora.cluster_interconnect.haip' on 'rhel12c2' succeeded
CRS-2673: Attempting to stop 'ora.cssd' on 'rhel12c2'
CRS-2677: Stop of 'ora.cssd' on 'rhel12c2' succeeded
CRS-2672: Attempting to start 'ora.cssdmonitor' on 'rhel12c2'
CRS-2672: Attempting to start 'ora.evmd' on 'rhel12c2'
CRS-2676: Start of 'ora.cssdmonitor' on 'rhel12c2' succeeded
CRS-2672: Attempting to start 'ora.cssd' on 'rhel12c2'
CRS-2672: Attempting to start 'ora.diskmon' on 'rhel12c2'
CRS-2676: Start of 'ora.diskmon' on 'rhel12c2' succeeded
CRS-2676: Start of 'ora.evmd' on 'rhel12c2' succeeded
CRS-2676: Start of 'ora.cssd' on 'rhel12c2' succeeded
CRS-2672: Attempting to start 'ora.ctssd' on 'rhel12c2'
CRS-2672: Attempting to start 'ora.cluster_interconnect.haip' on 'rhel12c2'
CRS-2676: Start of 'ora.ctssd' on 'rhel12c2' succeeded
CRS-2676: Start of 'ora.cluster_interconnect.haip' on 'rhel12c2' succeeded
CRS-2672: Attempting to start 'ora.asm' on 'rhel12c2'
CRS-2676: Start of 'ora.asm' on 'rhel12c2' succeeded
CRS-2672: Attempting to start 'ora.storage' on 'rhel12c2'
CRS-2676: Start of 'ora.storage' on 'rhel12c2' succeeded
CRS-2672: Attempting to start 'ora.crsd' on 'rhel12c2'
CRS-2676: Start of 'ora.crsd' on 'rhel12c2' succeeded
Oracle Grid Infrastructure restarted in node rhel12c2

End of the conversion the ASMCA will exit and console will have following output

Unable to establish connection to ASM instance.
 ORA-01034: ORACLE not available
 ASMCA will exit now.

This due to the restart of ASM and ASMCA unable to establish the connection. Rerun ASMCA and verify ASM instances are up an running. Notice the convert to flex ASM button is not there anymore.

List the new ASM configuration

$  srvctl config asm
ASM home: 
Password file: +NEWCLUSTERDG/orapwasm
ASM listener: LISTENER
ASM instance count: ALL
Cluster ASM listener: ASMNET1LSNR_ASM

The instance count is ALL because the option of running database version lower than 12.1 was selected. If this isn't the case the instance count would be

ASM instance count: 3

But only two ASM instances would be running.
The cluster ASM listener is available across the cluster

--------------------------------------------------------------------------------
Local Resources
--------------------------------------------------------------------------------
ora.ASMNET1LSNR_ASM.lsnr
               ONLINE  ONLINE       rhel12c1                 STABLE
               ONLINE  ONLINE       rhel12c2                 STABLE

Also verify the ASM flex mode with asmcmd

$ asmcmd
ASMCMD> showclustermode
ASM cluster : Flex mode enabled

The second phase is to convert the standard cluster mode to flex. This requires a cluster downtime.
To change the cluster mode to flex as root run the following(only on one of the nodes)

# crsctl set cluster mode flex
CRS-4933: Cluster mode set to "flex"; restart Oracle High Availability Services on all nodes for cluster to run in "flex" mode.

Shutdown clusterware on all nodes and start one node at a time

[root@rhel12c1 oracle]# crsctl stop crs
[root@rhel12c2 oracle]# crsctl stop crs

[root@rhel12c1 oracle]# crsctl start crs -wait
[root@rhel12c2 oracle]# crsctl start crs -wait

Once the cluster has started check the cluster mode and node roles

# crsctl get cluster mode status
Cluster is running in "flex" mode

[root@rhel12c1 oracle]# crsctl get node role config
Node 'rhel12c1' configured role is 'hub'

[grid@rhel12c2 ~]$ crsctl get node role config
Node 'rhel12c2' configured role is 'hub'

Make a note of ASM and network configurations as well

# srvctl config asm -detail
ASM home: 
Password file: +NEWCLUSTERDG/orapwasm
ASM listener: LISTENER
ASM is enabled.
ASM is individually enabled on nodes:
ASM is individually disabled on nodes:
ASM instance count: ALL
Cluster ASM listener: ASMNET1LSNR_ASM

oifcfg getif
eth0  192.168.0.0  global  public
eth1  192.168.1.0  global  cluster_interconnect,asm

This concludes the converting of a standard cluster to a flex cluster.

One of the key features of flex ASM is there's no one-one relationship between ASM instance and database instance. In a non flex ASM configuration failure of ASM instance would mean that database instance on that node will also fail. However with flex ASM in the event of an ASM instance failure the database instance is free to continue using a remaining ASM instance. This could be tested by stopping an ASM instance in one of the nodes

$ srvctl stop asm -node rhel12c1 -force

$ srvctl status asm -detail
ASM is running on rhel12c2
ASM is enabled.

ASM is running only one node 2. But both database instances would be up and running

ora.rac12c1.db
      1        ONLINE  ONLINE       rhel12c1                 Open,STABLE
      2        ONLINE  ONLINE       rhel12c2                 Open,STABLE

There's no restriction to making application connections to the db instance 1

SQL> select instance_name from v$instance;

INSTANCE_NAME
----------------
rac12c11

Connect to ASM instance on node 2 and check the asm clients which will list among others the db instance 1

SQL> select db_name,CLUSTER_NAME,STATUS,INSTANCE_NAME from v$asm_client;

DB_NAME  CLUSTER_NAME                    STATUS       INSTANCE_NAME
-------- ------------------------------- ------------ -----------------
_mgmtdb  rhel12c                         CONNECTED    -MGMTDB
+ASM     rhel12c                         CONNECTED    +ASM2
+ASM     rhel12c                         CONNECTED    +ASM2
rac12c1  rhel12c                         CONNECTED    rac12c11
rac12c1  rhel12c                         CONNECTED    rac12c11
rac12c1  rhel12c                         CONNECTED    rac12c12
rac12c1  rhel12c                         CONNECTED    rac12c12

Bringing back the stopped asm instance will not automatically relocate the db instance to it.

srvctl start asm -node rhel12c1
SQL>select db_name,CLUSTER_NAME,STATUS,INSTANCE_NAME from v$asm_client;
no rows selected

This has to be done manually with relocate command. Run this on node 2 by specifying instance_name:db_name

SQL> alter system relocate client 'rac12c11:rac12c1';
System altered.

The ASM alert log will capture the following during relocation

NOTE: Flex client id 0x10002 [rac12c11:rac12c1:rhel12c] attempting to reconnect
NOTE: registered owner id 0x10002 for rac12c11:rac12c1:rhel12c (reconnecting)
NOTE: Flex client rac12c11:rac12c1:rhel12c re-registered, osid 14767, mbr 0x0, asmb 6115 (reg:1427811453)
NOTE: client [rac12c11:rac12c1:rhel12c] declared 0 additional pending writes
Fri Aug 14 10:43:07 2015
NOTE: cleaning stale remote ownerid 0x10002 for client rac12c11:rac12c1:rhel12c (reg:1447305939)

Useful metalink notes
FAQ: Oracle Flex ASM 12c / 12.1 [ID 1573137.1]
Steps to Convert STANDARD ASM to FLEX ASM in 12C [ID 1915814.1]