For input string: SP2-0640: Not connected

Upgrade of a 11.1.0.7 instance to 11.2.0.4 fails with the following error message on DBUA.

Following errors could be seen when DBUA trace.log is examined.

[Thread-12] [ 2016-01-15 11:43:16.187 GMT ] [OracleHome.initOptions:1256]  ORA-27102: out of memory

oracle.sysman.assistants.util.sqlEngine.SQLFatalErrorException: ORA-27102: out of memory

        at oracle.sysman.assistants.util.sqlEngine.SQLEngine.executeImpl(SQLEngine.java:1658)
        at oracle.sysman.assistants.util.sqlEngine.SQLEngine.executeSql(SQLEngine.java:1925)
        at oracle.sysman.assistants.util.OracleHome.initOptions(OracleHome.java:1249)
        at oracle.sysman.assistants.dbma.backend.CompManager.doPreMigrationChecks(CompManager.java:2955)
        at oracle.sysman.assistants.dbma.backend.CompManager.obtainDatabaseInformation(CompManager.java:4269)
        at oracle.sysman.assistants.dbma.ui.DatabasesPage.doProcessing(DatabasesPage.java:811)
        at oracle.sysman.assistants.util.WaitDialog.run(WaitDialog.java:187)
        at java.lang.Thread.run(Thread.java:637)
..
[Thread-12] [ 2016-01-15 11:43:16.260 GMT ] [OracleHome.initOptions:1324]  Database Options queried: 1
java.lang.NumberFormatException: For input string: "SP2-0640: Not connected"
        at java.lang.NumberFormatException.forInputString(NumberFormatException.java:48)
        at java.lang.Integer.parseInt(Integer.java:447)
        at java.lang.Integer.parseInt(Integer.java:497)
        at oracle.sysman.assistants.dbma.backend.CompManager.doPreMigrationChecks(CompManager.java:2956)
        at oracle.sysman.assistants.dbma.backend.CompManager.obtainDatabaseInformation(CompManager.java:4269)
        at oracle.sysman.assistants.dbma.ui.DatabasesPage.doProcessing(DatabasesPage.java:811)
        at oracle.sysman.assistants.util.WaitDialog.run(WaitDialog.java:187)
        at java.lang.Thread.run(Thread.java:637)

Even though the latter stack trace matches the error output (For input string: "SP2-0640: Not connected") seen on DBUA it's not the root cause but the ORA-27102: out of memory. Key to solving the issue is to identify which instance runs out of the memory.
The current shm* parameters are

kernel.shmmax = 5370806272
kernel.shmall = 1311236
kernel.shmmni = 4096

ipcs  -lm

------ Shared Memory Limits --------
max number of segments = 4096
max seg size (kbytes) = 5244928
max total shared memory (kbytes) = 5244944
min seg size (bytes) = 1

And the database is using a one segment that's equal to the SGA size

ipcs -m

------ Shared Memory Segments --------
key        shmid      owner      perms      bytes      nattch     status
0x12982ecc 3112962    oracle    660        5370806272 36

SQL> show parameter sga

NAME                                 TYPE            VALUE
------------------------------------ --------------- ----------
sga_max_size                         big integer     5G
sga_target                           big integer     5G

As it is configured now there's sufficient shared segment to run the database. However it is not the database that's been upgraded that runs out of memory but a dummy database instance created by DBUA for a short period during information gathering phase. Information on this dummy instance is also logged on the trace.log

[OracleHome.initOptions:1248]  executing: startup nomount pfile='/opt/app/oracle/product/11.2.0/std3/dbs/initDBUA4304056.ora'

When the DBUA succeed all the files created for this dummy instance are deleted and there's no trace of this dummy instance whatsoever but if it fails the init file used will remain and could be examined.

cd /opt/app/oracle/product/11.2.0/std3/dbs/  
-rw-r----- 1 oracle oinstall     1536 Jan 15 11:43 orapwDBUA4304056
-rw-r----- 1 oracle oinstall      161 Jan 15 11:43 initDBUA4304056.ora

Looking at the init file the log directory could be identified.

more initDBUA4304056.ora
db_name=DBUA4304
db_unique_name=DBUA4304056
shared_pool_size=128m
cpu_count=1
diagnostic_dest=/opt/app/oracle/product/11.2.0/std3/log
_enable_NUMA_support=false

Within the log directory will be a diagnostic directory structure which will have the alert log and the trace file of the failed dummy instance

cd /opt/app/oracle/product/11.2.0/std3/log
cd diag/rdbms/dbua4304056/DBUA4304056/trace/
ls 
alert_DBUA4304056.log  DBUA4304056_ora_6420.trc

These files will have content similar to following

more DBUA4304056_ora_6420.trc

Switching to regular size pages for segment size 8388608
skgm warning: ENOSPC creating segment of size 0000000000800000
fix shm parameters in /etc/system or equivalent

more alert_DBUA4304056.log
Fri Jan 15 11:43:15 2016
The value of parameter shared_pool_size is below the required minimum
It has been reset to the minimum value
Starting ORACLE instance (normal)

As seen on the trace file, the dummy segment is trying acquire a segment of size 8388608. But for the DBUA to be successful it need to create at least 2 other segments. Following segment list captured during a successful DBUA run

ipcs -m

------ Shared Memory Segments --------
key        shmid      owner      perms      bytes      nattch     status
0x12982ecc 3112962    oracle    660        5370806272 37                  <-- DB for upgrade
0x00000000 4423684    oracle    777        13160      2          dest     <-- DBUA itself
0x00000000 4882437    oracle    640        8388608    1                   <-- Dummy instance
0x00000000 4915206    oracle    640        230686720  1                   <-- Dummy instance
0xbcf2a6a8 4947975    oracle    640        2097152    1                   <-- Dummy instance

Dummy instance trace file show this information

*** 2016-01-15 12:10:27.997
Switching to regular size pages for segment size 8388608
Switching to regular size pages for segment size 230686720
Switching to regular size pages for segment size 2097152
CELL communication is configured to use 0 interface(s):
kcrfwy: minimum sleep is 1922 usecs (overhead is 922 usecs)
Running with 1 strand for Non-Enterprise Edition
Running without dynamic strands for Non-Enterprise Edition
kgfmIsAppliance=0 (due to init)

The dummy instance's alert log shows the SGA size of it

****************** Large Pages Information *****************

Total Shared Global Region in Large Pages = 0 KB (0%)

Large Pages used by this instance: 0 (0 KB)
Large Pages unused system wide = 0 (0 KB) (alloc incr 4096 KB)
Large Pages configured system wide = 0 (0 KB)
Large Page size = 2048 KB

RECOMMENDATION:
  Total Shared Global Region size is 230 MB. For optimal performance,
  prior to the next instance restart increase the number
  of unused Large Pages by atleast 115 2048 KB Large Pages (230 MB)
  system wide to get 100% of the Shared
  Global Region allocated with Large pages
***********************************************************
...
Starting up:
Oracle Database 11g Release 11.2.0.3.0 - 64bit Production.
ORACLE_HOME = /opt/app/oracle/product/11.2.0/std3
Using parameter settings in client-side pfile /opt/app/oracle/product/11.2.0/std3/dbs/initDBUA0520596.ora on machine hpc1.domain.net
System parameters with non-default values:
  cpu_count                = 1
  _enable_NUMA_support     = FALSE
  shared_pool_size         = 148M
  db_name                  = "DBUA0520"
  db_unique_name           = "DBUA0520596"
  diagnostic_dest          = "/opt/app/oracle/product/11.2.0/std3/log"

With the increase demand for the shared memory segments the minimum kernel.shmall value must be at least 1370116 (theoretical value. The sum of all the segments size listed above / PAGE_SIZE). However it should be slightly larger than that in practice, in this case kernel.shmall had to be set for 1370123 before DBUA worked.
Point to remember is that if DBUA fails with above error then it's not the database being upgraded that ran out of memory. It is also good idea to set kernel.shmall to a higher value than SGA size.

Useful metalink notes
11.2.0.3 DBUA Fails : ORA-27102: Out Of Memory, "SP2-0640: Not Connected" [ID 1526424.1]
DBUA Fails With Error "SP2-0640 Not Connected" [ID 1384814.1]
Upon startup of Linux database get ORA-27102: out of memory Linux-X86_64 Error: 28: No space left on device [ID 301830.1]

Changing The Cluster Name

This post shows the steps for renaming the cluster. The cluster consists of two nodes and current resources are

$ crsctl stat res -t
--------------------------------------------------------------------------------
Name           Target  State        Server                   State details
--------------------------------------------------------------------------------
Local Resources
--------------------------------------------------------------------------------
ora.CLUSFS.dg
               ONLINE  ONLINE       rhel12c1                 STABLE
               ONLINE  ONLINE       rhel12c2                 STABLE
ora.DATA.dg
               ONLINE  ONLINE       rhel12c1                 STABLE
               ONLINE  ONLINE       rhel12c2                 STABLE
ora.FRA.dg
               ONLINE  ONLINE       rhel12c1                 STABLE
               ONLINE  ONLINE       rhel12c2                 STABLE
ora.LISTENER.lsnr
               ONLINE  ONLINE       rhel12c1                 STABLE
               ONLINE  ONLINE       rhel12c2                 STABLE
ora.asm
               ONLINE  ONLINE       rhel12c1                 Started,STABLE
               ONLINE  ONLINE       rhel12c2                 Started,STABLE
ora.net1.network
               ONLINE  ONLINE       rhel12c1                 STABLE
               ONLINE  ONLINE       rhel12c2                 STABLE
ora.ons
               ONLINE  ONLINE       rhel12c1                 STABLE
               ONLINE  ONLINE       rhel12c2                 STABLE
--------------------------------------------------------------------------------
Cluster Resources
--------------------------------------------------------------------------------
ora.LISTENER_SCAN1.lsnr
      1        ONLINE  ONLINE       rhel12c2                 STABLE
ora.LISTENER_SCAN2.lsnr
      1        ONLINE  ONLINE       rhel12c2                 STABLE
ora.LISTENER_SCAN3.lsnr
      1        ONLINE  ONLINE       rhel12c1                 STABLE
ora.MGMTLSNR
      1        ONLINE  ONLINE       rhel12c1                 169.254.75.103 192.1
                                                             68.1.87,STABLE
ora.cvu
      1        ONLINE  ONLINE       rhel12c2                 STABLE
ora.gns
      1        ONLINE  ONLINE       rhel12c2                 STABLE
ora.gns.vip
      1        ONLINE  ONLINE       rhel12c2                 STABLE
ora.mgmtdb
      1        ONLINE  ONLINE       rhel12c1                 Open,STABLE
ora.oc4j
      1        ONLINE  ONLINE       rhel12c2                 STABLE
ora.rhel12c1.vip
      1        ONLINE  ONLINE       rhel12c1                 STABLE
ora.rhel12c2.vip
      1        ONLINE  ONLINE       rhel12c2                 STABLE
ora.scan1.vip
      1        ONLINE  ONLINE       rhel12c2                 STABLE
ora.scan2.vip
      1        ONLINE  ONLINE       rhel12c2                 STABLE
ora.scan3.vip
      1        ONLINE  ONLINE       rhel12c1                 STABLE
ora.std12c1.db
      1        ONLINE  ONLINE       rhel12c1                 Open,STABLE
      2        ONLINE  ONLINE       rhel12c2                 Open,STABLE
--------------------------------------------------------------------------------

The cluster name is

$ cemutlo -n
rhel12c-cluster

This will be renamed to prod-cluster.
Renaming the cluster involves re-configuring the current cluster setup. To deconfigure the use rootcrs.pl -deconfig options. This is run on all remotes as root

[root@rhel12c2 grid]# $ORACLE_HOME/crs/install/rootcrs.pl -deconfig -force -verbose
Using configuration parameter file: /opt/app/12.1.0/grid2/crs/install/crsconfig_params
Network 1 exists
Subnet IPv4: 192.168.0.0/255.255.255.0/eth0, dhcp
Subnet IPv6:
Ping Targets:
Network is enabled
Network is individually enabled on nodes:
Network is individually disabled on nodes:
VIP exists: network number 1, hosting node rhel12c1
VIP IPv4 Address: -/rhel12c1-vip/192.168.0.90
VIP IPv6 Address:
VIP is enabled.
VIP is individually enabled on nodes:
VIP is individually disabled on nodes:
VIP exists: network number 1, hosting node rhel12c2
VIP IPv4 Address: -/rhel12c2-vip/192.168.0.91
VIP IPv6 Address:
VIP is enabled.
VIP is individually enabled on nodes:
VIP is individually disabled on nodes:
ONS exists: Local port 6100, remote port 6200, EM port 2016, Uses SSL false
ONS is enabled
ONS is individually enabled on nodes:
ONS is individually disabled on nodes:
CRS-2791: Starting shutdown of Oracle High Availability Services-managed resources on 'rhel12c2'
CRS-2673: Attempting to stop 'ora.crsd' on 'rhel12c2'
CRS-2790: Starting shutdown of Cluster Ready Services-managed resources on 'rhel12c2'
CRS-2673: Attempting to stop 'ora.std12c1.db' on 'rhel12c2'
CRS-2673: Attempting to stop 'ora.CLUSFS.dg' on 'rhel12c2'
CRS-2673: Attempting to stop 'ora.oc4j' on 'rhel12c2'
CRS-2677: Stop of 'ora.CLUSFS.dg' on 'rhel12c2' succeeded
CRS-2677: Stop of 'ora.std12c1.db' on 'rhel12c2' succeeded
CRS-2673: Attempting to stop 'ora.FRA.dg' on 'rhel12c2'
CRS-2673: Attempting to stop 'ora.DATA.dg' on 'rhel12c2'
CRS-2677: Stop of 'ora.DATA.dg' on 'rhel12c2' succeeded
CRS-2677: Stop of 'ora.FRA.dg' on 'rhel12c2' succeeded
CRS-2673: Attempting to stop 'ora.asm' on 'rhel12c2'
CRS-2677: Stop of 'ora.asm' on 'rhel12c2' succeeded
CRS-2677: Stop of 'ora.oc4j' on 'rhel12c2' succeeded
CRS-2672: Attempting to start 'ora.oc4j' on 'rhel12c1'
CRS-2676: Start of 'ora.oc4j' on 'rhel12c1' succeeded
CRS-2792: Shutdown of Cluster Ready Services-managed resources on 'rhel12c2' has completed
CRS-2677: Stop of 'ora.crsd' on 'rhel12c2' succeeded
CRS-2673: Attempting to stop 'ora.storage' on 'rhel12c2'
CRS-2673: Attempting to stop 'ora.mdnsd' on 'rhel12c2'
CRS-2673: Attempting to stop 'ora.gpnpd' on 'rhel12c2'
CRS-2673: Attempting to stop 'ora.drivers.acfs' on 'rhel12c2'
CRS-2677: Stop of 'ora.drivers.acfs' on 'rhel12c2' succeeded
CRS-2677: Stop of 'ora.storage' on 'rhel12c2' succeeded
CRS-2673: Attempting to stop 'ora.crf' on 'rhel12c2'
CRS-2673: Attempting to stop 'ora.ctssd' on 'rhel12c2'
CRS-2673: Attempting to stop 'ora.evmd' on 'rhel12c2'
CRS-2673: Attempting to stop 'ora.asm' on 'rhel12c2'
CRS-2677: Stop of 'ora.ctssd' on 'rhel12c2' succeeded
CRS-2677: Stop of 'ora.crf' on 'rhel12c2' succeeded
CRS-2677: Stop of 'ora.evmd' on 'rhel12c2' succeeded
CRS-2677: Stop of 'ora.gpnpd' on 'rhel12c2' succeeded
CRS-2677: Stop of 'ora.mdnsd' on 'rhel12c2' succeeded
CRS-2677: Stop of 'ora.asm' on 'rhel12c2' succeeded
CRS-2673: Attempting to stop 'ora.cluster_interconnect.haip' on 'rhel12c2'
CRS-2677: Stop of 'ora.cluster_interconnect.haip' on 'rhel12c2' succeeded
CRS-2673: Attempting to stop 'ora.cssd' on 'rhel12c2'
CRS-2677: Stop of 'ora.cssd' on 'rhel12c2' succeeded
CRS-2673: Attempting to stop 'ora.gipcd' on 'rhel12c2'
CRS-2677: Stop of 'ora.gipcd' on 'rhel12c2' succeeded
CRS-2793: Shutdown of Oracle High Availability Services-managed resources on 'rhel12c2' has completed
CRS-4133: Oracle High Availability Services has been stopped.
2015/10/30 14:05:37 CLSRSC-4006: Removing Oracle Trace File Analyzer (TFA) Collector.

2015/10/30 14:06:12 CLSRSC-4007: Successfully removed Oracle Trace File Analyzer (TFA) Collector.

2015/10/30 14:06:13 CLSRSC-336: Successfully deconfigured Oracle Clusterware stack on this node

Once all the remote nodes are deconfigured, run the same with -lastnode option on the local node.

[root@rhel12c1 grid2]# $ORACLE_HOME/crs/install/rootcrs.pl -deconfig -force -verbose -lastnode
Using configuration parameter file: /opt/app/12.1.0/grid2/crs/install/crsconfig_params
2015/10/30 14:07:10 CLSRSC-332: CRS resources for listeners are still configured

Network 1 exists
Subnet IPv4: 192.168.0.0/255.255.255.0/eth0, dhcp
Subnet IPv6:
Ping Targets:
Network is enabled
Network is individually enabled on nodes:
Network is individually disabled on nodes:
VIP exists: network number 1, hosting node rhel12c1
VIP IPv4 Address: -/rhel12c1-vip/192.168.0.90
VIP IPv6 Address:
VIP is enabled.
VIP is individually enabled on nodes:
VIP is individually disabled on nodes:
ONS exists: Local port 6100, remote port 6200, EM port 2016, Uses SSL false
ONS is enabled
ONS is individually enabled on nodes:
ONS is individually disabled on nodes:
CRS-2791: Starting shutdown of Oracle High Availability Services-managed resources on 'rhel12c1'
CRS-2673: Attempting to stop 'ora.crsd' on 'rhel12c1'
CRS-2790: Starting shutdown of Cluster Ready Services-managed resources on 'rhel12c1'
CRS-2673: Attempting to stop 'ora.mgmtdb' on 'rhel12c1'
CRS-2673: Attempting to stop 'ora.std12c1.db' on 'rhel12c1'
CRS-2677: Stop of 'ora.std12c1.db' on 'rhel12c1' succeeded
CRS-2673: Attempting to stop 'ora.DATA.dg' on 'rhel12c1'
CRS-2677: Stop of 'ora.mgmtdb' on 'rhel12c1' succeeded
CRS-2673: Attempting to stop 'ora.MGMTLSNR' on 'rhel12c1'
CRS-2677: Stop of 'ora.MGMTLSNR' on 'rhel12c1' succeeded
CRS-2677: Stop of 'ora.DATA.dg' on 'rhel12c1' succeeded
CRS-2673: Attempting to stop 'ora.CLUSFS.dg' on 'rhel12c1'
CRS-2673: Attempting to stop 'ora.FRA.dg' on 'rhel12c1'
CRS-2677: Stop of 'ora.CLUSFS.dg' on 'rhel12c1' succeeded
CRS-2677: Stop of 'ora.FRA.dg' on 'rhel12c1' succeeded
CRS-2673: Attempting to stop 'ora.asm' on 'rhel12c1'
CRS-2677: Stop of 'ora.asm' on 'rhel12c1' succeeded
CRS-2792: Shutdown of Cluster Ready Services-managed resources on 'rhel12c1' has completed
CRS-2677: Stop of 'ora.crsd' on 'rhel12c1' succeeded
CRS-2673: Attempting to stop 'ora.storage' on 'rhel12c1'
CRS-2673: Attempting to stop 'ora.mdnsd' on 'rhel12c1'
CRS-2673: Attempting to stop 'ora.gpnpd' on 'rhel12c1'
CRS-2673: Attempting to stop 'ora.drivers.acfs' on 'rhel12c1'
CRS-2677: Stop of 'ora.storage' on 'rhel12c1' succeeded
CRS-2677: Stop of 'ora.drivers.acfs' on 'rhel12c1' succeeded
CRS-2673: Attempting to stop 'ora.crf' on 'rhel12c1'
CRS-2673: Attempting to stop 'ora.ctssd' on 'rhel12c1'
CRS-2673: Attempting to stop 'ora.evmd' on 'rhel12c1'
CRS-2673: Attempting to stop 'ora.asm' on 'rhel12c1'
CRS-2677: Stop of 'ora.crf' on 'rhel12c1' succeeded
CRS-2677: Stop of 'ora.ctssd' on 'rhel12c1' succeeded
CRS-2677: Stop of 'ora.evmd' on 'rhel12c1' succeeded
CRS-2677: Stop of 'ora.gpnpd' on 'rhel12c1' succeeded
CRS-2677: Stop of 'ora.mdnsd' on 'rhel12c1' succeeded
CRS-2677: Stop of 'ora.asm' on 'rhel12c1' succeeded
CRS-2673: Attempting to stop 'ora.cluster_interconnect.haip' on 'rhel12c1'
CRS-2677: Stop of 'ora.cluster_interconnect.haip' on 'rhel12c1' succeeded
CRS-2673: Attempting to stop 'ora.cssd' on 'rhel12c1'
CRS-2677: Stop of 'ora.cssd' on 'rhel12c1' succeeded
CRS-2673: Attempting to stop 'ora.gipcd' on 'rhel12c1'
CRS-2677: Stop of 'ora.gipcd' on 'rhel12c1' succeeded
CRS-2793: Shutdown of Oracle High Availability Services-managed resources on 'rhel12c1' has completed
CRS-4133: Oracle High Availability Services has been stopped.
CRS-4123: Oracle High Availability Services has been started.
CRS-2672: Attempting to start 'ora.evmd' on 'rhel12c1'
CRS-2672: Attempting to start 'ora.mdnsd' on 'rhel12c1'
CRS-2676: Start of 'ora.mdnsd' on 'rhel12c1' succeeded
CRS-2676: Start of 'ora.evmd' on 'rhel12c1' succeeded
CRS-2672: Attempting to start 'ora.gpnpd' on 'rhel12c1'
CRS-2676: Start of 'ora.gpnpd' on 'rhel12c1' succeeded
CRS-2672: Attempting to start 'ora.cssdmonitor' on 'rhel12c1'
CRS-2672: Attempting to start 'ora.gipcd' on 'rhel12c1'
CRS-2676: Start of 'ora.cssdmonitor' on 'rhel12c1' succeeded
CRS-2676: Start of 'ora.gipcd' on 'rhel12c1' succeeded
CRS-2672: Attempting to start 'ora.cssd' on 'rhel12c1'
CRS-2672: Attempting to start 'ora.diskmon' on 'rhel12c1'
CRS-2676: Start of 'ora.diskmon' on 'rhel12c1' succeeded
CRS-2676: Start of 'ora.cssd' on 'rhel12c1' succeeded
ASM de-configuration trace file location: /tmp/asmcadc_clean2015-10-30_02-15-19-PM.log
ASM Clean Configuration START
ASM Clean Configuration END

ASM with SID +ASM1 deleted successfully. Check /tmp/asmcadc_clean2015-10-30_02-15-19-PM.log for details.

CRS-2791: Starting shutdown of Oracle High Availability Services-managed resources on 'rhel12c1'
CRS-2673: Attempting to stop 'ora.mdnsd' on 'rhel12c1'
CRS-2673: Attempting to stop 'ora.gpnpd' on 'rhel12c1'
CRS-2673: Attempting to stop 'ora.ctssd' on 'rhel12c1'
CRS-2673: Attempting to stop 'ora.evmd' on 'rhel12c1'
CRS-2673: Attempting to stop 'ora.cluster_interconnect.haip' on 'rhel12c1'
CRS-2673: Attempting to stop 'ora.drivers.acfs' on 'rhel12c1'
CRS-2677: Stop of 'ora.drivers.acfs' on 'rhel12c1' succeeded
CRS-2677: Stop of 'ora.cluster_interconnect.haip' on 'rhel12c1' succeeded
CRS-2677: Stop of 'ora.mdnsd' on 'rhel12c1' succeeded
CRS-2677: Stop of 'ora.gpnpd' on 'rhel12c1' succeeded
CRS-2677: Stop of 'ora.ctssd' on 'rhel12c1' succeeded
CRS-2677: Stop of 'ora.evmd' on 'rhel12c1' succeeded
CRS-2673: Attempting to stop 'ora.cssd' on 'rhel12c1'
CRS-2677: Stop of 'ora.cssd' on 'rhel12c1' succeeded
CRS-2673: Attempting to stop 'ora.gipcd' on 'rhel12c1'
CRS-2677: Stop of 'ora.gipcd' on 'rhel12c1' succeeded
CRS-2793: Shutdown of Oracle High Availability Services-managed resources on 'rhel12c1' has completed
CRS-4133: Oracle High Availability Services has been stopped.
2015/10/30 14:28:04 CLSRSC-4006: Removing Oracle Trace File Analyzer (TFA) Collector.

2015/10/30 14:28:25 CLSRSC-4007: Successfully removed Oracle Trace File Analyzer (TFA) Collector.

2015/10/30 14:28:26 CLSRSC-336: Successfully deconfigured Oracle Clusterware stack on this node

2015/10/30 14:28:26 CLSRSC-559: Ensure that the GPnP profile data under the 'gpnp' directory in /opt/app/12.1.0/grid2 is deleted on each node before using the software in the current Grid Infrastructure home for reconfiguration.

To configure the cluster again with desired cluster name run the following command as grid user

$ $GI_HOME/crs/config/config.sh

This opens the OUI. Select configure cluster option.

As this is a 12c cluster setup it could be either setup as a flex cluster or standard cluster.

Specify the new cluster name

This OUI will detect the data base related disk groups (Data and Flash) but the ASM diskgroup that contained the ocr/vote disk and mgmtdb is no longer there.

Create a new diskgroup to store the ocr and vote disks.

At the last step select install to begin the configuration.

During the configuration (install) root.sh execution will be prompted. However execution of root.sh failed with the following error

# /opt/app/12.1.0/grid2/root.sh
Performing root user operation.

The following environment variables are set as:
    ORACLE_OWNER= grid
    ORACLE_HOME=  /opt/app/12.1.0/grid2

Enter the full pathname of the local bin directory: [/usr/local/bin]:
The contents of "dbhome" have not changed. No need to overwrite.
The contents of "oraenv" have not changed. No need to overwrite.
The contents of "coraenv" have not changed. No need to overwrite.

Entries will be added to the /etc/oratab file as needed by
Database Configuration Assistant when a database is created
Finished running generic part of root script.
Now product-specific root actions will be performed.
Relinking oracle with rac_on option
Using configuration parameter file: /opt/app/12.1.0/grid2/crs/install/crsconfig_params
2015/10/30 14:44:46 CLSRSC-4001: Installing Oracle Trace File Analyzer (TFA) Collector.

2015/10/30 14:45:24 CLSRSC-4002: Successfully installed Oracle Trace File Analyzer (TFA) Collector.

2015/10/30 14:45:26 CLSRSC-363: User ignored prerequisites during installation

OLR initialization - successful
  root wallet
  root wallet cert
  root cert export
  peer wallet
  profile reader wallet
  pa wallet
  peer wallet keys
  pa wallet keys
  peer cert request
  pa cert request
  peer cert
  pa cert
  peer root cert TP
  profile reader root cert TP
  pa root cert TP
  peer pa cert TP
  pa peer cert TP
  profile reader pa cert TP
  profile reader peer cert TP
  peer user cert
  pa user cert
2015/10/30 14:46:31 CLSRSC-330: Adding Clusterware entries to file 'oracle-ohasd.conf'

CRS-4133: Oracle High Availability Services has been stopped.
CRS-4123: Oracle High Availability Services has been started.
CRS-4133: Oracle High Availability Services has been stopped.
CRS-4123: Oracle High Availability Services has been started.
CRS-2672: Attempting to start 'ora.evmd' on 'rhel12c1'
CRS-2672: Attempting to start 'ora.mdnsd' on 'rhel12c1'
CRS-2676: Start of 'ora.evmd' on 'rhel12c1' succeeded
CRS-2676: Start of 'ora.mdnsd' on 'rhel12c1' succeeded
CRS-2672: Attempting to start 'ora.gpnpd' on 'rhel12c1'
CRS-2676: Start of 'ora.gpnpd' on 'rhel12c1' succeeded
CRS-2672: Attempting to start 'ora.cssdmonitor' on 'rhel12c1'
CRS-2672: Attempting to start 'ora.gipcd' on 'rhel12c1'
CRS-2676: Start of 'ora.cssdmonitor' on 'rhel12c1' succeeded
CRS-2676: Start of 'ora.gipcd' on 'rhel12c1' succeeded
CRS-2672: Attempting to start 'ora.cssd' on 'rhel12c1'
CRS-2672: Attempting to start 'ora.diskmon' on 'rhel12c1'
CRS-2676: Start of 'ora.diskmon' on 'rhel12c1' succeeded
CRS-2676: Start of 'ora.cssd' on 'rhel12c1' succeeded

Disk Group CLUSFS creation failed with the following message:
ORA-15018: diskgroup cannot be created
ORA-15031: disk specification '/dev/oracleasm/ocr3' matches no disks
ORA-15014: path '/dev/oracleasm/ocr3' is not in the discovery set
ORA-15031: disk specification '/dev/oracleasm/ocr2' matches no disks
ORA-15014: path '/dev/oracleasm/ocr2' is not in the discovery set
ORA-15031: disk specification '/dev/oracleasm/ocr1' matches no disks
ORA-15014: path '/dev/oracleasm/ocr1' is not in the discovery set


2015/10/30 14:49:43 CLSRSC-184: Configuration of ASM failed

2015/10/30 14:49:45 CLSRSC-258: Failed to configure and start ASM

Died at /opt/app/12.1.0/grid2/crs/install/crsinstall.pm line 2017.
The command '/opt/app/12.1.0/grid2/perl/bin/perl -I/opt/app/12.1.0/grid2/perl/lib -I/opt/app/12.1.0/grid2/crs/install /opt/app/12.1.0/grid2/crs/install/rootcrs.pl ' execution failed

According to 431013.1 clearing the underlying disk would resolve this issue.

[root@rhel12c1 ~]# dd if=/dev/zero of=/dev/oracleasm/ocr3 bs=8192 count=1000
[root@rhel12c1 ~]# dd if=/dev/zero of=/dev/oracleasm/ocr2 bs=8192 count=1000
[root@rhel12c1 ~]# dd if=/dev/zero of=/dev/oracleasm/ocr1 bs=8192 count=1000

but this did not always work. The real reason seem to be the missing asm_diskstring. When the root.sh fail it leaves a empty asm instance (without any disk groups). Trying to create the same disk group fails with the same error

SQL> CREATE DISKGROUP CLUSFS NORMAL REDUNDANCY  DISK '/dev/oracleasm/ocr1',
'/dev/oracleasm/ocr2',
'/dev/oracleasm/ocr3' ATTRIBUTE 'compatible.asm'='12.1.0.0.0','au_size'='4M'  2    3  ;
CREATE DISKGROUP CLUSFS NORMAL REDUNDANCY  DISK '/dev/oracleasm/ocr1',
*
ERROR at line 1:
ORA-15018: diskgroup cannot be created
ORA-15031: disk specification '/dev/oracleasm/ocr3' matches no disks
ORA-15014: path '/dev/oracleasm/ocr3' is not in the discovery set
ORA-15031: disk specification '/dev/oracleasm/ocr2' matches no disks
ORA-15014: path '/dev/oracleasm/ocr2' is not in the discovery set
ORA-15031: disk specification '/dev/oracleasm/ocr1' matches no disks
ORA-15014: path '/dev/oracleasm/ocr1' is not in the discovery set

The asm_diskstring is empty

SQL> show parameter string

NAME                                 TYPE        VALUE
------------------------------------ ----------- ------------------------------
asm_diskstring                       string

However if asm_disktring was updated then diskgroup gets created without a problem

SQL>  alter system set asm_diskstring='/dev/oracleasm/*';

System altered.

SQL> CREATE DISKGROUP CLUSFS NORMAL REDUNDANCY  DISK '/dev/oracleasm/ocr1',
'/dev/oracleasm/ocr2',
'/dev/oracleasm/ocr3' ATTRIBUTE 'compatible.asm'='12.1.0.0.0','au_size'='4M'  2    3  ;

Diskgroup created.

So it seems the missing asm_diskstring is the cause of the problem. To fix this the $GI_HOME/crs/install/crsconfig_params file was updated with the asm_diskstring

ASM_DISCOVERY_STRING=/dev/oracleasm/*

Afterwards the root.sh was rerun and there was no issue.

[root@rhel12c1 grid]# /opt/app/12.1.0/grid2/root.sh
Performing root user operation.

The following environment variables are set as:
    ORACLE_OWNER= grid
    ORACLE_HOME=  /opt/app/12.1.0/grid2

Enter the full pathname of the local bin directory: [/usr/local/bin]:
The contents of "dbhome" have not changed. No need to overwrite.
The contents of "oraenv" have not changed. No need to overwrite.
The contents of "coraenv" have not changed. No need to overwrite.

Entries will be added to the /etc/oratab file as needed by
Database Configuration Assistant when a database is created
Finished running generic part of root script.
Now product-specific root actions will be performed.
Relinking oracle with rac_on option
Using configuration parameter file: /opt/app/12.1.0/grid2/crs/install/crsconfig_params
2015/10/30 16:06:21 CLSRSC-4001: Installing Oracle Trace File Analyzer (TFA) Collector.

2015/10/30 16:06:21 CLSRSC-4002: Successfully installed Oracle Trace File Analyzer (TFA) Collector.

2015/10/30 16:06:24 CLSRSC-363: User ignored prerequisites during installation

CRS-2791: Starting shutdown of Oracle High Availability Services-managed resources on 'rhel12c1'
CRS-2673: Attempting to stop 'ora.ctssd' on 'rhel12c1'
CRS-2673: Attempting to stop 'ora.evmd' on 'rhel12c1'
CRS-2673: Attempting to stop 'ora.asm' on 'rhel12c1'
CRS-2673: Attempting to stop 'ora.mdnsd' on 'rhel12c1'
CRS-2673: Attempting to stop 'ora.gpnpd' on 'rhel12c1'
CRS-2673: Attempting to stop 'ora.drivers.acfs' on 'rhel12c1'
CRS-2677: Stop of 'ora.drivers.acfs' on 'rhel12c1' succeeded
CRS-2677: Stop of 'ora.ctssd' on 'rhel12c1' succeeded
CRS-2677: Stop of 'ora.mdnsd' on 'rhel12c1' succeeded
CRS-2677: Stop of 'ora.gpnpd' on 'rhel12c1' succeeded
CRS-2677: Stop of 'ora.evmd' on 'rhel12c1' succeeded
CRS-2677: Stop of 'ora.asm' on 'rhel12c1' succeeded
CRS-2673: Attempting to stop 'ora.cluster_interconnect.haip' on 'rhel12c1'
CRS-2677: Stop of 'ora.cluster_interconnect.haip' on 'rhel12c1' succeeded
CRS-2673: Attempting to stop 'ora.cssd' on 'rhel12c1'
CRS-2677: Stop of 'ora.cssd' on 'rhel12c1' succeeded
CRS-2673: Attempting to stop 'ora.gipcd' on 'rhel12c1'
CRS-2677: Stop of 'ora.gipcd' on 'rhel12c1' succeeded
CRS-2793: Shutdown of Oracle High Availability Services-managed resources on 'rhel12c1' has completed
CRS-4133: Oracle High Availability Services has been stopped.
CRS-4123: Oracle High Availability Services has been started.


CRS-2672: Attempting to start 'ora.evmd' on 'rhel12c1'
CRS-2672: Attempting to start 'ora.mdnsd' on 'rhel12c1'
CRS-2676: Start of 'ora.evmd' on 'rhel12c1' succeeded
CRS-2676: Start of 'ora.mdnsd' on 'rhel12c1' succeeded
CRS-2672: Attempting to start 'ora.gpnpd' on 'rhel12c1'
CRS-2676: Start of 'ora.gpnpd' on 'rhel12c1' succeeded
CRS-2672: Attempting to start 'ora.cssdmonitor' on 'rhel12c1'
CRS-2672: Attempting to start 'ora.gipcd' on 'rhel12c1'
CRS-2676: Start of 'ora.cssdmonitor' on 'rhel12c1' succeeded
CRS-2676: Start of 'ora.gipcd' on 'rhel12c1' succeeded
CRS-2672: Attempting to start 'ora.cssd' on 'rhel12c1'
CRS-2672: Attempting to start 'ora.diskmon' on 'rhel12c1'
CRS-2676: Start of 'ora.diskmon' on 'rhel12c1' succeeded
CRS-2676: Start of 'ora.cssd' on 'rhel12c1' succeeded
CRS-2672: Attempting to start 'ora.cluster_interconnect.haip' on 'rhel12c1'
CRS-2672: Attempting to start 'ora.ctssd' on 'rhel12c1'
CRS-2676: Start of 'ora.ctssd' on 'rhel12c1' succeeded
CRS-2676: Start of 'ora.cluster_interconnect.haip' on 'rhel12c1' succeeded
CRS-2672: Attempting to start 'ora.asm' on 'rhel12c1'
CRS-2676: Start of 'ora.asm' on 'rhel12c1' succeeded
CRS-2672: Attempting to start 'ora.storage' on 'rhel12c1'
diskgroup CLUSFS not mounted ()
CRS-5017: The resource action "ora.storage start" encountered the following error:
Storage agent start action aborted. For details refer to "(:CLSN00107:)" in "/opt/app/oracle/diag/crs/rhel12c1/crs/trace/ohasd_orarootagent_root.trc".
CRS-2674: Start of 'ora.storage' on 'rhel12c1' failed
CRS-2679: Attempting to clean 'ora.storage' on 'rhel12c1'
CRS-2681: Clean of 'ora.storage' on 'rhel12c1' succeeded

ASM created and started successfully.

Disk Group CLUSFS created successfully.

CRS-2672: Attempting to start 'ora.crf' on 'rhel12c1'
CRS-2672: Attempting to start 'ora.storage' on 'rhel12c1'
CRS-2676: Start of 'ora.storage' on 'rhel12c1' succeeded
CRS-2676: Start of 'ora.crf' on 'rhel12c1' succeeded
CRS-2672: Attempting to start 'ora.crsd' on 'rhel12c1'
CRS-2676: Start of 'ora.crsd' on 'rhel12c1' succeeded
CRS-4256: Updating the profile
Successful addition of voting disk 92718a80064a4fdebf30cdd4ed314115.
Successful addition of voting disk 48f9d1873a394f4dbf63ce93cf8b8b03.
Successful addition of voting disk a96f1382267b4fbdbf084b7595fd3643.
Successfully replaced voting disk group with +CLUSFS.
CRS-4256: Updating the profile
CRS-4266: Voting file(s) successfully replaced
##  STATE    File Universal Id                File Name Disk group
--  -----    -----------------                --------- ---------
 1. ONLINE   92718a80064a4fdebf30cdd4ed314115 (/dev/oracleasm/ocr1) [CLUSFS]
 2. ONLINE   48f9d1873a394f4dbf63ce93cf8b8b03 (/dev/oracleasm/ocr2) [CLUSFS]
 3. ONLINE   a96f1382267b4fbdbf084b7595fd3643 (/dev/oracleasm/ocr3) [CLUSFS]
Located 3 voting disk(s).
CRS-2791: Starting shutdown of Oracle High Availability Services-managed resources on 'rhel12c1'
CRS-2673: Attempting to stop 'ora.crsd' on 'rhel12c1'
CRS-2677: Stop of 'ora.crsd' on 'rhel12c1' succeeded
CRS-2673: Attempting to stop 'ora.storage' on 'rhel12c1'
CRS-2673: Attempting to stop 'ora.gpnpd' on 'rhel12c1'
CRS-2673: Attempting to stop 'ora.drivers.acfs' on 'rhel12c1'
CRS-2673: Attempting to stop 'ora.mdnsd' on 'rhel12c1'
CRS-2677: Stop of 'ora.storage' on 'rhel12c1' succeeded
CRS-2673: Attempting to stop 'ora.asm' on 'rhel12c1'
CRS-2677: Stop of 'ora.drivers.acfs' on 'rhel12c1' succeeded
CRS-2677: Stop of 'ora.gpnpd' on 'rhel12c1' succeeded
CRS-2677: Stop of 'ora.mdnsd' on 'rhel12c1' succeeded
CRS-2677: Stop of 'ora.asm' on 'rhel12c1' succeeded
CRS-2673: Attempting to stop 'ora.cluster_interconnect.haip' on 'rhel12c1'
CRS-2677: Stop of 'ora.cluster_interconnect.haip' on 'rhel12c1' succeeded
CRS-2673: Attempting to stop 'ora.ctssd' on 'rhel12c1'
CRS-2673: Attempting to stop 'ora.evmd' on 'rhel12c1'
CRS-2677: Stop of 'ora.ctssd' on 'rhel12c1' succeeded
CRS-2677: Stop of 'ora.evmd' on 'rhel12c1' succeeded
CRS-2673: Attempting to stop 'ora.cssd' on 'rhel12c1'
CRS-2677: Stop of 'ora.cssd' on 'rhel12c1' succeeded
CRS-2673: Attempting to stop 'ora.crf' on 'rhel12c1'
CRS-2677: Stop of 'ora.crf' on 'rhel12c1' succeeded
CRS-2673: Attempting to stop 'ora.gipcd' on 'rhel12c1'
CRS-2677: Stop of 'ora.gipcd' on 'rhel12c1' succeeded
CRS-2793: Shutdown of Oracle High Availability Services-managed resources on 'rhel12c1' has completed
CRS-4133: Oracle High Availability Services has been stopped.
CRS-4123: Starting Oracle High Availability Services-managed resources
CRS-2672: Attempting to start 'ora.mdnsd' on 'rhel12c1'
CRS-2672: Attempting to start 'ora.evmd' on 'rhel12c1'
CRS-2676: Start of 'ora.mdnsd' on 'rhel12c1' succeeded
CRS-2676: Start of 'ora.evmd' on 'rhel12c1' succeeded
CRS-2672: Attempting to start 'ora.gpnpd' on 'rhel12c1'
CRS-2676: Start of 'ora.gpnpd' on 'rhel12c1' succeeded
CRS-2672: Attempting to start 'ora.gipcd' on 'rhel12c1'
CRS-2676: Start of 'ora.gipcd' on 'rhel12c1' succeeded
CRS-2672: Attempting to start 'ora.cssdmonitor' on 'rhel12c1'
CRS-2676: Start of 'ora.cssdmonitor' on 'rhel12c1' succeeded
CRS-2672: Attempting to start 'ora.cssd' on 'rhel12c1'
CRS-2672: Attempting to start 'ora.diskmon' on 'rhel12c1'
CRS-2676: Start of 'ora.diskmon' on 'rhel12c1' succeeded
CRS-2676: Start of 'ora.cssd' on 'rhel12c1' succeeded
CRS-2672: Attempting to start 'ora.cluster_interconnect.haip' on 'rhel12c1'
CRS-2672: Attempting to start 'ora.ctssd' on 'rhel12c1'
CRS-2676: Start of 'ora.ctssd' on 'rhel12c1' succeeded
CRS-2676: Start of 'ora.cluster_interconnect.haip' on 'rhel12c1' succeeded
CRS-2672: Attempting to start 'ora.asm' on 'rhel12c1'
CRS-2676: Start of 'ora.asm' on 'rhel12c1' succeeded
CRS-2672: Attempting to start 'ora.storage' on 'rhel12c1'
CRS-2676: Start of 'ora.storage' on 'rhel12c1' succeeded
CRS-2672: Attempting to start 'ora.crf' on 'rhel12c1'
CRS-2676: Start of 'ora.crf' on 'rhel12c1' succeeded
CRS-2672: Attempting to start 'ora.crsd' on 'rhel12c1'
CRS-2676: Start of 'ora.crsd' on 'rhel12c1' succeeded
CRS-6023: Starting Oracle Cluster Ready Services-managed resources
CRS-6017: Processing resource auto-start for servers: rhel12c1
CRS-6016: Resource auto-start has completed for server rhel12c1
CRS-6024: Completed start of Oracle Cluster Ready Services-managed resources
CRS-4123: Oracle High Availability Services has been started.
2015/10/30 16:33:07 CLSRSC-343: Successfully started Oracle Clusterware stack

CRS-2672: Attempting to start 'ora.net1.network' on 'rhel12c1'
CRS-2676: Start of 'ora.net1.network' on 'rhel12c1' succeeded
CRS-2672: Attempting to start 'ora.gns.vip' on 'rhel12c1'
CRS-2676: Start of 'ora.gns.vip' on 'rhel12c1' succeeded
CRS-2672: Attempting to start 'ora.gns' on 'rhel12c1'
CRS-2676: Start of 'ora.gns' on 'rhel12c1' succeeded
CRS-2672: Attempting to start 'ora.ASMNET1LSNR_ASM.lsnr' on 'rhel12c1'
CRS-2676: Start of 'ora.ASMNET1LSNR_ASM.lsnr' on 'rhel12c1' succeeded
CRS-2672: Attempting to start 'ora.asm' on 'rhel12c1'
CRS-2676: Start of 'ora.asm' on 'rhel12c1' succeeded
CRS-2672: Attempting to start 'ora.CLUSFS.dg' on 'rhel12c1'
CRS-2676: Start of 'ora.CLUSFS.dg' on 'rhel12c1' succeeded
CRS-2791: Starting shutdown of Oracle High Availability Services-managed resources on 'rhel12c1'
CRS-2673: Attempting to stop 'ora.crsd' on 'rhel12c1'
CRS-2790: Starting shutdown of Cluster Ready Services-managed resources on 'rhel12c1'
CRS-2673: Attempting to stop 'ora.rhel12c1.vip' on 'rhel12c1'
CRS-2673: Attempting to stop 'ora.CLUSFS.dg' on 'rhel12c1'
CRS-2673: Attempting to stop 'ora.LISTENER_SCAN3.lsnr' on 'rhel12c1'
CRS-2673: Attempting to stop 'ora.oc4j' on 'rhel12c1'
CRS-2673: Attempting to stop 'ora.gns' on 'rhel12c1'
CRS-2673: Attempting to stop 'ora.LISTENER_SCAN2.lsnr' on 'rhel12c1'
CRS-2673: Attempting to stop 'ora.LISTENER_SCAN1.lsnr' on 'rhel12c1'
CRS-2673: Attempting to stop 'ora.cvu' on 'rhel12c1'
CRS-2677: Stop of 'ora.CLUSFS.dg' on 'rhel12c1' succeeded
CRS-2673: Attempting to stop 'ora.asm' on 'rhel12c1'
CRS-2677: Stop of 'ora.cvu' on 'rhel12c1' succeeded
CRS-2677: Stop of 'ora.asm' on 'rhel12c1' succeeded
CRS-2673: Attempting to stop 'ora.ASMNET1LSNR_ASM.lsnr' on 'rhel12c1'
CRS-2677: Stop of 'ora.LISTENER_SCAN3.lsnr' on 'rhel12c1' succeeded
CRS-2673: Attempting to stop 'ora.scan3.vip' on 'rhel12c1'
CRS-2677: Stop of 'ora.LISTENER_SCAN2.lsnr' on 'rhel12c1' succeeded
CRS-2673: Attempting to stop 'ora.scan2.vip' on 'rhel12c1'
CRS-2677: Stop of 'ora.LISTENER_SCAN1.lsnr' on 'rhel12c1' succeeded
CRS-2673: Attempting to stop 'ora.scan1.vip' on 'rhel12c1'
CRS-2677: Stop of 'ora.ASMNET1LSNR_ASM.lsnr' on 'rhel12c1' succeeded
CRS-2677: Stop of 'ora.rhel12c1.vip' on 'rhel12c1' succeeded
CRS-2677: Stop of 'ora.gns' on 'rhel12c1' succeeded
CRS-2673: Attempting to stop 'ora.gns.vip' on 'rhel12c1'
CRS-2677: Stop of 'ora.scan1.vip' on 'rhel12c1' succeeded
CRS-2677: Stop of 'ora.scan2.vip' on 'rhel12c1' succeeded
CRS-2677: Stop of 'ora.scan3.vip' on 'rhel12c1' succeeded
CRS-2677: Stop of 'ora.gns.vip' on 'rhel12c1' succeeded
CRS-2677: Stop of 'ora.oc4j' on 'rhel12c1' succeeded
CRS-2673: Attempting to stop 'ora.ons' on 'rhel12c1'
CRS-2677: Stop of 'ora.ons' on 'rhel12c1' succeeded
CRS-2673: Attempting to stop 'ora.net1.network' on 'rhel12c1'
CRS-2677: Stop of 'ora.net1.network' on 'rhel12c1' succeeded
CRS-2792: Shutdown of Cluster Ready Services-managed resources on 'rhel12c1' has completed
CRS-2677: Stop of 'ora.crsd' on 'rhel12c1' succeeded
CRS-2673: Attempting to stop 'ora.ctssd' on 'rhel12c1'
CRS-2673: Attempting to stop 'ora.evmd' on 'rhel12c1'
CRS-2673: Attempting to stop 'ora.storage' on 'rhel12c1'
CRS-2673: Attempting to stop 'ora.mdnsd' on 'rhel12c1'
CRS-2673: Attempting to stop 'ora.gpnpd' on 'rhel12c1'
CRS-2673: Attempting to stop 'ora.drivers.acfs' on 'rhel12c1'
CRS-2677: Stop of 'ora.storage' on 'rhel12c1' succeeded
CRS-2673: Attempting to stop 'ora.asm' on 'rhel12c1'
CRS-2677: Stop of 'ora.drivers.acfs' on 'rhel12c1' succeeded
CRS-2677: Stop of 'ora.ctssd' on 'rhel12c1' succeeded
CRS-2677: Stop of 'ora.mdnsd' on 'rhel12c1' succeeded
CRS-2677: Stop of 'ora.gpnpd' on 'rhel12c1' succeeded
CRS-2677: Stop of 'ora.evmd' on 'rhel12c1' succeeded
CRS-2677: Stop of 'ora.asm' on 'rhel12c1' succeeded
CRS-2673: Attempting to stop 'ora.cluster_interconnect.haip' on 'rhel12c1'
CRS-2677: Stop of 'ora.cluster_interconnect.haip' on 'rhel12c1' succeeded
CRS-2673: Attempting to stop 'ora.cssd' on 'rhel12c1'
CRS-2677: Stop of 'ora.cssd' on 'rhel12c1' succeeded
CRS-2673: Attempting to stop 'ora.crf' on 'rhel12c1'
CRS-2677: Stop of 'ora.crf' on 'rhel12c1' succeeded
CRS-2673: Attempting to stop 'ora.gipcd' on 'rhel12c1'
CRS-2677: Stop of 'ora.gipcd' on 'rhel12c1' succeeded
CRS-2793: Shutdown of Oracle High Availability Services-managed resources on 'rhel12c1' has completed
CRS-4133: Oracle High Availability Services has been stopped.
CRS-4123: Starting Oracle High Availability Services-managed resources
CRS-2672: Attempting to start 'ora.mdnsd' on 'rhel12c1'
CRS-2672: Attempting to start 'ora.evmd' on 'rhel12c1'
CRS-2676: Start of 'ora.mdnsd' on 'rhel12c1' succeeded
CRS-2676: Start of 'ora.evmd' on 'rhel12c1' succeeded
CRS-2672: Attempting to start 'ora.gpnpd' on 'rhel12c1'
CRS-2676: Start of 'ora.gpnpd' on 'rhel12c1' succeeded
CRS-2672: Attempting to start 'ora.gipcd' on 'rhel12c1'
CRS-2676: Start of 'ora.gipcd' on 'rhel12c1' succeeded
CRS-2672: Attempting to start 'ora.cssdmonitor' on 'rhel12c1'
CRS-2676: Start of 'ora.cssdmonitor' on 'rhel12c1' succeeded
CRS-2672: Attempting to start 'ora.cssd' on 'rhel12c1'
CRS-2672: Attempting to start 'ora.diskmon' on 'rhel12c1'
CRS-2676: Start of 'ora.diskmon' on 'rhel12c1' succeeded
CRS-2676: Start of 'ora.cssd' on 'rhel12c1' succeeded
CRS-2672: Attempting to start 'ora.cluster_interconnect.haip' on 'rhel12c1'
CRS-2672: Attempting to start 'ora.ctssd' on 'rhel12c1'
CRS-2676: Start of 'ora.ctssd' on 'rhel12c1' succeeded
CRS-2676: Start of 'ora.cluster_interconnect.haip' on 'rhel12c1' succeeded
CRS-2672: Attempting to start 'ora.asm' on 'rhel12c1'
CRS-2676: Start of 'ora.asm' on 'rhel12c1' succeeded
CRS-2672: Attempting to start 'ora.storage' on 'rhel12c1'
CRS-2676: Start of 'ora.storage' on 'rhel12c1' succeeded
CRS-2672: Attempting to start 'ora.crf' on 'rhel12c1'
CRS-2676: Start of 'ora.crf' on 'rhel12c1' succeeded
CRS-2672: Attempting to start 'ora.crsd' on 'rhel12c1'
CRS-2676: Start of 'ora.crsd' on 'rhel12c1' succeeded
CRS-6023: Starting Oracle Cluster Ready Services-managed resources
CRS-2664: Resource 'ora.CLUSFS.dg' is already running on 'rhel12c1'
CRS-6017: Processing resource auto-start for servers: rhel12c1
CRS-2672: Attempting to start 'ora.oc4j' on 'rhel12c1'
CRS-2676: Start of 'ora.oc4j' on 'rhel12c1' succeeded
CRS-6016: Resource auto-start has completed for server rhel12c1
CRS-6024: Completed start of Oracle Cluster Ready Services-managed resources
CRS-4123: Oracle High Availability Services has been started.
2015/10/30 16:38:34 CLSRSC-325: Configure Oracle Grid Infrastructure for a Cluster ... succeeded

On the last node

[root@rhel12c2 grid]# /opt/app/12.1.0/grid2/root.sh
Performing root user operation.

The following environment variables are set as:
    ORACLE_OWNER= grid
    ORACLE_HOME=  /opt/app/12.1.0/grid2

Enter the full pathname of the local bin directory: [/usr/local/bin]:
The contents of "dbhome" have not changed. No need to overwrite.
The contents of "oraenv" have not changed. No need to overwrite.
The contents of "coraenv" have not changed. No need to overwrite.

Entries will be added to the /etc/oratab file as needed by
Database Configuration Assistant when a database is created
Finished running generic part of root script.
Now product-specific root actions will be performed.
Relinking oracle with rac_on option
Using configuration parameter file: /opt/app/12.1.0/grid2/crs/install/crsconfig_params
2015/10/30 16:43:33 CLSRSC-4001: Installing Oracle Trace File Analyzer (TFA) Collector.

2015/10/30 16:44:09 CLSRSC-4002: Successfully installed Oracle Trace File Analyzer (TFA) Collector.

2015/10/30 16:44:12 CLSRSC-363: User ignored prerequisites during installation

OLR initialization - successful
2015/10/30 16:45:41 CLSRSC-330: Adding Clusterware entries to file 'oracle-ohasd.conf'

CRS-4133: Oracle High Availability Services has been stopped.
CRS-4123: Oracle High Availability Services has been started.
CRS-4133: Oracle High Availability Services has been stopped.
CRS-4123: Oracle High Availability Services has been started.
CRS-2791: Starting shutdown of Oracle High Availability Services-managed resources on 'rhel12c2'
CRS-2673: Attempting to stop 'ora.drivers.acfs' on 'rhel12c2'
CRS-2677: Stop of 'ora.drivers.acfs' on 'rhel12c2' succeeded
CRS-2793: Shutdown of Oracle High Availability Services-managed resources on 'rhel12c2' has completed
CRS-4133: Oracle High Availability Services has been stopped.
CRS-4123: Starting Oracle High Availability Services-managed resources
CRS-2672: Attempting to start 'ora.mdnsd' on 'rhel12c2'
CRS-2672: Attempting to start 'ora.evmd' on 'rhel12c2'
CRS-2676: Start of 'ora.mdnsd' on 'rhel12c2' succeeded
CRS-2676: Start of 'ora.evmd' on 'rhel12c2' succeeded
CRS-2672: Attempting to start 'ora.gpnpd' on 'rhel12c2'
CRS-2676: Start of 'ora.gpnpd' on 'rhel12c2' succeeded
CRS-2672: Attempting to start 'ora.gipcd' on 'rhel12c2'
CRS-2676: Start of 'ora.gipcd' on 'rhel12c2' succeeded
CRS-2672: Attempting to start 'ora.cssdmonitor' on 'rhel12c2'
CRS-2676: Start of 'ora.cssdmonitor' on 'rhel12c2' succeeded
CRS-2672: Attempting to start 'ora.cssd' on 'rhel12c2'
CRS-2672: Attempting to start 'ora.diskmon' on 'rhel12c2'
CRS-2676: Start of 'ora.diskmon' on 'rhel12c2' succeeded
CRS-2676: Start of 'ora.cssd' on 'rhel12c2' succeeded
CRS-2672: Attempting to start 'ora.cluster_interconnect.haip' on 'rhel12c2'
CRS-2672: Attempting to start 'ora.ctssd' on 'rhel12c2'
CRS-2676: Start of 'ora.ctssd' on 'rhel12c2' succeeded
CRS-2672: Attempting to start 'ora.crf' on 'rhel12c2'
CRS-2676: Start of 'ora.crf' on 'rhel12c2' succeeded
CRS-2672: Attempting to start 'ora.crsd' on 'rhel12c2'
CRS-2676: Start of 'ora.crsd' on 'rhel12c2' succeeded
CRS-2676: Start of 'ora.cluster_interconnect.haip' on 'rhel12c2' succeeded
CRS-2672: Attempting to start 'ora.asm' on 'rhel12c2'
CRS-2676: Start of 'ora.asm' on 'rhel12c2' succeeded
CRS-6017: Processing resource auto-start for servers: rhel12c2
CRS-2672: Attempting to start 'ora.net1.network' on 'rhel12c2'
CRS-2672: Attempting to start 'ora.ASMNET1LSNR_ASM.lsnr' on 'rhel12c2'
CRS-2676: Start of 'ora.net1.network' on 'rhel12c2' succeeded
CRS-2672: Attempting to start 'ora.ons' on 'rhel12c2'
CRS-2676: Start of 'ora.ASMNET1LSNR_ASM.lsnr' on 'rhel12c2' succeeded
CRS-2672: Attempting to start 'ora.asm' on 'rhel12c2'
CRS-2676: Start of 'ora.ons' on 'rhel12c2' succeeded
CRS-2673: Attempting to stop 'ora.LISTENER_SCAN1.lsnr' on 'rhel12c1'
CRS-2677: Stop of 'ora.LISTENER_SCAN1.lsnr' on 'rhel12c1' succeeded
CRS-2673: Attempting to stop 'ora.scan1.vip' on 'rhel12c1'
CRS-2677: Stop of 'ora.scan1.vip' on 'rhel12c1' succeeded
CRS-2672: Attempting to start 'ora.scan1.vip' on 'rhel12c2'
CRS-2676: Start of 'ora.scan1.vip' on 'rhel12c2' succeeded
CRS-2672: Attempting to start 'ora.LISTENER_SCAN1.lsnr' on 'rhel12c2'
CRS-2676: Start of 'ora.LISTENER_SCAN1.lsnr' on 'rhel12c2' succeeded
CRS-2676: Start of 'ora.asm' on 'rhel12c2' succeeded
CRS-6016: Resource auto-start has completed for server rhel12c2
CRS-6024: Completed start of Oracle Cluster Ready Services-managed resources
CRS-4123: Oracle High Availability Services has been started.
2015/10/30 16:51:06 CLSRSC-343: Successfully started Oracle Clusterware stack

2015/10/30 16:51:28 CLSRSC-325: Configure Oracle Grid Infrastructure for a Cluster ... succeeded

At the end of the cluster configuration the cluster stack would be up and running

$ crsctl stat res -t
--------------------------------------------------------------------------------
Name           Target  State        Server                   State details
--------------------------------------------------------------------------------
Local Resources
--------------------------------------------------------------------------------
ora.ASMNET1LSNR_ASM.lsnr
               ONLINE  ONLINE       rhel12c1                 STABLE
               ONLINE  ONLINE       rhel12c2                 STABLE
ora.CLUSFS.dg
               ONLINE  ONLINE       rhel12c1                 STABLE
               ONLINE  ONLINE       rhel12c2                 STABLE
ora.LISTENER.lsnr
               ONLINE  ONLINE       rhel12c1                 STABLE
               ONLINE  ONLINE       rhel12c2                 STABLE
ora.net1.network
               ONLINE  ONLINE       rhel12c1                 STABLE
               ONLINE  ONLINE       rhel12c2                 STABLE
ora.ons
               ONLINE  ONLINE       rhel12c1                 STABLE
               ONLINE  ONLINE       rhel12c2                 STABLE
--------------------------------------------------------------------------------
Cluster Resources
--------------------------------------------------------------------------------
ora.LISTENER_SCAN1.lsnr
      1        ONLINE  ONLINE       rhel12c2                 STABLE
ora.LISTENER_SCAN2.lsnr
      1        ONLINE  ONLINE       rhel12c1                 STABLE
ora.LISTENER_SCAN3.lsnr
      1        ONLINE  ONLINE       rhel12c1                 STABLE
ora.MGMTLSNR
      1        ONLINE  ONLINE       rhel12c1                 169.254.229.238 192.
                                                             168.1.87,STABLE
ora.asm
      1        ONLINE  ONLINE       rhel12c1                 Started,STABLE
      2        ONLINE  ONLINE       rhel12c2                 Started,STABLE
      3        OFFLINE OFFLINE                               STABLE
ora.cvu
      1        ONLINE  ONLINE       rhel12c1                 STABLE
ora.gns
      1        ONLINE  ONLINE       rhel12c1                 STABLE
ora.gns.vip
      1        ONLINE  ONLINE       rhel12c1                 STABLE
ora.mgmtdb
      1        ONLINE  ONLINE       rhel12c1                 Open,STABLE
ora.oc4j
      1        ONLINE  ONLINE       rhel12c1                 STABLE
ora.rhel12c1.vip
      1        ONLINE  ONLINE       rhel12c1                 STABLE
ora.rhel12c2.vip
      1        ONLINE  ONLINE       rhel12c2                 STABLE
ora.scan1.vip
      1        ONLINE  ONLINE       rhel12c2                 STABLE
ora.scan2.vip
      1        ONLINE  ONLINE       rhel12c1                 STABLE
ora.scan3.vip
      1        ONLINE  ONLINE       rhel12c1                 STABLE
--------------------------------------------------------------------------------

with the new cluster name

 cemutlo -n
prod-cluster

One thing notable is that database related asm disk groups and database itself are not part of the cluster yet. For the cluster setup to be same as before the rename the database must be manually added to the cluster. Mount the asm disk groups data and fra

SQL> alter diskgroup fra mount;
Diskgroup altered.

SQL> alter diskgroup data mount ;
Diskgroup altered.

SQL>  select inst_id,name,state from gv$asm_diskgroup;

   INST_ID NAME                           STATE
---------- ------------------------------ -----------
         2 CLUSFS                         MOUNTED
         2 FRA                            MOUNTED
         2 DATA                           MOUNTED
         1 CLUSFS                         MOUNTED
         1 FRA                            MOUNTED
         1 DATA                           MOUNTED

As oracle user add the database to the cluster

[oracle@rhel12c1 ~]$ srvctl add database -db std12c1 -oraclehome $ORACLE_HOME -spfile "+DATA/STD12C1/PARAMETERFILE/spfilestd12c1.ora"
[oracle@rhel12c1 ~]$ srvctl add instance -db std12c1 -instance std12c11 -n rhel12c1
[oracle@rhel12c1 ~]$ srvctl add instance -db std12c1 -instance std12c12 -n rhel12c2
[oracle@rhel12c1 ~]$ srvctl start database -d std12c1

Database configuration is added to the cluster

$ srvctl config database -d std12c1
Database unique name: std12c1
Database name:
Oracle home: /opt/app/oracle/product/12.1.0/dbhome_2
Oracle user: oracle
Spfile: +DATA/STD12C1/PARAMETERFILE/spfilestd12c1.ora
Password file:
Domain:
Start options: open
Stop options: immediate
Database role: PRIMARY
Management policy: AUTOMATIC
Server pools:
Disk Groups: DATA,FRA
Mount point paths:
Services:
Type: RAC
Start concurrency:
Stop concurrency:
OSDBA group: dba
OSOPER group: oper
Database instances: std12c11,std12c12
Configured nodes: rhel12c1,rhel12c2
Database is administrator managed

Number of cluster resources are same as before the rename

$ crsctl stat res -t
--------------------------------------------------------------------------------
Name           Target  State        Server                   State details
--------------------------------------------------------------------------------
Local Resources
--------------------------------------------------------------------------------
ora.ASMNET1LSNR_ASM.lsnr
               ONLINE  ONLINE       rhel12c1                 STABLE
               ONLINE  ONLINE       rhel12c2                 STABLE
ora.CLUSFS.dg
               ONLINE  ONLINE       rhel12c1                 STABLE
               ONLINE  ONLINE       rhel12c2                 STABLE
ora.DATA.dg
               ONLINE  ONLINE       rhel12c1                 STABLE
               ONLINE  ONLINE       rhel12c2                 STABLE
ora.FRA.dg
               ONLINE  ONLINE       rhel12c1                 STABLE
               ONLINE  ONLINE       rhel12c2                 STABLE
ora.LISTENER.lsnr
               ONLINE  ONLINE       rhel12c1                 STABLE
               ONLINE  ONLINE       rhel12c2                 STABLE
ora.net1.network
               ONLINE  ONLINE       rhel12c1                 STABLE
               ONLINE  ONLINE       rhel12c2                 STABLE
ora.ons
               ONLINE  ONLINE       rhel12c1                 STABLE
               ONLINE  ONLINE       rhel12c2                 STABLE
--------------------------------------------------------------------------------
Cluster Resources
--------------------------------------------------------------------------------
ora.LISTENER_SCAN1.lsnr
      1        ONLINE  ONLINE       rhel12c2                 STABLE
ora.LISTENER_SCAN2.lsnr
      1        ONLINE  ONLINE       rhel12c1                 STABLE
ora.LISTENER_SCAN3.lsnr
      1        ONLINE  ONLINE       rhel12c1                 STABLE
ora.MGMTLSNR
      1        ONLINE  ONLINE       rhel12c1                 169.254.229.238 192.
                                                             168.1.87,STABLE
ora.asm
      1        ONLINE  ONLINE       rhel12c1                 Started,STABLE
      2        ONLINE  ONLINE       rhel12c2                 Started,STABLE
      3        OFFLINE OFFLINE                               STABLE
ora.cvu
      1        ONLINE  ONLINE       rhel12c1                 STABLE
ora.gns
      1        ONLINE  ONLINE       rhel12c1                 STABLE
ora.gns.vip
      1        ONLINE  ONLINE       rhel12c1                 STABLE
ora.mgmtdb
      1        ONLINE  ONLINE       rhel12c1                 Open,STABLE
ora.oc4j
      1        ONLINE  ONLINE       rhel12c1                 STABLE
ora.rhel12c1.vip
      1        ONLINE  ONLINE       rhel12c1                 STABLE
ora.rhel12c2.vip
      1        ONLINE  ONLINE       rhel12c2                 STABLE
ora.scan1.vip
      1        ONLINE  ONLINE       rhel12c2                 STABLE
ora.scan2.vip
      1        ONLINE  ONLINE       rhel12c1                 STABLE
ora.scan3.vip
      1        ONLINE  ONLINE       rhel12c1                 STABLE
ora.std12c1.db
      1        ONLINE  ONLINE       rhel12c1                 Open,STABLE
      2        ONLINE  ONLINE       rhel12c2                 Open,STABLE
--------------------------------------------------------------------------------

As the scan name changed along with the cluster name, update the remote_listener parameter value.

alter system set remote_listener='prod-cluster-scan.rac.domain.net:1521' scope=both sid='*';

This concludes the renaming of the cluster.

Useful metalink notes
How to Configure or Re-configure Grid Infrastructure With config.sh/config.bat [ID 1354258.1]
How to change the cluster name in RAC [ID 1967916.1]