Friday, January 1, 2016

Changing The Cluster Name

This post shows the steps for renaming the cluster. The cluster consists of two nodes and current resources are
$ crsctl stat res -t
--------------------------------------------------------------------------------
Name           Target  State        Server                   State details
--------------------------------------------------------------------------------
Local Resources
--------------------------------------------------------------------------------
ora.CLUSFS.dg
               ONLINE  ONLINE       rhel12c1                 STABLE
               ONLINE  ONLINE       rhel12c2                 STABLE
ora.DATA.dg
               ONLINE  ONLINE       rhel12c1                 STABLE
               ONLINE  ONLINE       rhel12c2                 STABLE
ora.FRA.dg
               ONLINE  ONLINE       rhel12c1                 STABLE
               ONLINE  ONLINE       rhel12c2                 STABLE
ora.LISTENER.lsnr
               ONLINE  ONLINE       rhel12c1                 STABLE
               ONLINE  ONLINE       rhel12c2                 STABLE
ora.asm
               ONLINE  ONLINE       rhel12c1                 Started,STABLE
               ONLINE  ONLINE       rhel12c2                 Started,STABLE
ora.net1.network
               ONLINE  ONLINE       rhel12c1                 STABLE
               ONLINE  ONLINE       rhel12c2                 STABLE
ora.ons
               ONLINE  ONLINE       rhel12c1                 STABLE
               ONLINE  ONLINE       rhel12c2                 STABLE
--------------------------------------------------------------------------------
Cluster Resources
--------------------------------------------------------------------------------
ora.LISTENER_SCAN1.lsnr
      1        ONLINE  ONLINE       rhel12c2                 STABLE
ora.LISTENER_SCAN2.lsnr
      1        ONLINE  ONLINE       rhel12c2                 STABLE
ora.LISTENER_SCAN3.lsnr
      1        ONLINE  ONLINE       rhel12c1                 STABLE
ora.MGMTLSNR
      1        ONLINE  ONLINE       rhel12c1                 169.254.75.103 192.1
                                                             68.1.87,STABLE
ora.cvu
      1        ONLINE  ONLINE       rhel12c2                 STABLE
ora.gns
      1        ONLINE  ONLINE       rhel12c2                 STABLE
ora.gns.vip
      1        ONLINE  ONLINE       rhel12c2                 STABLE
ora.mgmtdb
      1        ONLINE  ONLINE       rhel12c1                 Open,STABLE
ora.oc4j
      1        ONLINE  ONLINE       rhel12c2                 STABLE
ora.rhel12c1.vip
      1        ONLINE  ONLINE       rhel12c1                 STABLE
ora.rhel12c2.vip
      1        ONLINE  ONLINE       rhel12c2                 STABLE
ora.scan1.vip
      1        ONLINE  ONLINE       rhel12c2                 STABLE
ora.scan2.vip
      1        ONLINE  ONLINE       rhel12c2                 STABLE
ora.scan3.vip
      1        ONLINE  ONLINE       rhel12c1                 STABLE
ora.std12c1.db
      1        ONLINE  ONLINE       rhel12c1                 Open,STABLE
      2        ONLINE  ONLINE       rhel12c2                 Open,STABLE
--------------------------------------------------------------------------------
The cluster name is
$ cemutlo -n
rhel12c-cluster
This will be renamed to prod-cluster.
Renaming the cluster involves re-configuring the current cluster setup. To deconfigure the use rootcrs.pl -deconfig options. This is run on all remotes as root
[root@rhel12c2 grid]# $ORACLE_HOME/crs/install/rootcrs.pl -deconfig -force -verbose
Using configuration parameter file: /opt/app/12.1.0/grid2/crs/install/crsconfig_params
Network 1 exists
Subnet IPv4: 192.168.0.0/255.255.255.0/eth0, dhcp
Subnet IPv6:
Ping Targets:
Network is enabled
Network is individually enabled on nodes:
Network is individually disabled on nodes:
VIP exists: network number 1, hosting node rhel12c1
VIP IPv4 Address: -/rhel12c1-vip/192.168.0.90
VIP IPv6 Address:
VIP is enabled.
VIP is individually enabled on nodes:
VIP is individually disabled on nodes:
VIP exists: network number 1, hosting node rhel12c2
VIP IPv4 Address: -/rhel12c2-vip/192.168.0.91
VIP IPv6 Address:
VIP is enabled.
VIP is individually enabled on nodes:
VIP is individually disabled on nodes:
ONS exists: Local port 6100, remote port 6200, EM port 2016, Uses SSL false
ONS is enabled
ONS is individually enabled on nodes:
ONS is individually disabled on nodes:
CRS-2791: Starting shutdown of Oracle High Availability Services-managed resources on 'rhel12c2'
CRS-2673: Attempting to stop 'ora.crsd' on 'rhel12c2'
CRS-2790: Starting shutdown of Cluster Ready Services-managed resources on 'rhel12c2'
CRS-2673: Attempting to stop 'ora.std12c1.db' on 'rhel12c2'
CRS-2673: Attempting to stop 'ora.CLUSFS.dg' on 'rhel12c2'
CRS-2673: Attempting to stop 'ora.oc4j' on 'rhel12c2'
CRS-2677: Stop of 'ora.CLUSFS.dg' on 'rhel12c2' succeeded
CRS-2677: Stop of 'ora.std12c1.db' on 'rhel12c2' succeeded
CRS-2673: Attempting to stop 'ora.FRA.dg' on 'rhel12c2'
CRS-2673: Attempting to stop 'ora.DATA.dg' on 'rhel12c2'
CRS-2677: Stop of 'ora.DATA.dg' on 'rhel12c2' succeeded
CRS-2677: Stop of 'ora.FRA.dg' on 'rhel12c2' succeeded
CRS-2673: Attempting to stop 'ora.asm' on 'rhel12c2'
CRS-2677: Stop of 'ora.asm' on 'rhel12c2' succeeded
CRS-2677: Stop of 'ora.oc4j' on 'rhel12c2' succeeded
CRS-2672: Attempting to start 'ora.oc4j' on 'rhel12c1'
CRS-2676: Start of 'ora.oc4j' on 'rhel12c1' succeeded
CRS-2792: Shutdown of Cluster Ready Services-managed resources on 'rhel12c2' has completed
CRS-2677: Stop of 'ora.crsd' on 'rhel12c2' succeeded
CRS-2673: Attempting to stop 'ora.storage' on 'rhel12c2'
CRS-2673: Attempting to stop 'ora.mdnsd' on 'rhel12c2'
CRS-2673: Attempting to stop 'ora.gpnpd' on 'rhel12c2'
CRS-2673: Attempting to stop 'ora.drivers.acfs' on 'rhel12c2'
CRS-2677: Stop of 'ora.drivers.acfs' on 'rhel12c2' succeeded
CRS-2677: Stop of 'ora.storage' on 'rhel12c2' succeeded
CRS-2673: Attempting to stop 'ora.crf' on 'rhel12c2'
CRS-2673: Attempting to stop 'ora.ctssd' on 'rhel12c2'
CRS-2673: Attempting to stop 'ora.evmd' on 'rhel12c2'
CRS-2673: Attempting to stop 'ora.asm' on 'rhel12c2'
CRS-2677: Stop of 'ora.ctssd' on 'rhel12c2' succeeded
CRS-2677: Stop of 'ora.crf' on 'rhel12c2' succeeded
CRS-2677: Stop of 'ora.evmd' on 'rhel12c2' succeeded
CRS-2677: Stop of 'ora.gpnpd' on 'rhel12c2' succeeded
CRS-2677: Stop of 'ora.mdnsd' on 'rhel12c2' succeeded
CRS-2677: Stop of 'ora.asm' on 'rhel12c2' succeeded
CRS-2673: Attempting to stop 'ora.cluster_interconnect.haip' on 'rhel12c2'
CRS-2677: Stop of 'ora.cluster_interconnect.haip' on 'rhel12c2' succeeded
CRS-2673: Attempting to stop 'ora.cssd' on 'rhel12c2'
CRS-2677: Stop of 'ora.cssd' on 'rhel12c2' succeeded
CRS-2673: Attempting to stop 'ora.gipcd' on 'rhel12c2'
CRS-2677: Stop of 'ora.gipcd' on 'rhel12c2' succeeded
CRS-2793: Shutdown of Oracle High Availability Services-managed resources on 'rhel12c2' has completed
CRS-4133: Oracle High Availability Services has been stopped.
2015/10/30 14:05:37 CLSRSC-4006: Removing Oracle Trace File Analyzer (TFA) Collector.

2015/10/30 14:06:12 CLSRSC-4007: Successfully removed Oracle Trace File Analyzer (TFA) Collector.

2015/10/30 14:06:13 CLSRSC-336: Successfully deconfigured Oracle Clusterware stack on this node
Once all the remote nodes are deconfigured, run the same with -lastnode option on the local node.
[root@rhel12c1 grid2]# $ORACLE_HOME/crs/install/rootcrs.pl -deconfig -force -verbose -lastnode
Using configuration parameter file: /opt/app/12.1.0/grid2/crs/install/crsconfig_params
2015/10/30 14:07:10 CLSRSC-332: CRS resources for listeners are still configured

Network 1 exists
Subnet IPv4: 192.168.0.0/255.255.255.0/eth0, dhcp
Subnet IPv6:
Ping Targets:
Network is enabled
Network is individually enabled on nodes:
Network is individually disabled on nodes:
VIP exists: network number 1, hosting node rhel12c1
VIP IPv4 Address: -/rhel12c1-vip/192.168.0.90
VIP IPv6 Address:
VIP is enabled.
VIP is individually enabled on nodes:
VIP is individually disabled on nodes:
ONS exists: Local port 6100, remote port 6200, EM port 2016, Uses SSL false
ONS is enabled
ONS is individually enabled on nodes:
ONS is individually disabled on nodes:
CRS-2791: Starting shutdown of Oracle High Availability Services-managed resources on 'rhel12c1'
CRS-2673: Attempting to stop 'ora.crsd' on 'rhel12c1'
CRS-2790: Starting shutdown of Cluster Ready Services-managed resources on 'rhel12c1'
CRS-2673: Attempting to stop 'ora.mgmtdb' on 'rhel12c1'
CRS-2673: Attempting to stop 'ora.std12c1.db' on 'rhel12c1'
CRS-2677: Stop of 'ora.std12c1.db' on 'rhel12c1' succeeded
CRS-2673: Attempting to stop 'ora.DATA.dg' on 'rhel12c1'
CRS-2677: Stop of 'ora.mgmtdb' on 'rhel12c1' succeeded
CRS-2673: Attempting to stop 'ora.MGMTLSNR' on 'rhel12c1'
CRS-2677: Stop of 'ora.MGMTLSNR' on 'rhel12c1' succeeded
CRS-2677: Stop of 'ora.DATA.dg' on 'rhel12c1' succeeded
CRS-2673: Attempting to stop 'ora.CLUSFS.dg' on 'rhel12c1'
CRS-2673: Attempting to stop 'ora.FRA.dg' on 'rhel12c1'
CRS-2677: Stop of 'ora.CLUSFS.dg' on 'rhel12c1' succeeded
CRS-2677: Stop of 'ora.FRA.dg' on 'rhel12c1' succeeded
CRS-2673: Attempting to stop 'ora.asm' on 'rhel12c1'
CRS-2677: Stop of 'ora.asm' on 'rhel12c1' succeeded
CRS-2792: Shutdown of Cluster Ready Services-managed resources on 'rhel12c1' has completed
CRS-2677: Stop of 'ora.crsd' on 'rhel12c1' succeeded
CRS-2673: Attempting to stop 'ora.storage' on 'rhel12c1'
CRS-2673: Attempting to stop 'ora.mdnsd' on 'rhel12c1'
CRS-2673: Attempting to stop 'ora.gpnpd' on 'rhel12c1'
CRS-2673: Attempting to stop 'ora.drivers.acfs' on 'rhel12c1'
CRS-2677: Stop of 'ora.storage' on 'rhel12c1' succeeded
CRS-2677: Stop of 'ora.drivers.acfs' on 'rhel12c1' succeeded
CRS-2673: Attempting to stop 'ora.crf' on 'rhel12c1'
CRS-2673: Attempting to stop 'ora.ctssd' on 'rhel12c1'
CRS-2673: Attempting to stop 'ora.evmd' on 'rhel12c1'
CRS-2673: Attempting to stop 'ora.asm' on 'rhel12c1'
CRS-2677: Stop of 'ora.crf' on 'rhel12c1' succeeded
CRS-2677: Stop of 'ora.ctssd' on 'rhel12c1' succeeded
CRS-2677: Stop of 'ora.evmd' on 'rhel12c1' succeeded
CRS-2677: Stop of 'ora.gpnpd' on 'rhel12c1' succeeded
CRS-2677: Stop of 'ora.mdnsd' on 'rhel12c1' succeeded
CRS-2677: Stop of 'ora.asm' on 'rhel12c1' succeeded
CRS-2673: Attempting to stop 'ora.cluster_interconnect.haip' on 'rhel12c1'
CRS-2677: Stop of 'ora.cluster_interconnect.haip' on 'rhel12c1' succeeded
CRS-2673: Attempting to stop 'ora.cssd' on 'rhel12c1'
CRS-2677: Stop of 'ora.cssd' on 'rhel12c1' succeeded
CRS-2673: Attempting to stop 'ora.gipcd' on 'rhel12c1'
CRS-2677: Stop of 'ora.gipcd' on 'rhel12c1' succeeded
CRS-2793: Shutdown of Oracle High Availability Services-managed resources on 'rhel12c1' has completed
CRS-4133: Oracle High Availability Services has been stopped.
CRS-4123: Oracle High Availability Services has been started.
CRS-2672: Attempting to start 'ora.evmd' on 'rhel12c1'
CRS-2672: Attempting to start 'ora.mdnsd' on 'rhel12c1'
CRS-2676: Start of 'ora.mdnsd' on 'rhel12c1' succeeded
CRS-2676: Start of 'ora.evmd' on 'rhel12c1' succeeded
CRS-2672: Attempting to start 'ora.gpnpd' on 'rhel12c1'
CRS-2676: Start of 'ora.gpnpd' on 'rhel12c1' succeeded
CRS-2672: Attempting to start 'ora.cssdmonitor' on 'rhel12c1'
CRS-2672: Attempting to start 'ora.gipcd' on 'rhel12c1'
CRS-2676: Start of 'ora.cssdmonitor' on 'rhel12c1' succeeded
CRS-2676: Start of 'ora.gipcd' on 'rhel12c1' succeeded
CRS-2672: Attempting to start 'ora.cssd' on 'rhel12c1'
CRS-2672: Attempting to start 'ora.diskmon' on 'rhel12c1'
CRS-2676: Start of 'ora.diskmon' on 'rhel12c1' succeeded
CRS-2676: Start of 'ora.cssd' on 'rhel12c1' succeeded
ASM de-configuration trace file location: /tmp/asmcadc_clean2015-10-30_02-15-19-PM.log
ASM Clean Configuration START
ASM Clean Configuration END

ASM with SID +ASM1 deleted successfully. Check /tmp/asmcadc_clean2015-10-30_02-15-19-PM.log for details.

CRS-2791: Starting shutdown of Oracle High Availability Services-managed resources on 'rhel12c1'
CRS-2673: Attempting to stop 'ora.mdnsd' on 'rhel12c1'
CRS-2673: Attempting to stop 'ora.gpnpd' on 'rhel12c1'
CRS-2673: Attempting to stop 'ora.ctssd' on 'rhel12c1'
CRS-2673: Attempting to stop 'ora.evmd' on 'rhel12c1'
CRS-2673: Attempting to stop 'ora.cluster_interconnect.haip' on 'rhel12c1'
CRS-2673: Attempting to stop 'ora.drivers.acfs' on 'rhel12c1'
CRS-2677: Stop of 'ora.drivers.acfs' on 'rhel12c1' succeeded
CRS-2677: Stop of 'ora.cluster_interconnect.haip' on 'rhel12c1' succeeded
CRS-2677: Stop of 'ora.mdnsd' on 'rhel12c1' succeeded
CRS-2677: Stop of 'ora.gpnpd' on 'rhel12c1' succeeded
CRS-2677: Stop of 'ora.ctssd' on 'rhel12c1' succeeded
CRS-2677: Stop of 'ora.evmd' on 'rhel12c1' succeeded
CRS-2673: Attempting to stop 'ora.cssd' on 'rhel12c1'
CRS-2677: Stop of 'ora.cssd' on 'rhel12c1' succeeded
CRS-2673: Attempting to stop 'ora.gipcd' on 'rhel12c1'
CRS-2677: Stop of 'ora.gipcd' on 'rhel12c1' succeeded
CRS-2793: Shutdown of Oracle High Availability Services-managed resources on 'rhel12c1' has completed
CRS-4133: Oracle High Availability Services has been stopped.
2015/10/30 14:28:04 CLSRSC-4006: Removing Oracle Trace File Analyzer (TFA) Collector.

2015/10/30 14:28:25 CLSRSC-4007: Successfully removed Oracle Trace File Analyzer (TFA) Collector.

2015/10/30 14:28:26 CLSRSC-336: Successfully deconfigured Oracle Clusterware stack on this node

2015/10/30 14:28:26 CLSRSC-559: Ensure that the GPnP profile data under the 'gpnp' directory in /opt/app/12.1.0/grid2 is deleted on each node before using the software in the current Grid Infrastructure home for reconfiguration.
To configure the cluster again with desired cluster name run the following command as grid user
$ $GI_HOME/crs/config/config.sh
This opens the OUI. Select configure cluster option.
As this is a 12c cluster setup it could be either setup as a flex cluster or standard cluster.
Specify the new cluster name
This OUI will detect the data base related disk groups (Data and Flash) but the ASM diskgroup that contained the ocr/vote disk and mgmtdb is no longer there.
Create a new diskgroup to store the ocr and vote disks.
At the last step select install to begin the configuration.
During the configuration (install) root.sh execution will be prompted. However execution of root.sh failed with the following error
# /opt/app/12.1.0/grid2/root.sh
Performing root user operation.

The following environment variables are set as:
    ORACLE_OWNER= grid
    ORACLE_HOME=  /opt/app/12.1.0/grid2

Enter the full pathname of the local bin directory: [/usr/local/bin]:
The contents of "dbhome" have not changed. No need to overwrite.
The contents of "oraenv" have not changed. No need to overwrite.
The contents of "coraenv" have not changed. No need to overwrite.

Entries will be added to the /etc/oratab file as needed by
Database Configuration Assistant when a database is created
Finished running generic part of root script.
Now product-specific root actions will be performed.
Relinking oracle with rac_on option
Using configuration parameter file: /opt/app/12.1.0/grid2/crs/install/crsconfig_params
2015/10/30 14:44:46 CLSRSC-4001: Installing Oracle Trace File Analyzer (TFA) Collector.

2015/10/30 14:45:24 CLSRSC-4002: Successfully installed Oracle Trace File Analyzer (TFA) Collector.

2015/10/30 14:45:26 CLSRSC-363: User ignored prerequisites during installation

OLR initialization - successful
  root wallet
  root wallet cert
  root cert export
  peer wallet
  profile reader wallet
  pa wallet
  peer wallet keys
  pa wallet keys
  peer cert request
  pa cert request
  peer cert
  pa cert
  peer root cert TP
  profile reader root cert TP
  pa root cert TP
  peer pa cert TP
  pa peer cert TP
  profile reader pa cert TP
  profile reader peer cert TP
  peer user cert
  pa user cert
2015/10/30 14:46:31 CLSRSC-330: Adding Clusterware entries to file 'oracle-ohasd.conf'

CRS-4133: Oracle High Availability Services has been stopped.
CRS-4123: Oracle High Availability Services has been started.
CRS-4133: Oracle High Availability Services has been stopped.
CRS-4123: Oracle High Availability Services has been started.
CRS-2672: Attempting to start 'ora.evmd' on 'rhel12c1'
CRS-2672: Attempting to start 'ora.mdnsd' on 'rhel12c1'
CRS-2676: Start of 'ora.evmd' on 'rhel12c1' succeeded
CRS-2676: Start of 'ora.mdnsd' on 'rhel12c1' succeeded
CRS-2672: Attempting to start 'ora.gpnpd' on 'rhel12c1'
CRS-2676: Start of 'ora.gpnpd' on 'rhel12c1' succeeded
CRS-2672: Attempting to start 'ora.cssdmonitor' on 'rhel12c1'
CRS-2672: Attempting to start 'ora.gipcd' on 'rhel12c1'
CRS-2676: Start of 'ora.cssdmonitor' on 'rhel12c1' succeeded
CRS-2676: Start of 'ora.gipcd' on 'rhel12c1' succeeded
CRS-2672: Attempting to start 'ora.cssd' on 'rhel12c1'
CRS-2672: Attempting to start 'ora.diskmon' on 'rhel12c1'
CRS-2676: Start of 'ora.diskmon' on 'rhel12c1' succeeded
CRS-2676: Start of 'ora.cssd' on 'rhel12c1' succeeded

Disk Group CLUSFS creation failed with the following message:
ORA-15018: diskgroup cannot be created
ORA-15031: disk specification '/dev/oracleasm/ocr3' matches no disks
ORA-15014: path '/dev/oracleasm/ocr3' is not in the discovery set
ORA-15031: disk specification '/dev/oracleasm/ocr2' matches no disks
ORA-15014: path '/dev/oracleasm/ocr2' is not in the discovery set
ORA-15031: disk specification '/dev/oracleasm/ocr1' matches no disks
ORA-15014: path '/dev/oracleasm/ocr1' is not in the discovery set


2015/10/30 14:49:43 CLSRSC-184: Configuration of ASM failed

2015/10/30 14:49:45 CLSRSC-258: Failed to configure and start ASM

Died at /opt/app/12.1.0/grid2/crs/install/crsinstall.pm line 2017.
The command '/opt/app/12.1.0/grid2/perl/bin/perl -I/opt/app/12.1.0/grid2/perl/lib -I/opt/app/12.1.0/grid2/crs/install /opt/app/12.1.0/grid2/crs/install/rootcrs.pl ' execution failed
According to 431013.1 clearing the underlying disk would resolve this issue.
[root@rhel12c1 ~]# dd if=/dev/zero of=/dev/oracleasm/ocr3 bs=8192 count=1000
[root@rhel12c1 ~]# dd if=/dev/zero of=/dev/oracleasm/ocr2 bs=8192 count=1000
[root@rhel12c1 ~]# dd if=/dev/zero of=/dev/oracleasm/ocr1 bs=8192 count=1000
but this did not always work. The real reason seem to be the missing asm_diskstring. When the root.sh fail it leaves a empty asm instance (without any disk groups). Trying to create the same disk group fails with the same error
SQL> CREATE DISKGROUP CLUSFS NORMAL REDUNDANCY  DISK '/dev/oracleasm/ocr1',
'/dev/oracleasm/ocr2',
'/dev/oracleasm/ocr3' ATTRIBUTE 'compatible.asm'='12.1.0.0.0','au_size'='4M'  2    3  ;
CREATE DISKGROUP CLUSFS NORMAL REDUNDANCY  DISK '/dev/oracleasm/ocr1',
*
ERROR at line 1:
ORA-15018: diskgroup cannot be created
ORA-15031: disk specification '/dev/oracleasm/ocr3' matches no disks
ORA-15014: path '/dev/oracleasm/ocr3' is not in the discovery set
ORA-15031: disk specification '/dev/oracleasm/ocr2' matches no disks
ORA-15014: path '/dev/oracleasm/ocr2' is not in the discovery set
ORA-15031: disk specification '/dev/oracleasm/ocr1' matches no disks
ORA-15014: path '/dev/oracleasm/ocr1' is not in the discovery set
The asm_diskstring is empty
SQL> show parameter string

NAME                                 TYPE        VALUE
------------------------------------ ----------- ------------------------------
asm_diskstring                       string
However if asm_disktring was updated then diskgroup gets created without a problem
SQL>  alter system set asm_diskstring='/dev/oracleasm/*';

System altered.

SQL> CREATE DISKGROUP CLUSFS NORMAL REDUNDANCY  DISK '/dev/oracleasm/ocr1',
'/dev/oracleasm/ocr2',
'/dev/oracleasm/ocr3' ATTRIBUTE 'compatible.asm'='12.1.0.0.0','au_size'='4M'  2    3  ;

Diskgroup created.


So it seems the missing asm_diskstring is the cause of the problem. To fix this the $GI_HOME/crs/install/crsconfig_params file was updated with the asm_diskstring
ASM_DISCOVERY_STRING=/dev/oracleasm/*
Afterwards the root.sh was rerun and there was no issue.
[root@rhel12c1 grid]# /opt/app/12.1.0/grid2/root.sh
Performing root user operation.

The following environment variables are set as:
    ORACLE_OWNER= grid
    ORACLE_HOME=  /opt/app/12.1.0/grid2

Enter the full pathname of the local bin directory: [/usr/local/bin]:
The contents of "dbhome" have not changed. No need to overwrite.
The contents of "oraenv" have not changed. No need to overwrite.
The contents of "coraenv" have not changed. No need to overwrite.

Entries will be added to the /etc/oratab file as needed by
Database Configuration Assistant when a database is created
Finished running generic part of root script.
Now product-specific root actions will be performed.
Relinking oracle with rac_on option
Using configuration parameter file: /opt/app/12.1.0/grid2/crs/install/crsconfig_params
2015/10/30 16:06:21 CLSRSC-4001: Installing Oracle Trace File Analyzer (TFA) Collector.

2015/10/30 16:06:21 CLSRSC-4002: Successfully installed Oracle Trace File Analyzer (TFA) Collector.

2015/10/30 16:06:24 CLSRSC-363: User ignored prerequisites during installation

CRS-2791: Starting shutdown of Oracle High Availability Services-managed resources on 'rhel12c1'
CRS-2673: Attempting to stop 'ora.ctssd' on 'rhel12c1'
CRS-2673: Attempting to stop 'ora.evmd' on 'rhel12c1'
CRS-2673: Attempting to stop 'ora.asm' on 'rhel12c1'
CRS-2673: Attempting to stop 'ora.mdnsd' on 'rhel12c1'
CRS-2673: Attempting to stop 'ora.gpnpd' on 'rhel12c1'
CRS-2673: Attempting to stop 'ora.drivers.acfs' on 'rhel12c1'
CRS-2677: Stop of 'ora.drivers.acfs' on 'rhel12c1' succeeded
CRS-2677: Stop of 'ora.ctssd' on 'rhel12c1' succeeded
CRS-2677: Stop of 'ora.mdnsd' on 'rhel12c1' succeeded
CRS-2677: Stop of 'ora.gpnpd' on 'rhel12c1' succeeded
CRS-2677: Stop of 'ora.evmd' on 'rhel12c1' succeeded
CRS-2677: Stop of 'ora.asm' on 'rhel12c1' succeeded
CRS-2673: Attempting to stop 'ora.cluster_interconnect.haip' on 'rhel12c1'
CRS-2677: Stop of 'ora.cluster_interconnect.haip' on 'rhel12c1' succeeded
CRS-2673: Attempting to stop 'ora.cssd' on 'rhel12c1'
CRS-2677: Stop of 'ora.cssd' on 'rhel12c1' succeeded
CRS-2673: Attempting to stop 'ora.gipcd' on 'rhel12c1'
CRS-2677: Stop of 'ora.gipcd' on 'rhel12c1' succeeded
CRS-2793: Shutdown of Oracle High Availability Services-managed resources on 'rhel12c1' has completed
CRS-4133: Oracle High Availability Services has been stopped.
CRS-4123: Oracle High Availability Services has been started.


CRS-2672: Attempting to start 'ora.evmd' on 'rhel12c1'
CRS-2672: Attempting to start 'ora.mdnsd' on 'rhel12c1'
CRS-2676: Start of 'ora.evmd' on 'rhel12c1' succeeded
CRS-2676: Start of 'ora.mdnsd' on 'rhel12c1' succeeded
CRS-2672: Attempting to start 'ora.gpnpd' on 'rhel12c1'
CRS-2676: Start of 'ora.gpnpd' on 'rhel12c1' succeeded
CRS-2672: Attempting to start 'ora.cssdmonitor' on 'rhel12c1'
CRS-2672: Attempting to start 'ora.gipcd' on 'rhel12c1'
CRS-2676: Start of 'ora.cssdmonitor' on 'rhel12c1' succeeded
CRS-2676: Start of 'ora.gipcd' on 'rhel12c1' succeeded
CRS-2672: Attempting to start 'ora.cssd' on 'rhel12c1'
CRS-2672: Attempting to start 'ora.diskmon' on 'rhel12c1'
CRS-2676: Start of 'ora.diskmon' on 'rhel12c1' succeeded
CRS-2676: Start of 'ora.cssd' on 'rhel12c1' succeeded
CRS-2672: Attempting to start 'ora.cluster_interconnect.haip' on 'rhel12c1'
CRS-2672: Attempting to start 'ora.ctssd' on 'rhel12c1'
CRS-2676: Start of 'ora.ctssd' on 'rhel12c1' succeeded
CRS-2676: Start of 'ora.cluster_interconnect.haip' on 'rhel12c1' succeeded
CRS-2672: Attempting to start 'ora.asm' on 'rhel12c1'
CRS-2676: Start of 'ora.asm' on 'rhel12c1' succeeded
CRS-2672: Attempting to start 'ora.storage' on 'rhel12c1'
diskgroup CLUSFS not mounted ()
CRS-5017: The resource action "ora.storage start" encountered the following error:
Storage agent start action aborted. For details refer to "(:CLSN00107:)" in "/opt/app/oracle/diag/crs/rhel12c1/crs/trace/ohasd_orarootagent_root.trc".
CRS-2674: Start of 'ora.storage' on 'rhel12c1' failed
CRS-2679: Attempting to clean 'ora.storage' on 'rhel12c1'
CRS-2681: Clean of 'ora.storage' on 'rhel12c1' succeeded

ASM created and started successfully.

Disk Group CLUSFS created successfully.

CRS-2672: Attempting to start 'ora.crf' on 'rhel12c1'
CRS-2672: Attempting to start 'ora.storage' on 'rhel12c1'
CRS-2676: Start of 'ora.storage' on 'rhel12c1' succeeded
CRS-2676: Start of 'ora.crf' on 'rhel12c1' succeeded
CRS-2672: Attempting to start 'ora.crsd' on 'rhel12c1'
CRS-2676: Start of 'ora.crsd' on 'rhel12c1' succeeded
CRS-4256: Updating the profile
Successful addition of voting disk 92718a80064a4fdebf30cdd4ed314115.
Successful addition of voting disk 48f9d1873a394f4dbf63ce93cf8b8b03.
Successful addition of voting disk a96f1382267b4fbdbf084b7595fd3643.
Successfully replaced voting disk group with +CLUSFS.
CRS-4256: Updating the profile
CRS-4266: Voting file(s) successfully replaced
##  STATE    File Universal Id                File Name Disk group
--  -----    -----------------                --------- ---------
 1. ONLINE   92718a80064a4fdebf30cdd4ed314115 (/dev/oracleasm/ocr1) [CLUSFS]
 2. ONLINE   48f9d1873a394f4dbf63ce93cf8b8b03 (/dev/oracleasm/ocr2) [CLUSFS]
 3. ONLINE   a96f1382267b4fbdbf084b7595fd3643 (/dev/oracleasm/ocr3) [CLUSFS]
Located 3 voting disk(s).
CRS-2791: Starting shutdown of Oracle High Availability Services-managed resources on 'rhel12c1'
CRS-2673: Attempting to stop 'ora.crsd' on 'rhel12c1'
CRS-2677: Stop of 'ora.crsd' on 'rhel12c1' succeeded
CRS-2673: Attempting to stop 'ora.storage' on 'rhel12c1'
CRS-2673: Attempting to stop 'ora.gpnpd' on 'rhel12c1'
CRS-2673: Attempting to stop 'ora.drivers.acfs' on 'rhel12c1'
CRS-2673: Attempting to stop 'ora.mdnsd' on 'rhel12c1'
CRS-2677: Stop of 'ora.storage' on 'rhel12c1' succeeded
CRS-2673: Attempting to stop 'ora.asm' on 'rhel12c1'
CRS-2677: Stop of 'ora.drivers.acfs' on 'rhel12c1' succeeded
CRS-2677: Stop of 'ora.gpnpd' on 'rhel12c1' succeeded
CRS-2677: Stop of 'ora.mdnsd' on 'rhel12c1' succeeded
CRS-2677: Stop of 'ora.asm' on 'rhel12c1' succeeded
CRS-2673: Attempting to stop 'ora.cluster_interconnect.haip' on 'rhel12c1'
CRS-2677: Stop of 'ora.cluster_interconnect.haip' on 'rhel12c1' succeeded
CRS-2673: Attempting to stop 'ora.ctssd' on 'rhel12c1'
CRS-2673: Attempting to stop 'ora.evmd' on 'rhel12c1'
CRS-2677: Stop of 'ora.ctssd' on 'rhel12c1' succeeded
CRS-2677: Stop of 'ora.evmd' on 'rhel12c1' succeeded
CRS-2673: Attempting to stop 'ora.cssd' on 'rhel12c1'
CRS-2677: Stop of 'ora.cssd' on 'rhel12c1' succeeded
CRS-2673: Attempting to stop 'ora.crf' on 'rhel12c1'
CRS-2677: Stop of 'ora.crf' on 'rhel12c1' succeeded
CRS-2673: Attempting to stop 'ora.gipcd' on 'rhel12c1'
CRS-2677: Stop of 'ora.gipcd' on 'rhel12c1' succeeded
CRS-2793: Shutdown of Oracle High Availability Services-managed resources on 'rhel12c1' has completed
CRS-4133: Oracle High Availability Services has been stopped.
CRS-4123: Starting Oracle High Availability Services-managed resources
CRS-2672: Attempting to start 'ora.mdnsd' on 'rhel12c1'
CRS-2672: Attempting to start 'ora.evmd' on 'rhel12c1'
CRS-2676: Start of 'ora.mdnsd' on 'rhel12c1' succeeded
CRS-2676: Start of 'ora.evmd' on 'rhel12c1' succeeded
CRS-2672: Attempting to start 'ora.gpnpd' on 'rhel12c1'
CRS-2676: Start of 'ora.gpnpd' on 'rhel12c1' succeeded
CRS-2672: Attempting to start 'ora.gipcd' on 'rhel12c1'
CRS-2676: Start of 'ora.gipcd' on 'rhel12c1' succeeded
CRS-2672: Attempting to start 'ora.cssdmonitor' on 'rhel12c1'
CRS-2676: Start of 'ora.cssdmonitor' on 'rhel12c1' succeeded
CRS-2672: Attempting to start 'ora.cssd' on 'rhel12c1'
CRS-2672: Attempting to start 'ora.diskmon' on 'rhel12c1'
CRS-2676: Start of 'ora.diskmon' on 'rhel12c1' succeeded
CRS-2676: Start of 'ora.cssd' on 'rhel12c1' succeeded
CRS-2672: Attempting to start 'ora.cluster_interconnect.haip' on 'rhel12c1'
CRS-2672: Attempting to start 'ora.ctssd' on 'rhel12c1'
CRS-2676: Start of 'ora.ctssd' on 'rhel12c1' succeeded
CRS-2676: Start of 'ora.cluster_interconnect.haip' on 'rhel12c1' succeeded
CRS-2672: Attempting to start 'ora.asm' on 'rhel12c1'
CRS-2676: Start of 'ora.asm' on 'rhel12c1' succeeded
CRS-2672: Attempting to start 'ora.storage' on 'rhel12c1'
CRS-2676: Start of 'ora.storage' on 'rhel12c1' succeeded
CRS-2672: Attempting to start 'ora.crf' on 'rhel12c1'
CRS-2676: Start of 'ora.crf' on 'rhel12c1' succeeded
CRS-2672: Attempting to start 'ora.crsd' on 'rhel12c1'
CRS-2676: Start of 'ora.crsd' on 'rhel12c1' succeeded
CRS-6023: Starting Oracle Cluster Ready Services-managed resources
CRS-6017: Processing resource auto-start for servers: rhel12c1
CRS-6016: Resource auto-start has completed for server rhel12c1
CRS-6024: Completed start of Oracle Cluster Ready Services-managed resources
CRS-4123: Oracle High Availability Services has been started.
2015/10/30 16:33:07 CLSRSC-343: Successfully started Oracle Clusterware stack

CRS-2672: Attempting to start 'ora.net1.network' on 'rhel12c1'
CRS-2676: Start of 'ora.net1.network' on 'rhel12c1' succeeded
CRS-2672: Attempting to start 'ora.gns.vip' on 'rhel12c1'
CRS-2676: Start of 'ora.gns.vip' on 'rhel12c1' succeeded
CRS-2672: Attempting to start 'ora.gns' on 'rhel12c1'
CRS-2676: Start of 'ora.gns' on 'rhel12c1' succeeded
CRS-2672: Attempting to start 'ora.ASMNET1LSNR_ASM.lsnr' on 'rhel12c1'
CRS-2676: Start of 'ora.ASMNET1LSNR_ASM.lsnr' on 'rhel12c1' succeeded
CRS-2672: Attempting to start 'ora.asm' on 'rhel12c1'
CRS-2676: Start of 'ora.asm' on 'rhel12c1' succeeded
CRS-2672: Attempting to start 'ora.CLUSFS.dg' on 'rhel12c1'
CRS-2676: Start of 'ora.CLUSFS.dg' on 'rhel12c1' succeeded
CRS-2791: Starting shutdown of Oracle High Availability Services-managed resources on 'rhel12c1'
CRS-2673: Attempting to stop 'ora.crsd' on 'rhel12c1'
CRS-2790: Starting shutdown of Cluster Ready Services-managed resources on 'rhel12c1'
CRS-2673: Attempting to stop 'ora.rhel12c1.vip' on 'rhel12c1'
CRS-2673: Attempting to stop 'ora.CLUSFS.dg' on 'rhel12c1'
CRS-2673: Attempting to stop 'ora.LISTENER_SCAN3.lsnr' on 'rhel12c1'
CRS-2673: Attempting to stop 'ora.oc4j' on 'rhel12c1'
CRS-2673: Attempting to stop 'ora.gns' on 'rhel12c1'
CRS-2673: Attempting to stop 'ora.LISTENER_SCAN2.lsnr' on 'rhel12c1'
CRS-2673: Attempting to stop 'ora.LISTENER_SCAN1.lsnr' on 'rhel12c1'
CRS-2673: Attempting to stop 'ora.cvu' on 'rhel12c1'
CRS-2677: Stop of 'ora.CLUSFS.dg' on 'rhel12c1' succeeded
CRS-2673: Attempting to stop 'ora.asm' on 'rhel12c1'
CRS-2677: Stop of 'ora.cvu' on 'rhel12c1' succeeded
CRS-2677: Stop of 'ora.asm' on 'rhel12c1' succeeded
CRS-2673: Attempting to stop 'ora.ASMNET1LSNR_ASM.lsnr' on 'rhel12c1'
CRS-2677: Stop of 'ora.LISTENER_SCAN3.lsnr' on 'rhel12c1' succeeded
CRS-2673: Attempting to stop 'ora.scan3.vip' on 'rhel12c1'
CRS-2677: Stop of 'ora.LISTENER_SCAN2.lsnr' on 'rhel12c1' succeeded
CRS-2673: Attempting to stop 'ora.scan2.vip' on 'rhel12c1'
CRS-2677: Stop of 'ora.LISTENER_SCAN1.lsnr' on 'rhel12c1' succeeded
CRS-2673: Attempting to stop 'ora.scan1.vip' on 'rhel12c1'
CRS-2677: Stop of 'ora.ASMNET1LSNR_ASM.lsnr' on 'rhel12c1' succeeded
CRS-2677: Stop of 'ora.rhel12c1.vip' on 'rhel12c1' succeeded
CRS-2677: Stop of 'ora.gns' on 'rhel12c1' succeeded
CRS-2673: Attempting to stop 'ora.gns.vip' on 'rhel12c1'
CRS-2677: Stop of 'ora.scan1.vip' on 'rhel12c1' succeeded
CRS-2677: Stop of 'ora.scan2.vip' on 'rhel12c1' succeeded
CRS-2677: Stop of 'ora.scan3.vip' on 'rhel12c1' succeeded
CRS-2677: Stop of 'ora.gns.vip' on 'rhel12c1' succeeded
CRS-2677: Stop of 'ora.oc4j' on 'rhel12c1' succeeded
CRS-2673: Attempting to stop 'ora.ons' on 'rhel12c1'
CRS-2677: Stop of 'ora.ons' on 'rhel12c1' succeeded
CRS-2673: Attempting to stop 'ora.net1.network' on 'rhel12c1'
CRS-2677: Stop of 'ora.net1.network' on 'rhel12c1' succeeded
CRS-2792: Shutdown of Cluster Ready Services-managed resources on 'rhel12c1' has completed
CRS-2677: Stop of 'ora.crsd' on 'rhel12c1' succeeded
CRS-2673: Attempting to stop 'ora.ctssd' on 'rhel12c1'
CRS-2673: Attempting to stop 'ora.evmd' on 'rhel12c1'
CRS-2673: Attempting to stop 'ora.storage' on 'rhel12c1'
CRS-2673: Attempting to stop 'ora.mdnsd' on 'rhel12c1'
CRS-2673: Attempting to stop 'ora.gpnpd' on 'rhel12c1'
CRS-2673: Attempting to stop 'ora.drivers.acfs' on 'rhel12c1'
CRS-2677: Stop of 'ora.storage' on 'rhel12c1' succeeded
CRS-2673: Attempting to stop 'ora.asm' on 'rhel12c1'
CRS-2677: Stop of 'ora.drivers.acfs' on 'rhel12c1' succeeded
CRS-2677: Stop of 'ora.ctssd' on 'rhel12c1' succeeded
CRS-2677: Stop of 'ora.mdnsd' on 'rhel12c1' succeeded
CRS-2677: Stop of 'ora.gpnpd' on 'rhel12c1' succeeded
CRS-2677: Stop of 'ora.evmd' on 'rhel12c1' succeeded
CRS-2677: Stop of 'ora.asm' on 'rhel12c1' succeeded
CRS-2673: Attempting to stop 'ora.cluster_interconnect.haip' on 'rhel12c1'
CRS-2677: Stop of 'ora.cluster_interconnect.haip' on 'rhel12c1' succeeded
CRS-2673: Attempting to stop 'ora.cssd' on 'rhel12c1'
CRS-2677: Stop of 'ora.cssd' on 'rhel12c1' succeeded
CRS-2673: Attempting to stop 'ora.crf' on 'rhel12c1'
CRS-2677: Stop of 'ora.crf' on 'rhel12c1' succeeded
CRS-2673: Attempting to stop 'ora.gipcd' on 'rhel12c1'
CRS-2677: Stop of 'ora.gipcd' on 'rhel12c1' succeeded
CRS-2793: Shutdown of Oracle High Availability Services-managed resources on 'rhel12c1' has completed
CRS-4133: Oracle High Availability Services has been stopped.
CRS-4123: Starting Oracle High Availability Services-managed resources
CRS-2672: Attempting to start 'ora.mdnsd' on 'rhel12c1'
CRS-2672: Attempting to start 'ora.evmd' on 'rhel12c1'
CRS-2676: Start of 'ora.mdnsd' on 'rhel12c1' succeeded
CRS-2676: Start of 'ora.evmd' on 'rhel12c1' succeeded
CRS-2672: Attempting to start 'ora.gpnpd' on 'rhel12c1'
CRS-2676: Start of 'ora.gpnpd' on 'rhel12c1' succeeded
CRS-2672: Attempting to start 'ora.gipcd' on 'rhel12c1'
CRS-2676: Start of 'ora.gipcd' on 'rhel12c1' succeeded
CRS-2672: Attempting to start 'ora.cssdmonitor' on 'rhel12c1'
CRS-2676: Start of 'ora.cssdmonitor' on 'rhel12c1' succeeded
CRS-2672: Attempting to start 'ora.cssd' on 'rhel12c1'
CRS-2672: Attempting to start 'ora.diskmon' on 'rhel12c1'
CRS-2676: Start of 'ora.diskmon' on 'rhel12c1' succeeded
CRS-2676: Start of 'ora.cssd' on 'rhel12c1' succeeded
CRS-2672: Attempting to start 'ora.cluster_interconnect.haip' on 'rhel12c1'
CRS-2672: Attempting to start 'ora.ctssd' on 'rhel12c1'
CRS-2676: Start of 'ora.ctssd' on 'rhel12c1' succeeded
CRS-2676: Start of 'ora.cluster_interconnect.haip' on 'rhel12c1' succeeded
CRS-2672: Attempting to start 'ora.asm' on 'rhel12c1'
CRS-2676: Start of 'ora.asm' on 'rhel12c1' succeeded
CRS-2672: Attempting to start 'ora.storage' on 'rhel12c1'
CRS-2676: Start of 'ora.storage' on 'rhel12c1' succeeded
CRS-2672: Attempting to start 'ora.crf' on 'rhel12c1'
CRS-2676: Start of 'ora.crf' on 'rhel12c1' succeeded
CRS-2672: Attempting to start 'ora.crsd' on 'rhel12c1'
CRS-2676: Start of 'ora.crsd' on 'rhel12c1' succeeded
CRS-6023: Starting Oracle Cluster Ready Services-managed resources
CRS-2664: Resource 'ora.CLUSFS.dg' is already running on 'rhel12c1'
CRS-6017: Processing resource auto-start for servers: rhel12c1
CRS-2672: Attempting to start 'ora.oc4j' on 'rhel12c1'
CRS-2676: Start of 'ora.oc4j' on 'rhel12c1' succeeded
CRS-6016: Resource auto-start has completed for server rhel12c1
CRS-6024: Completed start of Oracle Cluster Ready Services-managed resources
CRS-4123: Oracle High Availability Services has been started.
2015/10/30 16:38:34 CLSRSC-325: Configure Oracle Grid Infrastructure for a Cluster ... succeeded
On the last node
[root@rhel12c2 grid]# /opt/app/12.1.0/grid2/root.sh
Performing root user operation.

The following environment variables are set as:
    ORACLE_OWNER= grid
    ORACLE_HOME=  /opt/app/12.1.0/grid2

Enter the full pathname of the local bin directory: [/usr/local/bin]:
The contents of "dbhome" have not changed. No need to overwrite.
The contents of "oraenv" have not changed. No need to overwrite.
The contents of "coraenv" have not changed. No need to overwrite.

Entries will be added to the /etc/oratab file as needed by
Database Configuration Assistant when a database is created
Finished running generic part of root script.
Now product-specific root actions will be performed.
Relinking oracle with rac_on option
Using configuration parameter file: /opt/app/12.1.0/grid2/crs/install/crsconfig_params
2015/10/30 16:43:33 CLSRSC-4001: Installing Oracle Trace File Analyzer (TFA) Collector.

2015/10/30 16:44:09 CLSRSC-4002: Successfully installed Oracle Trace File Analyzer (TFA) Collector.

2015/10/30 16:44:12 CLSRSC-363: User ignored prerequisites during installation

OLR initialization - successful
2015/10/30 16:45:41 CLSRSC-330: Adding Clusterware entries to file 'oracle-ohasd.conf'

CRS-4133: Oracle High Availability Services has been stopped.
CRS-4123: Oracle High Availability Services has been started.
CRS-4133: Oracle High Availability Services has been stopped.
CRS-4123: Oracle High Availability Services has been started.
CRS-2791: Starting shutdown of Oracle High Availability Services-managed resources on 'rhel12c2'
CRS-2673: Attempting to stop 'ora.drivers.acfs' on 'rhel12c2'
CRS-2677: Stop of 'ora.drivers.acfs' on 'rhel12c2' succeeded
CRS-2793: Shutdown of Oracle High Availability Services-managed resources on 'rhel12c2' has completed
CRS-4133: Oracle High Availability Services has been stopped.
CRS-4123: Starting Oracle High Availability Services-managed resources
CRS-2672: Attempting to start 'ora.mdnsd' on 'rhel12c2'
CRS-2672: Attempting to start 'ora.evmd' on 'rhel12c2'
CRS-2676: Start of 'ora.mdnsd' on 'rhel12c2' succeeded
CRS-2676: Start of 'ora.evmd' on 'rhel12c2' succeeded
CRS-2672: Attempting to start 'ora.gpnpd' on 'rhel12c2'
CRS-2676: Start of 'ora.gpnpd' on 'rhel12c2' succeeded
CRS-2672: Attempting to start 'ora.gipcd' on 'rhel12c2'
CRS-2676: Start of 'ora.gipcd' on 'rhel12c2' succeeded
CRS-2672: Attempting to start 'ora.cssdmonitor' on 'rhel12c2'
CRS-2676: Start of 'ora.cssdmonitor' on 'rhel12c2' succeeded
CRS-2672: Attempting to start 'ora.cssd' on 'rhel12c2'
CRS-2672: Attempting to start 'ora.diskmon' on 'rhel12c2'
CRS-2676: Start of 'ora.diskmon' on 'rhel12c2' succeeded
CRS-2676: Start of 'ora.cssd' on 'rhel12c2' succeeded
CRS-2672: Attempting to start 'ora.cluster_interconnect.haip' on 'rhel12c2'
CRS-2672: Attempting to start 'ora.ctssd' on 'rhel12c2'
CRS-2676: Start of 'ora.ctssd' on 'rhel12c2' succeeded
CRS-2672: Attempting to start 'ora.crf' on 'rhel12c2'
CRS-2676: Start of 'ora.crf' on 'rhel12c2' succeeded
CRS-2672: Attempting to start 'ora.crsd' on 'rhel12c2'
CRS-2676: Start of 'ora.crsd' on 'rhel12c2' succeeded
CRS-2676: Start of 'ora.cluster_interconnect.haip' on 'rhel12c2' succeeded
CRS-2672: Attempting to start 'ora.asm' on 'rhel12c2'
CRS-2676: Start of 'ora.asm' on 'rhel12c2' succeeded
CRS-6017: Processing resource auto-start for servers: rhel12c2
CRS-2672: Attempting to start 'ora.net1.network' on 'rhel12c2'
CRS-2672: Attempting to start 'ora.ASMNET1LSNR_ASM.lsnr' on 'rhel12c2'
CRS-2676: Start of 'ora.net1.network' on 'rhel12c2' succeeded
CRS-2672: Attempting to start 'ora.ons' on 'rhel12c2'
CRS-2676: Start of 'ora.ASMNET1LSNR_ASM.lsnr' on 'rhel12c2' succeeded
CRS-2672: Attempting to start 'ora.asm' on 'rhel12c2'
CRS-2676: Start of 'ora.ons' on 'rhel12c2' succeeded
CRS-2673: Attempting to stop 'ora.LISTENER_SCAN1.lsnr' on 'rhel12c1'
CRS-2677: Stop of 'ora.LISTENER_SCAN1.lsnr' on 'rhel12c1' succeeded
CRS-2673: Attempting to stop 'ora.scan1.vip' on 'rhel12c1'
CRS-2677: Stop of 'ora.scan1.vip' on 'rhel12c1' succeeded
CRS-2672: Attempting to start 'ora.scan1.vip' on 'rhel12c2'
CRS-2676: Start of 'ora.scan1.vip' on 'rhel12c2' succeeded
CRS-2672: Attempting to start 'ora.LISTENER_SCAN1.lsnr' on 'rhel12c2'
CRS-2676: Start of 'ora.LISTENER_SCAN1.lsnr' on 'rhel12c2' succeeded
CRS-2676: Start of 'ora.asm' on 'rhel12c2' succeeded
CRS-6016: Resource auto-start has completed for server rhel12c2
CRS-6024: Completed start of Oracle Cluster Ready Services-managed resources
CRS-4123: Oracle High Availability Services has been started.
2015/10/30 16:51:06 CLSRSC-343: Successfully started Oracle Clusterware stack

2015/10/30 16:51:28 CLSRSC-325: Configure Oracle Grid Infrastructure for a Cluster ... succeeded
At the end of the cluster configuration the cluster stack would be up and running
$ crsctl stat res -t
--------------------------------------------------------------------------------
Name           Target  State        Server                   State details
--------------------------------------------------------------------------------
Local Resources
--------------------------------------------------------------------------------
ora.ASMNET1LSNR_ASM.lsnr
               ONLINE  ONLINE       rhel12c1                 STABLE
               ONLINE  ONLINE       rhel12c2                 STABLE
ora.CLUSFS.dg
               ONLINE  ONLINE       rhel12c1                 STABLE
               ONLINE  ONLINE       rhel12c2                 STABLE
ora.LISTENER.lsnr
               ONLINE  ONLINE       rhel12c1                 STABLE
               ONLINE  ONLINE       rhel12c2                 STABLE
ora.net1.network
               ONLINE  ONLINE       rhel12c1                 STABLE
               ONLINE  ONLINE       rhel12c2                 STABLE
ora.ons
               ONLINE  ONLINE       rhel12c1                 STABLE
               ONLINE  ONLINE       rhel12c2                 STABLE
--------------------------------------------------------------------------------
Cluster Resources
--------------------------------------------------------------------------------
ora.LISTENER_SCAN1.lsnr
      1        ONLINE  ONLINE       rhel12c2                 STABLE
ora.LISTENER_SCAN2.lsnr
      1        ONLINE  ONLINE       rhel12c1                 STABLE
ora.LISTENER_SCAN3.lsnr
      1        ONLINE  ONLINE       rhel12c1                 STABLE
ora.MGMTLSNR
      1        ONLINE  ONLINE       rhel12c1                 169.254.229.238 192.
                                                             168.1.87,STABLE
ora.asm
      1        ONLINE  ONLINE       rhel12c1                 Started,STABLE
      2        ONLINE  ONLINE       rhel12c2                 Started,STABLE
      3        OFFLINE OFFLINE                               STABLE
ora.cvu
      1        ONLINE  ONLINE       rhel12c1                 STABLE
ora.gns
      1        ONLINE  ONLINE       rhel12c1                 STABLE
ora.gns.vip
      1        ONLINE  ONLINE       rhel12c1                 STABLE
ora.mgmtdb
      1        ONLINE  ONLINE       rhel12c1                 Open,STABLE
ora.oc4j
      1        ONLINE  ONLINE       rhel12c1                 STABLE
ora.rhel12c1.vip
      1        ONLINE  ONLINE       rhel12c1                 STABLE
ora.rhel12c2.vip
      1        ONLINE  ONLINE       rhel12c2                 STABLE
ora.scan1.vip
      1        ONLINE  ONLINE       rhel12c2                 STABLE
ora.scan2.vip
      1        ONLINE  ONLINE       rhel12c1                 STABLE
ora.scan3.vip
      1        ONLINE  ONLINE       rhel12c1                 STABLE
--------------------------------------------------------------------------------
with the new cluster name
 cemutlo -n
prod-cluster
One thing notable is that database related asm disk groups and database itself are not part of the cluster yet. For the cluster setup to be same as before the rename the database must be manually added to the cluster. Mount the asm disk groups data and fra
SQL> alter diskgroup fra mount;
Diskgroup altered.

SQL> alter diskgroup data mount ;
Diskgroup altered.

SQL>  select inst_id,name,state from gv$asm_diskgroup;

   INST_ID NAME                           STATE
---------- ------------------------------ -----------
         2 CLUSFS                         MOUNTED
         2 FRA                            MOUNTED
         2 DATA                           MOUNTED
         1 CLUSFS                         MOUNTED
         1 FRA                            MOUNTED
         1 DATA                           MOUNTED
As oracle user add the database to the cluster
[oracle@rhel12c1 ~]$ srvctl add database -db std12c1 -oraclehome $ORACLE_HOME -spfile "+DATA/STD12C1/PARAMETERFILE/spfilestd12c1.ora"
[oracle@rhel12c1 ~]$ srvctl add instance -db std12c1 -instance std12c11 -n rhel12c1
[oracle@rhel12c1 ~]$ srvctl add instance -db std12c1 -instance std12c12 -n rhel12c2
[oracle@rhel12c1 ~]$ srvctl start database -d std12c1
Database configuration is added to the cluster
$ srvctl config database -d std12c1
Database unique name: std12c1
Database name:
Oracle home: /opt/app/oracle/product/12.1.0/dbhome_2
Oracle user: oracle
Spfile: +DATA/STD12C1/PARAMETERFILE/spfilestd12c1.ora
Password file:
Domain:
Start options: open
Stop options: immediate
Database role: PRIMARY
Management policy: AUTOMATIC
Server pools:
Disk Groups: DATA,FRA
Mount point paths:
Services:
Type: RAC
Start concurrency:
Stop concurrency:
OSDBA group: dba
OSOPER group: oper
Database instances: std12c11,std12c12
Configured nodes: rhel12c1,rhel12c2
Database is administrator managed
Number of cluster resources are same as before the rename
$ crsctl stat res -t
--------------------------------------------------------------------------------
Name           Target  State        Server                   State details
--------------------------------------------------------------------------------
Local Resources
--------------------------------------------------------------------------------
ora.ASMNET1LSNR_ASM.lsnr
               ONLINE  ONLINE       rhel12c1                 STABLE
               ONLINE  ONLINE       rhel12c2                 STABLE
ora.CLUSFS.dg
               ONLINE  ONLINE       rhel12c1                 STABLE
               ONLINE  ONLINE       rhel12c2                 STABLE
ora.DATA.dg
               ONLINE  ONLINE       rhel12c1                 STABLE
               ONLINE  ONLINE       rhel12c2                 STABLE
ora.FRA.dg
               ONLINE  ONLINE       rhel12c1                 STABLE
               ONLINE  ONLINE       rhel12c2                 STABLE
ora.LISTENER.lsnr
               ONLINE  ONLINE       rhel12c1                 STABLE
               ONLINE  ONLINE       rhel12c2                 STABLE
ora.net1.network
               ONLINE  ONLINE       rhel12c1                 STABLE
               ONLINE  ONLINE       rhel12c2                 STABLE
ora.ons
               ONLINE  ONLINE       rhel12c1                 STABLE
               ONLINE  ONLINE       rhel12c2                 STABLE
--------------------------------------------------------------------------------
Cluster Resources
--------------------------------------------------------------------------------
ora.LISTENER_SCAN1.lsnr
      1        ONLINE  ONLINE       rhel12c2                 STABLE
ora.LISTENER_SCAN2.lsnr
      1        ONLINE  ONLINE       rhel12c1                 STABLE
ora.LISTENER_SCAN3.lsnr
      1        ONLINE  ONLINE       rhel12c1                 STABLE
ora.MGMTLSNR
      1        ONLINE  ONLINE       rhel12c1                 169.254.229.238 192.
                                                             168.1.87,STABLE
ora.asm
      1        ONLINE  ONLINE       rhel12c1                 Started,STABLE
      2        ONLINE  ONLINE       rhel12c2                 Started,STABLE
      3        OFFLINE OFFLINE                               STABLE
ora.cvu
      1        ONLINE  ONLINE       rhel12c1                 STABLE
ora.gns
      1        ONLINE  ONLINE       rhel12c1                 STABLE
ora.gns.vip
      1        ONLINE  ONLINE       rhel12c1                 STABLE
ora.mgmtdb
      1        ONLINE  ONLINE       rhel12c1                 Open,STABLE
ora.oc4j
      1        ONLINE  ONLINE       rhel12c1                 STABLE
ora.rhel12c1.vip
      1        ONLINE  ONLINE       rhel12c1                 STABLE
ora.rhel12c2.vip
      1        ONLINE  ONLINE       rhel12c2                 STABLE
ora.scan1.vip
      1        ONLINE  ONLINE       rhel12c2                 STABLE
ora.scan2.vip
      1        ONLINE  ONLINE       rhel12c1                 STABLE
ora.scan3.vip
      1        ONLINE  ONLINE       rhel12c1                 STABLE
ora.std12c1.db
      1        ONLINE  ONLINE       rhel12c1                 Open,STABLE
      2        ONLINE  ONLINE       rhel12c2                 Open,STABLE
--------------------------------------------------------------------------------
As the scan name changed along with the cluster name, update the remote_listener parameter value.
alter system set remote_listener='prod-cluster-scan.rac.domain.net:1521' scope=both sid='*';
This concludes the renaming of the cluster.

Useful metalink notes
How to Configure or Re-configure Grid Infrastructure With config.sh/config.bat [ID 1354258.1]
How to change the cluster name in RAC [ID 1967916.1]