Friday, June 21, 2019

Adding a Node to 18c RAC Using Cloning

This post shows steps of adding a node by way of cloning on 18c. On 19c, extending RAC by way of cloning is depreciated. However, the steps are still listed on 19c clusterware.
The post assumes all other pre-reqs related to adding a node have been completed. The new node is called rhel72.
1. As the first step create an image from an existing node. Steps in creating a gold image could be used for this.
2. Unzip gold image in the new node on the same directory structure as existing nodes.
3. Removing following files from the new node.
rm -rf $GI_HOME/network/admin/*.ora
rm -rf $GI_HOMEroot.sh*
4. Run gridsetup and select software only option.
Following is promoted about missing root.sh files. Ignore and click yes to continue.
Summary of software only setup.
When prompted execute the root scripts.
Root script output is shown below.
# /opt/app/18.x.0/grid/root.sh
Performing root user operation.

The following environment variables are set as:
    ORACLE_OWNER= grid
    ORACLE_HOME=  /opt/app/18.x.0/grid

Creating /etc/oratab file...
Entries will be added to the /etc/oratab file as needed by
Database Configuration Assistant when a database is created
Finished running generic part of root script.
Now product-specific root actions will be performed.

To configure Grid Infrastructure for a Cluster or Grid Infrastructure for a Stand-Alone Server execute the following command as grid user:
/opt/app/18.x.0/grid/gridSetup.sh
This command launches the Grid Infrastructure Setup Wizard. The wizard also supports silent operation, and the parameters can be passed through the response file that is available in the installation media.
Complete the GI software only install.

5. From an existing node run gridSetup with noCopy option and select add node.
on exisitng node run
gridsetup.sh -noCopy
The noCopy option will prevent GI binaries being copied to remote node being added, since it's already has software installed. Rest of the steps for adding the node is same as previous add node post.Run the root.sh on the new node when prompted. Output from root.sh on the new node is shown below.
# /opt/app/18.x.0/grid/root.sh
Performing root user operation.

The following environment variables are set as:
    ORACLE_OWNER= grid
    ORACLE_HOME=  /opt/app/18.x.0/grid

Enter the full pathname of the local bin directory: [/usr/local/bin]:
The contents of "dbhome" have not changed. No need to overwrite.
The contents of "oraenv" have not changed. No need to overwrite.
The contents of "coraenv" have not changed. No need to overwrite.

Entries will be added to the /etc/oratab file as needed by
Database Configuration Assistant when a database is created
Finished running generic part of root script.
Now product-specific root actions will be performed.
Relinking oracle with rac_on option
Using configuration parameter file: /opt/app/18.x.0/grid/crs/install/crsconfig_params
The log of current session can be found at:
  /opt/app/oracle/crsdata/rhel72/crsconfig/rootcrs_rhel72_2019-04-09_11-10-41AM.log
2019/04/09 11:10:51 CLSRSC-594: Executing installation step 1 of 20: 'SetupTFA'.
2019/04/09 11:10:51 CLSRSC-4001: Installing Oracle Trace File Analyzer (TFA) Collector.
2019/04/09 11:11:32 CLSRSC-4002: Successfully installed Oracle Trace File Analyzer (TFA) Collector.
2019/04/09 11:11:32 CLSRSC-594: Executing installation step 2 of 20: 'ValidateEnv'.
2019/04/09 11:11:43 CLSRSC-363: User ignored prerequisites during installation
2019/04/09 11:11:44 CLSRSC-594: Executing installation step 3 of 20: 'CheckFirstNode'.
2019/04/09 11:11:45 CLSRSC-594: Executing installation step 4 of 20: 'GenSiteGUIDs'.
2019/04/09 11:11:50 CLSRSC-594: Executing installation step 5 of 20: 'SaveParamFile'.
2019/04/09 11:11:57 CLSRSC-594: Executing installation step 6 of 20: 'SetupOSD'.
2019/04/09 11:11:57 CLSRSC-594: Executing installation step 7 of 20: 'CheckCRSConfig'.
2019/04/09 11:12:34 CLSRSC-17: Invalid GPnP setup
2019/04/09 11:12:43 CLSRSC-594: Executing installation step 8 of 20: 'SetupLocalGPNP'.
2019/04/09 11:12:45 CLSRSC-594: Executing installation step 9 of 20: 'CreateRootCert'.
2019/04/09 11:12:45 CLSRSC-594: Executing installation step 10 of 20: 'ConfigOLR'.
2019/04/09 11:12:56 CLSRSC-594: Executing installation step 11 of 20: 'ConfigCHMOS'.
2019/04/09 11:12:56 CLSRSC-594: Executing installation step 12 of 20: 'CreateOHASD'.
2019/04/09 11:12:59 CLSRSC-594: Executing installation step 13 of 20: 'ConfigOHASD'.
2019/04/09 11:12:59 CLSRSC-330: Adding Clusterware entries to file 'oracle-ohasd.service'
2019/04/09 11:14:46 CLSRSC-594: Executing installation step 14 of 20: 'InstallAFD'.
2019/04/09 11:14:50 CLSRSC-594: Executing installation step 15 of 20: 'InstallACFS'.
CRS-2791: Starting shutdown of Oracle High Availability Services-managed resources on 'rhel72'
CRS-2793: Shutdown of Oracle High Availability Services-managed resources on 'rhel72' has completed
CRS-4133: Oracle High Availability Services has been stopped.
CRS-4123: Oracle High Availability Services has been started.
2019/04/09 11:16:15 CLSRSC-594: Executing installation step 16 of 20: 'InstallKA'.
2019/04/09 11:16:17 CLSRSC-594: Executing installation step 17 of 20: 'InitConfig'.
CRS-2791: Starting shutdown of Oracle High Availability Services-managed resources on 'rhel72'
CRS-2793: Shutdown of Oracle High Availability Services-managed resources on 'rhel72' has completed
CRS-4133: Oracle High Availability Services has been stopped.
CRS-4123: Oracle High Availability Services has been started.
CRS-2791: Starting shutdown of Oracle High Availability Services-managed resources on 'rhel72'
CRS-2673: Attempting to stop 'ora.drivers.acfs' on 'rhel72'
CRS-2677: Stop of 'ora.drivers.acfs' on 'rhel72' succeeded
CRS-2793: Shutdown of Oracle High Availability Services-managed resources on 'rhel72' has completed
CRS-4133: Oracle High Availability Services has been stopped.
2019/04/09 11:16:31 CLSRSC-594: Executing installation step 18 of 20: 'StartCluster'.
CRS-4123: Starting Oracle High Availability Services-managed resources
CRS-2672: Attempting to start 'ora.mdnsd' on 'rhel72'
CRS-2672: Attempting to start 'ora.evmd' on 'rhel72'
CRS-2676: Start of 'ora.mdnsd' on 'rhel72' succeeded
CRS-2676: Start of 'ora.evmd' on 'rhel72' succeeded
CRS-2672: Attempting to start 'ora.gpnpd' on 'rhel72'
CRS-2676: Start of 'ora.gpnpd' on 'rhel72' succeeded
CRS-2672: Attempting to start 'ora.gipcd' on 'rhel72'
CRS-2676: Start of 'ora.gipcd' on 'rhel72' succeeded
CRS-2672: Attempting to start 'ora.crf' on 'rhel72'
CRS-2672: Attempting to start 'ora.cssdmonitor' on 'rhel72'
CRS-2676: Start of 'ora.cssdmonitor' on 'rhel72' succeeded
CRS-2672: Attempting to start 'ora.cssd' on 'rhel72'
CRS-2672: Attempting to start 'ora.diskmon' on 'rhel72'
CRS-2676: Start of 'ora.diskmon' on 'rhel72' succeeded
CRS-2676: Start of 'ora.crf' on 'rhel72' succeeded
CRS-2676: Start of 'ora.cssd' on 'rhel72' succeeded
CRS-2672: Attempting to start 'ora.cluster_interconnect.haip' on 'rhel72'
CRS-2672: Attempting to start 'ora.ctssd' on 'rhel72'
CRS-2676: Start of 'ora.ctssd' on 'rhel72' succeeded
CRS-2676: Start of 'ora.cluster_interconnect.haip' on 'rhel72' succeeded
CRS-2672: Attempting to start 'ora.asm' on 'rhel72'
CRS-2676: Start of 'ora.asm' on 'rhel72' succeeded
CRS-2672: Attempting to start 'ora.storage' on 'rhel72'
CRS-2676: Start of 'ora.storage' on 'rhel72' succeeded
CRS-2672: Attempting to start 'ora.crsd' on 'rhel72'
CRS-2676: Start of 'ora.crsd' on 'rhel72' succeeded
CRS-6017: Processing resource auto-start for servers: rhel72
CRS-2673: Attempting to stop 'ora.LISTENER_SCAN1.lsnr' on 'rhel71'
CRS-2672: Attempting to start 'ora.ASMNET1LSNR_ASM.lsnr' on 'rhel72'
CRS-2672: Attempting to start 'ora.ons' on 'rhel72'
CRS-2672: Attempting to start 'ora.chad' on 'rhel72'
CRS-2676: Start of 'ora.ASMNET1LSNR_ASM.lsnr' on 'rhel72' succeeded
CRS-2672: Attempting to start 'ora.asm' on 'rhel72'
CRS-2676: Start of 'ora.chad' on 'rhel72' succeeded
CRS-2677: Stop of 'ora.LISTENER_SCAN1.lsnr' on 'rhel71' succeeded
CRS-2673: Attempting to stop 'ora.scan1.vip' on 'rhel71'
CRS-2677: Stop of 'ora.scan1.vip' on 'rhel71' succeeded
CRS-2672: Attempting to start 'ora.scan1.vip' on 'rhel72'
CRS-2676: Start of 'ora.ons' on 'rhel72' succeeded
CRS-2676: Start of 'ora.scan1.vip' on 'rhel72' succeeded
CRS-2672: Attempting to start 'ora.LISTENER_SCAN1.lsnr' on 'rhel72'
CRS-2676: Start of 'ora.LISTENER_SCAN1.lsnr' on 'rhel72' succeeded
CRS-2676: Start of 'ora.asm' on 'rhel72' succeeded
CRS-2672: Attempting to start 'ora.DATA.dg' on 'rhel72'
CRS-2676: Start of 'ora.DATA.dg' on 'rhel72' succeeded
CRS-6016: Resource auto-start has completed for server rhel72
CRS-6024: Completed start of Oracle Cluster Ready Services-managed resources
CRS-4123: Oracle High Availability Services has been started.
2019/04/09 11:18:29 CLSRSC-343: Successfully started Oracle Clusterware stack
2019/04/09 11:18:29 CLSRSC-594: Executing installation step 19 of 20: 'ConfigNode'.
clscfg: EXISTING configuration version 5 detected.
clscfg: version 5 is 12c Release 2.
Successfully accumulated necessary OCR keys.
Creating OCR keys for user 'root', privgrp 'root'..
Operation successful.
2019/04/09 11:19:15 CLSRSC-594: Executing installation step 20 of 20: 'PostConfig'.
2019/04/09 11:19:28 CLSRSC-325: Configure Oracle Grid Infrastructure for a Cluster ... succeeded




6. Once the GI is cloned and node added, next is to clone the Oracle database software. For this too a gold image of the OH could be used.
7. Unzip the OH gold image on the new node following the same directory structure as existing nodes.
8. Run the clone script to configure and OH. On oracle documentation -noConfig is option is listed as part of the clone command. But on 18c this was not supported.
$ORACLE_HOME/perl/bin/perl clone.pl -silent -O 'CLUSTER_NODES={rhel71,rhel72}' -O LOCAL_NODE=rhel72 ORACLE_BASE=$ORACLE_BASE ORACLE_HOME=$ORACLE_HOME ORACLE_HOME_NAME=OraDB18Home1 -O -noConfig
Starting Oracle Universal Installer...

Checking Temp space: must be greater than 500 MB.   Actual 2595 MB    Passed
Checking swap space: must be greater than 500 MB.   Actual 3006 MB    Passed
Preparing to launch Oracle Universal Installer from /tmp/OraInstall2019-04-09_02-42-13PM. Please wait ...
[INS-04009] The argument [-noconfig] passed is not supported for the current context Clone
Run the clone command without the noConfig and cloning of OH completes without issue.
perl $ORACLE_HOME/clone/bin/clone.pl -silent ORACLE_HOME="/opt/app/oracle/product/18.x.0/dbhome_1"  ORACLE_HOME_NAME="OraDB18Home1" ORACLE_BASE="/opt/app/oracle" "CLUSTER_NODES={rhel71,rhel72}" LOCAL_NODE=rhel72
Starting Oracle Universal Installer...

Checking Temp space: must be greater than 500 MB.   Actual 3918 MB    Passed
Checking swap space: must be greater than 500 MB.   Actual 2974 MB    Passed
Preparing to launch Oracle Universal Installer from /tmp/OraInstall2019-04-09_03-26-55PM. Please wait ...You can find the log of this install session at:
 /opt/app/oraInventory/logs/cloneActions2019-04-09_03-26-55PM.log
..................................................   5% Done.
..................................................   10% Done.
..................................................   15% Done.
..................................................   20% Done.
..................................................   25% Done.
..................................................   30% Done.
..................................................   35% Done.
..................................................   40% Done.
..................................................   45% Done.
..................................................   50% Done.
..................................................   55% Done.
..................................................   60% Done.
..................................................   65% Done.
..................................................   70% Done.
..................................................   75% Done.
..................................................   80% Done.
..................................................   85% Done.
..........
Copy files in progress.

Copy files successful.

Link binaries in progress.
..........
Link binaries successful.

Setup files in progress.
..........
Setup files successful.

Setup Inventory in progress.

Setup Inventory successful.
..........
Finish Setup successful.
The cloning of OraDB18Home1 was successful.
Please check '/opt/app/oraInventory/logs/cloneActions2019-04-09_03-26-55PM.log' for more details.

Setup Oracle Base in progress.

Setup Oracle Base successful.
..................................................   95% Done.

As a root user, execute the following script(s):
        1. /opt/app/oracle/product/18.x.0/dbhome_1/root.sh

Execute /opt/app/oracle/product/18.x.0/dbhome_1/root.sh on the following nodes:
[rhel72]


..................................................   100% Done.

9. Run DBCA from an exiting node and select add instance option.