A! Help: rhel7

Showing posts with label rhel7. Show all posts

Wednesday, May 11, 2022

Upgrading Oracle Restart from 19c to 21c on RHEL 7

This post shows the steps for upgrading the grid infrastructure portion of 19c Oracle Restart to 21c.

1. The current Oracle restart configuration is as below.

Resource Name             Type                      Target             State              Host
-------------             ------                    -------            --------           ----------
ora.DATA.dg               ora.diskgroup.type        ONLINE             ONLINE             ip-172-31-10-91
ora.FRA.dg                ora.diskgroup.type        ONLINE             ONLINE             ip-172-31-10-91
ora.LISTENER.lsnr         ora.listener.type         ONLINE             ONLINE             ip-172-31-10-91
ora.asm                   ora.asm.type              ONLINE             ONLINE             ip-172-31-10-91
ora.cssd                  ora.cssd.type             ONLINE             ONLINE             ip-172-31-10-91
ora.diskmon               ora.diskmon.type          OFFLINE            OFFLINE
ora.evmd                  ora.evm.type              ONLINE             ONLINE             ip-172-31-10-91
ora.ons                   ora.ons.type              OFFLINE            OFFLINE
ora.testcdb.db            ora.database.type         ONLINE             ONLINE             ip-172-31-10-91

The software and release version of the HAS are as below.

crsctl query has releaseversion
Oracle High Availability Services release version on the local node is [19.0.0.0.0]

crsctl query has softwareversion
Oracle High Availability Services version on the local node is [19.0.0.0.0]

2. The current patch on 19c GI is 19.12.

$ORACLE_HOME/OPatch/opatch lspatches
32918050;TOMCAT RELEASE UPDATE 19.0.0.0.0 (32918050)
32904851;Database Release Update : 19.12.0.0.210720 (32904851)
32585572;DBWLM RELEASE UPDATE 19.0.0.0.0 (32585572)
32915586;ACFS RELEASE UPDATE 19.12.0.0.0 (32915586)
32916816;OCW RELEASE UPDATE 19.12.0.0.0 (32916816)

3. Make the directory for GI home and unzip the 21c installer to this directory.

mkdir -p /opt/app/oracle/product/21.x.0/grid
unzip LINUX.X64_213000_grid_home.zip -d /opt/app/oracle/product/21.x.0/grid

4. Run the cluster verification tool from 21c GI home with hacfg option. The failure listed below is due to 0 SWAP size on AWS node. This could be ignored.

./runcluvfy.sh stage -pre hacfg

Performing following verification checks ...

  Physical Memory ...PASSED
  Available Physical Memory ...PASSED
  Swap Size ...FAILED (PRVF-7573)
  Free Space: ip-172-31-10-91:/usr,ip-172-31-10-91:/var,ip-172-31-10-91:/etc,ip-172-31-10-91:/sbin,ip-172-31-10-91:/tmp ...PASSED
  User Existence: oracle ...
    Users With Same UID: 501 ...PASSED
  User Existence: oracle ...PASSED
  Group Existence: dba ...PASSED
  Group Existence: oinstall ...PASSED
  Group Membership: oinstall(Primary) ...PASSED
  Group Membership: dba ...PASSED
  Run Level ...PASSED
  Architecture ...PASSED
  OS Kernel Version ...PASSED
  OS Kernel Parameter: semmsl ...PASSED
  OS Kernel Parameter: semmns ...PASSED
  OS Kernel Parameter: semopm ...PASSED
  OS Kernel Parameter: semmni ...PASSED
  OS Kernel Parameter: shmmax ...PASSED
  OS Kernel Parameter: shmmni ...PASSED
  OS Kernel Parameter: shmall ...PASSED
  OS Kernel Parameter: file-max ...PASSED
  OS Kernel Parameter: ip_local_port_range ...PASSED
  OS Kernel Parameter: rmem_default ...PASSED
  OS Kernel Parameter: rmem_max ...PASSED
  OS Kernel Parameter: wmem_default ...PASSED
  OS Kernel Parameter: wmem_max ...PASSED
  OS Kernel Parameter: aio-max-nr ...PASSED
  Package: kmod-20-21 (x86_64) ...PASSED
  Package: kmod-libs-20-21 (x86_64) ...PASSED
  Package: binutils-2.23.52.0.1 ...PASSED
  Package: libgcc-4.8.2 (x86_64) ...PASSED
  Package: libstdc++-4.8.2 (x86_64) ...PASSED
  Package: sysstat-10.1.5 ...PASSED
  Package: ksh ...PASSED
  Package: make-3.82 ...PASSED
  Package: glibc-2.17 (x86_64) ...PASSED
  Package: glibc-devel-2.17 (x86_64) ...PASSED
  Package: libaio-0.3.109 (x86_64) ...PASSED
  Package: nfs-utils-1.2.3-15 ...PASSED
  Package: smartmontools-6.2-4 ...PASSED
  Package: net-tools-2.0-0.17 ...PASSED
  Package: libxcb-1.11 (x86_64) ...PASSED
  Package: libX11-1.6.3 (x86_64) ...PASSED
  Package: libXau-1.0.8 (x86_64) ...PASSED
  Package: libXi-1.7.4 (x86_64) ...PASSED
  Package: libXtst-1.2.2 (x86_64) ...PASSED
  Package: policycoreutils-python-2.5-17 ...PASSED
  Package: policycoreutils-2.5-17 ...PASSED
  Users With Same UID: 0 ...PASSED
  Current Group ID ...PASSED
  Root user consistency ...PASSED

Pre-check for Oracle Restart configuration was unsuccessful.


Failures were encountered during execution of CVU verification request "stage -pre hacfg".

Swap Size ...FAILED
ip-172-31-10-91: PRVF-7573 : Sufficient swap size is not available on node
                 "ip-172-31-10-91" [Required = 15.5115GB (1.6265024E7KB) ;
                 Found = 0.0 bytes]


CVU operation performed:      stage -pre hacfg
Date:                         Apr 25, 2022 12:56:04 PM
CVU home:                     /opt/app/oracle/product/21.x.0/grid
Grid home:                    /opt/app/oracle/product/19.x.0/grid
User:                         oracle
Operating system:             Linux3.10.0-1062.9.1.el7.x86_64

5. ASM will be upgraded as part of the upgrade process. Therefore stop the database before starting the GI upgrade. Upgrade process will prompt to stop the database during gridSetup run as well.

Stop the DB.

srvctl stop database -d testcdb

6. Run the gridsetup with '-applyRU' to apply the latest RU on 21c. One the patch is applied the OUI will appear.

./gridSetup.sh -applyRU /opt/installs/21c/psu_apr22/33859395
Preparing the home to patch...
Applying the patch /opt/installs/21c/psu_apr22/33859395...

Select upgrade GI

The unzip location is the GI home. Only Oracle base could be changed at install time.

During the pre-req check the following two items failed.

First item is related to ASM compatibility.

As shown on the message the asm.compatibility must be 11.2.0.2 or above. This was lower on the system used for this upgrade test. Upgrade the asm.compatibility to above 11.2.0.2.

NAME                           VALUE                          GROUP_NUMBER 
------------------------------ ------------------------------ ------------ 
compatible.asm                 19.0.0.0.0                                1 
compatible.rdbms               19.0.0.0.0                                1 
compatible.asm                 19.0.0.0.0                                2 
compatible.rdbms               19.0.0.0.0                                2

Once the asm.compatibility was updated this pre-req failure went away.
The second failure is related to disk discovery string.

The message states there's a mismatch between values in GnP profile and ASM instance. The asm disk discovery string could be at ASM instance level either with

SQL> alter system set asm_diskstring='/dev/oracleasm/cg*' scope=both;

dsset /dev/oracleasm/cg*

However, this did not resolve this pre-req failure and kept being flagged. Finally decided to ignore this pre-req failure and continue.
Summary page is shown below.

7. When prompted run the rootupgrade.sh.

# /opt/app/oracle/product/21.x.0/grid/rootupgrade.sh
Performing root user operation.

The following environment variables are set as:
    ORACLE_OWNER= oracle
    ORACLE_HOME=  /opt/app/oracle/product/21.x.0/grid

Enter the full pathname of the local bin directory: [/usr/local/bin]:
The contents of "dbhome" have not changed. No need to overwrite.
The file "oraenv" already exists in /usr/local/bin.  Overwrite it? (y/n)
[n]: y
   Copying oraenv to /usr/local/bin ...
The file "coraenv" already exists in /usr/local/bin.  Overwrite it? (y/n)
[n]: y
   Copying coraenv to /usr/local/bin ...

Entries will be added to the /etc/oratab file as needed by
Database Configuration Assistant when a database is created
Finished running generic part of root script.
Now product-specific root actions will be performed.
Using configuration parameter file: /opt/app/oracle/product/21.x.0/grid/crs/install/crsconfig_params
The log of current session can be found at:
  /opt/app/oracle/crsdata/ip-172-31-10-91/crsconfig/roothas_2022-04-25_01-50-58PM.log
2022/04/25 13:51:00 CLSRSC-595: Executing upgrade step 1 of 12: 'UpgPrechecks'.
2022/04/25 13:51:04 CLSRSC-595: Executing upgrade step 2 of 12: 'GetOldConfig'.
2022/04/25 13:51:06 CLSRSC-595: Executing upgrade step 3 of 12: 'GenSiteGUIDs'.
2022/04/25 13:51:06 CLSRSC-595: Executing upgrade step 4 of 12: 'SetupOSD'.
2022/04/25 13:51:07 CLSRSC-595: Executing upgrade step 5 of 12: 'PreUpgrade'.
2022/04/25 13:51:35 CLSRSC-595: Executing upgrade step 6 of 12: 'UpgradeAFD'.
2022/04/25 13:51:36 CLSRSC-595: Executing upgrade step 7 of 12: 'UpgradeOLR'.
clscfg: EXISTING configuration version 0 detected.
Creating OCR keys for user 'oracle', privgrp 'oinstall'..
Operation successful.
2022/04/25 13:51:42 CLSRSC-595: Executing upgrade step 8 of 12: 'UpgradeOCR'.
LOCAL ONLY MODE
Successfully accumulated necessary OCR keys.
Creating OCR keys for user 'root', privgrp 'root'..
Operation successful.
CRS-4664: Node ip-172-31-10-91 successfully pinned.
2022/04/25 13:51:45 CLSRSC-595: Executing upgrade step 9 of 12: 'CreateOHASD'.
2022/04/25 13:51:46 CLSRSC-595: Executing upgrade step 10 of 12: 'ConfigOHASD'.
2022/04/25 13:51:46 CLSRSC-329: Replacing Clusterware entries in file 'oracle-ohasd.service'
2022/04/25 13:52:03 CLSRSC-595: Executing upgrade step 11 of 12: 'UpgradeSIHA'.

ip-172-31-10-91     2022/04/25 13:52:55     /opt/app/oracle/crsdata/ip-172-31-10-91/olr/backup_20220425_135255.olr     498597253

ip-172-31-10-91     2021/08/27 12:17:16     /opt/app/oracle/crsdata/ip-172-31-10-91/olr/backup_20210827_121716.olr     1944883066
2022/04/25 13:52:56 CLSRSC-595: Executing upgrade step 12 of 12: 'InstallACFS'.
2022/04/25 13:53:54 CLSRSC-327: Successfully configured Oracle Restart for a standalone server

8. This would have successfully upgrade GI to 21c.

9. Current release and software version for HAS are as below.

crsctl query has releaseversion
Oracle High Availability Services release version on the local node is [21.0.0.0.0]

crsctl query has softwareversion
Oracle High Availability Services version on the local node is [21.0.0.0.0]

10. Run the cluvfy for post ugprade.

cluvfy stage -post hacfg

Performing following verification checks ...

  Oracle Restart Integrity ...PASSED
  OLR Integrity ...PASSED

Post-check for Oracle Restart configuration was successful.

CVU operation performed:      stage -post hacfg
Date:                         Apr 25, 2022 2:17:30 PM
CVU home:                     /opt/app/oracle/product/21.x.0/grid
Grid home:                    /opt/app/oracle/product/21.x.0/grid
User:                         oracle
Operating system:             Linux3.10.0-1062.9.1.el7.x86_64

Saturday, February 1, 2020

Upgrading Oracle Single Instance with ASM (Oracle Restart) from 11.2.0.4 to 19c (19.6) on RHEL 7

Oracle 19c is the last of the 12c series and is the long term support version. There's an earlier post which shows upgrading from 11.2.0.4 to 12.2.0.1. The 19c could be considered as 12.2.0.3. However, previous upgrade post was on a system running on RHEL 6. The 19c could only be installed on RHEL 7 (or higher when RHEL 8 gets certified). This post is based on upgrading a 11.2.0.4 Oracle restart environment on RHEL 7 to 19c. However, in this case the the Oracle restart setup is not role separated. Both GI and OH are installed under Oracle user.
The binaries used for 19c upgraded are gold images created from environment which had January 2020 PSU applied (19.6).
1. The current resources and software versions are as below.

Resource Name             Type                      Target             State              Host
-------------             ------                    -------            --------           ----------
ora.DATA.dg               ora.diskgroup.type        ONLINE             ONLINE             ip-172-31-7-244
ora.FRA.dg                ora.diskgroup.type        ONLINE             ONLINE             ip-172-31-7-244
ora.LISTENER.lsnr         ora.listener.type         ONLINE             ONLINE             ip-172-31-7-244
ora.asm                   ora.asm.type              ONLINE             ONLINE             ip-172-31-7-244
ora.cssd                  ora.cssd.type             ONLINE             ONLINE             ip-172-31-7-244
ora.diskmon               ora.diskmon.type          OFFLINE            OFFLINE
ora.evmd                  ora.evm.type              ONLINE             ONLINE             ip-172-31-7-244
ora.ons                   ora.ons.type              OFFLINE            OFFLINE
ora.test.db               ora.database.type         ONLINE             ONLINE             ip-172-31-7-244

$ crsctl query has releaseversion
Oracle High Availability Services release version on the local node is [11.2.0.4.0]

$ crsctl query has softwareversion
Oracle High Availability Services version on the local node is [11.2.0.4.0]

2. The 11.2.0.4 environment also has the latest available PSU applied. At the time of the post latest PSU was Jan-2020. MOS doc 2539751.1 list patches to apply before upgrade. In this upgrade path 28553832 and 23186035 were the only applicable patches as per doc. Installing the latest PSU, meant these patches were installed as part of the PSU. As such no need to do one-off patch instillations. Patch list shown below.

opatch lspatches

30503372;OJVM PATCH SET UPDATE 11.2.0.4.200114
30298532;Database Patch Set Update : 11.2.0.4.200114 (30298532)
29938455;OCW Patch Set Update : 11.2.0.4.191015 (29938455)

3. Make the directories for GI home unzip the gold image to these directories.

mkdir -p /opt/app/oracle/product/19.x.0/grid
unzip giGold_19.6.zip -d /opt/app/oracle/product/19.x.0/grid

4. Run the cluster verification tool from 19c GI home with hacfg option. The failure listed below is due to 0 SWAP size on AWS node. This could be ignored.

./runcluvfy.sh stage -pre hacfg

Verifying Physical Memory ...PASSED
Verifying Available Physical Memory ...PASSED
Verifying Swap Size ...FAILED (PRVF-7573)
Verifying Free Space: ip-172-31-7-244:/usr,ip-172-31-7-244:/var,ip-172-31-7-244:/etc,ip-172-31-7-244:/sbin,ip-172-31-7-244:/tmp ...PASSED
Verifying User Existence: oracle ...
  Verifying Users With Same UID: 501 ...PASSED
Verifying User Existence: oracle ...PASSED
Verifying Group Existence: dba ...PASSED
Verifying Group Membership: dba ...PASSED
Verifying Run Level ...PASSED
Verifying Architecture ...PASSED
Verifying OS Kernel Version ...PASSED
Verifying OS Kernel Parameter: semmsl ...PASSED
Verifying OS Kernel Parameter: semmns ...PASSED
Verifying OS Kernel Parameter: semopm ...PASSED
Verifying OS Kernel Parameter: semmni ...PASSED
Verifying OS Kernel Parameter: shmmax ...PASSED
Verifying OS Kernel Parameter: shmmni ...PASSED
Verifying OS Kernel Parameter: shmall ...PASSED
Verifying OS Kernel Parameter: file-max ...PASSED
Verifying OS Kernel Parameter: ip_local_port_range ...PASSED
Verifying OS Kernel Parameter: rmem_default ...PASSED
Verifying OS Kernel Parameter: rmem_max ...PASSED
Verifying OS Kernel Parameter: wmem_default ...PASSED
Verifying OS Kernel Parameter: wmem_max ...PASSED
Verifying OS Kernel Parameter: aio-max-nr ...PASSED
Verifying Package: kmod-20-21 (x86_64) ...PASSED
Verifying Package: kmod-libs-20-21 (x86_64) ...PASSED
Verifying Package: binutils-2.23.52.0.1 ...PASSED
Verifying Package: compat-libcap1-1.10 ...PASSED
Verifying Package: libgcc-4.8.2 (x86_64) ...PASSED
Verifying Package: libstdc++-4.8.2 (x86_64) ...PASSED
Verifying Package: libstdc++-devel-4.8.2 (x86_64) ...PASSED
Verifying Package: sysstat-10.1.5 ...PASSED
Verifying Package: ksh ...PASSED
Verifying Package: make-3.82 ...PASSED
Verifying Package: glibc-2.17 (x86_64) ...PASSED
Verifying Package: glibc-devel-2.17 (x86_64) ...PASSED
Verifying Package: libaio-0.3.109 (x86_64) ...PASSED
Verifying Package: libaio-devel-0.3.109 (x86_64) ...PASSED
Verifying Package: nfs-utils-1.2.3-15 ...PASSED
Verifying Package: smartmontools-6.2-4 ...PASSED
Verifying Package: net-tools-2.0-0.17 ...PASSED
Verifying Package: compat-libstdc++-33-3.2.3 (x86_64) ...PASSED
Verifying Package: libxcb-1.11 (x86_64) ...PASSED
Verifying Package: libX11-1.6.3 (x86_64) ...PASSED
Verifying Package: libXau-1.0.8 (x86_64) ...PASSED
Verifying Package: libXi-1.7.4 (x86_64) ...PASSED
Verifying Package: libXtst-1.2.2 (x86_64) ...PASSED
Verifying Users With Same UID: 0 ...PASSED
Verifying Current Group ID ...PASSED
Verifying Root user consistency ...PASSED

Pre-check for Oracle Restart configuration was unsuccessful.

Failures were encountered during execution of CVU verification request "stage -pre hacfg".

5. ASM will be upgraded as part of the upgrade process. Therefore stop the database before starting the GI upgrade. Upgrade process will prompt to stop the database during gridSetup run as well.

srvctl stop database -d test

6. Start the GI upgrade by running the gridSetup.sh from the grid home. Not all steps are shown below.

$ ./gridSetup.sh

Select upgrade GI.

Similar to previous 12c versions, unzip location is the GI home. Only Oracle base could be changed at install time.

GI Upgrade summary.

When prompted run rootupgrade.sh

/opt/app/oracle/product/19.x.0/grid/rootupgrade.sh
Performing root user operation.
...

Entries will be added to the /etc/oratab file as needed by
Database Configuration Assistant when a database is created
Finished running generic part of root script.
Now product-specific root actions will be performed.
Using configuration parameter file: /opt/app/oracle/product/19.x.0/grid/crs/install/crsconfig_params
The log of current session can be found at:
  /opt/app/oracle/crsdata/ip-172-31-7-244/crsconfig/roothas_2020-01-21_04-17-39PM.log
2020/01/21 16:17:40 CLSRSC-595: Executing upgrade step 1 of 12: 'UpgPrechecks'.
2020/01/21 16:17:42 CLSRSC-363: User ignored prerequisites during installation
2020/01/21 16:17:43 CLSRSC-595: Executing upgrade step 2 of 12: 'GetOldConfig'.
2020/01/21 16:17:45 CLSRSC-595: Executing upgrade step 3 of 12: 'GenSiteGUIDs'.
2020/01/21 16:17:45 CLSRSC-595: Executing upgrade step 4 of 12: 'SetupOSD'.
2020/01/21 16:17:46 CLSRSC-595: Executing upgrade step 5 of 12: 'PreUpgrade'.

ASM has been upgraded and started successfully.

2020/01/21 16:21:40 CLSRSC-595: Executing upgrade step 6 of 12: 'UpgradeAFD'.
2020/01/21 16:21:40 CLSRSC-595: Executing upgrade step 7 of 12: 'UpgradeOLR'.
clscfg: EXISTING configuration version 0 detected.
Creating OCR keys for user 'oracle', privgrp 'oinstall'..
Operation successful.
2020/01/21 16:21:48 CLSRSC-595: Executing upgrade step 8 of 12: 'UpgradeOCR'.
LOCAL ONLY MODE
Successfully accumulated necessary OCR keys.
Creating OCR keys for user 'root', privgrp 'root'..
Operation successful.
CRS-4664: Node ip-172-31-7-244 successfully pinned.
2020/01/21 16:21:51 CLSRSC-595: Executing upgrade step 9 of 12: 'CreateOHASD'.
2020/01/21 16:21:52 CLSRSC-595: Executing upgrade step 10 of 12: 'ConfigOHASD'.
2020/01/21 16:21:52 CLSRSC-329: Replacing Clusterware entries in file 'oracle-ohasd.service'
2020/01/21 16:22:40 CLSRSC-595: Executing upgrade step 11 of 12: 'UpgradeSIHA'.

ip-172-31-7-244     2020/01/21 16:24:11     /opt/app/oracle/crsdata/ip-172-31-7-244/olr/backup_20200121_162411.olr     1748936676

ip-172-31-7-244     2020/01/21 14:25:40     /opt/app/oracle/product/11.2.0/grid/cdata/ip-172-31-7-244/backup_20200121_142540.olr     -
2020/01/21 16:24:11 CLSRSC-595: Executing upgrade step 12 of 12: 'InstallACFS'.
2020/01/21 16:26:36 CLSRSC-327: Successfully configured Oracle Restart for a standalone server

Conclude the GI upgrade.

7. Verify the post upgrade versions and setup.

$ crsctl query has releaseversion
Oracle High Availability Services release version on the local node is [19.0.0.0.0]

$ crsctl query has softwareversion
Oracle High Availability Services version on the local node is [19.0.0.0.0]

cluvfy stage -post hacfg

Verifying Oracle Restart Integrity ...PASSED
Verifying OLR Integrity ...PASSED

Post-check for Oracle Restart configuration was successful.

8. Next step is to upgrade the oracle home. This is a simple installation of 19c software as DB is upgraded later. The 12.2 series introduced a new OS group for RAC DB management. If this is used then add the user group to both oracle and grid. Not adding the group to grid user result in an issue which is mentioned in 12.1.0.2 to 12.2.1.0 single instance upgrade post.

# groupadd racdba
# usermod -g oinstall -G dba,oper,asmoper,asmdba,asmadmin,backupdba,dgdba,kmdba,racdba oracle
# id oracle
uid=501(oracle) gid=502(oinstall) groups=502(oinstall),501(dba),503(oper),504(asmoper),505(asmdba),506(asmadmin),1001(backupdba),1002(dgdba),1003(kmdba),1004(racdba)

9. Unzip the OH gold image to 19c Oracle home.

mkdir -p /opt/app/oracle/product/19.x.0/dbhome_1
unzip ohGold_19.6.zip -d /opt/app/oracle/product/19.x.0/dbhome_1

10. Run the installer and select software only option. Rest of the steps are similar to installing a standalone OH.

11. When prompted run root.sh.

# /opt/app/oracle/product/19.x.0/dbhome_1/root.sh
Performing root user operation.

The following environment variables are set as:
    ORACLE_OWNER= oracle
    ORACLE_HOME=  /opt/app/oracle/product/19.x.0/dbhome_1

Enter the full pathname of the local bin directory: [/usr/local/bin]:
The contents of "dbhome" have not changed. No need to overwrite.
The contents of "oraenv" have not changed. No need to overwrite.
The contents of "coraenv" have not changed. No need to overwrite.

Entries will be added to the /etc/oratab file as needed by
Database Configuration Assistant when a database is created
Finished running generic part of root script.
Now product-specific root actions will be performed.
Oracle Trace File Analyzer (TFA - Standalone Mode) is available at :
    /opt/app/oracle/product/19.x.0/dbhome_1/bin/tfactl

Note :
1. tfactl will use TFA Service if that service is running and user has been granted access
2. tfactl will configure TFA Standalone Mode only if user has no access to TFA Service or TFA is not installed

12. Final step is upgrade the 11.2.0.4 DB to 19.6. Oracle has introduced AutoUpgrade Tool (2485457.1) to automate the upgrade process. However, in this case manual upgrade is done using dbua. Copy the preupgrade.jar (available on 19c OH rdbms/admin) to a temporary location run with terminal option.

$ORACLE_HOME/jdk/bin/java -jar preupgrade.jar TERMINAL
Report generated by Oracle Database Pre-Upgrade Information Tool Version
19.0.0.0.0 Build: 1 on 2020-01-22T11:03:49

Upgrade-To version: 19.0.0.0.0

=======================================
Status of the database prior to upgrade
=======================================
      Database Name:  TEST
     Container Name:  Not Applicable in Pre-12.1 database
       Container ID:  Not Applicable in Pre-12.1 database
            Version:  11.2.0.4.0
     DB Patch Level:  PSU 11.2.0.4.200114
         Compatible:  11.2.0.4.0
          Blocksize:  8192
           Platform:  Linux x86 64-bit
      Timezone File:  14
  Database log mode:  ARCHIVELOG
           Readonly:  FALSE
            Edition:  EE

  Oracle Component                       Upgrade Action    Current Status
  ----------------                       --------------    --------------
  Oracle Server                          [to be upgraded]  VALID
  Oracle Workspace Manager               [to be upgraded]  VALID
  Oracle Enterprise Manager Repository   [to be upgraded]  VALID
  Oracle Text                            [to be upgraded]  VALID
  Oracle XML Database                    [to be upgraded]  VALID

==============
BEFORE UPGRADE
==============

  REQUIRED ACTIONS
  ================
  1.  (AUTOFIXUP) Empty the RECYCLEBIN immediately before database upgrade.

      The database contains 2 objects in the recycle bin.

      The recycle bin must be completely empty before database upgrade.

  RECOMMENDED ACTIONS
  ===================
  2.  Remove the EM repository.

      - Copy the $ORACLE_HOME/rdbms/admin/emremove.sql script from the target
      19 ORACLE_HOME into the source 11.2.0.4.0 ORACLE_HOME.

      Step 1: If database control is configured, stop EM Database Control,
      using the following command

        $> emctl stop dbconsole

      Step 2: Connect to the database using the SYS account AS SYSDBA

        SET ECHO ON;
        SET SERVEROUTPUT ON;
        @emremove.sql

      Without the set echo and serveroutput commands, you will not be able to
      follow the progress of the script.

      The database has an Enterprise Manager Database Control repository.

      Starting with Oracle Database 12c, the local Enterprise Manager Database
      Control does not exist anymore. The repository will be removed from your
      database during the upgrade.  This step can be manually performed before
      the upgrade to reduce downtime.

  3.  (AUTOFIXUP) Directly grant ADMINISTER DATABASE TRIGGER privilege to the
      owner of the trigger or drop and re-create the trigger with a user that
      was granted directly with such. You can list those triggers using: SELECT
      OWNER, TRIGGER_NAME FROM DBA_TRIGGERS WHERE
      TRIM(BASE_OBJECT_TYPE)='DATABASE' AND OWNER NOT IN (SELECT GRANTEE FROM
      DBA_SYS_PRIVS WHERE PRIVILEGE='ADMINISTER DATABASE TRIGGER').

      There is one or more database triggers whose owner does not have the
      right privilege on the database.

      The creation of database triggers must be done by users granted with
      ADMINISTER DATABASE TRIGGER privilege. Privilege must have been granted
      directly.

  4.  (AUTOFIXUP) Gather statistics on fixed objects prior the upgrade.

      None of the fixed object tables have had stats collected.

      Gathering statistics on fixed objects, if none have been gathered yet, is
      recommended prior to upgrading.

      For information on managing optimizer statistics, refer to the 11.2.0.4
      Oracle Database Performance Tuning Guide.

  INFORMATION ONLY
  ================
  5.  To help you keep track of your tablespace allocations, the following
      AUTOEXTEND tablespaces are expected to successfully EXTEND during the
      upgrade process.

                                                 Min Size
      Tablespace                        Size     For Upgrade
      ----------                     ----------  -----------
      TEMP                                20 MB       150 MB
      UNDOTBS1                           380 MB       411 MB

      Minimum tablespace sizes for upgrade are estimates.

  6.  Check the Oracle Backup and Recovery User's Guide for information on how
      to manage an RMAN recovery catalog schema.

      If you are using a version of the recovery catalog schema that is older
      than that required by the RMAN client version, then you must upgrade the
      catalog schema.

      It is good practice to have the catalog schema the same or higher version
      than the RMAN client version you are using.

  ORACLE GENERATED FIXUP SCRIPT
  =============================
  All of the issues in database TEST
  which are identified above as BEFORE UPGRADE "(AUTOFIXUP)" can be resolved by
  executing the following

    SQL>@/opt/app/oracle/cfgtoollogs/test/preupgrade/preupgrade_fixups.sql

=============
AFTER UPGRADE
=============

  REQUIRED ACTIONS
  ================
  None

  RECOMMENDED ACTIONS
  ===================
  7.  Upgrade the database time zone file using the DBMS_DST package.

      The database is using time zone file version 14 and the target 19 release
      ships with time zone file version 32.

      Oracle recommends upgrading to the desired (latest) version of the time
      zone file.  For more information, refer to "Upgrading the Time Zone File
      and Timestamp with Time Zone Data" in the 19 Oracle Database
      Globalization Support Guide.

  8.  (AUTOFIXUP) Gather dictionary statistics after the upgrade using the
      command:

        EXECUTE DBMS_STATS.GATHER_DICTIONARY_STATS;

      Oracle recommends gathering dictionary statistics after upgrade.

      Dictionary statistics provide essential information to the Oracle
      optimizer to help it find efficient SQL execution plans. After a database
      upgrade, statistics need to be re-gathered as there can now be tables
      that have significantly changed during the upgrade or new tables that do
      not have statistics gathered yet.

  9.  Gather statistics on fixed objects after the upgrade and when there is a
      representative workload on the system using the command:

        EXECUTE DBMS_STATS.GATHER_FIXED_OBJECTS_STATS;

      This recommendation is given for all preupgrade runs.

      Fixed object statistics provide essential information to the Oracle
      optimizer to help it find efficient SQL execution plans.  Those
      statistics are specific to the Oracle Database release that generates
      them, and can be stale upon database upgrade.

      For information on managing optimizer statistics, refer to the 11.2.0.4
      Oracle Database Performance Tuning Guide.

  ORACLE GENERATED FIXUP SCRIPT
  =============================
  All of the issues in database TEST
  which are identified above as AFTER UPGRADE "(AUTOFIXUP)" can be resolved by
  executing the following

    SQL>@/opt/app/oracle/cfgtoollogs/test/preupgrade/postupgrade_fixups.sql


==================
PREUPGRADE SUMMARY
==================
  /opt/app/oracle/cfgtoollogs/test/preupgrade/preupgrade.log
  /opt/app/oracle/cfgtoollogs/test/preupgrade/preupgrade_fixups.sql
  /opt/app/oracle/cfgtoollogs/test/preupgrade/postupgrade_fixups.sql

Execute fixup scripts as indicated below:

Before upgrade:

Log into the database and execute the preupgrade fixups
@/opt/app/oracle/cfgtoollogs/test/preupgrade/preupgrade_fixups.sql

After the upgrade:

Log into the database and execute the postupgrade fixups
@/opt/app/oracle/cfgtoollogs/test/preupgrade/postupgrade_fixups.sql

Preupgrade complete: 2020-01-22T11:03:49

13. Any item marked with "(AUTOFIXUP)" could be fixed by running the preupgrade_fixups.sql. If an item is not marked (AUTOFIXUP) then manual intervention is needed.

SQL> @/opt/app/oracle/cfgtoollogs/test/preupgrade/preupgrade_fixups.sql
Executing Oracle PRE-Upgrade Fixup Script

Auto-Generated by:       Oracle Preupgrade Script
                         Version: 19.0.0.0.0 Build: 1
Generated on:            2020-01-22 11:03:47

For Source Database:     TEST
Source Database Version: 11.2.0.4.0
For Upgrade to Version:  19.0.0.0.0

Preup                             Preupgrade
Action                            Issue Is
Number  Preupgrade Check Name     Remedied    Further DBA Action
------  ------------------------  ----------  --------------------------------
    1.  purge_recyclebin          YES         None.
    2.  em_present                YES         None.
    3.  trgowner_no_admndbtrg     YES         None.
    4.  pre_fixed_objects         YES         None.
    5.  tablespaces_info          NO          Informational only.
                                              Further action is optional.
    6.  rman_recovery_version     NO          Informational only.
                                              Further action is optional.

The fixup scripts have been run and resolved what they can. However,
there are still issues originally identified by the preupgrade that
have not been remedied and are still present in the database.
Depending on the severity of the specific issue, and the nature of
the issue itself, that could mean that your database is not ready
for upgrade.  To resolve the outstanding issues, start by reviewing
the preupgrade_fixups.sql and searching it for the name of
the failed CHECK NAME or Preupgrade Action Number listed above.
There you will find the original corresponding diagnostic message
from the preupgrade which explains in more detail what still needs
to be done.

PL/SQL procedure successfully completed.

14. Run the DBUA from the 19c home to upgrade the DB.

The 19c uses timezone file version 32. It is possible to upgrade the timezone file at the same time as DB.

If guaranteed restore point is used as a backup option to recover from upgrade failure then this restore point must be dropped before advancing the compatible parameters on ASM and DB. Otherwise database mount will fail with

ORA-38880 signalled during: ALTER DATABASE MOUNT

38880, 00000, "Cannot advance compatibility from %s to %s due to guaranteed restore points"
// *Cause: Flashback database cannot undo the advance of database
//         compatibility.  Therefore, one cannot advance the compatibility
//         of the database while there are guaranteed restore points in the
//         database.
// *Action: Drop all guaranteed restore points first and retry, or delay
//          the advance of database compatibility to a later time.

Pre Uprade Summary

Post ugprade Summary

15. Run postupgrade_fixups.sql to complete the upgrade.

@/opt/app/oracle/cfgtoollogs/test/preupgrade/postupgrade_fixups.sql

Executing Oracle POST-Upgrade Fixup Script

Auto-Generated by:       Oracle Preupgrade Script
                         Version: 19.0.0.0.0 Build: 1
Generated on:            2020-01-22 11:13:31

For Source Database:     TEST
Source Database Version: 11.2.0.4.0
For Upgrade to Version:  19.0.0.0.0

Preup                             Preupgrade
Action                            Issue Is
Number  Preupgrade Check Name     Remedied    Further DBA Action
------  ------------------------  ----------  --------------------------------
    3.  old_time_zones_exist      YES         None.
    4.  post_dictionary           YES         None.
    5.  post_fixed_objects        NO          Informational only.
                                              Further action is optional.

The fixup scripts have been run and resolved what they can. However,
there are still issues originally identified by the preupgrade that
have not been remedied and are still present in the database.
Depending on the severity of the specific issue, and the nature of
the issue itself, that could mean that your database upgrade is not
fully complete.  To resolve the outstanding issues, start by reviewing
the postupgrade_fixups.sql and searching it for the name of
the failed CHECK NAME or Preupgrade Action Number listed above.
There you will find the original corresponding diagnostic message
from the preupgrade which explains in more detail what still needs
to be done.

PL/SQL procedure successfully completed.

Session altered.

16. Check the database components and versions.

COMP_NAME                           STATUS     VERSION         VERSION_FULL
----------------------------------- ---------- --------------- ---------------
Oracle Database Catalog Views       VALID      19.0.0.0.0      19.6.0.0.0
Oracle Database Packages and Types  VALID      19.0.0.0.0      19.6.0.0.0
Oracle Real Application Clusters    OPTION OFF 19.0.0.0.0      19.6.0.0.0
Oracle Workspace Manager            VALID      19.0.0.0.0      19.6.0.0.0
Oracle Text                         VALID      19.0.0.0.0      19.6.0.0.0
Oracle XML Database                 VALID      19.0.0.0.0      19.6.0.0.0

Registory history output.

ACTION_TIME                    ACTION     NAMESPACE  VERSION         COMMENTS
------------------------------ ---------- ---------- --------------- ------------------------------------------------------------
21-JAN-20 03.30.26.273084 PM   APPLY      SERVER     11.2.0.4        Patchset 11.2.0.2.0
21-JAN-20 04.07.12.218350 PM   APPLY      SERVER     11.2.0.4        PSU 11.2.0.4.200114
                               BOOTSTRAP  DATAPATCH  19              RDBMS_19.6.0.0.0DBRU_LINUX.X64_191217
22-JAN-20 11.50.55.143749 AM   RU_APPLY   SERVER     19.0.0.0.0      Patch applied on 19.6.0.0.0: Release_Update - 191217155004
22-JAN-20 11.52.39.733756 AM   UPGRADE    SERVER     19.0.0.0.0      Upgraded from 11.2.0.4.0 to 19.6.0.0.0

17. Check the timezone file version is 32.

SQL> select * from v$timezone_file;

FILENAME                VERSION     CON_ID
-------------------- ---------- ----------
timezlrg_32.dat              32          0

18. If satisfied with upgrade advance the compatible parameters. This is a irreversible change. On DB instance

SQL> alter system set compatible='19.0.0' scope=spfile;
shutdown immediate;

On ASM instance

SQL> alter diskgroup FRA SET attribute 'compatible.asm'='19.0.0.0.0';
SQL> alter diskgroup DATA  SET attribute 'compatible.asm'='19.0.0.0.0';
SQL> alter diskgroup fra set attribute 'compatible.rdbms'='19.0.0.0.0';
SQL> alter diskgroup data set attribute 'compatible.rdbms'='19.0.0.0.0';

SQL> select g.name,a.name,a.value from v$asm_diskgroup g, v$asm_attribute a where g.group_number=a.group_number and a.name like '%compat%';

NAME                 NAME                 VALUE
-------------------- -------------------- --------------------
DATA                 compatible.asm       19.0.0.0.0
DATA                 compatible.rdbms     19.0.0.0.0
FRA                  compatible.asm       19.0.0.0.0
FRA                  compatible.rdbms     19.0.0.0.0

19. Run orachk with post upgrade check option on the upgraded setup.
This concludes the upgrading from 11.2.0.4 to 19c.

Known Issues
If rootupgrade.sh fails with the following output

2020/01/23 12:33:30 CLSRSC-595: Executing upgrade step 1 of 12: 'UpgPrechecks'.
2020/01/23 12:33:33 CLSRSC-595: Executing upgrade step 2 of 12: 'GetOldConfig'.

CRS-6705: Oracle Clusterware Release Version ('19.0.0.0.0') does not match Software Version ('11.2.0.4.0'). Oracle Clusterware cannot be started.

CRS-4000: Command Start failed, or completed with errors.

2020/01/23 12:33:50 CLSRSC-318: Failed to start Oracle OHASD service

Died at /u01/app/oracle/product/19.0.0.0/grid/crs/install/crsupgrade.pm line 6269.

Check if the 11.2.0.4 HA stack is up and running at the time of the rootupgrade.sh execute.

High Versions Count resulting in poor performance
12.2 series has changed the default value for cursor obsolete threshold. This could lead to poor performance once upgraded to 19c (or any other 12.2 version). More on 2431353.1. To resolve any potential performance issues set the hiddne parameter to 11.2.0.4 value.

alter system set "_cursor_obsolete_threshold"=1024 scope=spfile;

Useful Metalink notes
Oracle 19c - Complete Checklist for Manual Upgrades to Non-CDB Oracle Database 19c [ID 2539778.1]
DATABASE UPGRADE TO 19C REPORTS ERROR "ORA-01403: NOT DATA FOUND" [ID 2579106.1]
Bug 30238715 - ORA-600 [qctfrc: csform] After Upgrade To 19c When Query Has Interval YEAR- MONTH Partition Type [ID 30238715.8]
Patches to apply before upgrading Oracle GI and DB to 19c or downgrading to previous release [ID 2539751.1]

Related Posts
Upgrading Oracle Restart from 19c to 21c on RHEL 7
Upgrading Oracle Restart from 18c (18.6) to 19c (19.3)
Upgrading Oracle Restart from 12.2.0.1 to 18c
Upgrading Oracle Single Instance with ASM (Oracle Restart) from 11.2.0.4 to 12.2.0.1
Upgrading Oracle Single Instance with ASM (Oracle Restart) from 12.1.0.2 to 12.2.0.1
Upgrading Single Instance on ASM from 11.2.0.3 to 11.2.0.4
Upgrading Grid Infrastructure Used for Single Instance from 11.2.0.4 to 12.1.0.2

Sunday, April 7, 2019

Installing 18c (18.3) RAC on RHEL 7 with Role Separation - Database

Following the 18c clusterware installation the next step is to install database software and create the database. The database software installation is not much different from that of 12.2 database software installation. Even though the oracle installation guides states gcc and gcc-c++ are not required for RHEL, the cluvfy still searches for gcc-c++.
1. Create the Oracle home for the database software and unzip the installer into the directory.

cd $ORACLE_BASE
mkdir -p product/18.x.0/dbhome_1
unzip LINUX.X64_180000_db_home.zip -d /opt/app/oracle/product/18.x.0/dbhome_1/

2. Run cluvfy to check the pre dbinst state. The failures related to shm could be ignored

cluvfy stage -pre dbinst -allnodes -r 18

Verifying Physical Memory ...PASSED
Verifying Available Physical Memory ...PASSED
Verifying Swap Size ...PASSED
Verifying Free Space: rhel72:/tmp ...PASSED
Verifying Free Space: rhel71:/tmp ...PASSED
Verifying User Existence: oracle ...
  Verifying Users With Same UID: 501 ...PASSED
Verifying User Existence: oracle ...PASSED
Verifying Group Existence: dba ...PASSED
Verifying Group Existence: oinstall ...PASSED
Verifying Group Membership: oinstall(Primary) ...PASSED
Verifying Group Membership: dba ...PASSED
Verifying Run Level ...PASSED
Verifying Hard Limit: maximum open file descriptors ...PASSED
Verifying Soft Limit: maximum open file descriptors ...PASSED
Verifying Hard Limit: maximum user processes ...PASSED
Verifying Soft Limit: maximum user processes ...PASSED
Verifying Soft Limit: maximum stack size ...PASSED
Verifying Architecture ...PASSED
Verifying OS Kernel Version ...PASSED
Verifying OS Kernel Parameter: semmsl ...PASSED
Verifying OS Kernel Parameter: semmns ...PASSED
Verifying OS Kernel Parameter: semopm ...PASSED
Verifying OS Kernel Parameter: semmni ...PASSED
Verifying OS Kernel Parameter: shmmax ...PASSED
Verifying OS Kernel Parameter: shmmni ...PASSED
Verifying OS Kernel Parameter: shmall ...PASSED
Verifying OS Kernel Parameter: file-max ...PASSED
Verifying OS Kernel Parameter: ip_local_port_range ...PASSED
Verifying OS Kernel Parameter: rmem_default ...PASSED
Verifying OS Kernel Parameter: rmem_max ...PASSED
Verifying OS Kernel Parameter: wmem_default ...PASSED
Verifying OS Kernel Parameter: wmem_max ...PASSED
Verifying OS Kernel Parameter: aio-max-nr ...PASSED
Verifying Package: binutils-2.23.52.0.1 ...PASSED
Verifying Package: compat-libcap1-1.10 ...PASSED
Verifying Package: libgcc-4.8.2 (x86_64) ...PASSED
Verifying Package: libstdc++-4.8.2 (x86_64) ...PASSED
Verifying Package: libstdc++-devel-4.8.2 (x86_64) ...PASSED
Verifying Package: sysstat-10.1.5 ...PASSED
Verifying Package: gcc-c++-4.8.2 ...PASSED
Verifying Package: ksh ...PASSED
Verifying Package: make-3.82 ...PASSED
Verifying Package: glibc-2.17 (x86_64) ...PASSED
Verifying Package: glibc-devel-2.17 (x86_64) ...PASSED
Verifying Package: libaio-0.3.109 (x86_64) ...PASSED
Verifying Package: libaio-devel-0.3.109 (x86_64) ...PASSED
Verifying Package: smartmontools-6.2-4 ...PASSED
Verifying Package: net-tools-2.0-0.17 ...PASSED
Verifying Package: compat-libstdc++-33-3.2.3 (x86_64) ...PASSED
Verifying Package: libxcb-1.11 (x86_64) ...PASSED
Verifying Package: libX11-1.6.3 (x86_64) ...PASSED
Verifying Package: libXau-1.0.8 (x86_64) ...PASSED
Verifying Package: libXi-1.7.4 (x86_64) ...PASSED
Verifying Package: libXtst-1.2.2 (x86_64) ...PASSED
Verifying Users With Same UID: 0 ...PASSED
Verifying Current Group ID ...PASSED
Verifying Root user consistency ...PASSED
Verifying Host name ...PASSED
Verifying Node Connectivity ...
  Verifying Hosts File ...PASSED
  Verifying Check that maximum (MTU) size packet goes through subnet ...PASSED
  Verifying subnet mask consistency for subnet "192.168.1.0" ...PASSED
  Verifying subnet mask consistency for subnet "192.168.0.0" ...PASSED
Verifying Node Connectivity ...PASSED
Verifying Multicast or broadcast check ...PASSED
Verifying User Mask ...PASSED
Verifying CRS Integrity ...
  Verifying Clusterware Version Consistency ...PASSED
Verifying CRS Integrity ...PASSED
Verifying Cluster Manager Integrity ...PASSED
Verifying Node Application Existence ...PASSED
Verifying Clock Synchronization ...PASSED
Verifying resolv.conf Integrity ...PASSED
Verifying Time zone consistency ...PASSED
Verifying Single Client Access Name (SCAN) ...
  Verifying DNS/NIS name service 'rac-scan.domain.net' ...
    Verifying Name Service Switch Configuration File Integrity ...PASSED
  Verifying DNS/NIS name service 'rac-scan.domain.net' ...PASSED
Verifying Single Client Access Name (SCAN) ...PASSED
Verifying VIP Subnet configuration check ...PASSED
Verifying Database Clusterware Version Compatibility ...PASSED
Verifying ASM storage privileges for the user: oracle ...
  Verifying Group Membership: asmdba ...PASSED
Verifying ASM storage privileges for the user: oracle ...PASSED
Verifying Daemon "proxyt" not configured and running ...PASSED
Verifying ACFS device special file ...PASSED
Verifying /dev/shm mounted as temporary file system ...FAILED (PRVE-0421)
Verifying Maximum locked memory check ...PASSED

Pre-check for database installation was unsuccessful on all the nodes.


Failures were encountered during execution of CVU verification request "stage -pre dbinst".

Verifying /dev/shm mounted as temporary file system ...FAILED
rhel72: PRVE-0421 : No entry exists in /etc/fstab for mounting /dev/shm

rhel71: PRVE-0421 : No entry exists in /etc/fstab for mounting /dev/shm

3. Execute runInstaller from the Oracle home and select software only install option.

Other steps are similar to 12.2. However, one noticeable change is that in 18c the space available for Oracle base is also checked as part of pre-reqs. Any warning raised for test system could be ignored.

4. Once the database software is installed proceed to creating the database. As this is a role separate setup, certain folders need write permission for the oinstall group so oracle user could write into them.

ls -l
total 0
drwxr-x---.  5 grid oinstall  45 Mar 20 11:50 admin
drwxr-x---.  2 grid oinstall   6 Mar 20 11:51 audit

[grid@rhel71 oracle]$ chmod 770 admin audit

[grid@rhel71 oracle]$ ls -l
total 0
drwxrwx---.  5 grid oinstall  45 Mar 20 11:50 admin
drwxrwx---.  2 grid oinstall   6 Mar 20 11:51 audit

chmod 770 $ORACLE_BASE/cfgtoollogs/dbca

5. Run cluvfy could to check pre-requisite for RAC DB (output is without verbose option).

cluvfy stage -pre dbcfg -allnodes -d /opt/app/oracle/product/18.x.0/dbhome_1

Verifying Physical Memory ...PASSED
Verifying Available Physical Memory ...PASSED
Verifying Swap Size ...PASSED
Verifying Free Space: rhel72:/tmp ...PASSED
Verifying Free Space: rhel71:/tmp ...PASSED
Verifying User Existence: oracle ...
  Verifying Users With Same UID: 501 ...PASSED
Verifying User Existence: oracle ...PASSED
Verifying Group Existence: dgdba ...PASSED
Verifying Group Existence: dba ...PASSED
Verifying Group Existence: racdba ...PASSED
Verifying Group Existence: backupdba ...PASSED
Verifying Group Existence: oper ...PASSED
Verifying Group Existence: oinstall ...PASSED
Verifying Group Membership: backupdba ...PASSED
Verifying Group Membership: racdba ...PASSED
Verifying Group Membership: dgdba ...PASSED
Verifying Group Membership: oinstall(Primary) ...PASSED
Verifying Group Membership: dba ...PASSED
Verifying Group Membership: oper ...PASSED
Verifying Run Level ...PASSED
Verifying Hard Limit: maximum open file descriptors ...PASSED
Verifying Soft Limit: maximum open file descriptors ...PASSED
Verifying Hard Limit: maximum user processes ...PASSED
Verifying Soft Limit: maximum user processes ...PASSED
Verifying Soft Limit: maximum stack size ...PASSED
Verifying Architecture ...PASSED
Verifying OS Kernel Version ...PASSED
Verifying OS Kernel Parameter: semmsl ...PASSED
Verifying OS Kernel Parameter: semmns ...PASSED
Verifying OS Kernel Parameter: semopm ...PASSED
Verifying OS Kernel Parameter: semmni ...PASSED
Verifying OS Kernel Parameter: shmmax ...PASSED
Verifying OS Kernel Parameter: shmmni ...PASSED
Verifying OS Kernel Parameter: shmall ...PASSED
Verifying OS Kernel Parameter: file-max ...PASSED
Verifying OS Kernel Parameter: ip_local_port_range ...PASSED
Verifying OS Kernel Parameter: rmem_default ...PASSED
Verifying OS Kernel Parameter: rmem_max ...PASSED
Verifying OS Kernel Parameter: wmem_default ...PASSED
Verifying OS Kernel Parameter: wmem_max ...PASSED
Verifying OS Kernel Parameter: aio-max-nr ...PASSED
Verifying Package: binutils-2.23.52.0.1 ...PASSED
Verifying Package: compat-libcap1-1.10 ...PASSED
Verifying Package: libgcc-4.8.2 (x86_64) ...PASSED
Verifying Package: libstdc++-4.8.2 (x86_64) ...PASSED
Verifying Package: libstdc++-devel-4.8.2 (x86_64) ...PASSED
Verifying Package: sysstat-10.1.5 ...PASSED
Verifying Package: gcc-c++-4.8.2 ...PASSED
Verifying Package: ksh ...PASSED
Verifying Package: make-3.82 ...PASSED
Verifying Package: glibc-2.17 (x86_64) ...PASSED
Verifying Package: glibc-devel-2.17 (x86_64) ...PASSED
Verifying Package: libaio-0.3.109 (x86_64) ...PASSED
Verifying Package: libaio-devel-0.3.109 (x86_64) ...PASSED
Verifying Package: smartmontools-6.2-4 ...PASSED
Verifying Package: net-tools-2.0-0.17 ...PASSED
Verifying Package: compat-libstdc++-33-3.2.3 (x86_64) ...PASSED
Verifying Package: libxcb-1.11 (x86_64) ...PASSED
Verifying Package: libX11-1.6.3 (x86_64) ...PASSED
Verifying Package: libXau-1.0.8 (x86_64) ...PASSED
Verifying Package: libXi-1.7.4 (x86_64) ...PASSED
Verifying Package: libXtst-1.2.2 (x86_64) ...PASSED
Verifying Current Group ID ...PASSED
Verifying CRS Integrity ...
  Verifying Clusterware Version Consistency ...PASSED
Verifying CRS Integrity ...PASSED
Verifying Node Application Existence ...PASSED
Verifying Time zone consistency ...PASSED
Verifying Single Client Access Name (SCAN) ...
  Verifying DNS/NIS name service 'rac-scan.domain.net' ...
    Verifying Name Service Switch Configuration File Integrity ...PASSED
  Verifying DNS/NIS name service 'rac-scan.domain.net' ...PASSED
Verifying Single Client Access Name (SCAN) ...PASSED
Verifying ASM Integrity ...
  Verifying Node Connectivity ...
    Verifying Hosts File ...PASSED
    Verifying Check that maximum (MTU) size packet goes through subnet ...PASSED
    Verifying subnet mask consistency for subnet "192.168.1.0" ...PASSED
    Verifying subnet mask consistency for subnet "192.168.0.0" ...PASSED
  Verifying Node Connectivity ...PASSED
Verifying ASM Integrity ...PASSED
Verifying Database Clusterware Version Compatibility ...PASSED
Verifying File system mount options for path ORACLE_HOME ...PASSED
Verifying /dev/shm mounted as temporary file system ...FAILED (PRVE-0421)

Pre-check for database configuration was unsuccessful on all the nodes.


Failures were encountered during execution of CVU verification request "stage -pre dbcfg".

Verifying /dev/shm mounted as temporary file system ...FAILED
rhel72: PRVE-0421 : No entry exists in /etc/fstab for mounting /dev/shm

rhel71: PRVE-0421 : No entry exists in /etc/fstab for mounting /dev/shm

6. Create the database using dbca.

7. Create PDBs are required and associate services with them so PDBs are auto started. Assuming PDB named pdb18c is created, add a database service associated with PDB with

srvctl add service -db ent18c -pdb pdb18c -service pdbsrv -preferred "ent18c1,ent18c2"
srvctl start service -db ent18c -s pdbsrv

8. Finally, the stopping of the ASM proxy instances has no relevance to the "flex" functioning of DB instances. If an ASM instances in one node goes down, these DB instances are able to use a remote ASM instance to access the disks. This could be verified with the following.

[grid@rhel71 ~]$  srvctl status asm -detail
ASM is running on rhel71,rhel72
ASM is enabled.
ASM instance +ASM1 is running on node rhel71
Number of connected clients: 2
Client names: ent18c1:ent18c:rhel7-cluster rhel71.domain.net:_OCR:rhel7-cluster
ASM instance +ASM2 is running on node rhel72
Number of connected clients: 3
Client names: -MGMTDB:_mgmtdb:rhel7-cluster ent18c2:ent18c:rhel7-cluster rhel72.domain.net:_OCR:rhel7-cluster

The above output shows the clients connected to each ASM instance on each node. Currently ASM1 has 2 clients while ASM2 has 3 clients, *OCR is available as client on both ASM instances.
Then an ASM instance is stopped.

srvctl stop asm  -node rhel72  -force

When one of the ASM instances is stopped forcefully those clients start using the service of the available ASM instance

srvctl status asm -detail
ASM is running on rhel71
ASM is enabled.
ASM instance +ASM1 is running on node rhel71
Number of connected clients: 4
Client names: -MGMTDB:_mgmtdb:rhel7-cluster ent18c1:ent18c:rhel7-cluster ent18c2:ent18c:rhel7-cluster rhel71.domain.net:_OCR:rhel7-cluster

As seen above, clients that previously clients of ASM2 instance are now using ASM1 instance to access ASM disks. (total client count is 4 not 5 due to not duplicating the OCR* client).
When ASM instance is restarted the clients automatically select the local ASM instance over remote ASM instances.

srvctl start asm  -node rhel72
srvctl status asm -detail
ASM is running on rhel71,rhel72
ASM is enabled.
ASM instance +ASM1 is running on node rhel71
Number of connected clients: 2
Client names: ent18c1:ent18c:rhel7-cluster rhel71.domain.net:_OCR:rhel7-cluster
ASM instance +ASM2 is running on node rhel72
Number of connected clients: 3
Client names: -MGMTDB:_mgmtdb:rhel7-cluster ent18c2:ent18c:rhel7-cluster rhel72.domain.net:_OCR:rhel7-cluster

This concludes the setting up RAC DB on 18c.

Related Posts
Installing 12cR2 (12.2.0.1) RAC on RHEL 6 with Role Separation - Database
Installing 12c (12.1.0.1) RAC on RHEL 6 with Role Separation - Database Software
Installing 12c (12.1.0.1) RAC on RHEL 6 with Role Separation - Creating CDB & PDB

Monday, April 1, 2019

Installing 18c (18.3) RAC on RHEL 7 with Role Separation - Clusterware

This post is not a step by step guide to installing 18c RAC but list the main differences in the installation process compared to previous versions and things to look out for. Similar to 12cR2 clusterware the 18c cluster installation is no longer done through runInstaller. The downloaded Oracle Grid Infrastructure image files are unzipped into the folder path where GI is to be installed. This only need to be done on the local node only. During installation, the software is copied and installed on all other nodes in the cluster. Unlike 12cR1, it is possible to specify a separate disk group for GIMR at the installation time. Oracle documentation says "You cannot migrate the GIMR from one disk group to another later". 12cR1 also mentioned the same but there were ways to move GIMR to different disk group after the installation. It has been not verified whether the same techniques could be used for 18c as well.
1. The OS version and kernel versions used for this RAC setup is shown below. Unlike the previous post on 12cR2 this post for 18c uses RHEL7

cat /etc/redhat-release
Red Hat Enterprise Linux Server release 7.4 (Maipo)

uname -r
3.10.0-693.el7.x86_64

2. Create the user groups and users similar to previous 12.1 installation. As said in an earlier post, 12.2 introduce a new user group for RAC management. Following from install guide "You must designate a group as the OSRACDBA group during database installation. Members of this group are granted the SYSRAC privileges to perform day–to–day administration of Oracle databases on an Oracle RAC cluster".

groupadd dba
groupadd oinstall
groupadd oper
groupadd asmoper
groupadd asmdba
groupadd asmadmin
groupadd backupdba
groupadd dgdba
groupadd kmdba
groupadd racdba

useradd -g oinstall -G dba,oper,asmdba,backupdba,dgdba,kmdba,racdba oracle
useradd -g oinstall -G asmadmin,asmdba,asmoper,dba grid

For user equivelancy ensure that user id and group id are same across the cluster nodes. The option -g could be used to ensure that same user/group id is used for creating users across all nodes. The nodes in this case are named rhel71 and rhel72.

[oracle@rhel71 ~]$ id oracle
uid=501(oracle) gid=501(oinstall) groups=501(oinstall),1000(dba),1002(oper),1004(asmdba),1006(backupdba),1007(dgdba),1008(kmdba),1009(racdba)
[oracle@rhel71 ~]$ ssh rhel72 id oracle
uid=501(oracle) gid=501(oinstall) groups=501(oinstall),1000(dba),1002(oper),1004(asmdba),1006(backupdba),1007(dgdba),1008(kmdba),1009(racdba)

[oracle@rhel71 ~]$ id grid
uid=502(grid) gid=501(oinstall) groups=501(oinstall),1000(dba),1003(asmoper),1004(asmdba),1005(asmadmin)
[oracle@rhel71 ~]$ ssh rhel72 id grid
uid=502(grid) gid=501(oinstall) groups=501(oinstall),1000(dba),1003(asmoper),1004(asmdba),1005(asmadmin)

3. If a separate disk group is used for GIMR then it needs to be at least 35GB in size. Following from installation doc "during installation of an Oracle Standalone Cluster, if you create the MGMT disk group for Grid Infrastructure Management Repository (GIMR), then the installer requires that you use a disk group with at least 35 GB of available space". A 40GB disk was created to store the GIMR and udev rules were created for granting necessary ownership and permission.

cat /etc/udev/rules.d/99-sd.rules
KERNEL=="sd*1", SUBSYSTEM=="block", PROGRAM=="/usr/lib/udev/scsi_id -g -u -d /dev/$parent",RESULT=="1ATA_VBOX_HARDDISK_VB1a5c8787-e082e7a7",SYMLINK+="oracleasm/ocr1",OWNER="grid",GROUP="asmadmin",MODE="0660"
KERNEL=="sd*1", SUBSYSTEM=="block", PROGRAM=="/usr/lib/udev/scsi_id -g -u -d /dev/$parent",RESULT=="1ATA_VBOX_HARDDISK_VB1a72679c-ab23e15f",SYMLINK+="oracleasm/ocr2",OWNER="grid",GROUP="asmadmin",MODE="0660"
KERNEL=="sd*1", SUBSYSTEM=="block", PROGRAM=="/usr/lib/udev/scsi_id -g -u -d /dev/$parent",RESULT=="1ATA_VBOX_HARDDISK_VB8f43e75c-3e0bec6d",SYMLINK+="oracleasm/ocr3",OWNER="grid",GROUP="asmadmin",MODE="0660"

KERNEL=="sd*1", SUBSYSTEM=="block", PROGRAM=="/usr/lib/udev/scsi_id -g -u -d /dev/$parent",RESULT=="1ATA_VBOX_HARDDISK_VBf8475ab7-d401ec8b",SYMLINK+="oracleasm/gimr",OWNER="grid",GROUP="asmadmin",MODE="0660"

KERNEL=="sd*1", SUBSYSTEM=="block", PROGRAM=="/usr/lib/udev/scsi_id -g -u -d /dev/$parent",RESULT=="1ATA_VBOX_HARDDISK_VB34650f33-5ecdc029",SYMLINK+="oracleasm/data1",OWNER="grid",GROUP="asmadmin",MODE="0660"

KERNEL=="sd*1", SUBSYSTEM=="block", PROGRAM=="/usr/lib/udev/scsi_id -g -u -d /dev/$parent",RESULT=="1ATA_VBOX_HARDDISK_VBdb057b08-1733da4b",SYMLINK+="oracleasm/fra1",OWNER="grid",GROUP="asmadmin",MODE="0660"

4. Create the GI installation directory and unzip the downloaded install media into the directory.

mkdir -p /opt/app/18.x.0/grid
chown grid:oinstall /opt/app/18.x.0/grid
unzip ./LINUX.X64_180000_grid_home.zip -d /opt/app/18.x.0/grid

5. Run cluvfy to check the pre-reqs checks. The failure related to shm shown at the end could be ignored as per 2065603.1 and 25907259.8

[grid@rhel71 grid]$ ./runcluvfy.sh stage -pre crsinst -n rhel71,rhel72 -r 18 -osdba asmdba -osoper asmoper -asm -asmgrp asmadmin -presence local -asmdev /dev/oracleasm/data1,/dev/oracleasm/fra1,/dev/oracleasm/ocr1,/dev/oracleasm/ocr2,/dev/oracleasm/ocr3,/dev/oracleasm/gimr  -crshome /opt/app/18.x.0/grid -fixup  -verbose

Verifying Physical Memory ...
  Node Name     Available                 Required                  Status
  ------------  ------------------------  ------------------------  ----------
  rhel72        2.7806GB (2915620.0KB)    8GB (8388608.0KB)         failed
  rhel71        2.7806GB (2915620.0KB)    8GB (8388608.0KB)         failed
Verifying Physical Memory ...FAILED (PRVF-7530)
Verifying Available Physical Memory ...
  Node Name     Available                 Required                  Status
  ------------  ------------------------  ------------------------  ----------
  rhel72        2.6736GB (2803492.0KB)    50MB (51200.0KB)          passed
  rhel71        2.4872GB (2608032.0KB)    50MB (51200.0KB)          passed
Verifying Available Physical Memory ...PASSED
Verifying Swap Size ...
  Node Name     Available                 Required                  Status
  ------------  ------------------------  ------------------------  ----------
  rhel72        3.7246GB (3905532.0KB)    2.7806GB (2915620.0KB)    passed
  rhel71        3.7246GB (3905532.0KB)    2.7806GB (2915620.0KB)    passed
Verifying Swap Size ...PASSED
Verifying Free Space: rhel72:/usr,rhel72:/var,rhel72:/etc,rhel72:/opt/app/18.x.0/grid,rhel72:/sbin,rhel72:/tmp ...
  Path              Node Name     Mount point   Available     Required      Status
  ----------------  ------------  ------------  ------------  ------------  ------------
  /usr              rhel72        /             28.9385GB     25MB          passed
  /var              rhel72        /             28.9385GB     5MB           passed
  /etc              rhel72        /             28.9385GB     25MB          passed
  /opt/app/18.x.0/grid  rhel72        /             28.9385GB     6.9GB         passed
  /sbin             rhel72        /             28.9385GB     10MB          passed
  /tmp              rhel72        /             28.9385GB     1GB           passed
Verifying Free Space: rhel72:/usr,rhel72:/var,rhel72:/etc,rhel72:/opt/app/18.x.0/grid,rhel72:/sbin,rhel72:/tmp ...PASSED
Verifying Free Space: rhel71:/usr,rhel71:/var,rhel71:/etc,rhel71:/opt/app/18.x.0/grid,rhel71:/sbin,rhel71:/tmp ...
  Path              Node Name     Mount point   Available     Required      Status
  ----------------  ------------  ------------  ------------  ------------  ------------
  /usr              rhel71        /             18.9439GB     25MB          passed
  /var              rhel71        /             18.9439GB     5MB           passed
  /etc              rhel71        /             18.9439GB     25MB          passed
  /opt/app/18.x.0/grid  rhel71        /             18.9439GB     6.9GB         passed
  /sbin             rhel71        /             18.9439GB     10MB          passed
  /tmp              rhel71        /             18.9439GB     1GB           passed
Verifying Free Space: rhel71:/usr,rhel71:/var,rhel71:/etc,rhel71:/opt/app/18.x.0/grid,rhel71:/sbin,rhel71:/tmp ...PASSED
Verifying User Existence: grid ...
  Node Name     Status                    Comment
  ------------  ------------------------  ------------------------
  rhel72        passed                    exists(502)
  rhel71        passed                    exists(502)

  Verifying Users With Same UID: 502 ...PASSED
Verifying User Existence: grid ...PASSED
Verifying Group Existence: asmadmin ...
  Node Name     Status                    Comment
  ------------  ------------------------  ------------------------
  rhel72        passed                    exists
  rhel71        passed                    exists
Verifying Group Existence: asmadmin ...PASSED
Verifying Group Existence: asmoper ...
  Node Name     Status                    Comment
  ------------  ------------------------  ------------------------
  rhel72        passed                    exists
  rhel71        passed                    exists
Verifying Group Existence: asmoper ...PASSED
Verifying Group Existence: asmdba ...
  Node Name     Status                    Comment
  ------------  ------------------------  ------------------------
  rhel72        passed                    exists
  rhel71        passed                    exists
Verifying Group Existence: asmdba ...PASSED
Verifying Group Existence: oinstall ...
  Node Name     Status                    Comment
  ------------  ------------------------  ------------------------
  rhel72        passed                    exists
  rhel71        passed                    exists
Verifying Group Existence: oinstall ...PASSED
Verifying Group Membership: asmdba ...
  Node Name         User Exists   Group Exists  User in Group  Status
  ----------------  ------------  ------------  ------------  ----------------
  rhel72            yes           yes           yes           passed
  rhel71            yes           yes           yes           passed
Verifying Group Membership: asmdba ...PASSED
Verifying Group Membership: asmadmin ...
  Node Name         User Exists   Group Exists  User in Group  Status
  ----------------  ------------  ------------  ------------  ----------------
  rhel72            yes           yes           yes           passed
  rhel71            yes           yes           yes           passed
Verifying Group Membership: asmadmin ...PASSED
Verifying Group Membership: oinstall(Primary) ...
  Node Name         User Exists   Group Exists  User in Group  Primary       Status
  ----------------  ------------  ------------  ------------  ------------  ------------
  rhel72            yes           yes           yes           yes           passed
  rhel71            yes           yes           yes           yes           passed
Verifying Group Membership: oinstall(Primary) ...PASSED
Verifying Group Membership: asmoper ...
  Node Name         User Exists   Group Exists  User in Group  Status
  ----------------  ------------  ------------  ------------  ----------------
  rhel72            yes           yes           yes           passed
  rhel71            yes           yes           yes           passed
Verifying Group Membership: asmoper ...PASSED
Verifying Run Level ...
  Node Name     run level                 Required                  Status
  ------------  ------------------------  ------------------------  ----------
  rhel72        3                         3,5                       passed
  rhel71        3                         3,5                       passed
Verifying Run Level ...PASSED
Verifying Hard Limit: maximum open file descriptors ...
  Node Name         Type          Available     Required      Status
  ----------------  ------------  ------------  ------------  ----------------
  rhel72            hard          65536         65536         passed
  rhel71            hard          65536         65536         passed
Verifying Hard Limit: maximum open file descriptors ...PASSED
Verifying Soft Limit: maximum open file descriptors ...
  Node Name         Type          Available     Required      Status
  ----------------  ------------  ------------  ------------  ----------------
  rhel72            soft          65536         1024          passed
  rhel71            soft          65536         1024          passed
Verifying Soft Limit: maximum open file descriptors ...PASSED
Verifying Hard Limit: maximum user processes ...
  Node Name         Type          Available     Required      Status
  ----------------  ------------  ------------  ------------  ----------------
  rhel72            hard          16384         16384         passed
  rhel71            hard          16384         16384         passed
Verifying Hard Limit: maximum user processes ...PASSED
Verifying Soft Limit: maximum user processes ...
  Node Name         Type          Available     Required      Status
  ----------------  ------------  ------------  ------------  ----------------
  rhel72            soft          16384         2047          passed
  rhel71            soft          16384         2047          passed
Verifying Soft Limit: maximum user processes ...PASSED
Verifying Soft Limit: maximum stack size ...
  Node Name         Type          Available     Required      Status
  ----------------  ------------  ------------  ------------  ----------------
  rhel72            soft          10240         10240         passed
  rhel71            soft          10240         10240         passed
Verifying Soft Limit: maximum stack size ...PASSED
Verifying Architecture ...
  Node Name     Available                 Required                  Status
  ------------  ------------------------  ------------------------  ----------
  rhel72        x86_64                    x86_64                    passed
  rhel71        x86_64                    x86_64                    passed
Verifying Architecture ...PASSED
Verifying OS Kernel Version ...
  Node Name     Available                 Required                  Status
  ------------  ------------------------  ------------------------  ----------
  rhel72        3.10.0-693.el7.x86_64     3.10.0                    passed
  rhel71        3.10.0-693.el7.x86_64     3.10.0                    passed
Verifying OS Kernel Version ...PASSED
Verifying OS Kernel Parameter: semmsl ...
  Node Name         Current       Configured    Required      Status        Comment
  ----------------  ------------  ------------  ------------  ------------  ------------
  rhel71            3010          3010          250           passed
  rhel72            3010          3010          250           passed
Verifying OS Kernel Parameter: semmsl ...PASSED
Verifying OS Kernel Parameter: semmns ...
  Node Name         Current       Configured    Required      Status        Comment
  ----------------  ------------  ------------  ------------  ------------  ------------
  rhel71            385280        385280        32000         passed
  rhel72            385280        385280        32000         passed
Verifying OS Kernel Parameter: semmns ...PASSED
Verifying OS Kernel Parameter: semopm ...
  Node Name         Current       Configured    Required      Status        Comment
  ----------------  ------------  ------------  ------------  ------------  ------------
  rhel71            3010          3010          100           passed
  rhel72            3010          3010          100           passed
Verifying OS Kernel Parameter: semopm ...PASSED
Verifying OS Kernel Parameter: semmni ...
  Node Name         Current       Configured    Required      Status        Comment
  ----------------  ------------  ------------  ------------  ------------  ------------
  rhel71            128           128           128           passed
  rhel72            128           128           128           passed
Verifying OS Kernel Parameter: semmni ...PASSED
Verifying OS Kernel Parameter: shmmax ...
  Node Name         Current       Configured    Required      Status        Comment
  ----------------  ------------  ------------  ------------  ------------  ------------
  rhel71            4398046511104  4398046511104  1492797440    passed
  rhel72            4398046511104  4398046511104  1492797440    passed
Verifying OS Kernel Parameter: shmmax ...PASSED
Verifying OS Kernel Parameter: shmmni ...
  Node Name         Current       Configured    Required      Status        Comment
  ----------------  ------------  ------------  ------------  ------------  ------------
  rhel71            4096          4096          4096          passed
  rhel72            4096          4096          4096          passed
Verifying OS Kernel Parameter: shmmni ...PASSED
Verifying OS Kernel Parameter: shmall ...
  Node Name         Current       Configured    Required      Status        Comment
  ----------------  ------------  ------------  ------------  ------------  ------------
  rhel71            1073741824    1073741824    1073741824    passed
  rhel72            1073741824    1073741824    1073741824    passed
Verifying OS Kernel Parameter: shmall ...PASSED
Verifying OS Kernel Parameter: file-max ...
  Node Name         Current       Configured    Required      Status        Comment
  ----------------  ------------  ------------  ------------  ------------  ------------
  rhel71            6815744       6815744       6815744       passed
  rhel72            6815744       6815744       6815744       passed
Verifying OS Kernel Parameter: file-max ...PASSED
Verifying OS Kernel Parameter: ip_local_port_range ...
  Node Name         Current       Configured    Required      Status        Comment
  ----------------  ------------  ------------  ------------  ------------  ------------
  rhel71            between 9000 & 65500  between 9000 & 65500  between 9000 & 65535  passed
  rhel72            between 9000 & 65500  between 9000 & 65500  between 9000 & 65535  passed
Verifying OS Kernel Parameter: ip_local_port_range ...PASSED
Verifying OS Kernel Parameter: rmem_default ...
  Node Name         Current       Configured    Required      Status        Comment
  ----------------  ------------  ------------  ------------  ------------  ------------
  rhel71            4194304       4194304       262144        passed
  rhel72            4194304       4194304       262144        passed
Verifying OS Kernel Parameter: rmem_default ...PASSED
Verifying OS Kernel Parameter: rmem_max ...
  Node Name         Current       Configured    Required      Status        Comment
  ----------------  ------------  ------------  ------------  ------------  ------------
  rhel71            4194304       4194304       4194304       passed
  rhel72            4194304       4194304       4194304       passed
Verifying OS Kernel Parameter: rmem_max ...PASSED
Verifying OS Kernel Parameter: wmem_default ...
  Node Name         Current       Configured    Required      Status        Comment
  ----------------  ------------  ------------  ------------  ------------  ------------
  rhel71            1048576       1048576       262144        passed
  rhel72            1048576       1048576       262144        passed
Verifying OS Kernel Parameter: wmem_default ...PASSED
Verifying OS Kernel Parameter: wmem_max ...
  Node Name         Current       Configured    Required      Status        Comment
  ----------------  ------------  ------------  ------------  ------------  ------------
  rhel71            2097152       2097152       1048576       passed
  rhel72            2097152       2097152       1048576       passed
Verifying OS Kernel Parameter: wmem_max ...PASSED
Verifying OS Kernel Parameter: aio-max-nr ...
  Node Name         Current       Configured    Required      Status        Comment
  ----------------  ------------  ------------  ------------  ------------  ------------
  rhel71            3145728       3145728       1048576       passed
  rhel72            3145728       3145728       1048576       passed
Verifying OS Kernel Parameter: aio-max-nr ...PASSED
Verifying Package: binutils-2.23.52.0.1 ...
  Node Name     Available                 Required                  Status
  ------------  ------------------------  ------------------------  ----------
  rhel72        binutils-2.25.1-31.base.el7  binutils-2.23.52.0.1      passed
  rhel71        binutils-2.25.1-31.base.el7  binutils-2.23.52.0.1      passed
Verifying Package: binutils-2.23.52.0.1 ...PASSED
Verifying Package: compat-libcap1-1.10 ...
  Node Name     Available                 Required                  Status
  ------------  ------------------------  ------------------------  ----------
  rhel72        compat-libcap1-1.10-7.el7  compat-libcap1-1.10       passed
  rhel71        compat-libcap1-1.10-7.el7  compat-libcap1-1.10       passed
Verifying Package: compat-libcap1-1.10 ...PASSED
Verifying Package: libgcc-4.8.2 (x86_64) ...
  Node Name     Available                 Required                  Status
  ------------  ------------------------  ------------------------  ----------
  rhel72        libgcc(x86_64)-4.8.5-16.el7  libgcc(x86_64)-4.8.2      passed
  rhel71        libgcc(x86_64)-4.8.5-16.el7  libgcc(x86_64)-4.8.2      passed
Verifying Package: libgcc-4.8.2 (x86_64) ...PASSED
Verifying Package: libstdc++-4.8.2 (x86_64) ...
  Node Name     Available                 Required                  Status
  ------------  ------------------------  ------------------------  ----------
  rhel72        libstdc++(x86_64)-4.8.5-16.el7  libstdc++(x86_64)-4.8.2   passed
  rhel71        libstdc++(x86_64)-4.8.5-16.el7  libstdc++(x86_64)-4.8.2   passed
Verifying Package: libstdc++-4.8.2 (x86_64) ...PASSED
Verifying Package: libstdc++-devel-4.8.2 (x86_64) ...
  Node Name     Available                 Required                  Status
  ------------  ------------------------  ------------------------  ----------
  rhel72        libstdc++-devel(x86_64)-4.8.5-16.el7  libstdc++-devel(x86_64)-4.8.2  passed
  rhel71        libstdc++-devel(x86_64)-4.8.5-16.el7  libstdc++-devel(x86_64)-4.8.2  passed
Verifying Package: libstdc++-devel-4.8.2 (x86_64) ...PASSED
Verifying Package: sysstat-10.1.5 ...
  Node Name     Available                 Required                  Status
  ------------  ------------------------  ------------------------  ----------
  rhel72        sysstat-10.1.5-12.el7     sysstat-10.1.5            passed
  rhel71        sysstat-10.1.5-12.el7     sysstat-10.1.5            passed
Verifying Package: sysstat-10.1.5 ...PASSED
Verifying Package: gcc-c++-4.8.2 ...
  Node Name     Available                 Required                  Status
  ------------  ------------------------  ------------------------  ----------
  rhel72        gcc-c++-4.8.5-16.el7      gcc-c++-4.8.2             passed
  rhel71        gcc-c++-4.8.5-16.el7      gcc-c++-4.8.2             passed
Verifying Package: gcc-c++-4.8.2 ...PASSED
Verifying Package: ksh ...
  Node Name     Available                 Required                  Status
  ------------  ------------------------  ------------------------  ----------
  rhel72        ksh                       ksh                       passed
  rhel71        ksh                       ksh                       passed
Verifying Package: ksh ...PASSED
Verifying Package: make-3.82 ...
  Node Name     Available                 Required                  Status
  ------------  ------------------------  ------------------------  ----------
  rhel72        make-3.82-23.el7          make-3.82                 passed
  rhel71        make-3.82-23.el7          make-3.82                 passed
Verifying Package: make-3.82 ...PASSED
Verifying Package: glibc-2.17 (x86_64) ...
  Node Name     Available                 Required                  Status
  ------------  ------------------------  ------------------------  ----------
  rhel72        glibc(x86_64)-2.17-196.el7  glibc(x86_64)-2.17        passed
  rhel71        glibc(x86_64)-2.17-196.el7  glibc(x86_64)-2.17        passed
Verifying Package: glibc-2.17 (x86_64) ...PASSED
Verifying Package: glibc-devel-2.17 (x86_64) ...
  Node Name     Available                 Required                  Status
  ------------  ------------------------  ------------------------  ----------
  rhel72        glibc-devel(x86_64)-2.17-196.el7  glibc-devel(x86_64)-2.17  passed
  rhel71        glibc-devel(x86_64)-2.17-196.el7  glibc-devel(x86_64)-2.17  passed
Verifying Package: glibc-devel-2.17 (x86_64) ...PASSED
Verifying Package: libaio-0.3.109 (x86_64) ...
  Node Name     Available                 Required                  Status
  ------------  ------------------------  ------------------------  ----------
  rhel72        libaio(x86_64)-0.3.109-13.el7  libaio(x86_64)-0.3.109    passed
  rhel71        libaio(x86_64)-0.3.109-13.el7  libaio(x86_64)-0.3.109    passed
Verifying Package: libaio-0.3.109 (x86_64) ...PASSED
Verifying Package: libaio-devel-0.3.109 (x86_64) ...
  Node Name     Available                 Required                  Status
  ------------  ------------------------  ------------------------  ----------
  rhel72        libaio-devel(x86_64)-0.3.109-13.el7  libaio-devel(x86_64)-0.3.109  passed
  rhel71        libaio-devel(x86_64)-0.3.109-13.el7  libaio-devel(x86_64)-0.3.109  passed
Verifying Package: libaio-devel-0.3.109 (x86_64) ...PASSED
Verifying Package: nfs-utils-1.2.3-15 ...
  Node Name     Available                 Required                  Status
  ------------  ------------------------  ------------------------  ----------
  rhel72        nfs-utils-1.3.0-0.48.el7  nfs-utils-1.2.3-15        passed
  rhel71        nfs-utils-1.3.0-0.48.el7  nfs-utils-1.2.3-15        passed
Verifying Package: nfs-utils-1.2.3-15 ...PASSED
Verifying Package: smartmontools-6.2-4 ...
  Node Name     Available                 Required                  Status
  ------------  ------------------------  ------------------------  ----------
  rhel72        smartmontools-6.2-8.el7   smartmontools-6.2-4       passed
  rhel71        smartmontools-6.2-8.el7   smartmontools-6.2-4       passed
Verifying Package: smartmontools-6.2-4 ...PASSED
Verifying Package: net-tools-2.0-0.17 ...
  Node Name     Available                 Required                  Status
  ------------  ------------------------  ------------------------  ----------
  rhel72        net-tools-2.0-0.22.20131004git.el7  net-tools-2.0-0.17        passed
  rhel71        net-tools-2.0-0.22.20131004git.el7  net-tools-2.0-0.17        passed
Verifying Package: net-tools-2.0-0.17 ...PASSED
Verifying Package: compat-libstdc++-33-3.2.3 (x86_64) ...
  Node Name     Available                 Required                  Status
  ------------  ------------------------  ------------------------  ----------
  rhel72        compat-libstdc++-33(x86_64)-3.2.3-72.el7  compat-libstdc++-33(x86_64)-3.2.3  passed
  rhel71        compat-libstdc++-33(x86_64)-3.2.3-72.el7  compat-libstdc++-33(x86_64)-3.2.3  passed
Verifying Package: compat-libstdc++-33-3.2.3 (x86_64) ...PASSED
Verifying Package: libxcb-1.11 (x86_64) ...
  Node Name     Available                 Required                  Status
  ------------  ------------------------  ------------------------  ----------
  rhel72        libxcb(x86_64)-1.12-1.el7  libxcb(x86_64)-1.11       passed
  rhel71        libxcb(x86_64)-1.12-1.el7  libxcb(x86_64)-1.11       passed
Verifying Package: libxcb-1.11 (x86_64) ...PASSED
Verifying Package: libX11-1.6.3 (x86_64) ...
  Node Name     Available                 Required                  Status
  ------------  ------------------------  ------------------------  ----------
  rhel72        libX11(x86_64)-1.6.5-1.el7  libX11(x86_64)-1.6.3      passed
  rhel71        libX11(x86_64)-1.6.5-1.el7  libX11(x86_64)-1.6.3      passed
Verifying Package: libX11-1.6.3 (x86_64) ...PASSED
Verifying Package: libXau-1.0.8 (x86_64) ...
  Node Name     Available                 Required                  Status
  ------------  ------------------------  ------------------------  ----------
  rhel72        libXau(x86_64)-1.0.8-2.1.el7  libXau(x86_64)-1.0.8      passed
  rhel71        libXau(x86_64)-1.0.8-2.1.el7  libXau(x86_64)-1.0.8      passed
Verifying Package: libXau-1.0.8 (x86_64) ...PASSED
Verifying Package: libXi-1.7.4 (x86_64) ...
  Node Name     Available                 Required                  Status
  ------------  ------------------------  ------------------------  ----------
  rhel72        libXi(x86_64)-1.7.9-1.el7  libXi(x86_64)-1.7.4       passed
  rhel71        libXi(x86_64)-1.7.9-1.el7  libXi(x86_64)-1.7.4       passed
Verifying Package: libXi-1.7.4 (x86_64) ...PASSED
Verifying Package: libXtst-1.2.2 (x86_64) ...
  Node Name     Available                 Required                  Status
  ------------  ------------------------  ------------------------  ----------
  rhel72        libXtst(x86_64)-1.2.3-1.el7  libXtst(x86_64)-1.2.2     passed
  rhel71        libXtst(x86_64)-1.2.3-1.el7  libXtst(x86_64)-1.2.2     passed
Verifying Package: libXtst-1.2.2 (x86_64) ...PASSED
Verifying Port Availability for component "Oracle Notification Service (ONS)" ...
  Node Name         Port Number   Protocol      Available     Status
  ----------------  ------------  ------------  ------------  ----------------
  rhel72            6200          TCP           yes           successful
  rhel71            6200          TCP           yes           successful
  rhel72            6100          TCP           yes           successful
  rhel71            6100          TCP           yes           successful
Verifying Port Availability for component "Oracle Notification Service (ONS)" ...PASSED
Verifying Port Availability for component "Oracle Cluster Synchronization Services (CSSD)" ...
  Node Name         Port Number   Protocol      Available     Status
  ----------------  ------------  ------------  ------------  ----------------
  rhel72            42424         TCP           yes           successful
  rhel71            42424         TCP           yes           successful
Verifying Port Availability for component "Oracle Cluster Synchronization Services (CSSD)" ...PASSED
Verifying Users With Same UID: 0 ...PASSED
Verifying Current Group ID ...PASSED
Verifying Root user consistency ...
  Node Name                             Status
  ------------------------------------  ------------------------
  rhel72                                passed
  rhel71                                passed
Verifying Root user consistency ...PASSED
Verifying Package: cvuqdisk-1.0.10-1 ...
  Node Name     Available                 Required                  Status
  ------------  ------------------------  ------------------------  ----------
  rhel72        cvuqdisk-1.0.10-1         cvuqdisk-1.0.10-1         passed
  rhel71        cvuqdisk-1.0.10-1         cvuqdisk-1.0.10-1         passed
Verifying Package: cvuqdisk-1.0.10-1 ...PASSED
Verifying Host name ...PASSED
Verifying Node Connectivity ...
  Verifying Hosts File ...
  Node Name                             Status
  ------------------------------------  ------------------------
  rhel71                                passed
  rhel72                                passed
  Verifying Hosts File ...PASSED

Interface information for node "rhel71"

 Name   IP Address      Subnet          Gateway         Def. Gateway    HW Address        MTU
 ------ --------------- --------------- --------------- --------------- ----------------- ------
 enp0s3 192.168.0.93    192.168.0.0     0.0.0.0         192.168.0.102   08:00:27:BD:32:39 1500
 enp0s8 192.168.1.93    192.168.1.0     0.0.0.0         192.168.0.102   08:00:27:BD:A9:0A 1500

Interface information for node "rhel72"

 Name   IP Address      Subnet          Gateway         Def. Gateway    HW Address        MTU
 ------ --------------- --------------- --------------- --------------- ----------------- ------
 enp0s3 192.168.0.94    192.168.0.0     0.0.0.0         192.168.0.102   08:00:27:31:0F:1F 1500
 enp0s8 192.168.1.94    192.168.1.0     0.0.0.0         192.168.0.102   08:00:27:98:22:A6 1500

Check: MTU consistency of the subnet "192.168.1.0".

  Node              Name          IP Address    Subnet        MTU
  ----------------  ------------  ------------  ------------  ----------------
  rhel71            enp0s8        192.168.1.93  192.168.1.0   1500
  rhel72            enp0s8        192.168.1.94  192.168.1.0   1500

Check: MTU consistency of the subnet "192.168.0.0".

  Node              Name          IP Address    Subnet        MTU
  ----------------  ------------  ------------  ------------  ----------------
  rhel71            enp0s3        192.168.0.93  192.168.0.0   1500
  rhel72            enp0s3        192.168.0.94  192.168.0.0   1500
  Verifying Check that maximum (MTU) size packet goes through subnet ...PASSED

  Source                          Destination                     Connected?
  ------------------------------  ------------------------------  ----------------
  rhel71[enp0s8:192.168.1.93]     rhel72[enp0s8:192.168.1.94]     yes

  Source                          Destination                     Connected?
  ------------------------------  ------------------------------  ----------------
  rhel71[enp0s3:192.168.0.93]     rhel72[enp0s3:192.168.0.94]     yes
  Verifying subnet mask consistency for subnet "192.168.1.0" ...PASSED
  Verifying subnet mask consistency for subnet "192.168.0.0" ...PASSED
Verifying Node Connectivity ...PASSED
Verifying Multicast or broadcast check ...
Checking subnet "192.168.1.0" for multicast communication with multicast group "224.0.0.251"
Verifying Multicast or broadcast check ...PASSED
Verifying Device Checks for ASM ...
  Verifying Package: cvuqdisk-1.0.10-1 ...PASSED
  Verifying ASM device sharedness check ...
    Verifying Shared Storage Accessibility:/dev/oracleasm/gimr,/dev/oracleasm/fra1,/dev/oracleasm/data1,/dev/oracleasm/ocr3,/dev/oracleasm/ocr1,/dev/oracleasm/ocr2 ...PASSED

  Device                                Device Type
  ------------------------------------  ------------------------
  /dev/oracleasm/ocr3                   Disk
  /dev/oracleasm/data1                  Disk
  /dev/oracleasm/ocr2                   Disk
  /dev/oracleasm/fra1                   Disk
  /dev/oracleasm/ocr1                   Disk
  /dev/oracleasm/gimr                   Disk
  Verifying ASM device sharedness check ...PASSED
  Verifying Access Control List check ...
Checking consistency of device owner across all nodes...
Consistency check of device owner for "/dev/oracleasm/gimr" PASSED
Consistency check of device owner for "/dev/oracleasm/fra1" PASSED
Consistency check of device owner for "/dev/oracleasm/ocr3" PASSED
Consistency check of device owner for "/dev/oracleasm/data1" PASSED
Consistency check of device owner for "/dev/oracleasm/ocr1" PASSED
Consistency check of device owner for "/dev/oracleasm/ocr2" PASSED

Checking consistency of device group across all nodes...
Consistency check of device group for "/dev/oracleasm/gimr" PASSED
Consistency check of device group for "/dev/oracleasm/fra1" PASSED
Consistency check of device group for "/dev/oracleasm/ocr3" PASSED
Consistency check of device group for "/dev/oracleasm/data1" PASSED
Consistency check of device group for "/dev/oracleasm/ocr1" PASSED
Consistency check of device group for "/dev/oracleasm/ocr2" PASSED

Checking consistency of device permissions across all nodes...
Consistency check of device permissions for "/dev/oracleasm/gimr" PASSED
Consistency check of device permissions for "/dev/oracleasm/fra1" PASSED
Consistency check of device permissions for "/dev/oracleasm/ocr3" PASSED
Consistency check of device permissions for "/dev/oracleasm/data1" PASSED
Consistency check of device permissions for "/dev/oracleasm/ocr1" PASSED
Consistency check of device permissions for "/dev/oracleasm/ocr2" PASSED
  Verifying Access Control List check ...PASSED
  Verifying I/O scheduler ...PASSED
Verifying Device Checks for ASM ...PASSED
Verifying Network Time Protocol (NTP) ...
  Verifying '/etc/ntp.conf' ...
  Node Name                             File exists?
  ------------------------------------  ------------------------
  rhel72                                no
  rhel71                                no

  Verifying '/etc/ntp.conf' ...PASSED
  Verifying '/etc/chrony.conf' ...
  Node Name                             File exists?
  ------------------------------------  ------------------------
  rhel72                                no
  rhel71                                no

  Verifying '/etc/chrony.conf' ...PASSED
  Verifying '/var/run/ntpd.pid' ...
  Node Name                             File exists?
  ------------------------------------  ------------------------
  rhel72                                no
  rhel71                                no

  Verifying '/var/run/ntpd.pid' ...PASSED
  Verifying '/var/run/chronyd.pid' ...
  Node Name                             File exists?
  ------------------------------------  ------------------------
  rhel72                                no
  rhel71                                no

  Verifying '/var/run/chronyd.pid' ...PASSED
Verifying Network Time Protocol (NTP) ...PASSED
Verifying Same core file name pattern ...PASSED
Verifying User Mask ...
  Node Name     Available                 Required                  Comment
  ------------  ------------------------  ------------------------  ----------
  rhel72        0022                      0022                      passed
  rhel71        0022                      0022                      passed
Verifying User Mask ...PASSED
Verifying User Not In Group "root": grid ...
  Node Name     Status                    Comment
  ------------  ------------------------  ------------------------
  rhel72        passed                    does not exist
  rhel71        passed                    does not exist
Verifying User Not In Group "root": grid ...PASSED
Verifying Time zone consistency ...PASSED
Verifying Time offset between nodes ...PASSED
Verifying resolv.conf Integrity ...
  Node Name                             Status
  ------------------------------------  ------------------------
  rhel71                                passed
  rhel72                                passed

checking response for name "rhel72" from each of the name servers specified in
"/etc/resolv.conf"

  Node Name     Source                    Comment                   Status
  ------------  ------------------------  ------------------------  ----------
  rhel72        192.168.0.66              IPv4                      passed

checking response for name "rhel71" from each of the name servers specified in
"/etc/resolv.conf"

  Node Name     Source                    Comment                   Status
  ------------  ------------------------  ------------------------  ----------
  rhel71        192.168.0.66              IPv4                      passed
Verifying resolv.conf Integrity ...PASSED
Verifying DNS/NIS name service ...PASSED
Verifying Domain Sockets ...PASSED
Verifying /boot mount ...PASSED
Verifying File system mount options for path GI_HOME ...PASSED
Verifying Daemon "avahi-daemon" not configured and running ...
  Node Name     Configured                Status
  ------------  ------------------------  ------------------------
  rhel72        no                        passed
  rhel71        no                        passed

  Node Name     Running?                  Status
  ------------  ------------------------  ------------------------
  rhel72        no                        passed
  rhel71        no                        passed
Verifying Daemon "avahi-daemon" not configured and running ...PASSED
Verifying Daemon "proxyt" not configured and running ...
  Node Name     Configured                Status
  ------------  ------------------------  ------------------------
  rhel72        no                        passed
  rhel71        no                        passed

  Node Name     Running?                  Status
  ------------  ------------------------  ------------------------
  rhel72        no                        passed
  rhel71        no                        passed
Verifying Daemon "proxyt" not configured and running ...PASSED
Verifying Grid Infrastructure home path: /opt/app/18.x.0/grid ...
  Verifying '/opt/app/18.x.0/grid' ...PASSED
Verifying Grid Infrastructure home path: /opt/app/18.x.0/grid ...PASSED
Verifying User Equivalence ...PASSED
Verifying /dev/shm mounted as temporary file system ...FAILED (PRVE-0421)
Verifying File system mount options for path /var ...PASSED
Verifying zeroconf check ...PASSED
Verifying ASM Filter Driver configuration ...PASSED

Pre-check for cluster services setup was unsuccessful on all the nodes.


Failures were encountered during execution of CVU verification request "stage -pre crsinst".

Verifying Physical Memory ...FAILED
rhel72: PRVF-7530 : Sufficient physical memory is not available on node
        "rhel72" [Required physical memory = 8GB (8388608.0KB)]

rhel71: PRVF-7530 : Sufficient physical memory is not available on node
        "rhel71" [Required physical memory = 8GB (8388608.0KB)]

Verifying /dev/shm mounted as temporary file system ...FAILED
rhel72: PRVE-0421 : No entry exists in /etc/fstab for mounting /dev/shm

rhel71: PRVE-0421 : No entry exists in /etc/fstab for mounting /dev/shm

6. From the GI_HOME directory run gridSetup.sh to begin the installation. Select new cluster configuration option.

Select standard cluster (this would create a flex cluster which is the default for all standard clusters).

Set cluster name and scan name. No GNS is setup or used for this installation.

Add node and node VIP names that will be used in the cluster configuration. Importantly all node roles are set as HUB nodes.

Select the network interfaces for public and ASM/Private interconnect.

The OCR and vote disks are stored in ASM but ASM could be configured either block devices or using NFS. This setup uses block devices.

A separate disk group is used to store GIMR.

Create the disk group for OCR files.

Create disk group for GIMR

Set the oracle base location. The GI install location is immutable.

The failed pre-reqs are ignorable for test setup.

Summary

7. When prompted run the root script. Following output from running root.sh on the first node.

./root.sh
Performing root user operation.

Creating /etc/oratab file...
Entries will be added to the /etc/oratab file as needed by
Database Configuration Assistant when a database is created
Finished running generic part of root script.
Now product-specific root actions will be performed.
Relinking oracle with rac_on option
Using configuration parameter file: /opt/app/18.x.0/grid/crs/install/crsconfig_params
The log of current session can be found at:
  /opt/app/oracle/crsdata/rhel71/crsconfig/rootcrs_rhel71_2019-03-19_02-59-46PM.log
2019/03/19 15:00:12 CLSRSC-594: Executing installation step 1 of 20: 'SetupTFA'.
2019/03/19 15:00:12 CLSRSC-4001: Installing Oracle Trace File Analyzer (TFA) Collector.
2019/03/19 15:00:56 CLSRSC-4002: Successfully installed Oracle Trace File Analyzer (TFA) Collector.
2019/03/19 15:00:56 CLSRSC-594: Executing installation step 2 of 20: 'ValidateEnv'.
2019/03/19 15:00:57 CLSRSC-363: User ignored prerequisites during installation
2019/03/19 15:00:57 CLSRSC-594: Executing installation step 3 of 20: 'CheckFirstNode'.
2019/03/19 15:01:01 CLSRSC-594: Executing installation step 4 of 20: 'GenSiteGUIDs'.
2019/03/19 15:01:07 CLSRSC-594: Executing installation step 5 of 20: 'SaveParamFile'.
2019/03/19 15:01:28 CLSRSC-594: Executing installation step 6 of 20: 'SetupOSD'.
2019/03/19 15:01:28 CLSRSC-594: Executing installation step 7 of 20: 'CheckCRSConfig'.
2019/03/19 15:01:28 CLSRSC-594: Executing installation step 8 of 20: 'SetupLocalGPNP'.
2019/03/19 15:02:17 CLSRSC-594: Executing installation step 9 of 20: 'CreateRootCert'.
2019/03/19 15:02:30 CLSRSC-594: Executing installation step 10 of 20: 'ConfigOLR'.
2019/03/19 15:02:58 CLSRSC-594: Executing installation step 11 of 20: 'ConfigCHMOS'.
2019/03/19 15:02:58 CLSRSC-594: Executing installation step 12 of 20: 'CreateOHASD'.
2019/03/19 15:03:17 CLSRSC-594: Executing installation step 13 of 20: 'ConfigOHASD'.
2019/03/19 15:03:17 CLSRSC-330: Adding Clusterware entries to file 'oracle-ohasd.service'
2019/03/19 15:04:47 CLSRSC-594: Executing installation step 14 of 20: 'InstallAFD'.
2019/03/19 15:05:05 CLSRSC-594: Executing installation step 15 of 20: 'InstallACFS'.
CRS-2791: Starting shutdown of Oracle High Availability Services-managed resources on 'rhel71'
CRS-2793: Shutdown of Oracle High Availability Services-managed resources on 'rhel71' has completed
CRS-4133: Oracle High Availability Services has been stopped.
CRS-4123: Oracle High Availability Services has been started.
2019/03/19 15:06:37 CLSRSC-594: Executing installation step 16 of 20: 'InstallKA'.
2019/03/19 15:06:55 CLSRSC-594: Executing installation step 17 of 20: 'InitConfig'.
CRS-2791: Starting shutdown of Oracle High Availability Services-managed resources on 'rhel71'
CRS-2793: Shutdown of Oracle High Availability Services-managed resources on 'rhel71' has completed
CRS-4133: Oracle High Availability Services has been stopped.
CRS-4123: Oracle High Availability Services has been started.
CRS-2672: Attempting to start 'ora.evmd' on 'rhel71'
CRS-2672: Attempting to start 'ora.mdnsd' on 'rhel71'
CRS-2676: Start of 'ora.mdnsd' on 'rhel71' succeeded
CRS-2676: Start of 'ora.evmd' on 'rhel71' succeeded
CRS-2672: Attempting to start 'ora.gpnpd' on 'rhel71'
CRS-2676: Start of 'ora.gpnpd' on 'rhel71' succeeded
CRS-2672: Attempting to start 'ora.cssdmonitor' on 'rhel71'
CRS-2672: Attempting to start 'ora.gipcd' on 'rhel71'
CRS-2676: Start of 'ora.cssdmonitor' on 'rhel71' succeeded
CRS-2676: Start of 'ora.gipcd' on 'rhel71' succeeded
CRS-2672: Attempting to start 'ora.cssd' on 'rhel71'
CRS-2672: Attempting to start 'ora.diskmon' on 'rhel71'
CRS-2676: Start of 'ora.diskmon' on 'rhel71' succeeded
CRS-2676: Start of 'ora.cssd' on 'rhel71' succeeded

[INFO] [DBT-30001] Disk groups created successfully. Check /opt/app/oracle/cfgtoollogs/asmca/asmca-190319PM030741.log for details.


2019/03/19 15:08:59 CLSRSC-482: Running command: '/opt/app/18.x.0/grid/bin/ocrconfig -upgrade grid oinstall'
CRS-2672: Attempting to start 'ora.crf' on 'rhel71'
CRS-2672: Attempting to start 'ora.storage' on 'rhel71'
CRS-2676: Start of 'ora.storage' on 'rhel71' succeeded
CRS-2676: Start of 'ora.crf' on 'rhel71' succeeded
CRS-2672: Attempting to start 'ora.crsd' on 'rhel71'
CRS-2676: Start of 'ora.crsd' on 'rhel71' succeeded
CRS-4256: Updating the profile
Successful addition of voting disk c500a19f07ec4f62bf8ac318166a95ae.
Successful addition of voting disk 9100c10cb7bb4fd1bffbe97b8dc91245.
Successful addition of voting disk 300c2bf8b7094fa6bfaf11e49f826c7e.
Successfully replaced voting disk group with +OCRDG.
CRS-4256: Updating the profile
CRS-4266: Voting file(s) successfully replaced
##  STATE    File Universal Id                File Name Disk group
--  -----    -----------------                --------- ---------
 1. ONLINE   c500a19f07ec4f62bf8ac318166a95ae (/dev/oracleasm/ocr1) [OCRDG]
 2. ONLINE   9100c10cb7bb4fd1bffbe97b8dc91245 (/dev/oracleasm/ocr2) [OCRDG]
 3. ONLINE   300c2bf8b7094fa6bfaf11e49f826c7e (/dev/oracleasm/ocr3) [OCRDG]
Located 3 voting disk(s).
CRS-2791: Starting shutdown of Oracle High Availability Services-managed resources on 'rhel71'
CRS-2673: Attempting to stop 'ora.crsd' on 'rhel71'
CRS-2677: Stop of 'ora.crsd' on 'rhel71' succeeded
CRS-2673: Attempting to stop 'ora.storage' on 'rhel71'
CRS-2673: Attempting to stop 'ora.crf' on 'rhel71'
CRS-2673: Attempting to stop 'ora.drivers.acfs' on 'rhel71'
CRS-2673: Attempting to stop 'ora.mdnsd' on 'rhel71'
CRS-2677: Stop of 'ora.crf' on 'rhel71' succeeded
CRS-2677: Stop of 'ora.storage' on 'rhel71' succeeded
CRS-2673: Attempting to stop 'ora.asm' on 'rhel71'
CRS-2677: Stop of 'ora.drivers.acfs' on 'rhel71' succeeded
CRS-2677: Stop of 'ora.mdnsd' on 'rhel71' succeeded
CRS-2677: Stop of 'ora.asm' on 'rhel71' succeeded
CRS-2673: Attempting to stop 'ora.cluster_interconnect.haip' on 'rhel71'
CRS-2677: Stop of 'ora.cluster_interconnect.haip' on 'rhel71' succeeded
CRS-2673: Attempting to stop 'ora.ctssd' on 'rhel71'
CRS-2673: Attempting to stop 'ora.evmd' on 'rhel71'
CRS-2677: Stop of 'ora.ctssd' on 'rhel71' succeeded
CRS-2677: Stop of 'ora.evmd' on 'rhel71' succeeded
CRS-2673: Attempting to stop 'ora.cssd' on 'rhel71'
CRS-2677: Stop of 'ora.cssd' on 'rhel71' succeeded
CRS-2673: Attempting to stop 'ora.gipcd' on 'rhel71'
CRS-2673: Attempting to stop 'ora.gpnpd' on 'rhel71'
CRS-2677: Stop of 'ora.gipcd' on 'rhel71' succeeded
CRS-2677: Stop of 'ora.gpnpd' on 'rhel71' succeeded
CRS-2793: Shutdown of Oracle High Availability Services-managed resources on 'rhel71' has completed
CRS-4133: Oracle High Availability Services has been stopped.
2019/03/19 15:13:36 CLSRSC-594: Executing installation step 18 of 20: 'StartCluster'.
CRS-4123: Starting Oracle High Availability Services-managed resources
CRS-2672: Attempting to start 'ora.mdnsd' on 'rhel71'
CRS-2672: Attempting to start 'ora.evmd' on 'rhel71'
CRS-2676: Start of 'ora.evmd' on 'rhel71' succeeded
CRS-2676: Start of 'ora.mdnsd' on 'rhel71' succeeded
CRS-2672: Attempting to start 'ora.gpnpd' on 'rhel71'
CRS-2676: Start of 'ora.gpnpd' on 'rhel71' succeeded
CRS-2672: Attempting to start 'ora.gipcd' on 'rhel71'
CRS-2676: Start of 'ora.gipcd' on 'rhel71' succeeded
CRS-2672: Attempting to start 'ora.crf' on 'rhel71'
CRS-2672: Attempting to start 'ora.cssdmonitor' on 'rhel71'
CRS-2676: Start of 'ora.cssdmonitor' on 'rhel71' succeeded
CRS-2672: Attempting to start 'ora.cssd' on 'rhel71'
CRS-2672: Attempting to start 'ora.diskmon' on 'rhel71'
CRS-2676: Start of 'ora.diskmon' on 'rhel71' succeeded
CRS-2676: Start of 'ora.crf' on 'rhel71' succeeded
CRS-2676: Start of 'ora.cssd' on 'rhel71' succeeded
CRS-2672: Attempting to start 'ora.cluster_interconnect.haip' on 'rhel71'
CRS-2672: Attempting to start 'ora.ctssd' on 'rhel71'
CRS-2676: Start of 'ora.ctssd' on 'rhel71' succeeded
CRS-2676: Start of 'ora.cluster_interconnect.haip' on 'rhel71' succeeded
CRS-2672: Attempting to start 'ora.asm' on 'rhel71'
CRS-2676: Start of 'ora.asm' on 'rhel71' succeeded
CRS-2672: Attempting to start 'ora.storage' on 'rhel71'
CRS-2676: Start of 'ora.storage' on 'rhel71' succeeded
CRS-2672: Attempting to start 'ora.crsd' on 'rhel71'
CRS-2676: Start of 'ora.crsd' on 'rhel71' succeeded
CRS-6023: Starting Oracle Cluster Ready Services-managed resources
CRS-6017: Processing resource auto-start for servers: rhel71
CRS-6016: Resource auto-start has completed for server rhel71
CRS-6024: Completed start of Oracle Cluster Ready Services-managed resources
CRS-4123: Oracle High Availability Services has been started.
2019/03/19 15:16:19 CLSRSC-343: Successfully started Oracle Clusterware stack
2019/03/19 15:16:19 CLSRSC-594: Executing installation step 19 of 20: 'ConfigNode'.
CRS-2672: Attempting to start 'ora.ASMNET1LSNR_ASM.lsnr' on 'rhel71'
CRS-2676: Start of 'ora.ASMNET1LSNR_ASM.lsnr' on 'rhel71' succeeded
CRS-2672: Attempting to start 'ora.asm' on 'rhel71'
CRS-2676: Start of 'ora.asm' on 'rhel71' succeeded
CRS-2672: Attempting to start 'ora.OCRDG.dg' on 'rhel71'
CRS-2676: Start of 'ora.OCRDG.dg' on 'rhel71' succeeded
2019/03/19 15:19:48 CLSRSC-594: Executing installation step 20 of 20: 'PostConfig'.

[INFO] [DBT-30001] Disk groups created successfully. Check /opt/app/oracle/cfgtoollogs/asmca/asmca-190319PM031955.log for details.


2019/03/19 15:23:06 CLSRSC-325: Configure Oracle Grid Infrastructure for a Cluster ... succeeded

Root script run on the second (last) node.

./root.sh
Performing root user operation.

Creating /etc/oratab file...
Entries will be added to the /etc/oratab file as needed by
Database Configuration Assistant when a database is created
Finished running generic part of root script.
Now product-specific root actions will be performed.
Relinking oracle with rac_on option
Using configuration parameter file: /opt/app/18.x.0/grid/crs/install/crsconfig_params
The log of current session can be found at:
  /opt/app/oracle/crsdata/rhel72/crsconfig/rootcrs_rhel72_2019-03-19_03-25-21PM.log
2019/03/19 15:25:38 CLSRSC-594: Executing installation step 1 of 20: 'SetupTFA'.
2019/03/19 15:25:38 CLSRSC-4001: Installing Oracle Trace File Analyzer (TFA) Collector.
2019/03/19 15:26:17 CLSRSC-4002: Successfully installed Oracle Trace File Analyzer (TFA) Collector.
2019/03/19 15:26:17 CLSRSC-594: Executing installation step 2 of 20: 'ValidateEnv'.
2019/03/19 15:26:17 CLSRSC-363: User ignored prerequisites during installation
2019/03/19 15:26:17 CLSRSC-594: Executing installation step 3 of 20: 'CheckFirstNode'.
2019/03/19 15:26:19 CLSRSC-594: Executing installation step 4 of 20: 'GenSiteGUIDs'.
2019/03/19 15:26:19 CLSRSC-594: Executing installation step 5 of 20: 'SaveParamFile'.
2019/03/19 15:26:23 CLSRSC-594: Executing installation step 6 of 20: 'SetupOSD'.
2019/03/19 15:26:23 CLSRSC-594: Executing installation step 7 of 20: 'CheckCRSConfig'.
2019/03/19 15:26:25 CLSRSC-594: Executing installation step 8 of 20: 'SetupLocalGPNP'.
2019/03/19 15:26:28 CLSRSC-594: Executing installation step 9 of 20: 'CreateRootCert'.
2019/03/19 15:26:28 CLSRSC-594: Executing installation step 10 of 20: 'ConfigOLR'.
2019/03/19 15:26:31 CLSRSC-594: Executing installation step 11 of 20: 'ConfigCHMOS'.
2019/03/19 15:26:31 CLSRSC-594: Executing installation step 12 of 20: 'CreateOHASD'.
2019/03/19 15:26:34 CLSRSC-594: Executing installation step 13 of 20: 'ConfigOHASD'.
2019/03/19 15:26:35 CLSRSC-330: Adding Clusterware entries to file 'oracle-ohasd.service'
2019/03/19 15:27:53 CLSRSC-594: Executing installation step 14 of 20: 'InstallAFD'.
2019/03/19 15:27:56 CLSRSC-594: Executing installation step 15 of 20: 'InstallACFS'.
CRS-2791: Starting shutdown of Oracle High Availability Services-managed resources on 'rhel72'
CRS-2793: Shutdown of Oracle High Availability Services-managed resources on 'rhel72' has completed
CRS-4133: Oracle High Availability Services has been stopped.
CRS-4123: Oracle High Availability Services has been started.
2019/03/19 15:29:13 CLSRSC-594: Executing installation step 16 of 20: 'InstallKA'.
2019/03/19 15:29:15 CLSRSC-594: Executing installation step 17 of 20: 'InitConfig'.
CRS-2791: Starting shutdown of Oracle High Availability Services-managed resources on 'rhel72'
CRS-2793: Shutdown of Oracle High Availability Services-managed resources on 'rhel72' has completed
CRS-4133: Oracle High Availability Services has been stopped.
CRS-4123: Oracle High Availability Services has been started.
CRS-2791: Starting shutdown of Oracle High Availability Services-managed resources on 'rhel72'
CRS-2673: Attempting to stop 'ora.drivers.acfs' on 'rhel72'
CRS-2677: Stop of 'ora.drivers.acfs' on 'rhel72' succeeded
CRS-2793: Shutdown of Oracle High Availability Services-managed resources on 'rhel72' has completed
CRS-4133: Oracle High Availability Services has been stopped.
2019/03/19 15:29:28 CLSRSC-594: Executing installation step 18 of 20: 'StartCluster'.
CRS-4123: Starting Oracle High Availability Services-managed resources
CRS-2672: Attempting to start 'ora.mdnsd' on 'rhel72'
CRS-2672: Attempting to start 'ora.evmd' on 'rhel72'
CRS-2676: Start of 'ora.mdnsd' on 'rhel72' succeeded
CRS-2676: Start of 'ora.evmd' on 'rhel72' succeeded
CRS-2672: Attempting to start 'ora.gpnpd' on 'rhel72'
CRS-2676: Start of 'ora.gpnpd' on 'rhel72' succeeded
CRS-2672: Attempting to start 'ora.gipcd' on 'rhel72'
CRS-2676: Start of 'ora.gipcd' on 'rhel72' succeeded
CRS-2672: Attempting to start 'ora.crf' on 'rhel72'
CRS-2672: Attempting to start 'ora.cssdmonitor' on 'rhel72'
CRS-2676: Start of 'ora.cssdmonitor' on 'rhel72' succeeded
CRS-2672: Attempting to start 'ora.cssd' on 'rhel72'
CRS-2672: Attempting to start 'ora.diskmon' on 'rhel72'
CRS-2676: Start of 'ora.diskmon' on 'rhel72' succeeded
CRS-2676: Start of 'ora.crf' on 'rhel72' succeeded
CRS-2676: Start of 'ora.cssd' on 'rhel72' succeeded
CRS-2672: Attempting to start 'ora.cluster_interconnect.haip' on 'rhel72'
CRS-2672: Attempting to start 'ora.ctssd' on 'rhel72'
CRS-2676: Start of 'ora.ctssd' on 'rhel72' succeeded
CRS-2672: Attempting to start 'ora.crsd' on 'rhel72'
CRS-2676: Start of 'ora.crsd' on 'rhel72' succeeded
CRS-2676: Start of 'ora.cluster_interconnect.haip' on 'rhel72' succeeded
CRS-2672: Attempting to start 'ora.asm' on 'rhel72'
CRS-2676: Start of 'ora.asm' on 'rhel72' succeeded
CRS-6017: Processing resource auto-start for servers: rhel72
CRS-2673: Attempting to stop 'ora.LISTENER_SCAN1.lsnr' on 'rhel71'
CRS-2672: Attempting to start 'ora.ons' on 'rhel72'
CRS-2672: Attempting to start 'ora.ASMNET1LSNR_ASM.lsnr' on 'rhel72'
CRS-2677: Stop of 'ora.LISTENER_SCAN1.lsnr' on 'rhel71' succeeded
CRS-2673: Attempting to stop 'ora.scan1.vip' on 'rhel71'
CRS-2677: Stop of 'ora.scan1.vip' on 'rhel71' succeeded
CRS-2672: Attempting to start 'ora.scan1.vip' on 'rhel72'
CRS-2676: Start of 'ora.ASMNET1LSNR_ASM.lsnr' on 'rhel72' succeeded
CRS-2672: Attempting to start 'ora.asm' on 'rhel72'
CRS-2676: Start of 'ora.scan1.vip' on 'rhel72' succeeded
CRS-2672: Attempting to start 'ora.LISTENER_SCAN1.lsnr' on 'rhel72'
CRS-2676: Start of 'ora.ons' on 'rhel72' succeeded
CRS-2676: Start of 'ora.LISTENER_SCAN1.lsnr' on 'rhel72' succeeded
CRS-2676: Start of 'ora.asm' on 'rhel72' succeeded
CRS-2672: Attempting to start 'ora.proxy_advm' on 'rhel71'
CRS-2672: Attempting to start 'ora.proxy_advm' on 'rhel72'
CRS-2676: Start of 'ora.proxy_advm' on 'rhel71' succeeded
CRS-2676: Start of 'ora.proxy_advm' on 'rhel72' succeeded
CRS-6016: Resource auto-start has completed for server rhel72
CRS-6024: Completed start of Oracle Cluster Ready Services-managed resources
CRS-4123: Oracle High Availability Services has been started.
2019/03/19 15:32:28 CLSRSC-343: Successfully started Oracle Clusterware stack
2019/03/19 15:32:29 CLSRSC-594: Executing installation step 19 of 20: 'ConfigNode'.
2019/03/19 15:33:03 CLSRSC-594: Executing installation step 20 of 20: 'PostConfig'.
2019/03/19 15:33:13 CLSRSC-325: Configure Oracle Grid Infrastructure for a Cluster ... succeeded

8. Once the root scripts are run click OK on the configuration dialogue box to complete the creation of GIMR and other configuration assistants. At the end this clusterware installation would be completed.

9. Run cluvfy with post crsinst to check the installation.

[grid@rhel71 ~]$ cluvfy stage -post crsinst -allnodes

Verifying Node Connectivity ...
  Verifying Hosts File ...PASSED
  Verifying Check that maximum (MTU) size packet goes through subnet ...PASSED
  Verifying subnet mask consistency for subnet "192.168.1.0" ...PASSED
  Verifying subnet mask consistency for subnet "192.168.0.0" ...PASSED
Verifying Node Connectivity ...PASSED
Verifying Multicast or broadcast check ...PASSED
Verifying ASM filter driver configuration consistency ...PASSED
Verifying Time zone consistency ...PASSED
Verifying Cluster Manager Integrity ...PASSED
Verifying User Mask ...PASSED
Verifying Cluster Integrity ...PASSED
Verifying OCR Integrity ...PASSED
Verifying CRS Integrity ...
  Verifying Clusterware Version Consistency ...PASSED
Verifying CRS Integrity ...PASSED
Verifying Node Application Existence ...PASSED
Verifying Single Client Access Name (SCAN) ...
  Verifying DNS/NIS name service 'rac-scan.domain.net' ...
    Verifying Name Service Switch Configuration File Integrity ...PASSED
  Verifying DNS/NIS name service 'rac-scan.domain.net' ...PASSED
Verifying Single Client Access Name (SCAN) ...PASSED
Verifying OLR Integrity ...PASSED
Verifying Voting Disk ...PASSED
Verifying ASM Integrity ...
  Verifying Node Connectivity ...
    Verifying Hosts File ...PASSED
    Verifying Check that maximum (MTU) size packet goes through subnet ...PASSED
    Verifying subnet mask consistency for subnet "192.168.1.0" ...PASSED
    Verifying subnet mask consistency for subnet "192.168.0.0" ...PASSED
  Verifying Node Connectivity ...PASSED
Verifying ASM Integrity ...PASSED
Verifying Device Checks for ASM ...
  Verifying Access Control List check ...PASSED
  Verifying I/O scheduler ...PASSED
Verifying Device Checks for ASM ...PASSED
Verifying ASM disk group free space ...PASSED
Verifying User Not In Group "root": grid ...PASSED
Verifying Clock Synchronization ...PASSED
Verifying VIP Subnet configuration check ...PASSED
Verifying Network configuration consistency checks ...PASSED
Verifying File system mount options for path GI_HOME ...PASSED
Verifying Access control attributes for /var/tmp/.oracle ...PASSED
Verifying Access control attributes for %OCRCONFIGDIR%/maps ...PASSED

Post-check for cluster services setup was successful.

10. There are few cluster and local resources that are new in 18c compared to 12.2 Most of them relate the rapid home provisioning feature and are in disable state.

 crsctl stat res -t
--------------------------------------------------------------------------------
Name           Target  State        Server                   State details
--------------------------------------------------------------------------------
Local Resources
--------------------------------------------------------------------------------
ora.GIMRDG.GHCHKPT.advm
               OFFLINE OFFLINE      rhel71                   STABLE
...
ora.gimrdg.ghchkpt.acfs
               OFFLINE OFFLINE      rhel71                   STABLE
               OFFLINE OFFLINE      rhel72                   STABLE
ora.helper
               OFFLINE OFFLINE      rhel71                   IDLE,STABLE
               OFFLINE OFFLINE      rhel72                   IDLE,STABLE
ora.proxy_advm
               ONLINE  ONLINE       rhel71                   STABLE
               ONLINE  ONLINE       rhel72                   STABLE
--------------------------------------------------------------------------------
Cluster Resources
--------------------------------------------------------------------------------
...
ora.rhpserver
      1        OFFLINE OFFLINE                               STABLE
--------------------------------------------------------------------------------

For example clusterware installation create a pre-configued ASM volume group call GHCHKPT which is to be used with rapid home provisioning framework.

ASMCMD> volinfo --all
Diskgroup Name: GIMRDG

         Volume Name: GHCHKPT
         Volume Device: /dev/asm/ghchkpt-493
         State: DISABLED
         Size (MB): 1536
         Resize Unit (MB): 512
         Redundancy: UNPROT
         Stripe Columns: 8
         Stripe Width (K): 1024
         Usage: ACFS
         Mountpath: /opt/oracle/rhp_images/chkbase

Because of this volume group the proxy ASM instance will have an active and online status. These would be visible as APX{node number} instances. Oracle docs states "an Oracle ASM proxy instance is an Oracle instance running on a Hub node with a direct Oracle ASM client. An Oracle ASM proxy instance provides support for Oracle Automatic Storage Management Cluster File System (Oracle ACFS) and Oracle ASM Dynamic Volume Manager (Oracle ADVM)". If no other ASM volume groups are to be created or online then ASM proxy instances could be stopped and disabled.

[grid@rhel71 ~]$ srvctl stop asm -proxy -node rhel71
[grid@rhel71 ~]$ srvctl stop asm -proxy -node rhel72
[grid@rhel71 ~]$ srvctl status asm -proxy
ADVM proxy is not running

The full list of resources after the installation is shown below (before proxy instances were disabled).

Resource Name                       Type                      Target             State              Host
-------------                       ------                    -------            --------           ----------
ora.ASMNET1LSNR_ASM.lsnr            ora.asm_listener.type     ONLINE             ONLINE             rhel71
ora.ASMNET1LSNR_ASM.lsnr            ora.asm_listener.type     ONLINE             ONLINE             rhel72
ora.GIMRDG.GHCHKPT.advm             ora.volume.type           OFFLINE            OFFLINE
ora.GIMRDG.GHCHKPT.advm             ora.volume.type           OFFLINE            OFFLINE
ora.GIMRDG.dg                       ora.diskgroup.type        ONLINE             ONLINE             rhel71
ora.GIMRDG.dg                       ora.diskgroup.type        ONLINE             ONLINE             rhel72
ora.LISTENER.lsnr                   ora.listener.type         ONLINE             ONLINE             rhel71
ora.LISTENER.lsnr                   ora.listener.type         ONLINE             ONLINE             rhel72
ora.LISTENER_SCAN1.lsnr             ora.scan_listener.type    ONLINE             ONLINE             rhel72
ora.LISTENER_SCAN2.lsnr             ora.scan_listener.type    ONLINE             ONLINE             rhel71
ora.LISTENER_SCAN3.lsnr             ora.scan_listener.type    ONLINE             ONLINE             rhel71
ora.MGMTLSNR                        ora.mgmtlsnr.type         ONLINE             ONLINE             rhel71
ora.OCRDG.dg                        ora.diskgroup.type        ONLINE             ONLINE             rhel71
ora.OCRDG.dg                        ora.diskgroup.type        ONLINE             ONLINE             rhel72
ora.asm                             ora.asm.type              ONLINE             ONLINE             rhel71
ora.asm                             ora.asm.type              ONLINE             ONLINE             rhel72
ora.asm                             ora.asm.type              OFFLINE            OFFLINE
ora.chad                            ora.chad.type             ONLINE             ONLINE             rhel71
ora.chad                            ora.chad.type             ONLINE             ONLINE             rhel72
ora.cvu                             ora.cvu.type              ONLINE             ONLINE             rhel71
ora.gimrdg.ghchkpt.acfs             ora.acfs.type             OFFLINE            OFFLINE
ora.gimrdg.ghchkpt.acfs             ora.acfs.type             OFFLINE            OFFLINE
ora.helper                          ora.helper.type           OFFLINE            OFFLINE
ora.helper                          ora.helper.type           OFFLINE            OFFLINE
ora.mgmtdb                          ora.mgmtdb.type           ONLINE             ONLINE             rhel71
ora.net1.network                    ora.network.type          ONLINE             ONLINE             rhel71
ora.net1.network                    ora.network.type          ONLINE             ONLINE             rhel72
ora.ons                             ora.ons.type              ONLINE             ONLINE             rhel71
ora.ons                             ora.ons.type              ONLINE             ONLINE             rhel72
ora.proxy_advm                      ora.proxy_advm.type       ONLINE             ONLINE             rhel71
ora.proxy_advm                      ora.proxy_advm.type       ONLINE             ONLINE             rhel72
ora.qosmserver                      ora.qosmserver.type       ONLINE             ONLINE             rhel71
ora.rhel71.vip                      ora.cluster_vip_net1.type ONLINE             ONLINE             rhel71
ora.rhel72.vip                      ora.cluster_vip_net1.type ONLINE             ONLINE             rhel72
ora.rhpserver                       ora.rhpserver.type        OFFLINE            OFFLINE
ora.scan1.vip                       ora.scan_vip.type         ONLINE             ONLINE             rhel72
ora.scan2.vip                       ora.scan_vip.type         ONLINE             ONLINE             rhel71
ora.scan3.vip                       ora.scan_vip.type         ONLINE             ONLINE             rhel71

11. The default backup location for OCR is the disk group used for GIMR

ocrconfig -showbackuploc
The Oracle Cluster Registry backup location is [+GIMRDG].

12. The clusermode and versions are as follows.

$ crsctl get cluster mode status
Cluster is running in "flex" mode

$crsctl get node role config -all
Node 'rhel71' configured role is 'hub'
Node 'rhel72' configured role is 'hub'

$ crsctl query crs activeversion
Oracle Clusterware active version on the cluster is [18.0.0.0.0]

The next step is to install the database software and craete the RAC DB.

Related Posts
Installing 12cR2 (12.2.0.1) RAC on RHEL 6 with Role Separation - Clusterware
Installing 12c (12.1.0.2) Flex Cluster on RHEL 6 with Role Separation
Installing 12c (12.1.0.1) RAC on RHEL 6 with Role Separation - Clusterware
Installing 11gR2 (11.2.0.3) GI with Role Separation on RHEL 6
Installing 11gR2 (11.2.0.3) GI with Role Separation on OEL 6
Installing 11gR2 Standalone Server with ASM and Role Separation on RHEL 6
11gR2 Standalone Data Guard (with ASM and Role Separation)

Update on 2020-10-01
Following error was given when intsalling a role separated Oracle restart on 19c. Error was thrown during the Oracle home installaton.

 ./runInstaller -silent -responseFile /home/oracle/db_install.rsp -ignorePrereqFailure
Launching Oracle Database Setup Wizard...

[FATAL] [INS-30215] Installer has detected that Oracle Grid Infrastructure owner "grid" does not belong to the OS group "racdba" on local node.
   CAUSE: The OS group "racdba" does not exist or Oracle Grid Infrastructure owner "grid" is not a member of the OS group "racdba" on local node.
   ACTION: Ensure that OS group "racdba" exists and add Oracle Grid Infrastructure owner "grid" as a member of the OS group "racdba" on local node or specify a different OS group in which Oracle Grid Infrastructure owner is a member.
*ADDITIONAL INFORMATION:*
 - PRVF-7566 : User "grid" does not belong to group "racdba" on node "ip-172-31-2-77"
 - Cause:  The specified user is not a member of the specified group on the specified node.
 - Action:  Make the user a member of the group on the specified node.

It seems with 19c grid user should also be a memeber of the racdba group.

A! Help

Labels

Wednesday, May 11, 2022

Upgrading Oracle Restart from 19c to 21c on RHEL 7

Saturday, February 1, 2020

Upgrading Oracle Single Instance with ASM (Oracle Restart) from 11.2.0.4 to 19c (19.6) on RHEL 7

Sunday, April 7, 2019

Installing 18c (18.3) RAC on RHEL 7 with Role Separation - Database

Monday, April 1, 2019

Installing 18c (18.3) RAC on RHEL 7 with Role Separation - Clusterware

About Me

Downloads

Quick Response

Popular

Blog Archive

Total Pageviews

Followers

Oracle Documentation