Saturday, September 21, 2019

Oracle Extended Cluster Setup on 19c

Extended cluster or stretch cluster are setups where each cluster node span a considerable geographical distance. They would require specialist network and storage setup which makes the distance between the nodes transparent to the database. Each node in a geographic site would have local storage and a copy of the data. How data is copied from one site to another is where the specialist hardware/network/storage comes in. The white paper here gives a good explanation of extended clusters and points to consider when implementing an Oracle stretch cluster.
Until 12.2 the GI didn't have a specialist option for installing a extended cluster. Once the hardware is setup in a transparent way the GI was installed as one would in a standard cluster where all nodes reside on the same location. But with 12.2 a new option was introduced which allowed to specially select the extended cluster option. This meant GI is now aware of the extended cluster setup.
The blog post shows steps of installing a extended cluster using 19.3 GI.
To configure an extended cluster, select the cluster configuration and then select the option to configure as an extended cluster. On extended cluster each node would be associated with a site. However an additional quorum site must also be specified in order to proceed with the configuration. In this case alpha and beta are the sites where the cluster nodes will reside. But specifying only those sites will give an error.
As per the error message shown it seems extend cluster can have up to 5 sites. So in total 3 sites are specified. Alpha and Beta for nodes and Omega for the quorum site.
The quorum sites could be just a storage only site (no Oracle GI is installed here) that is in equal distance to other two sites. Oracle even allows NFS to be used for the quorum site.
Nothing special need to be done for the SCAN setup compared to a standard cluster.
As mentioned above each node in an extended cluster configuration will be assigned a site. In this case each node is assigned to sites alpha and beta.
Setup the public and private inter-connect network interfaces. In an extended cluster configuration the private interconnect speeds could restrict the distance between sites so that performance doesn't degrade beyond acceptable level.
Choose ASM for storage, as ASM it provides host based mirroring. In this setup the GIMR installation was not configured.
To store OCR and vote disk in ASM, the ASM diskgroup must be of extended, high, normal or flex redundancy type. External redundancy is not an option in an extended cluster. Oracle recommends using extended redundancy for the this disk group. Extended redundancy would require three failure groups per site in addition to a quorum site. A lesser number of failure groups would not allow to proceed.
To create failure groups select specify failure groups button. Define three failure groups per site and mark the quorum site failure group by selecting it.
When specifying disk for the diskgroup allocate the respective failure group.
Below is the summary of the GI setup. It list the sites on the extended cluster and each node associated with it.
When prompted run the root scripts. There's nothing on the root script output to indicate an extended cluster setup. Root script output from site alpha node rhel71.
# /opt/app/19.x.0/grid/root.sh
Performing root user operation.

The following environment variables are set as:
    ORACLE_OWNER= oracle
    ORACLE_HOME=  /opt/app/19.x.0/grid

Enter the full pathname of the local bin directory: [/usr/local/bin]:
The contents of "dbhome" have not changed. No need to overwrite.
The contents of "oraenv" have not changed. No need to overwrite.
The contents of "coraenv" have not changed. No need to overwrite.


Creating /etc/oratab file...
Entries will be added to the /etc/oratab file as needed by
Database Configuration Assistant when a database is created
Finished running generic part of root script.
Now product-specific root actions will be performed.
Relinking oracle with rac_on option
Using configuration parameter file: /opt/app/19.x.0/grid/crs/install/crsconfig_params
The log of current session can be found at:
  /opt/app/oracle/crsdata/rhel71/crsconfig/rootcrs_rhel71_2019-08-21_08-36-05AM.log
2019/08/21 08:36:31 CLSRSC-594: Executing installation step 1 of 19: 'SetupTFA'.
2019/08/21 08:36:31 CLSRSC-594: Executing installation step 2 of 19: 'ValidateEnv'.
2019/08/21 08:36:31 CLSRSC-363: User ignored prerequisites during installation
2019/08/21 08:36:31 CLSRSC-594: Executing installation step 3 of 19: 'CheckFirstNode'.
2019/08/21 08:36:37 CLSRSC-594: Executing installation step 4 of 19: 'GenSiteGUIDs'.
2019/08/21 08:36:40 CLSRSC-594: Executing installation step 5 of 19: 'SetupOSD'.
2019/08/21 08:36:40 CLSRSC-594: Executing installation step 6 of 19: 'CheckCRSConfig'.
2019/08/21 08:36:41 CLSRSC-594: Executing installation step 7 of 19: 'SetupLocalGPNP'.
2019/08/21 08:37:41 CLSRSC-4002: Successfully installed Oracle Trace File Analyzer (TFA) Collector.
2019/08/21 08:37:42 CLSRSC-594: Executing installation step 8 of 19: 'CreateRootCert'.
2019/08/21 08:37:52 CLSRSC-594: Executing installation step 9 of 19: 'ConfigOLR'.
2019/08/21 08:38:21 CLSRSC-594: Executing installation step 10 of 19: 'ConfigCHMOS'.
2019/08/21 08:38:21 CLSRSC-594: Executing installation step 11 of 19: 'CreateOHASD'.
2019/08/21 08:38:35 CLSRSC-594: Executing installation step 12 of 19: 'ConfigOHASD'.
2019/08/21 08:38:36 CLSRSC-330: Adding Clusterware entries to file 'oracle-ohasd.service'
2019/08/21 08:39:59 CLSRSC-594: Executing installation step 13 of 19: 'InstallAFD'.
2019/08/21 08:40:14 CLSRSC-594: Executing installation step 14 of 19: 'InstallACFS'.
2019/08/21 08:41:55 CLSRSC-594: Executing installation step 15 of 19: 'InstallKA'.
2019/08/21 08:42:10 CLSRSC-594: Executing installation step 16 of 19: 'InitConfig'.

ASM has been created and started successfully.

[DBT-30001] Disk groups created successfully. Check /opt/app/oracle/cfgtoollogs/asmca/asmca-190821AM084259.log for details.

2019/08/21 08:44:36 CLSRSC-482: Running command: '/opt/app/19.x.0/grid/bin/ocrconfig -upgrade oracle oinstall'
CRS-4256: Updating the profile
Successful addition of voting disk a06f85aec09b4fa4bf90ce54a22f9ad4.
Successful addition of voting disk d335f6377ec04f96bfca7598c7fa632b.
Successful addition of voting disk d977e270ea0c4f6cbffd9b3c705aff33.
Successful addition of voting disk 1f89e026ce3b4f43bfffba60ceca2e64.
Successful addition of voting disk b8f67fe289bc4f5fbfdac97c76844452.
Successful addition of voting disk 9092fa74e0a34ff6bfae1bfa43c0f6b0.
Successful addition of voting disk 245d2ef7d3384f13bf39dc54343bcc91.
Successfully replaced voting disk group with +OCRDG.
CRS-4256: Updating the profile
CRS-4266: Voting file(s) successfully replaced
##  STATE    File Universal Id                File Name Disk group
--  -----    -----------------                --------- ---------
 1. ONLINE   a06f85aec09b4fa4bf90ce54a22f9ad4 (/dev/oracleasm/quo1) [OCRDG]
 2. ONLINE   d335f6377ec04f96bfca7598c7fa632b (/dev/oracleasm/bocr3) [OCRDG]
 3. ONLINE   d977e270ea0c4f6cbffd9b3c705aff33 (/dev/oracleasm/bocr1) [OCRDG]
 4. ONLINE   1f89e026ce3b4f43bfffba60ceca2e64 (/dev/oracleasm/bocr2) [OCRDG]
 5. ONLINE   b8f67fe289bc4f5fbfdac97c76844452 (/dev/oracleasm/aocr3) [OCRDG]
 6. ONLINE   9092fa74e0a34ff6bfae1bfa43c0f6b0 (/dev/oracleasm/aocr1) [OCRDG]
 7. ONLINE   245d2ef7d3384f13bf39dc54343bcc91 (/dev/oracleasm/aocr2) [OCRDG]
Located 7 voting disk(s).
2019/08/21 08:47:50 CLSRSC-594: Executing installation step 17 of 19: 'StartCluster'.
2019/08/21 08:49:23 CLSRSC-343: Successfully started Oracle Clusterware stack
2019/08/21 08:49:24 CLSRSC-594: Executing installation step 18 of 19: 'ConfigNode'.
2019/08/21 08:53:20 CLSRSC-594: Executing installation step 19 of 19: 'PostConfig'.
2019/08/21 08:54:26 CLSRSC-325: Configure Oracle Grid Infrastructure for a Cluster ... succeeded
Root script output from site beta node rhel72.
# /opt/app/19.x.0/grid/root.sh
Performing root user operation.

The following environment variables are set as:
    ORACLE_OWNER= oracle
    ORACLE_HOME=  /opt/app/19.x.0/grid

Enter the full pathname of the local bin directory: [/usr/local/bin]:
The contents of "dbhome" have not changed. No need to overwrite.
The contents of "oraenv" have not changed. No need to overwrite.
The contents of "coraenv" have not changed. No need to overwrite.


Creating /etc/oratab file...
Entries will be added to the /etc/oratab file as needed by
Database Configuration Assistant when a database is created
Finished running generic part of root script.
Now product-specific root actions will be performed.
Relinking oracle with rac_on option
Using configuration parameter file: /opt/app/19.x.0/grid/crs/install/crsconfig_params
The log of current session can be found at:
  /opt/app/oracle/crsdata/rhel72/crsconfig/rootcrs_rhel72_2019-08-21_08-55-15AM.log
2019/08/21 08:55:33 CLSRSC-594: Executing installation step 1 of 19: 'SetupTFA'.
2019/08/21 08:55:33 CLSRSC-594: Executing installation step 2 of 19: 'ValidateEnv'.
2019/08/21 08:55:33 CLSRSC-363: User ignored prerequisites during installation
2019/08/21 08:55:33 CLSRSC-594: Executing installation step 3 of 19: 'CheckFirstNode'.
2019/08/21 08:55:36 CLSRSC-594: Executing installation step 4 of 19: 'GenSiteGUIDs'.
2019/08/21 08:55:37 CLSRSC-594: Executing installation step 5 of 19: 'SetupOSD'.
2019/08/21 08:55:37 CLSRSC-594: Executing installation step 6 of 19: 'CheckCRSConfig'.
2019/08/21 08:55:40 CLSRSC-594: Executing installation step 7 of 19: 'SetupLocalGPNP'.
2019/08/21 08:55:43 CLSRSC-594: Executing installation step 8 of 19: 'CreateRootCert'.
2019/08/21 08:55:43 CLSRSC-594: Executing installation step 9 of 19: 'ConfigOLR'.
2019/08/21 08:55:58 CLSRSC-594: Executing installation step 10 of 19: 'ConfigCHMOS'.
2019/08/21 08:55:58 CLSRSC-594: Executing installation step 11 of 19: 'CreateOHASD'.
2019/08/21 08:56:02 CLSRSC-594: Executing installation step 12 of 19: 'ConfigOHASD'.
2019/08/21 08:56:03 CLSRSC-330: Adding Clusterware entries to file 'oracle-ohasd.service'
2019/08/21 08:56:36 CLSRSC-4002: Successfully installed Oracle Trace File Analyzer (TFA) Collector.
2019/08/21 08:57:18 CLSRSC-594: Executing installation step 13 of 19: 'InstallAFD'.
2019/08/21 08:57:22 CLSRSC-594: Executing installation step 14 of 19: 'InstallACFS'.
2019/08/21 08:58:55 CLSRSC-594: Executing installation step 15 of 19: 'InstallKA'.
2019/08/21 08:58:59 CLSRSC-594: Executing installation step 16 of 19: 'InitConfig'.
2019/08/21 08:59:20 CLSRSC-594: Executing installation step 17 of 19: 'StartCluster'.
2019/08/21 09:00:35 CLSRSC-343: Successfully started Oracle Clusterware stack
2019/08/21 09:00:36 CLSRSC-594: Executing installation step 18 of 19: 'ConfigNode'.
2019/08/21 09:01:08 CLSRSC-594: Executing installation step 19 of 19: 'PostConfig'.
2019/08/21 09:01:25 CLSRSC-325: Configure Oracle Grid Infrastructure for a Cluster ... succeeded
Completes the cluster setup.



At the end of the setup check if the cluster is extended.
$  crsctl get cluster mode status
Cluster is running in "flex" mode

$ crsctl get cluster extended
CRS-6577: The cluster is extended.
Query all cluster sites
$ crsctl query cluster site -all
Site 'alpha' identified by GUID '0dc406e8d13acf67bf0fc86402d1573a' in state 'ENABLED' contains nodes 'rhel71' and disks 'OCRDG_0004(OCRDG),OCRDG_0005(OCRDG),OCRDG_0006(OCRDG)'.
Site 'beta' identified by GUID 'fdafc618b7bc4f9eff995d061d05be6f' in state 'ENABLED' contains nodes 'rhel72' and disks 'OCRDG_0001(OCRDG),OCRDG_0002(OCRDG),OCRDG_0003(OCRDG)'.
Site 'omega' identified by GUID '1884d0cc77367fdabfa889ad2efb84dc' in state 'ENABLED' contains disks 'OCRDG_0000(OCRDG)' and no nodes .
The cluster would consists of 7 vote disks
crsctl query css votedisk
##  STATE    File Universal Id                File Name Disk group
--  -----    -----------------                --------- ---------
 1. ONLINE   a06f85aec09b4fa4bf90ce54a22f9ad4 (/dev/oracleasm/quo1) [OCRDG]
 2. ONLINE   d335f6377ec04f96bfca7598c7fa632b (/dev/oracleasm/bocr3) [OCRDG]
 3. ONLINE   d977e270ea0c4f6cbffd9b3c705aff33 (/dev/oracleasm/bocr1) [OCRDG]
 4. ONLINE   1f89e026ce3b4f43bfffba60ceca2e64 (/dev/oracleasm/bocr2) [OCRDG]
 5. ONLINE   b8f67fe289bc4f5fbfdac97c76844452 (/dev/oracleasm/aocr3) [OCRDG]
 6. ONLINE   9092fa74e0a34ff6bfae1bfa43c0f6b0 (/dev/oracleasm/aocr1) [OCRDG]
 7. ONLINE   245d2ef7d3384f13bf39dc54343bcc91 (/dev/oracleasm/aocr2) [OCRDG]
Located 7 voting disk(s).
Even though for ASM diskgroup where OCR and vote disk were stored was created using extended redundancy it is possible to create a normal redundancy disk group specifying disks for each site. Once the scenarios where oracle suggest the use of normal redundancy is in extended cluster configuration. Run asmca and select normal redundancy for the disk group and specify disk from each site.
crsctl query cluster site -all
Site 'alpha' identified by GUID '0dc406e8d13acf67bf0fc86402d1573a' in state 'ENABLED' contains nodes 'rhel71' and disks 'DATA_0000(DATA),FRA_0000(FRA),OCRDG_0004(OCRDG),OCRDG_0005(OCRDG),OCRDG_0006(OCRDG)'.
Site 'beta' identified by GUID 'fdafc618b7bc4f9eff995d061d05be6f' in state 'ENABLED' contains nodes 'rhel72' and disks 'DATA_0001(DATA),FRA_0001(FRA),OCRDG_0001(OCRDG),OCRDG_0002(OCRDG),OCRDG_0003(OCRDG)'.
Site 'omega' identified by GUID '1884d0cc77367fdabfa889ad2efb84dc' in state 'ENABLED' contains disks 'OCRDG_0000(OCRDG)' and no nodes .
Even though asmca gives the option of external redundancy, create a disk with this redundancy would result in an error.
Another point of concern is preferred read option for ASM diskgroup. It seems for extended cluster this is enable by default
   INST_ID DISKGROUP_ ATTRIBUTE_NAME                 VALUE
---------- ---------- ------------------------------ ----------
         1 DATA       preferred_read.enabled         TRUE
         1 FRA        preferred_read.enabled         TRUE
         1 OCRDG      preferred_read.enabled         TRUE
         2 DATA       preferred_read.enabled         TRUE
         2 FRA        preferred_read.enabled         TRUE
         2 OCRDG      preferred_read.enabled         TRUE
As per Oracle doc "In an Oracle extended cluster, which contains nodes that span multiple physically separated sites, the PREFERRED_READ.ENABLED disk group attribute controls whether preferred read functionality is enabled for a disk group. If preferred read functionality is enabled, then this functionality enables an instance to determine and read from disks at the same site as itself, which can improve performance".
Also is seems setting ASM_PREFERRED_READ_FAILURE_GROUPS is availble for backward compatibility. Following from oracle doc "Whether or not PREFERRED_READ.ENABLED has been enabled, preferred read can be set at the failure group level on an Oracle ASM instance or a client instance in a cluster with the ASM_PREFERRED_READ_FAILURE_GROUPS initialization parameter, which is available for backward compatibility".
Following shows that each ASM instances has set the perferred read disk that is local to the site.
SQL>  select inst_id,site_name,name,failgroup,preferred_read,path from gv$asm_disk order by 1,2,3,4;

   INST_ID SITE_NAME                      NAME                           FAILGROUP                      P PATH
---------- ------------------------------ ------------------------------ ------------------------------ - ----------------------------------------
         1 alpha                          DATA_0000                      DATA_0000                      Y /dev/oracleasm/adata1
           alpha                          FRA_0000                       FRA_0000                       Y /dev/oracleasm/afra1
           alpha                          OCRDG_0004                     ALPHAFG3                       Y /dev/oracleasm/aocr3
           alpha                          OCRDG_0005                     ALPHAFG2                       Y /dev/oracleasm/aocr2
           alpha                          OCRDG_0006                     ALPHAFG1                       Y /dev/oracleasm/aocr1
           beta                           DATA_0001                      DATA_0001                      N /dev/oracleasm/bdata1
           beta                           FRA_0001                       FRA_0001                       N /dev/oracleasm/bfra1
           beta                           OCRDG_0001                     BETAFG3                        N /dev/oracleasm/bocr3
           beta                           OCRDG_0002                     BETAFG2                        N /dev/oracleasm/bocr2
           beta                           OCRDG_0003                     BETAFG1                        N /dev/oracleasm/bocr1
           omega                          OCRDG_0000                     OMEGAFG                        N /dev/oracleasm/quo1

   INST_ID SITE_NAME                      NAME                           FAILGROUP                      P PATH
---------- ------------------------------ ------------------------------ ------------------------------ - ----------------------------------------
         2 alpha                          DATA_0000                      DATA_0000                      N /dev/oracleasm/adata1
           alpha                          FRA_0000                       FRA_0000                       N /dev/oracleasm/afra1
           alpha                          OCRDG_0004                     ALPHAFG3                       N /dev/oracleasm/aocr3
           alpha                          OCRDG_0005                     ALPHAFG2                       N /dev/oracleasm/aocr2
           alpha                          OCRDG_0006                     ALPHAFG1                       N /dev/oracleasm/aocr1
           beta                           DATA_0001                      DATA_0001                      Y /dev/oracleasm/bdata1
           beta                           FRA_0001                       FRA_0001                       Y /dev/oracleasm/bfra1
           beta                           OCRDG_0001                     BETAFG3                        Y /dev/oracleasm/bocr3
           beta                           OCRDG_0002                     BETAFG2                        Y /dev/oracleasm/bocr2
           beta                           OCRDG_0003                     BETAFG1                        Y /dev/oracleasm/bocr1
           omega                          OCRDG_0000                     OMEGAFG                        N /dev/oracleasm/quo1

There was no speical setup for installing and creating a database on the extendd cluster. Steps for these were same as installing a standard 19c cluster.

Related Posts
Installing 19c (19.3) RAC on RHEL 7 Using Response File
Installing 18c (18.3) RAC on RHEL 7 with Role Separation - Clusterware
Installing 12cR2 (12.2.0.1) RAC on RHEL 6 with Role Separation - Clusterware
Installing 12c (12.1.0.2) Flex Cluster on RHEL 6 with Role Separation
Installing 12c (12.1.0.1) RAC on RHEL 6 with Role Separation - Clusterware
Installing 11gR2 (11.2.0.3) GI with Role Separation on RHEL 6
Installing 11gR2 (11.2.0.3) GI with Role Separation on OEL 6
Installing 11gR2 Standalone Server with ASM and Role Separation on RHEL 6
11gR2 Standalone Data Guard (with ASM and Role Separation)

Saturday, September 7, 2019

Installing 19c (19.3) RAC on RHEL 7 Using Response File

This post shows the steps for installing 19c RAC using response file (silent installation). This post is not a step by step guide to installing 19c RAC. The majority of the pre-reqs are same as that of 18c RAC installation. For detail pre-reqs refer the Oracle 19c clusterware installation documentations.
There few notable differences on 19c RAC compared to previous versions. One of the main ones is RAC is no longer supported on standard edition 2. With 19c release RAC becomes effectively a enterprise edition only.
In terms of installation two of the notable differences are setting up GIMR has become optional. It is possible to install RAC without GIMR. Secondly the support for storing OCR and vote disks on shared file system is back. This was de-supported in 12.2.
This setup keeps in with the same configurations as previous RAC installation. That is, OCR and vote disks are stored in ASM and RAC is installed as a standard cluster.
Assuming the pre-reqs are completed, check the clusterware instillation readiness with cluvfy.
./runcluvfy.sh stage -pre crsinst -n rhel71,rhel72 -r 19 -osdba asmdba -osoper asmoper -asm -asmgrp asmadmin -presence local -asmdev /dev/oracleasm/ocr1,/dev/oracleasm/ocr2,/dev/oracleasm/ocr3,/dev/oracleasm/gimr,/dev/oracleasm/data1,/dev/oracleasm/fra1 -crshome /opt/app/19.x.0/grid -fixup  -orainv oinstall -method root

Verifying Physical Memory ...FAILED (PRVF-7530)
Verifying Available Physical Memory ...PASSED
Verifying Swap Size ...FAILED (PRVF-7573)
Verifying Free Space: rhel72:/usr,rhel72:/var,rhel72:/etc,rhel72:/opt/app/19.x.0/grid,rhel72:/sbin,rhel72:/tmp ...PASSED
Verifying Free Space: rhel71:/usr,rhel71:/var,rhel71:/etc,rhel71:/opt/app/19.x.0/grid,rhel71:/sbin,rhel71:/tmp ...PASSED
Verifying User Existence: grid ...
  Verifying Users With Same UID: 502 ...PASSED
Verifying User Existence: grid ...PASSED
Verifying Group Existence: asmadmin ...PASSED
Verifying Group Existence: asmoper ...PASSED
Verifying Group Existence: asmdba ...PASSED
Verifying Group Existence: oinstall ...PASSED
Verifying Group Membership: asmdba ...PASSED
Verifying Group Membership: asmadmin ...PASSED
Verifying Group Membership: oinstall(Primary) ...PASSED
Verifying Group Membership: asmoper ...PASSED
Verifying Run Level ...PASSED
Verifying Hard Limit: maximum open file descriptors ...PASSED
Verifying Soft Limit: maximum open file descriptors ...PASSED
Verifying Hard Limit: maximum user processes ...PASSED
Verifying Soft Limit: maximum user processes ...PASSED
Verifying Soft Limit: maximum stack size ...PASSED
Verifying Architecture ...PASSED
Verifying OS Kernel Version ...PASSED
Verifying OS Kernel Parameter: semmsl ...PASSED
Verifying OS Kernel Parameter: semmns ...PASSED
Verifying OS Kernel Parameter: semopm ...PASSED
Verifying OS Kernel Parameter: semmni ...PASSED
Verifying OS Kernel Parameter: shmmax ...PASSED
Verifying OS Kernel Parameter: shmmni ...PASSED
Verifying OS Kernel Parameter: shmall ...PASSED
Verifying OS Kernel Parameter: file-max ...PASSED
Verifying OS Kernel Parameter: ip_local_port_range ...PASSED
Verifying OS Kernel Parameter: rmem_default ...PASSED
Verifying OS Kernel Parameter: rmem_max ...PASSED
Verifying OS Kernel Parameter: wmem_default ...PASSED
Verifying OS Kernel Parameter: wmem_max ...PASSED
Verifying OS Kernel Parameter: aio-max-nr ...PASSED
Verifying Package: kmod-20-21 (x86_64) ...PASSED
Verifying Package: kmod-libs-20-21 (x86_64) ...PASSED
Verifying Package: binutils-2.23.52.0.1 ...PASSED
Verifying Package: compat-libcap1-1.10 ...PASSED
Verifying Package: libgcc-4.8.2 (x86_64) ...PASSED
Verifying Package: libstdc++-4.8.2 (x86_64) ...PASSED
Verifying Package: libstdc++-devel-4.8.2 (x86_64) ...PASSED
Verifying Package: sysstat-10.1.5 ...PASSED
Verifying Package: gcc-c++-4.8.2 ...PASSED
Verifying Package: ksh ...PASSED
Verifying Package: make-3.82 ...PASSED
Verifying Package: glibc-2.17 (x86_64) ...PASSED
Verifying Package: glibc-devel-2.17 (x86_64) ...PASSED
Verifying Package: libaio-0.3.109 (x86_64) ...PASSED
Verifying Package: libaio-devel-0.3.109 (x86_64) ...PASSED
Verifying Package: nfs-utils-1.2.3-15 ...PASSED
Verifying Package: smartmontools-6.2-4 ...PASSED
Verifying Package: net-tools-2.0-0.17 ...PASSED
Verifying Package: compat-libstdc++-33-3.2.3 (x86_64) ...PASSED
Verifying Package: libxcb-1.11 (x86_64) ...PASSED
Verifying Package: libX11-1.6.3 (x86_64) ...PASSED
Verifying Package: libXau-1.0.8 (x86_64) ...PASSED
Verifying Package: libXi-1.7.4 (x86_64) ...PASSED
Verifying Package: libXtst-1.2.2 (x86_64) ...PASSED
Verifying Port Availability for component "Oracle Notification Service (ONS)" ...PASSED
Verifying Port Availability for component "Oracle Cluster Synchronization Services (CSSD)" ...PASSED
Verifying Users With Same UID: 0 ...PASSED
Verifying Current Group ID ...PASSED
Verifying Root user consistency ...PASSED
Verifying Package: cvuqdisk-1.0.10-1 ...PASSED
Verifying Host name ...PASSED
Verifying Node Connectivity ...
  Verifying Hosts File ...PASSED
  Verifying Check that maximum (MTU) size packet goes through subnet ...PASSED
  Verifying subnet mask consistency for subnet "192.168.1.0" ...PASSED
  Verifying subnet mask consistency for subnet "192.168.0.0" ...PASSED
Verifying Node Connectivity ...PASSED
Verifying Multicast or broadcast check ...PASSED
Verifying Device Checks for ASM ...
  Verifying Package: cvuqdisk-1.0.10-1 ...PASSED
  Verifying ASM device sharedness check ...
    Verifying Shared Storage Accessibility:/dev/sdd1,/dev/sde1,/dev/sdf1,/dev/sdg1,/dev/sdb1,/dev/sdc1 ...PASSED
  Verifying ASM device sharedness check ...PASSED
  Verifying Access Control List check ...PASSED
  Verifying I/O scheduler ...PASSED
Verifying Device Checks for ASM ...PASSED
Verifying Network Time Protocol (NTP) ...PASSED
Verifying Same core file name pattern ...PASSED
Verifying User Mask ...PASSED
Verifying User Not In Group "root": grid ...PASSED
Verifying Time zone consistency ...PASSED
Verifying Time offset between nodes ...PASSED
Verifying resolv.conf Integrity ...PASSED
Verifying DNS/NIS name service ...PASSED
Verifying Domain Sockets ...PASSED
Verifying /boot mount ...PASSED
Verifying File system mount options for path GI_HOME ...PASSED
Verifying Daemon "avahi-daemon" not configured and running ...PASSED
Verifying Daemon "proxyt" not configured and running ...PASSED
Verifying Grid Infrastructure home path: /opt/app/19.x.0/grid ...
  Verifying '/opt/app/19.x.0/grid' ...PASSED
Verifying Grid Infrastructure home path: /opt/app/19.x.0/grid ...PASSED
Verifying User Equivalence ...PASSED
Verifying RPM Package Manager database ...PASSED
Verifying /dev/shm mounted as temporary file system ...FAILED (PRVE-0421)
Verifying File system mount options for path /var ...PASSED
Verifying DefaultTasksMax parameter ...PASSED
Verifying zeroconf check ...PASSED
Verifying ASM Filter Driver configuration ...PASSED

Pre-check for cluster services setup was unsuccessful on all the nodes.


Failures were encountered during execution of CVU verification request "stage -pre crsinst".

Verifying Physical Memory ...FAILED
rhel72: PRVF-7530 : Sufficient physical memory is not available on node
        "rhel72" [Required physical memory = 8GB (8388608.0KB)]

rhel71: PRVF-7530 : Sufficient physical memory is not available on node
        "rhel71" [Required physical memory = 8GB (8388608.0KB)]

Verifying Swap Size ...FAILED
rhel72: PRVF-7573 : Sufficient swap size is not available on node "rhel72"
        [Required = 4.6848GB (4912356.0KB) ; Found = 3.7246GB (3905532.0KB)]

rhel71: PRVF-7573 : Sufficient swap size is not available on node "rhel71"
        [Required = 4.6848GB (4912368.0KB) ; Found = 3.7246GB (3905532.0KB)]

Verifying RPM Package Manager database ...INFORMATION
PRVG-11250 : The check "RPM Package Manager database" was not performed because
it needs 'root' user privileges.

Verifying /dev/shm mounted as temporary file system ...FAILED
rhel72: PRVE-0421 : No entry exists in /etc/fstab for mounting /dev/shm

rhel71: PRVE-0421 : No entry exists in /etc/fstab for mounting /dev/shm


CVU operation performed:      stage -pre crsinst
Date:                         Jun 13, 2019 8:08:49 AM
CVU home:                     /opt/app/19.x.0/grid/
User:                         grid

The memory related failures are ignored (as this is a test system). To fix /dev/shm related failure refer 2065603.1.
GI installation provide several resposne (*.rsp) files that could be used to install grid. Though content of these files are the same they are located in different folders and named differently. Copy one of the following files to temporary location to be modified for the installation.
$GI_HOME/install/response/gridsetup.rsp
or
$GI_HOME/inventory/response/grid_install.rsp
Open the file copied to the temporary location and fill out section A-G "To register and configure 'Grid Infrastructure for Cluster'". Below is the content of the response file used for this setup.
###############################################################################
## Copyright(c) Oracle Corporation 1998,2019. All rights reserved.           ##
##                                                                           ##
## Specify values for the variables listed below to customize                ##
## your installation.                                                        ##
##                                                                           ##
## Each variable is associated with a comment. The comment                   ##
## can help to populate the variables with the appropriate                   ##
## values.                                                                   ##
##                                                                           ##
## IMPORTANT NOTE: This file contains plain text passwords and               ##
## should be secured to have read permission only by oracle user             ##
## or db administrator who owns this installation.                           ##
##                                                                           ##
###############################################################################

###############################################################################
##                                                                           ##
## Instructions to fill this response file                                   ##
## To register and configure 'Grid Infrastructure for Cluster'               ##
##  - Fill out sections A,B,C,D,E,F and G                                    ##
##  - Fill out section G if OCR and voting disk should be placed on ASM      ##
##                                                                           ##
## To register and configure 'Grid Infrastructure for Standalone server'     ##
##  - Fill out sections A,B and G                                            ##
##                                                                           ##
## To register software for 'Grid Infrastructure'                            ##
##  - Fill out sections A,B and D                                            ##
##  - Provide the cluster nodes in section D when choosing CRS_SWONLY as     ##
##    installation option in section A                                       ##
##                                                                           ##
## To upgrade clusterware and/or Automatic storage management of earlier     ##
## releases                                                                  ##
##  - Fill out sections A,B,C,D and H                                        ##
##                                                                           ##
## To add more nodes to the cluster                                          ##
##  - Fill out sections A and D                                              ##
##  - Provide the cluster nodes in section D when choosing CRS_ADDNODE as    ##
##    installation option in section A                                       ##
##                                                                           ##
###############################################################################

#------------------------------------------------------------------------------
# Do not change the following system generated value. 
#------------------------------------------------------------------------------
oracle.install.responseFileVersion=/oracle/install/rspfmt_crsinstall_response_schema_v19.0.0

###############################################################################
#                                                                             #
#                          SECTION A - BASIC                                  #
#                                                                             #
###############################################################################


#-------------------------------------------------------------------------------
# Specify the location which holds the inventory files.
# This is an optional parameter if installing on  
# Windows based Operating System.
#-------------------------------------------------------------------------------
INVENTORY_LOCATION=/opt/app/oraInventory

#-------------------------------------------------------------------------------
# Specify the installation option.
# Allowed values: CRS_CONFIG or HA_CONFIG or UPGRADE or CRS_SWONLY or HA_SWONLY
#   - CRS_CONFIG  : To register home and configure Grid Infrastructure for cluster
#   - HA_CONFIG   : To register home and configure Grid Infrastructure for stand alone server
#   - UPGRADE     : To register home and upgrade clusterware software of earlier release
#   - CRS_SWONLY  : To register Grid Infrastructure Software home (can be configured for cluster 
#                   or stand alone server later)
#   - HA_SWONLY   : To register Grid Infrastructure Software home (can be configured for stand 
#                   alone server later. This is only supported on Windows.)
#   - CRS_ADDNODE : To add more nodes to the cluster
#   - CRS_DELETE_NODE : To delete nodes to the cluster
#-------------------------------------------------------------------------------
oracle.install.option=CRS_CONFIG

#-------------------------------------------------------------------------------
# Specify the complete path of the Oracle Base.
#-------------------------------------------------------------------------------
ORACLE_BASE=/opt/app/oracle

################################################################################
#                                                                              #
#                              SECTION B - GROUPS                              #
#                                                                              #
#   The following three groups need to be assigned for all GI installations.   #
#   OSDBA and OSOPER can be the same or different.  OSASM must be different    #
#   than the other two.                                                        #
#   The value to be specified for OSDBA, OSOPER and OSASM group is only for    #
#   Unix based Operating System.                                               #
#   These groups are not required for upgrades, as they will be determined     #
#   from the Oracle home to upgrade.                                           #
#                                                                              #
################################################################################
#-------------------------------------------------------------------------------
# The OSDBA_GROUP is the OS group which is to be granted SYSDBA privileges.
#-------------------------------------------------------------------------------
oracle.install.asm.OSDBA=asmdba

#-------------------------------------------------------------------------------
# The OSOPER_GROUP is the OS group which is to be granted SYSOPER privileges.
# The value to be specified for OSOPER group is optional.
# Value should not be provided if configuring Client Cluster - i.e. storageOption=CLIENT_ASM_STORAGE.
#-------------------------------------------------------------------------------
oracle.install.asm.OSOPER=asmoper

#-------------------------------------------------------------------------------
# The OSASM_GROUP is the OS group which is to be granted SYSASM privileges. This
# must be different than the previous two.
#-------------------------------------------------------------------------------
oracle.install.asm.OSASM=asmadmin

################################################################################
#                                                                              #
#                           SECTION C - SCAN                                   #
#                                                                              #
################################################################################
#-------------------------------------------------------------------------------
# Specify the type of SCAN configuration for the cluster
# Allowed values : LOCAL_SCAN and SHARED_SCAN
#-------------------------------------------------------------------------------
oracle.install.crs.config.scanType=LOCAL_SCAN

#-------------------------------------------------------------------------------
# Applicable only if SHARED_SCAN is being configured for cluster
# Specify the path to the SCAN client data file
#-------------------------------------------------------------------------------
oracle.install.crs.config.SCANClientDataFile=

#-------------------------------------------------------------------------------
# Specify a name for SCAN
# Applicable if LOCAL_SCAN is being configured for the cluster
# If you choose to configure the cluster with GNS with Auto assigned Node VIPs(DHCP),then the scanName should be specified in the format of 'SCAN name.Cluster name.GNS sub-domain'
#-------------------------------------------------------------------------------
oracle.install.crs.config.gpnp.scanName=rac-scan.domain.net

#-------------------------------------------------------------------------------
# Specify a unused port number for SCAN service
#-------------------------------------------------------------------------------

oracle.install.crs.config.gpnp.scanPort=1521

################################################################################
#                                                                              #
#                           SECTION D - CLUSTER & GNS                         #
#                                                                              #
################################################################################
#-------------------------------------------------------------------------------
# Specify the required cluster configuration
# Allowed values: STANDALONE, DOMAIN, MEMBERDB, MEMBERAPP
#-------------------------------------------------------------------------------
oracle.install.crs.config.ClusterConfiguration=STANDALONE

#-------------------------------------------------------------------------------
# Specify 'true' if you would like to configure the cluster as Extended, else
# specify 'false'
#
# Applicable only for STANDALONE and DOMAIN cluster configuration
#-------------------------------------------------------------------------------
oracle.install.crs.config.configureAsExtendedCluster=false


#-------------------------------------------------------------------------------
# Specify the Member Cluster Manifest file
#
# Applicable only for MEMBERDB and MEMBERAPP cluster configuration
#-------------------------------------------------------------------------------
oracle.install.crs.config.memberClusterManifestFile=

#-------------------------------------------------------------------------------
# Specify a name for the Cluster you are creating.
#
# The maximum length allowed for clustername is 63 characters. The name can be 
# any combination of lower and uppercase alphabets (A - Z), (0 - 9) and hyphens (-).
#
# Applicable only for STANDALONE and DOMAIN cluster configuration
#-------------------------------------------------------------------------------
oracle.install.crs.config.clusterName=rhel7-19c-clust

#-------------------------------------------------------------------------------
# Applicable only for STANDALONE, DOMAIN, MEMBERDB cluster configuration.
# Specify 'true' if you would like to configure Grid Naming Service(GNS), else
# specify 'false'
#-------------------------------------------------------------------------------
oracle.install.crs.config.gpnp.configureGNS=false

#-------------------------------------------------------------------------------
# Applicable only for STANDALONE and DOMAIN cluster configuration if you choose to configure GNS.
# Specify 'true' if you would like to assign SCAN name VIP and Node VIPs by DHCP
# , else specify 'false'
#-------------------------------------------------------------------------------
oracle.install.crs.config.autoConfigureClusterNodeVIP=false

#-------------------------------------------------------------------------------
# Applicable only if you choose to configure GNS.
# Specify the type of GNS configuration for cluster
# Allowed values are: CREATE_NEW_GNS and USE_SHARED_GNS
# Only USE_SHARED_GNS value is allowed for MEMBERDB cluster configuration.
#-------------------------------------------------------------------------------
oracle.install.crs.config.gpnp.gnsOption=

#-------------------------------------------------------------------------------
# Applicable only if SHARED_GNS is being configured for cluster
# Specify the path to the GNS client data file
#-------------------------------------------------------------------------------
oracle.install.crs.config.gpnp.gnsClientDataFile=

#-------------------------------------------------------------------------------
# Applicable only for STANDALONE and DOMAIN cluster configuration if you choose to 
# configure GNS for this cluster oracle.install.crs.config.gpnp.gnsOption=CREATE_NEW_GNS
# Specify the GNS subdomain and an unused virtual hostname for GNS service
#-------------------------------------------------------------------------------
oracle.install.crs.config.gpnp.gnsSubDomain=
oracle.install.crs.config.gpnp.gnsVIPAddress=

#-------------------------------------------------------------------------------
# Specify the list of sites - only if configuring an Extended Cluster
#-------------------------------------------------------------------------------
oracle.install.crs.config.sites=

#-------------------------------------------------------------------------------
# Specify the list of nodes that have to be configured to be part of the cluster.
#
# The list should a comma-separated list of tuples.  Each tuple should be a
# colon-separated string that contains
# - 1 field if you have chosen CRS_SWONLY as installation option, or
# - 1 field if configuring an Application Cluster, or
# - 3 fields if configuring a Flex Cluster
# - 3 fields if adding more nodes to the configured cluster, or
# - 4 fields if configuring an Extended Cluster
# 
# The fields should be ordered as follows:
# 1. The first field should be the public node name.
# 2. The second field should be the virtual host name
#    (Should be specified as AUTO if you have chosen 'auto configure for VIP'
#     i.e. autoConfigureClusterNodeVIP=true)
# 3. The third field indicates the site designation for the node. To be specified only if configuring an Extended Cluster.
# Only the 1st field is applicable if you have chosen CRS_SWONLY as installation option
# Only the 1st field is applicable if configuring an Application Cluster
#
# Examples
# For registering GI for a cluster software: oracle.install.crs.config.clusterNodes=node1,node2
# For adding more nodes to the configured cluster: oracle.install.crs.config.clusterNodes=node1:node1-vip,node2:node2-vip
# For configuring Application Cluster: oracle.install.crs.config.clusterNodes=node1,node2
# For configuring Flex Cluster: oracle.install.crs.config.clusterNodes=node1:node1-vip,node2:node2-vip
# For configuring Extended Cluster: oracle.install.crs.config.clusterNodes=node1:node1-vip:site1,node2:node2-vip:site2
# You can specify a range of nodes in the tuple using colon separated fields of format
# hostnameprefix:lowerbound-upperbound:hostnamesuffix:vipsuffix:role of node
#
#-------------------------------------------------------------------------------
oracle.install.crs.config.clusterNodes=rhel71.domain.net:rhel71-vip,rhel72.domain.net:rhel72-vip

#-------------------------------------------------------------------------------
# The value should be a comma separated strings where each string is as shown below
# InterfaceName:SubnetAddress:InterfaceType
# where InterfaceType can be either "1", "2", "3", "4", or "5"
# InterfaceType stand for the following values
#   - 1 : PUBLIC
#   - 2 : PRIVATE
#   - 3 : DO NOT USE
#   - 4 : ASM
#   - 5 : ASM & PRIVATE
#
# For example: eth0:140.87.24.0:1,eth1:10.2.1.0:2,eth2:140.87.52.0:3
#
#-------------------------------------------------------------------------------
oracle.install.crs.config.networkInterfaceList=enp0s3:192.168.0.0:1,enp0s8:192.168.1.0:5

#------------------------------------------------------------------------------
# Specify 'true' if you would like to configure Grid Infrastructure Management
# Repository (GIMR), else specify 'false'.
# This option is only applicable when CRS_CONFIG is chosen as install option,
# and STANDALONE is chosen as cluster configuration.
#------------------------------------------------------------------------------
oracle.install.crs.configureGIMR=true

#------------------------------------------------------------------------------
# Create a separate ASM DiskGroup to store GIMR data.
# Specify 'true' if you would like to separate GIMR data with clusterware data, 
# else specify 'false'
# Value should be 'true' for DOMAIN cluster configurations
# Value can be true/false for STANDALONE cluster configurations.
#------------------------------------------------------------------------------
oracle.install.asm.configureGIMRDataDG=true

################################################################################
#                                                                              #
#                              SECTION E - STORAGE                             #
#                                                                              #
################################################################################

#-------------------------------------------------------------------------------
# Specify the type of storage to use for Oracle Cluster Registry(OCR) and Voting
# Disks files. Only applicable for Standalone and MemberDB cluster.
#   - FLEX_ASM_STORAGE
#   - CLIENT_ASM_STORAGE
#   - FILE_SYSTEM_STORAGE
#
# Option FILE_SYSTEM_STORAGE is only for STANDALONE cluster configuration.
#-------------------------------------------------------------------------------
oracle.install.crs.config.storageOption=FLEX_ASM_STORAGE

#-------------------------------------------------------------------------------
# These properties are applicable only if FILE_SYSTEM_STORAGE is chosen for 
# storing OCR and voting disk
# Specify the location(s) for OCR and voting disks
# Three(3) or one(1) location(s) should be specified for OCR and voting disk, 
# separated by commas.
# Example:
#     For Unix based Operating System:
#     oracle.install.crs.config.sharedFileSystemStorage.votingDiskLocations=/oradbocfs/storage/vdsk1,/oradbocfs/storage/vdsk2,/oradbocfs/storage/vdsk3
#     oracle.install.crs.config.sharedFileSystemStorage.ocrLocations=/oradbocfs/storage/ocr1,/oradbocfs/storage/ocr2,/oradbocfs/storage/ocr3
#     For Windows based Operating System OCR/VDSK on shared storage is not supported.
#-------------------------------------------------------------------------------
oracle.install.crs.config.sharedFileSystemStorage.votingDiskLocations=
oracle.install.crs.config.sharedFileSystemStorage.ocrLocations=                
################################################################################
#                                                                              #
#                               SECTION F - IPMI                               #
#                                                                              #
################################################################################

#-------------------------------------------------------------------------------
# Specify 'true' if you would like to configure Intelligent Power Management interface
# (IPMI), else specify 'false'
#-------------------------------------------------------------------------------
oracle.install.crs.config.useIPMI=false

#-------------------------------------------------------------------------------
# Applicable only if you choose to configure IPMI
# i.e. oracle.install.crs.config.useIPMI=true
# Specify the username and password for using IPMI service
#-------------------------------------------------------------------------------
oracle.install.crs.config.ipmi.bmcUsername=
oracle.install.crs.config.ipmi.bmcPassword=
################################################################################
#                                                                              #
#                                SECTION G - ASM                               #
#                                                                              #
################################################################################


#-------------------------------------------------------------------------------
# Password for SYS user of Oracle ASM
#-------------------------------------------------------------------------------
oracle.install.asm.SYSASMPassword=test1234

#-------------------------------------------------------------------------------
# The ASM DiskGroup
#
# Example: oracle.install.asm.diskGroup.name=data
#
#-------------------------------------------------------------------------------
oracle.install.asm.diskGroup.name=OCRDG

#-------------------------------------------------------------------------------
# Redundancy level to be used by ASM.
# It can be one of the following  
#   - NORMAL
#   - HIGH
#   - EXTERNAL
#   - FLEX#   - EXTENDED (required if oracle.install.crs.config.ClusterConfiguration=EXTENDED)
# Example: oracle.install.asm.diskGroup.redundancy=NORMAL
#
#-------------------------------------------------------------------------------
oracle.install.asm.diskGroup.redundancy=NORMAL

#-------------------------------------------------------------------------------
# Allocation unit size to be used by ASM.
# It can be one of the following values
#   - 1
#   - 2
#   - 4
#   - 8
#   - 16
# Example: oracle.install.asm.diskGroup.AUSize=4
# size unit is MB
#
#-------------------------------------------------------------------------------
oracle.install.asm.diskGroup.AUSize=4

#-------------------------------------------------------------------------------
# Failure Groups for the disk group
# If configuring for Extended cluster specify as list of "failure group name:site"
# tuples.
# Else just specify as list of failure group names
#-------------------------------------------------------------------------------
oracle.install.asm.diskGroup.FailureGroups=

#-------------------------------------------------------------------------------
# List of disks and their failure groups to create a ASM DiskGroup
# (Use this if each of the disks have an associated failure group)
# Failure Groups are not required if oracle.install.asm.diskGroup.redundancy=EXTERNAL
# Example:
#     For Unix based Operating System:
#     oracle.install.asm.diskGroup.disksWithFailureGroupNames=/oracle/asm/disk1,FGName,/oracle/asm/disk2,FGName
#     For Windows based Operating System:
#     oracle.install.asm.diskGroup.disksWithFailureGroupNames=\\.\ORCLDISKDATA0,FGName,\\.\ORCLDISKDATA1,FGName
#
#-------------------------------------------------------------------------------
oracle.install.asm.diskGroup.disksWithFailureGroupNames=/dev/oracleasm/ocr1,,/dev/oracleasm/ocr2,,/dev/oracleasm/ocr3,

#-------------------------------------------------------------------------------
# List of disks to create a ASM DiskGroup
# (Use this variable only if failure groups configuration is not required)
# Example:
#     For Unix based Operating System:
#     oracle.install.asm.diskGroup.disks=/oracle/asm/disk1,/oracle/asm/disk2
#     For Windows based Operating System:
#     oracle.install.asm.diskGroup.disks=\\.\ORCLDISKDATA0,\\.\ORCLDISKDATA1
#
#-------------------------------------------------------------------------------
oracle.install.asm.diskGroup.disks=/dev/oracleasm/ocr1,/dev/oracleasm/ocr2,/dev/oracleasm/ocr3

#-------------------------------------------------------------------------------
# List of failure groups to be marked as QUORUM.
# Quorum failure groups contain only voting disk data, no user data is stored
# Example:
# oracle.install.asm.diskGroup.quorumFailureGroupNames=FGName1,FGName2
#-------------------------------------------------------------------------------
oracle.install.asm.diskGroup.quorumFailureGroupNames=
#-------------------------------------------------------------------------------
# The disk discovery string to be used to discover the disks used create a ASM DiskGroup
#
# Example:
#     For Unix based Operating System:
#     oracle.install.asm.diskGroup.diskDiscoveryString=/oracle/asm/*
#     For Windows based Operating System:
#     oracle.install.asm.diskGroup.diskDiscoveryString=\\.\ORCLDISK*
#
#-------------------------------------------------------------------------------
oracle.install.asm.diskGroup.diskDiscoveryString=/dev/oracleasm/*

#-------------------------------------------------------------------------------
# Password for ASMSNMP account
# ASMSNMP account is used by Oracle Enterprise Manager to monitor Oracle ASM instances
#-------------------------------------------------------------------------------
oracle.install.asm.monitorPassword=test1234

#-------------------------------------------------------------------------------
# GIMR Storage data ASM DiskGroup
# Applicable only when 
# oracle.install.asm.configureGIMRDataDG=true
# Example: oracle.install.asm.GIMRDG.name=MGMT
#
#-------------------------------------------------------------------------------
oracle.install.asm.gimrDG.name=GIMRDG

#-------------------------------------------------------------------------------
# Redundancy level to be used by ASM.
# It can be one of the following  
#   - NORMAL
#   - HIGH
#   - EXTERNAL
#   - FLEX#   - EXTENDED (only if oracle.install.crs.config.ClusterConfiguration=EXTENDED)
# Example: oracle.install.asm.gimrDG.redundancy=NORMAL
#
#-------------------------------------------------------------------------------
oracle.install.asm.gimrDG.redundancy=EXTERNAL

#-------------------------------------------------------------------------------
# Allocation unit size to be used by ASM.
# It can be one of the following values
#   - 1
#   - 2
#   - 4
#   - 8
#   - 16
# Example: oracle.install.asm.gimrDG.AUSize=4
# size unit is MB
#
#-------------------------------------------------------------------------------
oracle.install.asm.gimrDG.AUSize=4

#-------------------------------------------------------------------------------
# Failure Groups for the GIMR storage data ASM disk group
# If configuring for Extended cluster specify as list of "failure group name:site"
# tuples.
# Else just specify as list of failure group names
#-------------------------------------------------------------------------------
oracle.install.asm.gimrDG.FailureGroups=

#-------------------------------------------------------------------------------
# List of disks and their failure groups to create GIMR data ASM DiskGroup
# (Use this if each of the disks have an associated failure group)
# Failure Groups are not required if oracle.install.asm.gimrDG.redundancy=EXTERNAL
# Example:
#     For Unix based Operating System:
#     oracle.install.asm.gimrDG.disksWithFailureGroupNames=/oracle/asm/disk1,FGName,/oracle/asm/disk2,FGName
#     For Windows based Operating System:
#     oracle.install.asm.gimrDG.disksWithFailureGroupNames=\\.\ORCLDISKDATA0,FGName,\\.\ORCLDISKDATA1,FGName
#
#-------------------------------------------------------------------------------
oracle.install.asm.gimrDG.disksWithFailureGroupNames=/dev/oracleasm/gimr,

#-------------------------------------------------------------------------------
# List of disks to create GIMR data ASM DiskGroup
# (Use this variable only if failure groups configuration is not required)
# Example:
#     For Unix based Operating System:
#     oracle.install.asm.gimrDG.disks=/oracle/asm/disk1,/oracle/asm/disk2
#     For Windows based Operating System:
#     oracle.install.asm.gimrDG.disks=\\.\ORCLDISKDATA0,\\.\ORCLDISKDATA1
#
#-------------------------------------------------------------------------------
oracle.install.asm.gimrDG.disks=/dev/oracleasm/gimr

#-------------------------------------------------------------------------------
# List of failure groups to be marked as QUORUM.
# Quorum failure groups contain only voting disk data, no user data is stored
# Example:
# oracle.install.asm.gimrDG.quorumFailureGroupNames=FGName1,FGName2
#-------------------------------------------------------------------------------
oracle.install.asm.gimrDG.quorumFailureGroupNames=

#-------------------------------------------------------------------------------
# Configure AFD - ASM Filter Driver
# Applicable only for FLEX_ASM_STORAGE option
# Specify 'true' if you want to configure AFD, else specify 'false'
#-------------------------------------------------------------------------------
oracle.install.asm.configureAFD=false
#-------------------------------------------------------------------------------
# Configure RHPS - Rapid Home Provisioning Service
# Applicable only for DOMAIN cluster configuration
# Specify 'true' if you want to configure RHP service, else specify 'false'
#-------------------------------------------------------------------------------
oracle.install.crs.configureRHPS=false

################################################################################
#                                                                              #
#                             SECTION H - UPGRADE                              #
#                                                                              #
################################################################################
#-------------------------------------------------------------------------------
# Specify whether to ignore down nodes during upgrade operation.
# Value should be 'true' to ignore down nodes otherwise specify 'false'
#-------------------------------------------------------------------------------
oracle.install.crs.config.ignoreDownNodes=false                
################################################################################
#                                                                              #
#                               MANAGEMENT OPTIONS                             #
#                                                                              #
################################################################################

#-------------------------------------------------------------------------------
# Specify the management option to use for managing Oracle Grid Infrastructure
# Options are:
# 1. CLOUD_CONTROL - If you want to manage your Oracle Grid Infrastructure with Enterprise Manager Cloud Control.
# 2. NONE   -If you do not want to manage your Oracle Grid Infrastructure with Enterprise Manager Cloud Control.
#-------------------------------------------------------------------------------
oracle.install.config.managementOption=NONE

#-------------------------------------------------------------------------------
# Specify the OMS host to connect to Cloud Control.
# Applicable only when oracle.install.config.managementOption=CLOUD_CONTROL
#-------------------------------------------------------------------------------
oracle.install.config.omsHost=

#-------------------------------------------------------------------------------
# Specify the OMS port to connect to Cloud Control.
# Applicable only when oracle.install.config.managementOption=CLOUD_CONTROL
#-------------------------------------------------------------------------------
oracle.install.config.omsPort=0

#-------------------------------------------------------------------------------
# Specify the EM Admin user name to use to connect to Cloud Control.
# Applicable only when oracle.install.config.managementOption=CLOUD_CONTROL
#-------------------------------------------------------------------------------
oracle.install.config.emAdminUser=

#-------------------------------------------------------------------------------
# Specify the EM Admin password to use to connect to Cloud Control.
# Applicable only when oracle.install.config.managementOption=CLOUD_CONTROL
#-------------------------------------------------------------------------------
oracle.install.config.emAdminPassword=
################################################################################
#                                                                              #
#                      Root script execution configuration                     #
#                                                                              #
################################################################################

#-------------------------------------------------------------------------------------------------------
# Specify the root script execution mode.
#
#   - true  : To execute the root script automatically by using the appropriate configuration methods.
#   - false : To execute the root script manually.
#
# If this option is selected, password should be specified on the console.
#-------------------------------------------------------------------------------------------------------
oracle.install.crs.rootconfig.executeRootScript=false

#--------------------------------------------------------------------------------------
# Specify the configuration method to be used for automatic root script execution.
#
# Following are the possible choices:
#   - ROOT
#   - SUDO
#--------------------------------------------------------------------------------------
oracle.install.crs.rootconfig.configMethod=
#--------------------------------------------------------------------------------------
# Specify the absolute path of the sudo program.
#
# Applicable only when SUDO configuration method was chosen.
#--------------------------------------------------------------------------------------
oracle.install.crs.rootconfig.sudoPath=

#--------------------------------------------------------------------------------------
# Specify the name of the user who is in the sudoers list. 
# Applicable only when SUDO configuration method was chosen.
# Note:For Grid Infrastructure for Standalone server installations,the sudo user name must be the username of the user performing the installation.
#--------------------------------------------------------------------------------------
oracle.install.crs.rootconfig.sudoUserName=
#--------------------------------------------------------------------------------------
# Specify the nodes batch map.
#
# This should be a comma separated list of node:batch pairs.
# During upgrade, you can sequence the automatic execution of root scripts
# by pooling the nodes into batches. 
# A maximum of three batches can be specified. 
# Installer will execute the root scripts on all the nodes in one batch before
# proceeding to next batch.
# Root script execution on the local node must be in Batch 1.
#
# Examples: 
# 1. oracle.install.crs.config.batchinfo=Node1:1,Node2:2,Node3:2,Node4:3
# 2. oracle.install.crs.config.batchinfo=Node1:1,Node2:2,Node3:2,Node4:2
# 3. oracle.install.crs.config.batchinfo=Node1:1,Node2:1,Node3:2,Node4:3
#
# Applicable only for UPGRADE install option. 
#--------------------------------------------------------------------------------------
oracle.install.crs.config.batchinfo=
################################################################################
#                                                                              #
#                           APPLICATION CLUSTER OPTIONS                        #
#                                                                              #
################################################################################

#-------------------------------------------------------------------------------
# Specify the Virtual hostname to configure virtual access for your Application
# The value to be specified for Virtual hostname is optional.
#-------------------------------------------------------------------------------
oracle.install.crs.app.applicationAddress=
#################################################################################
#          #
#       DELETE NODE OPTIONS     #
#          #
#################################################################################

#--------------------------------------------------------------------------------
# Specify the node names to delete nodes from cluster. 
# Delete node will be performed only for the remote nodes from the cluster.
#--------------------------------------------------------------------------------
oracle.install.crs.deleteNode.nodes=

Start the grid installation processes by running gridSetup.sh and specifying the response file. The ignorePrereqFailure is used here due the failure of memory related pre-reqs.
./gridSetup.sh -silent -responseFile /media/grid19c.rsp -ignorePrereqFailure
Launching Oracle Grid Infrastructure Setup Wizard...

[WARNING] [INS-13014] Target environment does not meet some optional requirements.
   CAUSE: Some of the optional prerequisites are not met. See logs for details. gridSetupActions2019-06-13_08-14-04AM.log
   ACTION: Identify the list of failed prerequisite checks from the log: gridSetupActions2019-06-13_08-14-04AM.log. Then either from the log file or from installation manual find the appropriate configuration to meet the prerequisites and fix it manually.
The response file for this session can be found at:
 /opt/app/19.x.0/grid/install/response/grid_2019-06-13_08-14-04AM.rsp

You can find the log of this install session at:
 /tmp/GridSetupActions2019-06-13_08-14-04AM/gridSetupActions2019-06-13_08-14-04AM.log

As a root user, execute the following script(s):
        1. /opt/app/oraInventory/orainstRoot.sh
        2. /opt/app/19.x.0/grid/root.sh

Execute /opt/app/oraInventory/orainstRoot.sh on the following nodes:
[rhel71, rhel72]
Execute /opt/app/19.x.0/grid/root.sh on the following nodes:
[rhel71, rhel72]

Run the script on the local node first. After successful completion, you can start the script in parallel on all other nodes.

Successfully Setup Software with warning(s).
As install user, execute the following command to complete the configuration.
        /opt/app/19.x.0/grid/gridSetup.sh -executeConfigTools -responseFile /media/grid19c.rsp [-silent]

When prompted run the root scripts. Unlike a GUI based installation, in a silent mode installation the root script output is written a logfile and not shown on the console. Root script outptu from first node.
cat /opt/app/19.x.0/grid/install/root_rhel71.domain.net_2019-06-13_08-25-41-362238834.log
Performing root user operation.

The following environment variables are set as:
    ORACLE_OWNER= grid
    ORACLE_HOME=  /opt/app/19.x.0/grid
   Copying dbhome to /usr/local/bin ...
   Copying oraenv to /usr/local/bin ...
   Copying coraenv to /usr/local/bin ...


Creating /etc/oratab file...
Entries will be added to the /etc/oratab file as needed by
Database Configuration Assistant when a database is created
Finished running generic part of root script.
Now product-specific root actions will be performed.
Relinking oracle with rac_on option
Using configuration parameter file: /opt/app/19.x.0/grid/crs/install/crsconfig_params
The log of current session can be found at:
  /opt/app/oracle/crsdata/rhel71/crsconfig/rootcrs_rhel71_2019-06-13_08-26-09AM.log
2019/06/13 08:26:35 CLSRSC-594: Executing installation step 1 of 19: 'SetupTFA'.
2019/06/13 08:26:35 CLSRSC-594: Executing installation step 2 of 19: 'ValidateEnv'.
2019/06/13 08:26:35 CLSRSC-363: User ignored prerequisites during installation
2019/06/13 08:26:36 CLSRSC-594: Executing installation step 3 of 19: 'CheckFirstNode'.
2019/06/13 08:26:40 CLSRSC-594: Executing installation step 4 of 19: 'GenSiteGUIDs'.
2019/06/13 08:26:42 CLSRSC-594: Executing installation step 5 of 19: 'SetupOSD'.
2019/06/13 08:26:42 CLSRSC-594: Executing installation step 6 of 19: 'CheckCRSConfig'.
2019/06/13 08:26:44 CLSRSC-594: Executing installation step 7 of 19: 'SetupLocalGPNP'.
2019/06/13 08:27:39 CLSRSC-4002: Successfully installed Oracle Trace File Analyzer (TFA) Collector.
2019/06/13 08:27:42 CLSRSC-594: Executing installation step 8 of 19: 'CreateRootCert'.
2019/06/13 08:27:52 CLSRSC-594: Executing installation step 9 of 19: 'ConfigOLR'.
2019/06/13 08:28:19 CLSRSC-594: Executing installation step 10 of 19: 'ConfigCHMOS'.
2019/06/13 08:28:19 CLSRSC-594: Executing installation step 11 of 19: 'CreateOHASD'.
2019/06/13 08:28:31 CLSRSC-594: Executing installation step 12 of 19: 'ConfigOHASD'.
2019/06/13 08:28:32 CLSRSC-330: Adding Clusterware entries to file 'oracle-ohasd.service'
2019/06/13 08:29:48 CLSRSC-594: Executing installation step 13 of 19: 'InstallAFD'.
2019/06/13 08:30:03 CLSRSC-594: Executing installation step 14 of 19: 'InstallACFS'.
2019/06/13 08:31:31 CLSRSC-594: Executing installation step 15 of 19: 'InstallKA'.
2019/06/13 08:31:43 CLSRSC-594: Executing installation step 16 of 19: 'InitConfig'.

ASM has been created and started successfully.

[DBT-30001] Disk groups created successfully. Check /opt/app/oracle/cfgtoollogs/asmca/asmca-190613AM083230.log for details.

2019/06/13 08:33:51 CLSRSC-482: Running command: '/opt/app/19.x.0/grid/bin/ocrconfig -upgrade grid oinstall'
CRS-4256: Updating the profile
Successful addition of voting disk a379bd1a1f614fcfbf2bc03dc4edd382.
Successful addition of voting disk e6caeaf494054f35bf6144b6ae79fd31.
Successful addition of voting disk cf400723672e4f0ebf65845059245bf1.
Successfully replaced voting disk group with +OCRDG.
CRS-4256: Updating the profile
CRS-4266: Voting file(s) successfully replaced
##  STATE    File Universal Id                File Name Disk group
--  -----    -----------------                --------- ---------
 1. ONLINE   a379bd1a1f614fcfbf2bc03dc4edd382 (/dev/oracleasm/ocr3) [OCRDG]
 2. ONLINE   e6caeaf494054f35bf6144b6ae79fd31 (/dev/oracleasm/ocr1) [OCRDG]
 3. ONLINE   cf400723672e4f0ebf65845059245bf1 (/dev/oracleasm/ocr2) [OCRDG]
Located 3 voting disk(s).
2019/06/13 08:36:34 CLSRSC-594: Executing installation step 17 of 19: 'StartCluster'.
2019/06/13 08:38:05 CLSRSC-343: Successfully started Oracle Clusterware stack
2019/06/13 08:38:05 CLSRSC-594: Executing installation step 18 of 19: 'ConfigNode'.
2019/06/13 08:41:34 CLSRSC-594: Executing installation step 19 of 19: 'PostConfig'.

[INFO] [DBT-30001] Disk groups created successfully. Check /opt/app/oracle/cfgtoollogs/asmca/asmca-190613AM084151.log for details.


2019/06/13 08:44:51 CLSRSC-325: Configure Oracle Grid Infrastructure for a Cluster ... succeeded

Root script outptu from last node.
cat /opt/app/19.x.0/grid/install/root_rhel72.domain.net_2019-06-13_08-45-22-859282472.log
Performing root user operation.

The following environment variables are set as:
    ORACLE_OWNER= grid
    ORACLE_HOME=  /opt/app/19.x.0/grid
   Copying dbhome to /usr/local/bin ...
   Copying oraenv to /usr/local/bin ...
   Copying coraenv to /usr/local/bin ...


Creating /etc/oratab file...
Entries will be added to the /etc/oratab file as needed by
Database Configuration Assistant when a database is created
Finished running generic part of root script.
Now product-specific root actions will be performed.
Relinking oracle with rac_on option
Using configuration parameter file: /opt/app/19.x.0/grid/crs/install/crsconfig_params
The log of current session can be found at:
  /opt/app/oracle/crsdata/rhel72/crsconfig/rootcrs_rhel72_2019-06-13_08-45-55AM.log
2019/06/13 08:46:15 CLSRSC-594: Executing installation step 1 of 19: 'SetupTFA'.
2019/06/13 08:46:15 CLSRSC-594: Executing installation step 2 of 19: 'ValidateEnv'.
2019/06/13 08:46:15 CLSRSC-363: User ignored prerequisites during installation
2019/06/13 08:46:16 CLSRSC-594: Executing installation step 3 of 19: 'CheckFirstNode'.
2019/06/13 08:46:19 CLSRSC-594: Executing installation step 4 of 19: 'GenSiteGUIDs'.
2019/06/13 08:46:19 CLSRSC-594: Executing installation step 5 of 19: 'SetupOSD'.
2019/06/13 08:46:19 CLSRSC-594: Executing installation step 6 of 19: 'CheckCRSConfig'.
2019/06/13 08:46:22 CLSRSC-594: Executing installation step 7 of 19: 'SetupLocalGPNP'.
2019/06/13 08:46:24 CLSRSC-594: Executing installation step 8 of 19: 'CreateRootCert'.
2019/06/13 08:46:24 CLSRSC-594: Executing installation step 9 of 19: 'ConfigOLR'.
2019/06/13 08:46:38 CLSRSC-594: Executing installation step 10 of 19: 'ConfigCHMOS'.
2019/06/13 08:46:39 CLSRSC-594: Executing installation step 11 of 19: 'CreateOHASD'.
2019/06/13 08:46:42 CLSRSC-594: Executing installation step 12 of 19: 'ConfigOHASD'.
2019/06/13 08:46:42 CLSRSC-330: Adding Clusterware entries to file 'oracle-ohasd.service'
2019/06/13 08:47:15 CLSRSC-4002: Successfully installed Oracle Trace File Analyzer (TFA) Collector.
2019/06/13 08:47:48 CLSRSC-594: Executing installation step 13 of 19: 'InstallAFD'.
2019/06/13 08:47:52 CLSRSC-594: Executing installation step 14 of 19: 'InstallACFS'.
2019/06/13 08:49:10 CLSRSC-594: Executing installation step 15 of 19: 'InstallKA'.
2019/06/13 08:49:13 CLSRSC-594: Executing installation step 16 of 19: 'InitConfig'.
2019/06/13 08:49:31 CLSRSC-594: Executing installation step 17 of 19: 'StartCluster'.
2019/06/13 08:50:36 CLSRSC-343: Successfully started Oracle Clusterware stack
2019/06/13 08:50:36 CLSRSC-594: Executing installation step 18 of 19: 'ConfigNode'.
2019/06/13 08:51:11 CLSRSC-594: Executing installation step 19 of 19: 'PostConfig'.
2019/06/13 08:51:31 CLSRSC-325: Configure Oracle Grid Infrastructure for a Cluster ... succeeded

Finally run the configuration tools script.
/opt/app/19.x.0/grid/gridSetup.sh -executeConfigTools -responseFile /media/grid19c.rsp -silent
Launching Oracle Grid Infrastructure Setup Wizard...

You can find the logs of this session at:
/opt/app/oraInventory/logs/GridSetupActions2019-06-13_08-52-50AM

You can find the log of this install session at:
 /opt/app/oraInventory/logs/UpdateNodeList2019-06-13_08-52-50AM.log
Successfully Configured Software.
Check the clusterware post installation setup with cluvfy.
cluvfy stage -post crsinst -allnodes

Verifying Node Connectivity ...
  Verifying Hosts File ...PASSED
  Verifying Check that maximum (MTU) size packet goes through subnet ...PASSED
  Verifying subnet mask consistency for subnet "192.168.1.0" ...PASSED
  Verifying subnet mask consistency for subnet "192.168.0.0" ...PASSED
Verifying Node Connectivity ...PASSED
Verifying Multicast or broadcast check ...PASSED
Verifying ASM filter driver configuration consistency ...PASSED
Verifying Time zone consistency ...PASSED
Verifying Cluster Manager Integrity ...PASSED
Verifying User Mask ...PASSED
Verifying Cluster Integrity ...PASSED
Verifying OCR Integrity ...PASSED
Verifying CRS Integrity ...
  Verifying Clusterware Version Consistency ...PASSED
Verifying CRS Integrity ...PASSED
Verifying Node Application Existence ...PASSED
Verifying Single Client Access Name (SCAN) ...
  Verifying DNS/NIS name service 'rac-scan.domain.net' ...
    Verifying Name Service Switch Configuration File Integrity ...PASSED
  Verifying DNS/NIS name service 'rac-scan.domain.net' ...PASSED
Verifying Single Client Access Name (SCAN) ...PASSED
Verifying OLR Integrity ...PASSED
Verifying Voting Disk ...PASSED
Verifying ASM Integrity ...PASSED
Verifying ASM disk group free space ...PASSED
Verifying User Not In Group "root": grid ...PASSED
Verifying Clock Synchronization ...PASSED
Verifying VIP Subnet configuration check ...PASSED
Verifying Network configuration consistency checks ...PASSED
Verifying File system mount options for path GI_HOME ...PASSED
Verifying Access control attributes for /var/tmp/.oracle ...PASSED
Verifying Access control attributes for %OCRCONFIGDIR%/maps ...PASSED

Post-check for cluster services setup was successful.

CVU operation performed:      stage -post crsinst
Date:                         Jun 13, 2019 9:49:15 AM
CVU home:                     /opt/app/19.x.0/grid/
User:                         grid

In the 19c RAC setup, unlike in the 18c RAC installation the asm proxy instance, acfs volumes are by default in offline state.
Resource Name                                 Type                      Target             State              Host
-------------                                 ------                    -------            --------           ----------
ora.ASMNET1LSNR_ASM.lsnr(ora.asmgroup)        ora.asm_listener.type     ONLINE             ONLINE             rhel71
ora.ASMNET1LSNR_ASM.lsnr(ora.asmgroup)        ora.asm_listener.type     ONLINE             ONLINE             rhel72
ora.ASMNET1LSNR_ASM.lsnr(ora.asmgroup)        ora.asm_listener.type     OFFLINE            OFFLINE
ora.DATA.dg(ora.asmgroup)                     ora.diskgroup.type        ONLINE             ONLINE             rhel71
ora.DATA.dg(ora.asmgroup)                     ora.diskgroup.type        ONLINE             ONLINE             rhel72
ora.DATA.dg(ora.asmgroup)                     ora.diskgroup.type        ONLINE             OFFLINE
ora.FRA.dg(ora.asmgroup)                      ora.diskgroup.type        ONLINE             ONLINE             rhel71
ora.FRA.dg(ora.asmgroup)                      ora.diskgroup.type        ONLINE             ONLINE             rhel72
ora.FRA.dg(ora.asmgroup)                      ora.diskgroup.type        ONLINE             OFFLINE
ora.GIMRDG.GHCHKPT.advm                       ora.volume.type           OFFLINE            OFFLINE
ora.GIMRDG.GHCHKPT.advm                       ora.volume.type           OFFLINE            OFFLINE
ora.GIMRDG.dg(ora.asmgroup)                   ora.diskgroup.type        ONLINE             ONLINE             rhel71
ora.GIMRDG.dg(ora.asmgroup)                   ora.diskgroup.type        ONLINE             ONLINE             rhel72
ora.GIMRDG.dg(ora.asmgroup)                   ora.diskgroup.type        OFFLINE            OFFLINE
ora.LISTENER.lsnr                             ora.listener.type         ONLINE             ONLINE             rhel71
ora.LISTENER.lsnr                             ora.listener.type         ONLINE             ONLINE             rhel72
ora.LISTENER_SCAN1.lsnr                       ora.scan_listener.type    ONLINE             ONLINE             rhel72
ora.LISTENER_SCAN2.lsnr                       ora.scan_listener.type    ONLINE             ONLINE             rhel71
ora.LISTENER_SCAN3.lsnr                       ora.scan_listener.type    ONLINE             ONLINE             rhel71
ora.MGMTLSNR                                  ora.mgmtlsnr.type         ONLINE             ONLINE             rhel71
ora.OCRDG.dg(ora.asmgroup)                    ora.diskgroup.type        ONLINE             ONLINE             rhel71
ora.OCRDG.dg(ora.asmgroup)                    ora.diskgroup.type        ONLINE             ONLINE             rhel72
ora.OCRDG.dg(ora.asmgroup)                    ora.diskgroup.type        OFFLINE            OFFLINE
ora.asm(ora.asmgroup)                         ora.asm.type              ONLINE             ONLINE             rhel71
ora.asm(ora.asmgroup)                         ora.asm.type              ONLINE             ONLINE             rhel72
ora.asm(ora.asmgroup)                         ora.asm.type              OFFLINE            OFFLINE
ora.asmnet1.asmnetwork(ora.asmgroup)          ora.asm_network.type      ONLINE             ONLINE             rhel71
ora.asmnet1.asmnetwork(ora.asmgroup)          ora.asm_network.type      ONLINE             ONLINE             rhel72
ora.asmnet1.asmnetwork(ora.asmgroup)          ora.asm_network.type      OFFLINE            OFFLINE
ora.chad                                      ora.chad.type             ONLINE             ONLINE             rhel71
ora.chad                                      ora.chad.type             ONLINE             ONLINE             rhel72
ora.cvu                                       ora.cvu.type              ONLINE             ONLINE             rhel71
ora.ent19c.db                                 ora.database.type         ONLINE             ONLINE             rhel71
ora.ent19c.db                                 ora.database.type         ONLINE             ONLINE             rhel72
ora.gimrdg.ghchkpt.acfs                       ora.acfs.type             OFFLINE            OFFLINE
ora.gimrdg.ghchkpt.acfs                       ora.acfs.type             OFFLINE            OFFLINE
ora.helper                                    ora.helper.type           OFFLINE            OFFLINE
ora.helper                                    ora.helper.type           OFFLINE            OFFLINE
ora.mgmtdb                                    ora.mgmtdb.type           ONLINE             ONLINE             rhel71
ora.net1.network                              ora.network.type          ONLINE             ONLINE             rhel71
ora.net1.network                              ora.network.type          ONLINE             ONLINE             rhel72
ora.ons                                       ora.ons.type              ONLINE             ONLINE             rhel71
ora.ons                                       ora.ons.type              ONLINE             ONLINE             rhel72
ora.proxy_advm                                ora.proxy_advm.type       OFFLINE            OFFLINE
ora.proxy_advm                                ora.proxy_advm.type       OFFLINE            OFFLINE
ora.qosmserver                                ora.qosmserver.type       ONLINE             ONLINE             rhel71
ora.rhel71.vip                                ora.cluster_vip_net1.type ONLINE             ONLINE             rhel71
ora.rhel72.vip                                ora.cluster_vip_net1.type ONLINE             ONLINE             rhel72
ora.rhpserver                                 ora.rhpserver.type        OFFLINE            OFFLINE
ora.scan1.vip                                 ora.scan_vip.type         ONLINE             ONLINE             rhel72
ora.scan2.vip                                 ora.scan_vip.type         ONLINE             ONLINE             rhel71
ora.scan3.vip                                 ora.scan_vip.type         ONLINE             ONLINE             rhel71

As the last step in GI installation, create other disk groups that would be used for the database. In this case two disk groups called +data and +fra are created.
asmca -silent -createDiskGroup -diskGroupName data -disk /dev/oracleasm/data1 -redundancy EXTERNAL -au_size 4 -compatible.asm 19.0.0.0.0 -compatible.rdbms 19.0.0.0.0
asmca -silent -createDiskGroup -diskGroupName fra -disk /dev/oracleasm/fra1 -redundancy EXTERNAL -au_size 4 -compatible.asm 19.0.0.0.0 -compatible.rdbms 19.0.0.0.0


The next phase is to install the database software. Check the datadbase software pre-reqs with cluvfy.
cluvfy stage -pre dbinst -allnodes -r 19

Verifying Physical Memory ...PASSED
Verifying Available Physical Memory ...PASSED
Verifying Swap Size ...PASSED
Verifying Free Space: rhel72:/tmp ...PASSED
Verifying Free Space: rhel71:/tmp ...PASSED
Verifying User Existence: oracle ...
  Verifying Users With Same UID: 1001 ...PASSED
Verifying User Existence: oracle ...PASSED
Verifying Group Existence: dba ...PASSED
Verifying Group Existence: oinstall ...PASSED
Verifying Group Membership: oinstall(Primary) ...PASSED
Verifying Group Membership: dba ...PASSED
Verifying Run Level ...PASSED
Verifying Hard Limit: maximum open file descriptors ...PASSED
Verifying Soft Limit: maximum open file descriptors ...PASSED
Verifying Hard Limit: maximum user processes ...PASSED
Verifying Soft Limit: maximum user processes ...PASSED
Verifying Soft Limit: maximum stack size ...PASSED
Verifying Architecture ...PASSED
Verifying OS Kernel Version ...PASSED
Verifying OS Kernel Parameter: semmsl ...PASSED
Verifying OS Kernel Parameter: semmns ...PASSED
Verifying OS Kernel Parameter: semopm ...PASSED
Verifying OS Kernel Parameter: semmni ...PASSED
Verifying OS Kernel Parameter: shmmax ...PASSED
Verifying OS Kernel Parameter: shmmni ...PASSED
Verifying OS Kernel Parameter: shmall ...PASSED
Verifying OS Kernel Parameter: file-max ...PASSED
Verifying OS Kernel Parameter: ip_local_port_range ...PASSED
Verifying OS Kernel Parameter: rmem_default ...PASSED
Verifying OS Kernel Parameter: rmem_max ...PASSED
Verifying OS Kernel Parameter: wmem_default ...PASSED
Verifying OS Kernel Parameter: wmem_max ...PASSED
Verifying OS Kernel Parameter: aio-max-nr ...PASSED
Verifying Package: kmod-20-21 (x86_64) ...PASSED
Verifying Package: kmod-libs-20-21 (x86_64) ...PASSED
Verifying Package: binutils-2.23.52.0.1 ...PASSED
Verifying Package: compat-libcap1-1.10 ...PASSED
Verifying Package: libgcc-4.8.2 (x86_64) ...PASSED
Verifying Package: libstdc++-4.8.2 (x86_64) ...PASSED
Verifying Package: libstdc++-devel-4.8.2 (x86_64) ...PASSED
Verifying Package: sysstat-10.1.5 ...PASSED
Verifying Package: gcc-c++-4.8.2 ...PASSED
Verifying Package: ksh ...PASSED
Verifying Package: make-3.82 ...PASSED
Verifying Package: glibc-2.17 (x86_64) ...PASSED
Verifying Package: glibc-devel-2.17 (x86_64) ...PASSED
Verifying Package: libaio-0.3.109 (x86_64) ...PASSED
Verifying Package: libaio-devel-0.3.109 (x86_64) ...PASSED
Verifying Package: smartmontools-6.2-4 ...PASSED
Verifying Package: net-tools-2.0-0.17 ...PASSED
Verifying Package: compat-libstdc++-33-3.2.3 (x86_64) ...PASSED
Verifying Package: libxcb-1.11 (x86_64) ...PASSED
Verifying Package: libX11-1.6.3 (x86_64) ...PASSED
Verifying Package: libXau-1.0.8 (x86_64) ...PASSED
Verifying Package: libXi-1.7.4 (x86_64) ...PASSED
Verifying Package: libXtst-1.2.2 (x86_64) ...PASSED
Verifying Users With Same UID: 0 ...PASSED
Verifying Current Group ID ...PASSED
Verifying Root user consistency ...PASSED
Verifying Host name ...PASSED
Verifying Node Connectivity ...
  Verifying Hosts File ...PASSED
  Verifying Check that maximum (MTU) size packet goes through subnet ...PASSED
  Verifying subnet mask consistency for subnet "192.168.1.0" ...PASSED
  Verifying subnet mask consistency for subnet "192.168.0.0" ...PASSED
Verifying Node Connectivity ...PASSED
Verifying Multicast or broadcast check ...PASSED
Verifying User Mask ...PASSED
Verifying CRS Integrity ...
  Verifying Clusterware Version Consistency ...PASSED
Verifying CRS Integrity ...PASSED
Verifying Cluster Manager Integrity ...PASSED
Verifying Node Application Existence ...PASSED
Verifying Clock Synchronization ...PASSED
Verifying resolv.conf Integrity ...PASSED
Verifying Time zone consistency ...PASSED
Verifying Single Client Access Name (SCAN) ...
  Verifying DNS/NIS name service 'rac-scan' ...
    Verifying Name Service Switch Configuration File Integrity ...PASSED
  Verifying DNS/NIS name service 'rac-scan' ...PASSED
Verifying Single Client Access Name (SCAN) ...PASSED
Verifying Database Clusterware Version Compatibility ...PASSED
Verifying ASM storage privileges for the user: oracle ...
  Verifying Group Membership: asmdba ...PASSED
Verifying ASM storage privileges for the user: oracle ...PASSED
Verifying Daemon "proxyt" not configured and running ...PASSED
Verifying ACFS device special file ...PASSED
Verifying /dev/shm mounted as temporary file system ...PASSED
Verifying Maximum locked memory check ...FAILED (PRVE-0059)

Pre-check for database installation was unsuccessful on all the nodes.


Failures were encountered during execution of CVU verification request "stage -pre dbinst".

Verifying Maximum locked memory check ...FAILED
rhel72: PRVE-0059 : no default entry or entry specific to user "oracle" was
        found in the configuration file "/etc/security/limits.conf" when
        checking the maximum locked memory "HARD" limit on node
        "rhel72.domain.net"

rhel71: PRVE-0059 : no default entry or entry specific to user "oracle" was
        found in the configuration file "/etc/security/limits.conf" when
        checking the maximum locked memory "HARD" limit on node
        "rhel71.domain.net"

The memlock related failure is ignorable at this stage. This could be set either before database creation or after database is created (in later case it would requrie a restart for SGA to use the large pages).
Similar to GI home, Oracle home also provides several resposne files that could be edited and used for installing the RAC DB software. Copy one of the following to temporary location and modify to reflect the cluster being setup.
$ORACLE_HOME/install/response/db_install.rsp
or
$ORACLE_HOME/inventory/response/db_install.rsp
The following text shows the response file used in this setup. Resposne file content is edited such that it will do a software only installation.
####################################################################
## Copyright(c) Oracle Corporation 1998,2019. All rights reserved.##
##                                                                ##
## Specify values for the variables listed below to customize     ##
## your installation.                                             ##
##                                                                ##
## Each variable is associated with a comment. The comment        ##
## can help to populate the variables with the appropriate        ##
## values.                                                        ##
##                                                                ##
## IMPORTANT NOTE: This file contains plain text passwords and    ##
## should be secured to have read permission only by oracle user  ##
## or db administrator who owns this installation.                ##
##                                                                ##
####################################################################


#------------------------------------------------------------------------------
# Do not change the following system generated value. 
#------------------------------------------------------------------------------
oracle.install.responseFileVersion=/oracle/install/rspfmt_dbinstall_response_schema_v19.0.0

#-------------------------------------------------------------------------------
# Specify the installation option.
# It can be one of the following:
#   - INSTALL_DB_SWONLY
#   - INSTALL_DB_AND_CONFIG
#-------------------------------------------------------------------------------
oracle.install.option=INSTALL_DB_SWONLY

#-------------------------------------------------------------------------------
# Specify the Unix group to be set for the inventory directory.  
#-------------------------------------------------------------------------------
UNIX_GROUP_NAME=oinstall

#-------------------------------------------------------------------------------
# Specify the location which holds the inventory files.
# This is an optional parameter if installing on
# Windows based Operating System.
#-------------------------------------------------------------------------------
INVENTORY_LOCATION=/opt/app/oraInventory

#-------------------------------------------------------------------------------
# Specify the complete path of the Oracle Base. 
#-------------------------------------------------------------------------------
ORACLE_BASE=/opt/app/oracle

#-------------------------------------------------------------------------------
# Specify the installation edition of the component.                     
#                                                             
# The value should contain only one of these choices.  
      
#   - EE     : Enterprise Edition 
                               
#   - SE2     : Standard Edition 2


#-------------------------------------------------------------------------------

oracle.install.db.InstallEdition=EE
###############################################################################
#                                                                             #
# PRIVILEGED OPERATING SYSTEM GROUPS                                          #
# ------------------------------------------                                  #
# Provide values for the OS groups to which SYSDBA and SYSOPER privileges     #
# needs to be granted. If the install is being performed as a member of the   #
# group "dba", then that will be used unless specified otherwise below.       #
#                                                                             #
# The value to be specified for OSDBA and OSOPER group is only for UNIX based #
# Operating System.                                                           #
#                                                                             #
###############################################################################

#------------------------------------------------------------------------------
# The OSDBA_GROUP is the OS group which is to be granted SYSDBA privileges.
#-------------------------------------------------------------------------------
oracle.install.db.OSDBA_GROUP=dba

#------------------------------------------------------------------------------
# The OSOPER_GROUP is the OS group which is to be granted SYSOPER privileges.
# The value to be specified for OSOPER group is optional.
#------------------------------------------------------------------------------
oracle.install.db.OSOPER_GROUP=oper

#------------------------------------------------------------------------------
# The OSBACKUPDBA_GROUP is the OS group which is to be granted SYSBACKUP privileges.
#------------------------------------------------------------------------------
oracle.install.db.OSBACKUPDBA_GROUP=backupdba

#------------------------------------------------------------------------------
# The OSDGDBA_GROUP is the OS group which is to be granted SYSDG privileges.
#------------------------------------------------------------------------------
oracle.install.db.OSDGDBA_GROUP=dgdba

#------------------------------------------------------------------------------
# The OSKMDBA_GROUP is the OS group which is to be granted SYSKM privileges.
#------------------------------------------------------------------------------
oracle.install.db.OSKMDBA_GROUP=kmdba

#------------------------------------------------------------------------------
# The OSRACDBA_GROUP is the OS group which is to be granted SYSRAC privileges.
#------------------------------------------------------------------------------
oracle.install.db.OSRACDBA_GROUP=racdba
################################################################################
#                                                                              #
#                      Root script execution configuration                     #
#                                                                              #
################################################################################

#-------------------------------------------------------------------------------------------------------
# Specify the root script execution mode.
#
#   - true  : To execute the root script automatically by using the appropriate configuration methods.
#   - false : To execute the root script manually.
#
# If this option is selected, password should be specified on the console.
#-------------------------------------------------------------------------------------------------------
oracle.install.db.rootconfig.executeRootScript=false

#--------------------------------------------------------------------------------------
# Specify the configuration method to be used for automatic root script execution.
#
# Following are the possible choices:
#   - ROOT
#   - SUDO
#--------------------------------------------------------------------------------------
oracle.install.db.rootconfig.configMethod=
#--------------------------------------------------------------------------------------
# Specify the absolute path of the sudo program.
#
# Applicable only when SUDO configuration method was chosen.
#--------------------------------------------------------------------------------------
oracle.install.db.rootconfig.sudoPath=

#--------------------------------------------------------------------------------------
# Specify the name of the user who is in the sudoers list. 
# Applicable only when SUDO configuration method was chosen.
# Note:For Single Instance database installations,the sudo user name must be the username of the user installing the database.
#--------------------------------------------------------------------------------------
oracle.install.db.rootconfig.sudoUserName=

###############################################################################
#                                                                             #
#                               Grid Options                                  #
#                                                                             #
###############################################################################

#------------------------------------------------------------------------------
# Value is required only if the specified install option is INSTALL_DB_SWONLY
# 
# Specify the cluster node names selected during the installation.
# 
# Example : oracle.install.db.CLUSTER_NODES=node1,node2
#------------------------------------------------------------------------------
oracle.install.db.CLUSTER_NODES=rhel71,rhel72

###############################################################################
#                                                                             #
#                        Database Configuration Options                       #
#                                                                             #
###############################################################################

#-------------------------------------------------------------------------------
# Specify the type of database to create.
# It can be one of the following:
#   - GENERAL_PURPOSE                       
#   - DATA_WAREHOUSE 
# GENERAL_PURPOSE: A starter database designed for general purpose use or transaction-heavy applications.
# DATA_WAREHOUSE : A starter database optimized for data warehousing applications.
#-------------------------------------------------------------------------------
oracle.install.db.config.starterdb.type=GENERAL_PURPOSE

#-------------------------------------------------------------------------------
# Specify the Starter Database Global Database Name. 
#-------------------------------------------------------------------------------
oracle.install.db.config.starterdb.globalDBName=

#-------------------------------------------------------------------------------
# Specify the Starter Database SID.
#-------------------------------------------------------------------------------
oracle.install.db.config.starterdb.SID=

#-------------------------------------------------------------------------------
# Specify whether the database should be configured as a Container database.
# The value can be either "true" or "false". If left blank it will be assumed
# to be "false".
#-------------------------------------------------------------------------------
oracle.install.db.ConfigureAsContainerDB=false

#-------------------------------------------------------------------------------
# Specify the  Pluggable Database name for the pluggable database in Container Database.
#-------------------------------------------------------------------------------
oracle.install.db.config.PDBName=

#-------------------------------------------------------------------------------
# Specify the Starter Database character set.
#                                               
#  One of the following
#  AL32UTF8, WE8ISO8859P15, WE8MSWIN1252, EE8ISO8859P2,
#  EE8MSWIN1250, NE8ISO8859P10, NEE8ISO8859P4, BLT8MSWIN1257,
#  BLT8ISO8859P13, CL8ISO8859P5, CL8MSWIN1251, AR8ISO8859P6,
#  AR8MSWIN1256, EL8ISO8859P7, EL8MSWIN1253, IW8ISO8859P8,
#  IW8MSWIN1255, JA16EUC, JA16EUCTILDE, JA16SJIS, JA16SJISTILDE,
#  KO16MSWIN949, ZHS16GBK, TH8TISASCII, ZHT32EUC, ZHT16MSWIN950,
#  ZHT16HKSCS, WE8ISO8859P9, TR8MSWIN1254, VN8MSWIN1258
#-------------------------------------------------------------------------------
oracle.install.db.config.starterdb.characterSet=

#------------------------------------------------------------------------------
# This variable should be set to true if Automatic Memory Management 
# in Database is desired.
# If Automatic Memory Management is not desired, and memory allocation
# is to be done manually, then set it to false.
#------------------------------------------------------------------------------
oracle.install.db.config.starterdb.memoryOption=false

#-------------------------------------------------------------------------------
# Specify the total memory allocation for the database. Value(in MB) should be
# at least 256 MB, and should not exceed the total physical memory available 
# on the system.
# Example: oracle.install.db.config.starterdb.memoryLimit=512
#-------------------------------------------------------------------------------
oracle.install.db.config.starterdb.memoryLimit=

#-------------------------------------------------------------------------------
# This variable controls whether to load Example Schemas onto
# the starter database or not.
# The value can be either "true" or "false". If left blank it will be assumed
# to be "false".
#-------------------------------------------------------------------------------
oracle.install.db.config.starterdb.installExampleSchemas=false

###############################################################################
#                                                                             #
# Passwords can be supplied for the following four schemas in the       #
# starter database:                  #
#   SYS                                                                       #
#   SYSTEM                                                                    #
#   DBSNMP (used by Enterprise Manager)                                       #
#                                                                             #
# Same password can be used for all accounts (not recommended)         #
# or different passwords for each account can be provided (recommended)       #
#                                                                             #
###############################################################################

#------------------------------------------------------------------------------
# This variable holds the password that is to be used for all schemas in the
# starter database.
#-------------------------------------------------------------------------------
oracle.install.db.config.starterdb.password.ALL=

#-------------------------------------------------------------------------------
# Specify the SYS password for the starter database.
#-------------------------------------------------------------------------------
oracle.install.db.config.starterdb.password.SYS=

#-------------------------------------------------------------------------------
# Specify the SYSTEM password for the starter database.
#-------------------------------------------------------------------------------
oracle.install.db.config.starterdb.password.SYSTEM=

#-------------------------------------------------------------------------------
# Specify the DBSNMP password for the starter database.
# Applicable only when oracle.install.db.config.starterdb.managementOption=CLOUD_CONTROL
#-------------------------------------------------------------------------------
oracle.install.db.config.starterdb.password.DBSNMP=

#-------------------------------------------------------------------------------
# Specify the PDBADMIN password required for creation of Pluggable Database in the Container Database.
#-------------------------------------------------------------------------------
oracle.install.db.config.starterdb.password.PDBADMIN=

#-------------------------------------------------------------------------------
# Specify the management option to use for managing the database.
# Options are:
# 1. CLOUD_CONTROL - If you want to manage your database with Enterprise Manager Cloud Control along with Database Express.
# 2. DEFAULT   -If you want to manage your database using the default Database Express option.
#-------------------------------------------------------------------------------
oracle.install.db.config.starterdb.managementOption=DEFAULT

#-------------------------------------------------------------------------------
# Specify the OMS host to connect to Cloud Control.
# Applicable only when oracle.install.db.config.starterdb.managementOption=CLOUD_CONTROL
#-------------------------------------------------------------------------------
oracle.install.db.config.starterdb.omsHost=

#-------------------------------------------------------------------------------
# Specify the OMS port to connect to Cloud Control.
# Applicable only when oracle.install.db.config.starterdb.managementOption=CLOUD_CONTROL
#-------------------------------------------------------------------------------
oracle.install.db.config.starterdb.omsPort=0

#-------------------------------------------------------------------------------
# Specify the EM Admin user name to use to connect to Cloud Control.
# Applicable only when oracle.install.db.config.starterdb.managementOption=CLOUD_CONTROL
#-------------------------------------------------------------------------------
oracle.install.db.config.starterdb.emAdminUser=

#-------------------------------------------------------------------------------
# Specify the EM Admin password to use to connect to Cloud Control.
# Applicable only when oracle.install.db.config.starterdb.managementOption=CLOUD_CONTROL
#-------------------------------------------------------------------------------
oracle.install.db.config.starterdb.emAdminPassword=

###############################################################################
#                                                                             #
# SPECIFY RECOVERY OPTIONS                                                #
# ------------------------------------                                #
# Recovery options for the database can be mentioned using the entries below  #
#                                                                             #
###############################################################################

#------------------------------------------------------------------------------
# This variable is to be set to false if database recovery is not required. Else 
# this can be set to true.
#-------------------------------------------------------------------------------
oracle.install.db.config.starterdb.enableRecovery=false

#-------------------------------------------------------------------------------
# Specify the type of storage to use for the database.
# It can be one of the following:
#   - FILE_SYSTEM_STORAGE
#   - ASM_STORAGE
#-------------------------------------------------------------------------------
oracle.install.db.config.starterdb.storageType=

#-------------------------------------------------------------------------------
# Specify the database file location which is a directory for datafiles, control
# files, redo logs.         
#
# Applicable only when oracle.install.db.config.starterdb.storage=FILE_SYSTEM_STORAGE 
#-------------------------------------------------------------------------------
oracle.install.db.config.starterdb.fileSystemStorage.dataLocation=

#-------------------------------------------------------------------------------
# Specify the recovery location.
#
# Applicable only when oracle.install.db.config.starterdb.storage=FILE_SYSTEM_STORAGE 
#-------------------------------------------------------------------------------
oracle.install.db.config.starterdb.fileSystemStorage.recoveryLocation=

#-------------------------------------------------------------------------------
# Specify the existing ASM disk groups to be used for storage.
#
# Applicable only when oracle.install.db.config.starterdb.storageType=ASM_STORAGE
#-------------------------------------------------------------------------------
oracle.install.db.config.asm.diskGroup=

#-------------------------------------------------------------------------------
# Specify the password for ASMSNMP user of the ASM instance.                 
#
# Applicable only when oracle.install.db.config.starterdb.storage=ASM_STORAGE 
#-------------------------------------------------------------------------------
oracle.install.db.config.asm.ASMSNMPPassword=

Execute the installer specifying the response file. The ignorePrereqFailure is used here due to pre-req failures being ignored as this is a test system.
$ORACLE_HOME/runInstaller -silent -responseFile /media/db_install.rsp -ignorePrereqFailure
Launching Oracle Database Setup Wizard...

[WARNING] [INS-13013] Target environment does not meet some mandatory requirements.
   CAUSE: Some of the mandatory prerequisites are not met. See logs for details. /opt/app/oraInventory/logs/InstallActions2019-06-13_10-22-10AM/installActions2019-06-13_10-22-10AM.log
   ACTION: Identify the list of failed prerequisite checks from the log: /opt/app/oraInventory/logs/InstallActions2019-06-13_10-22-10AM/installActions2019-06-13_10-22-10AM.log. Then either from the log file or from installation manual find the appropriate configuration to meet the prerequisites and fix it manually.
The response file for this session can be found at:
 /opt/app/oracle/product/19.x.0/dbhome_1/install/response/db_2019-06-13_10-22-10AM.rsp

You can find the log of this install session at:
 /opt/app/oraInventory/logs/InstallActions2019-06-13_10-22-10AM/installActions2019-06-13_10-22-10AM.log

As a root user, execute the following script(s):
        1. /opt/app/oracle/product/19.x.0/dbhome_1/root.sh

Execute /opt/app/oracle/product/19.x.0/dbhome_1/root.sh on the following nodes:
[rhel71, rhel72]


Successfully Setup Software with warning(s).

The last phase is the creation of the RAC database. In a role separated setup change the permission following folders.
cd $ORACLE_BASE
chmod 770 audit admin

cd $ORACLE_BASE/cfgtoollogs
chmod 770 dbca sqlpatch
Check database configuraiton pre-reqs with cluvfy.
cluvfy stage -pre dbcfg -allnodes -d $ORACLE_HOME

Verifying Physical Memory ...PASSED
Verifying Available Physical Memory ...PASSED
Verifying Swap Size ...PASSED
Verifying Free Space: rhel72:/tmp ...PASSED
Verifying Free Space: rhel71:/tmp ...PASSED
Verifying User Existence: oracle ...
  Verifying Users With Same UID: 1001 ...PASSED
Verifying User Existence: oracle ...PASSED
Verifying Group Existence: dgdba ...PASSED
Verifying Group Existence: dba ...PASSED
Verifying Group Existence: racdba ...PASSED
Verifying Group Existence: backupdba ...PASSED
Verifying Group Existence: oper ...PASSED
Verifying Group Existence: oinstall ...PASSED
Verifying Group Membership: backupdba ...PASSED
Verifying Group Membership: racdba ...PASSED
Verifying Group Membership: dgdba ...PASSED
Verifying Group Membership: oinstall(Primary) ...PASSED
Verifying Group Membership: dba ...PASSED
Verifying Group Membership: oper ...PASSED
Verifying Run Level ...PASSED
Verifying Hard Limit: maximum open file descriptors ...PASSED
Verifying Soft Limit: maximum open file descriptors ...PASSED
Verifying Hard Limit: maximum user processes ...PASSED
Verifying Soft Limit: maximum user processes ...PASSED
Verifying Soft Limit: maximum stack size ...PASSED
Verifying Architecture ...PASSED
Verifying OS Kernel Version ...PASSED
Verifying OS Kernel Parameter: semmsl ...PASSED
Verifying OS Kernel Parameter: semmns ...PASSED
Verifying OS Kernel Parameter: semopm ...PASSED
Verifying OS Kernel Parameter: semmni ...PASSED
Verifying OS Kernel Parameter: shmmax ...PASSED
Verifying OS Kernel Parameter: shmmni ...PASSED
Verifying OS Kernel Parameter: shmall ...PASSED
Verifying OS Kernel Parameter: file-max ...PASSED
Verifying OS Kernel Parameter: ip_local_port_range ...PASSED
Verifying OS Kernel Parameter: rmem_default ...PASSED
Verifying OS Kernel Parameter: rmem_max ...PASSED
Verifying OS Kernel Parameter: wmem_default ...PASSED
Verifying OS Kernel Parameter: wmem_max ...PASSED
Verifying OS Kernel Parameter: aio-max-nr ...PASSED
Verifying Package: kmod-20-21 (x86_64) ...PASSED
Verifying Package: kmod-libs-20-21 (x86_64) ...PASSED
Verifying Package: binutils-2.23.52.0.1 ...PASSED
Verifying Package: compat-libcap1-1.10 ...PASSED
Verifying Package: libgcc-4.8.2 (x86_64) ...PASSED
Verifying Package: libstdc++-4.8.2 (x86_64) ...PASSED
Verifying Package: libstdc++-devel-4.8.2 (x86_64) ...PASSED
Verifying Package: sysstat-10.1.5 ...PASSED
Verifying Package: gcc-c++-4.8.2 ...PASSED
Verifying Package: ksh ...PASSED
Verifying Package: make-3.82 ...PASSED
Verifying Package: glibc-2.17 (x86_64) ...PASSED
Verifying Package: glibc-devel-2.17 (x86_64) ...PASSED
Verifying Package: libaio-0.3.109 (x86_64) ...PASSED
Verifying Package: libaio-devel-0.3.109 (x86_64) ...PASSED
Verifying Package: smartmontools-6.2-4 ...PASSED
Verifying Package: net-tools-2.0-0.17 ...PASSED
Verifying Package: compat-libstdc++-33-3.2.3 (x86_64) ...PASSED
Verifying Package: libxcb-1.11 (x86_64) ...PASSED
Verifying Package: libX11-1.6.3 (x86_64) ...PASSED
Verifying Package: libXau-1.0.8 (x86_64) ...PASSED
Verifying Package: libXi-1.7.4 (x86_64) ...PASSED
Verifying Package: libXtst-1.2.2 (x86_64) ...PASSED
Verifying Current Group ID ...PASSED
Verifying CRS Integrity ...
  Verifying Clusterware Version Consistency ...PASSED
Verifying CRS Integrity ...PASSED
Verifying Node Application Existence ...PASSED
Verifying Time zone consistency ...PASSED
Verifying Single Client Access Name (SCAN) ...
  Verifying DNS/NIS name service 'rac-scan' ...
    Verifying Name Service Switch Configuration File Integrity ...PASSED
  Verifying DNS/NIS name service 'rac-scan' ...PASSED
Verifying Single Client Access Name (SCAN) ...PASSED
Verifying ASM Integrity ...
  Verifying Node Connectivity ...
    Verifying Hosts File ...PASSED
    Verifying Check that maximum (MTU) size packet goes through subnet ...PASSED
    Verifying subnet mask consistency for subnet "192.168.1.0" ...PASSED
    Verifying subnet mask consistency for subnet "192.168.0.0" ...PASSED
  Verifying Node Connectivity ...PASSED
Verifying ASM Integrity ...PASSED
Verifying Database Clusterware Version Compatibility ...PASSED
Verifying Maximum locked memory check ...FAILED (PRVE-0059)
Verifying File system mount options for path ORACLE_HOME ...FAILED (PRVE-0007)
Verifying /dev/shm mounted as temporary file system ...PASSED

Pre-check for database configuration was unsuccessful on all the nodes.


Failures were encountered during execution of CVU verification request "stage -pre dbcfg".

Verifying Maximum locked memory check ...FAILED
rhel72: PRVE-0059 : no default entry or entry specific to user "oracle" was
        found in the configuration file "/etc/security/limits.conf" when
        checking the maximum locked memory "HARD" limit on node
        "rhel72.domain.net"

rhel71: PRVE-0059 : no default entry or entry specific to user "oracle" was
        found in the configuration file "/etc/security/limits.conf" when
        checking the maximum locked memory "HARD" limit on node
        "rhel71.domain.net"

Verifying File system mount options for path ORACLE_HOME ...FAILED
rhel72: PRVE-0007 : Could not find executable
        "/tmp/CVU_19.0.0.0.0_oracle/checkFSMountOptions.sh"

rhel71: PRVE-0007 : Could not find executable
        "/tmp/CVU_19.0.0.0.0_oracle/checkFSMountOptions.sh"

The pre-reqs failurs are ignored. The databae creation could be done using dbca in silent mode and with the use of a custom template. One way to create a custom template is by modifying the $ORACLE_HOME/assistants/dbca/templates/New_Database.dbt file. Following shows the content of the custom template created by modifying the aforementioned file.
 cat testdb.dbt
<DatabaseTemplate name="Test Database" description="" version="19.0.0.0.0">
   <CommonAttributes>
      <option name="OMS" value="false" includeInPDBs="false"/>
      <option name="JSERVER" value="false" includeInPDBs="false"/>
      <option name="SPATIAL" value="false" includeInPDBs="false"/>
      <option name="IMEDIA" value="false" includeInPDBs="false"/>
      <option name="ORACLE_TEXT" value="true" includeInPDBs="true">
         <tablespace id="SYSAUX"/>
      </option>
      <option name="CWMLITE" value="false" includeInPDBs="false">
         <tablespace id="SYSAUX"/>
      </option>
      <option name="SAMPLE_SCHEMA" value="false" includeInPDBs="false"/>
      <option name="APEX" value="false" includeInPDBs="false"/>
      <option name="DV" value="false" includeInPDBs="false"/>
   </CommonAttributes>
   <Variables/>
   <CustomScripts Execute="false"/>
   <InitParamAttributes>
      <InitParams>
         <initParam name="db_block_size" value="8" unit="KB"/>
         <initParam name="open_cursors" value="300"/>
         <initParam name="undo_tablespace" value="UNDOTBS1"/>
         <initParam name="control_files" value="("{ORACLE_BASE}/oradata/{DB_UNIQUE_NAME}/control01.ctl", "{ORACLE_BASE}/fast_recovery_area/{DB_UNIQUE_NAME}/control02.ctl")"/>

         <initParam name="compatible" value="19.0.0"/>
         <initParam name="audit_file_dest" value="{ORACLE_BASE}/admin/{DB_UNIQUE_NAME}/adump"/>
         <initParam name="audit_trail" value="OS"/>
         <initParam name="diagnostic_dest" value="{ORACLE_BASE}"/>
         <initParam name="remote_login_passwordfile" value="EXCLUSIVE"/>
         <initParam name="dispatchers" value="(PROTOCOL=TCP) (SERVICE={SID}XDB)"/>
      </InitParams>
      <MiscParams>
         <percentageMemTOSGA>40</percentageMemTOSGA>
         <archiveLogMode>false</archiveLogMode>
         <initParamFileName>{ORACLE_BASE}/admin/{DB_UNIQUE_NAME}/pfile/init.ora</initParamFileName>
      </MiscParams>
      <SPfile useSPFile="true">{ORACLE_HOME}/dbs/spfile{SID}.ora</SPfile>
   </InitParamAttributes>
   <StorageAttributes>
      <ControlfileAttributes id="Controlfile">
         <maxDatafiles>100</maxDatafiles>
         <maxLogfiles>16</maxLogfiles>
         <maxLogMembers>3</maxLogMembers>
         <maxLogHistory>1</maxLogHistory>
         <maxInstances>8</maxInstances>
         <image name="control01.ctl" filepath="{ORACLE_BASE}/oradata/{DB_UNIQUE_NAME}/"/>
         <image name="control02.ctl" filepath="{ORACLE_BASE}/fast_recovery_area/{DB_UNIQUE_NAME}/"/>

      </ControlfileAttributes>
      <DatafileAttributes id="{ORACLE_BASE}/oradata/{DB_UNIQUE_NAME}/sysaux01.dbf">
         <tablespace>SYSAUX</tablespace>
         <temporary>false</temporary>
         <online>true</online>
         <status>0</status>
         <size unit="MB">550</size>
         <reuse>true</reuse>
         <autoExtend>true</autoExtend>
         <increment unit="KB">10240</increment>
         <maxSize unit="MB">-1</maxSize>
      </DatafileAttributes>
      <DatafileAttributes id="{ORACLE_BASE}/oradata/{DB_UNIQUE_NAME}/users01.dbf">
         <tablespace>USERS</tablespace>
         <temporary>false</temporary>
         <online>true</online>
         <status>0</status>
         <size unit="MB">5</size>
         <reuse>true</reuse>
         <autoExtend>true</autoExtend>
         <increment unit="KB">1280</increment>
         <maxSize unit="MB">-1</maxSize>
      </DatafileAttributes>
      <DatafileAttributes id="{ORACLE_BASE}/oradata/{DB_UNIQUE_NAME}/system01.dbf">
         <tablespace>SYSTEM</tablespace>
         <temporary>false</temporary>
         <online>true</online>
         <status>0</status>
         <size unit="MB">700</size>
         <reuse>true</reuse>
         <autoExtend>true</autoExtend>
         <increment unit="KB">10240</increment>
         <maxSize unit="MB">-1</maxSize>
      </DatafileAttributes>
      <DatafileAttributes id="{ORACLE_BASE}/oradata/{DB_UNIQUE_NAME}/temp01.dbf">
         <tablespace>TEMP</tablespace>
         <temporary>false</temporary>
         <online>true</online>
         <status>0</status>
         <size unit="MB">20</size>
         <reuse>true</reuse>
         <autoExtend>true</autoExtend>
         <increment unit="KB">640</increment>
         <maxSize unit="MB">-1</maxSize>
      </DatafileAttributes>
      <DatafileAttributes id="{ORACLE_BASE}/oradata/{DB_UNIQUE_NAME}/undotbs01.dbf">
         <tablespace>UNDOTBS1</tablespace>
         <temporary>false</temporary>
         <online>true</online>
         <status>0</status>
         <size unit="MB">200</size>
         <reuse>true</reuse>
         <autoExtend>true</autoExtend>
         <increment unit="KB">5120</increment>
         <maxSize unit="MB">-1</maxSize>
      </DatafileAttributes>
      <TablespaceAttributes id="SYSAUX">
         <online>true</online>
         <offlineMode>1</offlineMode>
         <readOnly>false</readOnly>
         <temporary>false</temporary>
         <defaultTemp>false</defaultTemp>
         <undo>false</undo>
         <local>true</local>
         <blockSize>-1</blockSize>
         <allocation>1</allocation>
         <uniAllocSize unit="KB">-1</uniAllocSize>
         <initSize unit="KB">64</initSize>
         <increment unit="KB">64</increment>
         <incrementPercent>50</incrementPercent>
         <minExtends>1</minExtends>
         <maxExtends>4096</maxExtends>
         <minExtendsSize unit="KB">64</minExtendsSize>
         <logging>true</logging>
         <recoverable>false</recoverable>
         <maxFreeSpace>0</maxFreeSpace>
         <bigfile>false</bigfile>
         <datafilesList>
            <TablespaceDatafileAttributes id="{ORACLE_BASE}/oradata/{DB_UNIQUE_NAME}/sysaux01.dbf"/>
         </datafilesList>
      </TablespaceAttributes>
      <TablespaceAttributes id="USERS">
         <online>true</online>
         <offlineMode>1</offlineMode>
         <readOnly>false</readOnly>
         <temporary>false</temporary>
         <defaultTemp>false</defaultTemp>
         <undo>false</undo>
         <local>true</local>
         <blockSize>-1</blockSize>
         <allocation>1</allocation>
         <uniAllocSize unit="KB">-1</uniAllocSize>
         <initSize unit="KB">128</initSize>
         <increment unit="KB">128</increment>
         <incrementPercent>0</incrementPercent>
         <minExtends>1</minExtends>
         <maxExtends>4096</maxExtends>
         <minExtendsSize unit="KB">128</minExtendsSize>
         <logging>true</logging>
         <recoverable>false</recoverable>
         <maxFreeSpace>0</maxFreeSpace>
         <bigfile>false</bigfile>
         <datafilesList>
            <TablespaceDatafileAttributes id="{ORACLE_BASE}/oradata/{DB_UNIQUE_NAME}/users01.dbf"/>
         </datafilesList>
      </TablespaceAttributes>
      <TablespaceAttributes id="SYSTEM">
         <online>true</online>
         <offlineMode>1</offlineMode>
         <readOnly>false</readOnly>
         <temporary>false</temporary>
         <defaultTemp>false</defaultTemp>
         <undo>false</undo>
         <local>true</local>
         <blockSize>-1</blockSize>
         <allocation>3</allocation>
         <uniAllocSize unit="KB">-1</uniAllocSize>
         <initSize unit="KB">64</initSize>
         <increment unit="KB">64</increment>
         <incrementPercent>50</incrementPercent>
         <minExtends>1</minExtends>
         <maxExtends>-1</maxExtends>
         <minExtendsSize unit="KB">64</minExtendsSize>
         <logging>true</logging>
         <recoverable>false</recoverable>
         <maxFreeSpace>0</maxFreeSpace>
         <bigfile>false</bigfile>
         <datafilesList>
            <TablespaceDatafileAttributes id="{ORACLE_BASE}/oradata/{DB_UNIQUE_NAME}/system01.dbf"/>
         </datafilesList>
      </TablespaceAttributes>
      <TablespaceAttributes id="TEMP">
         <online>true</online>
         <offlineMode>1</offlineMode>
         <readOnly>false</readOnly>
         <temporary>true</temporary>
         <defaultTemp>true</defaultTemp>
         <undo>false</undo>
         <local>true</local>
         <blockSize>-1</blockSize>
         <allocation>1</allocation>
         <uniAllocSize unit="KB">-1</uniAllocSize>
         <initSize unit="KB">64</initSize>
         <increment unit="KB">64</increment>
         <incrementPercent>0</incrementPercent>
         <minExtends>1</minExtends>
         <maxExtends>0</maxExtends>
         <minExtendsSize unit="KB">64</minExtendsSize>
         <logging>true</logging>
         <recoverable>false</recoverable>
         <maxFreeSpace>0</maxFreeSpace>
         <bigfile>false</bigfile>
         <datafilesList>
            <TablespaceDatafileAttributes id="{ORACLE_BASE}/oradata/{DB_UNIQUE_NAME}/temp01.dbf"/>
         </datafilesList>
      </TablespaceAttributes>
      <TablespaceAttributes id="UNDOTBS1">
         <online>true</online>
         <offlineMode>1</offlineMode>
         <readOnly>false</readOnly>
         <temporary>false</temporary>
         <defaultTemp>false</defaultTemp>
         <undo>true</undo>
         <local>true</local>
         <blockSize>-1</blockSize>
         <allocation>1</allocation>
         <uniAllocSize unit="KB">-1</uniAllocSize>
         <initSize unit="KB">512</initSize>
         <increment unit="KB">512</increment>
         <incrementPercent>50</incrementPercent>
         <minExtends>8</minExtends>
         <maxExtends>4096</maxExtends>
         <minExtendsSize unit="KB">512</minExtendsSize>
         <logging>true</logging>
         <recoverable>false</recoverable>
         <maxFreeSpace>0</maxFreeSpace>
         <bigfile>false</bigfile>
         <datafilesList>
            <TablespaceDatafileAttributes id="{ORACLE_BASE}/oradata/{DB_UNIQUE_NAME}/undotbs01.dbf"/>
         </datafilesList>
      </TablespaceAttributes>
      <RedoLogGroupAttributes id="1">
         <reuse>false</reuse>
         <fileSize unit="KB">204800</fileSize>
         <Thread>1</Thread>
         <member ordinal="0" memberName="redo01.log" filepath="{ORACLE_BASE}/oradata/{DB_UNIQUE_NAME}/"/>
      </RedoLogGroupAttributes>
      <RedoLogGroupAttributes id="2">
         <reuse>false</reuse>
         <fileSize unit="KB">204800</fileSize>
         <Thread>1</Thread>
         <member ordinal="0" memberName="redo02.log" filepath="{ORACLE_BASE}/oradata/{DB_UNIQUE_NAME}/"/>
      </RedoLogGroupAttributes>
      <RedoLogGroupAttributes id="3">
         <reuse>false</reuse>
         <fileSize unit="KB">204800</fileSize>
         <Thread>1</Thread>
         <member ordinal="0" memberName="redo03.log" filepath="{ORACLE_BASE}/oradata/{DB_UNIQUE_NAME}/"/>
      </RedoLogGroupAttributes>
   </StorageAttributes>
</DatabaseTemplate>

Run the dbca specifying the template and other parameters as command options.
dbca -createDatabase -gdbName racext -templateName $ORACLE_HOME/assistants/dbca/templates/testdb.dbt -characterSet AL32UTF8 -emConfiguration DBEXPRESS 
-storageType ASM -asmsnmpPassword testASM1234 
-diskGroupName DATA -recoveryGroupName FRA 
-nodelist rhel71,rhel72 -sysPassword racextDB1234 
-systemPassword racextDB1234 
-createAsContainerDatabase false 
-memoryMgmtType AUTO_SGA 
-enableArchive false -useOMF true 
-adminManaged -nationalCharacterSet AL16UTF16 
-databaseConfigType RAC -silent
[WARNING] [DBT-09102] Target environment does not meet some optional requirements.
   CAUSE: Some of the optional prerequisites are not met. See logs for details.
   ACTION: Find the appropriate configuration from the log file or from the installation guide to meet the prerequisites and fix this manually.
Prepare for db operation
8% complete
Creating and starting Oracle instance
10% complete
11% complete
15% complete
Creating database files
16% complete
17% complete
23% complete
Creating data dictionary views
25% complete
30% complete
31% complete
32% complete
37% complete
39% complete
42% complete
46% complete
Oracle Text
47% complete
48% complete
52% complete
54% complete
Creating cluster database views
55% complete
69% complete
Completing Database Creation
73% complete
76% complete
77% complete
Executing Post Configuration Actions
100% complete
Database creation complete. For details check the logfiles at:
 /opt/app/oracle/cfgtoollogs/dbca/racext.
Database Information:
Global Database Name:racext
System Identifier(SID) Prefix:racext
Look at the log file "/opt/app/oracle/cfgtoollogs/dbca/racext/racext0.log" for further details.

Check the databae status at the end
srvctl status database -db racext
Instance racext1 is running on node rhel71
Instance racext2 is running on node rhel72
This concludes the installing 19c RAC.

Related Posts
Installing 18c (18.3) RAC on RHEL 7 with Role Separation - Clusterware
Installing 12cR2 (12.2.0.1) RAC on RHEL 6 with Role Separation - Clusterware
Installing 12c (12.1.0.2) Flex Cluster on RHEL 6 with Role Separation
Installing 12c (12.1.0.1) RAC on RHEL 6 with Role Separation - Clusterware
Installing 11gR2 (11.2.0.3) GI with Role Separation on RHEL 6
Installing 11gR2 (11.2.0.3) GI with Role Separation on OEL 6
Installing 11gR2 Standalone Server with ASM and Role Separation on RHEL 6
11gR2 Standalone Data Guard (with ASM and Role Separation)

Following screenshots shows the same done using the OUI. OUI also gives the option of saving the setup inputs in a repsonse which could be later used in silent instalations.
In step the private interface use defautls only for "private". In previous versions this would default to "ASM & private". This is due to the fact now that OCR and vote disks could be located in shared file system and GIMR is optional. Therefore it is possible to have a RAC without ASM (if data and recovery area is also located in a NFS).
If ASM is used for OCR and GIMR change the use to "ASM & private".

Database software setup (not all steps are shown)

Database Setup (not all steps shown)