Sunday, April 28, 2019

Add or Remove Database Options with chopt

chopt tool allows easier way of enabling or disabling certain database options after the database software installation. chopt does the binary relinking as part of enabling or disabling. Following shows an oracle database installation where real application testing is enabled (output generated using script here).
RAC             [    INSTALLED     ]
RAT             [    INSTALLED     ]
OLS             [  NOT INSTALLED   ]
DV              [  NOT INSTALLED   ]
ASM             [  NOT INSTALLED   ]
OLAP            [    INSTALLED     ]
PART            [    INSTALLED     ]
CTX             [    INSTALLED     ]
chopt is used to disable RAT. Shutdown the database first and then use chopt.
srvctl stop database -db en18c

chopt disable rat

Writing to /opt/app/oracle/product/18.x.0/dbhome_1/install/disable_rat_2019-04-09_15-36-37pm.log...
/usr/bin/make -f /opt/app/oracle/product/18.x.0/dbhome_1/rdbms/lib/ins_rdbms.mk rat_off ORACLE_HOME=/opt/app/oracle/product/18.x.0/dbhome_1
/usr/bin/make -f /opt/app/oracle/product/18.x.0/dbhome_1/rdbms/lib/ins_rdbms.mk ioracle ORACLE_HOME=/opt/app/oracle/product/18.x.0/dbhome_1

srvctl start database -db en18c


Checking the output for enabled components after the change shows RAT is not installed any more.
RAC             [    INSTALLED     ]
RAT             [  NOT INSTALLED   ]
OLS             [  NOT INSTALLED   ]
DV              [  NOT INSTALLED   ]
ASM             [  NOT INSTALLED   ]
OLAP            [    INSTALLED     ]
PART            [    INSTALLED     ]
CTX             [    INSTALLED     ]
Currently chopt could be used to enable or disable Oracle Advanced Analytics, Oracle OLAP, Oracle Partitioning and Oracle Real Application Testing options in 12.1 and above. For 11.2 the list include Oracle Data Mining RDBMS Files, Oracle Database Vault option, Oracle Label Security, Oracle OLAP, Oracle Partitioning and Oracle Real Application Testing.

Monday, April 22, 2019

Creating Gold Images on 18c

Both gridSetup.sh in grid infrastructure and runInstaller in Oracle database software has the option to create gold images from an installation. This facilitate standardization and uniformity. Moreover each time a patch is applied a new gold image could be created which will include the patched binaries. So any new home created from the gold image will also include the patch. This post shows steps for creating gold image for grid infrastructure and oracle database.

Creating Gold Image of Grid Infrastructure
Gold image from a grid infrastructure setup is created using gridSetup.sh with the createGoldImage option. By default this expect X-windows to be available. If not the execution will fail.
./gridSetup.sh -createGoldImage -destinationLocation /media/giGold
ERROR: Unable to verify the graphical display setup. This application requires X display. Make sure that xdpyinfo exist under PATH variable.

Secondly the destination location must be at least 23GB, even though the final zip file generated could be 5-6 GB (depending on excluding certain files). Execution will fail if there isn't sufficient space
./gridSetup.sh -createGoldImage -destinationLocation /media/giGold -silent
Launching Oracle Grid Infrastructure Setup Wizard...

[FATAL] [INS-32707] The specified destination location (/media/giGold) does not have enough free space.
   ACTION: Provide a destination location path with at least (22,694) MB of free space.
If there's no risk of patches being rolled back, then the .patch_storage folder could be excluded from the gold image. This will reduce the size of the final zip file created. Other candidates for exclusions are files that have the server hostname in the name.
 ./gridSetup.sh -createGoldImage -destinationLocation /media/giGold -silent -exclFiles $GI_HOME/.patch_storage
Launching Oracle Database Setup Wizard...

Successfully Setup Software.
Gold Image location: /media/giGold/grid_home_2019-03-26_12-20-38PM.zip
Any new grid infrastructure installation done using the gold image will have the patches.
$ /opt/app/oracle/product/18.x.0/grid/OPatch/opatch lspatches
28864607;ACFS RELEASE UPDATE 18.5.0.0.0 (28864607)
28864593;OCW RELEASE UPDATE 18.5.0.0.0 (28864593)
28822489;Database Release Update : 18.5.0.0.190115 (28822489)
28547619;TOMCAT RELEASE UPDATE 18.0.0.0.0 (28547619)
28435192;DBWLM RELEASE UPDATE 18.0.0.0.0 (28435192)
27908644;UPDATE 18.3 DATABASE CLIENT JDK IN ORACLE HOME TO JDK8U171
27923415;OJVM RELEASE UPDATE: 18.3.0.0.180717 (27923415)
The gold image doesn't zip all the files in the current installation. Some directories such as log directories are excluded automatically. In previous versions when installing from a gold image, these missing folders caused issue. But this appear to be fixed on 18c and no issues were encountered when a new installation was done using the gold image.




Creating Gold Image of Oracle Database
The gold image for Oracle database software is created using runInstaller. There are two runInstallers inside an OH. One is in $OH/runInstaller and other is in $OH/oui/bin/runInstaller. This gold image creating option is only available in $OH/runInstaller.
./runInstaller -createGoldImage -destinationLocation /media/ohGold -silent -exclFiles $ORACLE_HOME/.patch_storage
Launching Oracle Database Setup Wizard...

Successfully Setup Software.
Gold Image location: /media/ohGold/db_home_2019-03-26_03-35-48PM.zip
As mentioned earlier patch storage could be excluded from the gold image. Any new database installation created using the gold image will have the patches already applied.
/opt/app/oracle/product/18.x.0/dbhome_1/OPatch/opatch lspatches
28864593;OCW RELEASE UPDATE 18.5.0.0.0 (28864593)
28822489;Database Release Update : 18.5.0.0.190115 (28822489)
27908644;UPDATE 18.3 DATABASE CLIENT JDK IN ORACLE HOME TO JDK8U171
27923415;OJVM RELEASE UPDATE: 18.3.0.0.180717 (27923415)
Any new database created will also have the post patch scripts applied.
SQL> select PATCH_ID,PATCH_TYPE,ACTION,STATUS,SOURCE_VERSION,TARGET_VERSION,ACTION_TIME from dba_registry_sqlpatch;

  PATCH_ID PATCH_TYPE ACTION          STATUS                    SOURCE_VERSION  TARGET_VERSION  ACTION_TIME
---------- ---------- --------------- ------------------------- --------------- --------------- ------------------------------
  28822489 RU         APPLY           SUCCESS                   18.1.0.0.0      18.5.0.0.0      27-MAR-19 12.10.44.167872 PM
  27923415 INTERIM    APPLY           SUCCESS                   18.1.0.0.0      18.1.0.0.0      27-MAR-19 12.10.44.171197 PM

Update on 2021-02-15

It appears creating gold image excluding .patch_storage could cause issues down the line even if explicity patch rollback is not done. For example, a gold image was created from a GI home which had RU 19.5 and one-off patch 29013832 installed (needed to fix an issue on AFD). The installation using the gold image was fine until RU 19.10 was applied. During GI home patch the opatchauto command failed with and following could be seen on the patch logs
[Feb 15, 2021 3:25:56 PM] [INFO]    Patch 29013832:
                                    Copy Action: Directory "/opt/app/oracle/product/19.x.0/grid/.patch_storage/29013832_Oct_25_2019_13_06_10" does not exists or is not readable.
                                    'oracle.rdbms, 19.0.0.0.0': Cannot copy file from 'asmcmdafd.pm' to '/opt/app/oracle/product/19.x.0/grid/lib/asmcmdafd.pm'
[Feb 15, 2021 3:25:56 PM] [INFO]    Prerequisite check "CheckRollbackable" on auto-rollback patches failed.
                                    The details are:

                                    Patch 29013832:
                                    Copy Action: Directory "/opt/app/oracle/product/19.x.0/grid/.patch_storage/29013832_Oct_25_2019_13_06_10" does not exists or is not readable.
                                    'oracle.rdbms, 19.0.0.0.0': Cannot copy file from 'asmcmdafd.pm' to '/opt/app/oracle/product/19.x.0/grid/lib/asmcmdafd.pm'
[Feb 15, 2021 3:25:56 PM] [SEVERE]  OUI-67073:UtilSession failed: Prerequisite check "CheckRollbackable" on auto-rollback patches failed.

It appears RU 19.10 does an implicity rollback of the patch 29013832 (as it is included in RU 19.10) but the old version of (19.5) is not available.
As old patch directory (29013832_Oct_25_2019_13_06_10) could not be copied over from an earlier setup, had to recover from this situation by reinstalling GI as software only and then configuring HAS.
So it maybe best to create the gold images with .patch_storage folder though that can add several GBs to the final size of the gold image.

Monday, April 15, 2019

DIA-49802: missing read, write, or execute permission on specified ADR home directory

Executing an 18c (18.5) Oracle restart reconfiguration after a hostname change resulted in following error.
$ORACLE_HOME/perl/bin/perl $ORACLE_HOME/crs/install/roothas.pl
Using configuration parameter file: /opt/app/oracle/product/18.0.0/grid/crs/install/crsconfig_params
The log of current session can be found at:
  /opt/app/oracle/crsdata/ip-172-31-1-113/crsconfig/roothas_2019-03-27_08-25-04AM.log
2019/03/27 08:25:06 CLSRSC-363: User ignored prerequisites during installation
Oracle Clusterware infrastructure error in OCRCONFIG (OS PID 8212): CLSD/ADR initialization failed with return value -1
1: clskec:has:CLSU:910 4 args[clsdAdrInit_CLSK_err][mod=clsdadr.c][loc=(:CLSD00281:)][msg=clsdAdrInit: Additional diagnostic data returned by the ADR component for dbgc_init_all failure:
 DIA-49802: missing read, write, or execute permission on specified ADR home directory [/opt/app/oracle/diag/crs/ip-172-31-1-113/crs/log]
DIA-49801: actual permissions [rwxrwx---], expected minimum permissions [rwxrwxrwx] for effective user [oracle]
DIA-48188: user missing read, write, or exec permission on specified directory
Linux-x86_64 Error: 13: Permission denied
Additional information: 2
Additional information: 511
Additional information: 16888
([all diagnostic data retrieved from ADR])]
2: clskec:has:CLSU:910 4 args[clsdAdrInit_CLSK_err][mod=clsdadr.c][loc=(:CLSD00050:)][msg=clsdAdrInit: call to dbgc_init_all failed. facility:[CRS] product:[CRS] line number:[1610] return code: [ORA-49802] Oracle Base: [/opt/app/oracle]  Product Type: [CRS]  Host Name: [ip-172-31-1-113]  Instance ID: [crs]  User Name: [oracle]]

Oracle Clusterware infrastructure error in CLSCFG (OS PID 8223): CLSD/ADR initialization failed with return value -1
1: clskec:has:CLSU:910 4 args[clsdAdrInit_CLSK_err][mod=clsdadr.c][loc=(:CLSD00281:)][msg=clsdAdrInit: Additional diagnostic data returned by the ADR component for dbgc_init_all failure:
 DIA-49802: missing read, write, or execute permission on specified ADR home directory [/opt/app/oracle/diag/crs/ip-172-31-1-113/crs/log]
DIA-49801: actual permissions [rwxrwx---], expected minimum permissions [rwxrwxrwx] for effective user [oracle]
DIA-48188: user missing read, write, or exec permission on specified directory
Linux-x86_64 Error: 13: Permission denied
Additional information: 2
Additional information: 511
Additional information: 16888
([all diagnostic data retrieved from ADR])]
2: clskec:has:CLSU:910 4 args[clsdAdrInit_CLSK_err][mod=clsdadr.c][loc=(:CLSD00050:)][msg=clsdAdrInit: call to dbgc_init_all failed. facility:[CRS] product:[CRS] line number:[1610] return code: [ORA-49802] Oracle Base: [/opt/app/oracle]  Product Type: [CRS]  Host Name: [ip-172-31-1-113]  Instance ID: [crs]  User Name: [oracle]]

LOCAL ADD MODE
Creating OCR keys for user 'oracle', privgrp 'oinstall'..
Operation successful.
LOCAL ONLY MODE
Successfully accumulated necessary OCR keys.
Creating OCR keys for user 'root', privgrp 'root'..
Operation successful.
CRS-4664: Node ip-172-31-1-113 successfully pinned.
2019/03/27 08:26:01 CLSRSC-330: Adding Clusterware entries to file 'oracle-ohasd.conf'

2019/03/27 08:28:17 CLSRSC-214: Failed to start the resource 'ohasd'
Failed to start the Clusterware. Last 20 lines of the alert log follow:
2019-03-27 08:26:28.832
[client(8848)]CRS-8500:Oracle Clusterware OHASD process is starting with operating system process ID 8848
2019-03-27 08:26:28.845
[client(8848)]CRS-2112:The OLR service started on node ip-172-31-1-113.
2019-03-27 08:26:31.155
[client(8900)]CRS-8500:Oracle Clusterware OHASD process is starting with operating system process ID 8900
2019-03-27 08:26:31.168
[client(8900)]CRS-2112:The OLR service started on node ip-172-31-1-113.
2019-03-27 08:26:33.140
[client(8950)]CRS-8500:Oracle Clusterware OHASD process is starting with operating system process ID 8950
2019-03-27 08:26:33.152
[client(8950)]CRS-2112:The OLR service started on node ip-172-31-1-113.
2019-03-27 08:26:35.195
[client(9000)]CRS-8500:Oracle Clusterware OHASD process is starting with operating system process ID 9000
2019-03-27 08:26:35.208
[client(9000)]CRS-2112:The OLR service started on node ip-172-31-1-113.
2019-03-27 08:26:37.231
[client(9050)]CRS-8500:Oracle Clusterware OHASD process is starting with operating system process ID 9050
2019-03-27 08:26:37.243
[client(9050)]CRS-2112:The OLR service started on node ip-172-31-1-113.

2019/03/27 08:28:17 CLSRSC-318: Failed to start Oracle OHASD service
Died at /opt/app/oracle/product/18.0.0/grid/crs/install/crsinstall.pm line 3226.

It appears during the reconfiguration, which is executed as root user the folders inside $ORACLE_BASE/diag/crs/`hostname -s`/crs gets created with root ownership but are written to as oracle (or grid if role separation is used) user.
cd /opt/app/oracle/diag/crs/ip-172-31-1-113/crs
[root@ip-172-31-1-113 crs]# ls -l

drwxrwx---. 2 root root  4096 Mar 27 08:24 alert
drwxrwx---. 2 root root  4096 Mar 27 08:24 cdump
drwxrwx---. 2 root root  4096 Mar 27 08:24 incpkg
drwxrwx---. 2 root root  4096 Mar 27 08:24 lck
drwxrwx---. 4 root root  4096 Mar 27 08:29 log
drwxrwx---. 2 root root  4096 Mar 27 08:24 metadata
drwxrwx---. 2 root root  4096 Mar 27 08:24 metadata_dgif
drwxrwx---. 2 root root  4096 Mar 27 08:24 metadata_pv
drwxrwx---. 2 root root  4096 Mar 27 08:24 stage
drwxrwx---. 2 root root  4096 Mar 27 08:24 sweep
drwxrwx---. 2 root root 12288 Mar 27 08:33 trace


This issue only happens during reconfiguration. On a new installation the ownership is set to the user installing the grid software, either oracle or grid. Changing the ownership to oracle (or grid) user in these directories resolved the issue.
chown -R oracle:oinstall *

[root@ip-172-31-1-113 crs]# ls -l
drwxrwx---. 2 oracle oinstall  4096 Mar 27 08:24 alert
drwxrwx---. 2 oracle oinstall  4096 Mar 27 08:24 cdump
drwxrwx---. 2 oracle oinstall  4096 Mar 27 08:24 incident
drwxrwx---. 2 oracle oinstall  4096 Mar 27 08:24 incpkg
drwxrwx---. 2 oracle oinstall  4096 Mar 27 08:24 lck
drwxrwx---. 4 oracle oinstall  4096 Mar 27 08:29 log
drwxrwx---. 2 oracle oinstall  4096 Mar 27 08:24 metadata
drwxrwx---. 2 oracle oinstall  4096 Mar 27 08:24 metadata_dgif
drwxrwx---. 2 oracle oinstall  4096 Mar 27 08:24 metadata_pv
drwxrwx---. 2 oracle oinstall  4096 Mar 27 08:24 stage
drwxrwx---. 2 oracle oinstall  4096 Mar 27 08:24 sweep
drwxrwx---. 2 oracle oinstall 20480 Mar 27 08:41 trace
Re-run the Oracle restart configuration again.
$ORACLE_HOME/perl/bin/perl $ORACLE_HOME/crs/install/roothas.pl
Using configuration parameter file: /opt/app/oracle/product/18.0.0/grid/crs/install/crsconfig_params
The log of current session can be found at:
  /opt/app/oracle/crsdata/ip-172-31-1-113/crsconfig/roothas_2019-03-27_08-38-56AM.log
2019/03/27 08:38:59 CLSRSC-363: User ignored prerequisites during installation
LOCAL ADD MODE
Creating OCR keys for user 'oracle', privgrp 'oinstall'..
Operation successful.
LOCAL ONLY MODE
Successfully accumulated necessary OCR keys.
Creating OCR keys for user 'root', privgrp 'root'..
Operation successful.
CRS-4664: Node ip-172-31-1-113 successfully pinned.
2019/03/27 08:39:28 CLSRSC-330: Adding Clusterware entries to file 'oracle-ohasd.conf'
CRS-2791: Starting shutdown of Oracle High Availability Services-managed resources on 'ip-172-31-1-113'
CRS-2673: Attempting to stop 'ora.evmd' on 'ip-172-31-1-113'
CRS-2677: Stop of 'ora.evmd' on 'ip-172-31-1-113' succeeded
CRS-2793: Shutdown of Oracle High Availability Services-managed resources on 'ip-172-31-1-113' has completed
CRS-4133: Oracle High Availability Services has been stopped.
CRS-4123: Oracle High Availability Services has been started.

ip-172-31-1-113     2019/03/27 08:42:19     /opt/app/oracle/product/18.0.0/grid/cdata/ip-172-31-1-113/backup_20190327_084219.olr     2532936542
2019/03/27 08:42:53 CLSRSC-327: Successfully configured Oracle Restart for a standalone server

Related Post
Changing Hostname in a Standalone DB Configuration with ASM

Sunday, April 7, 2019

Installing 18c (18.3) RAC on RHEL 7 with Role Separation - Database

Following the 18c clusterware installation the next step is to install database software and create the database. The database software installation is not much different from that of 12.2 database software installation. Even though the oracle installation guides states gcc and gcc-c++ are not required for RHEL, the cluvfy still searches for gcc-c++.
1. Create the Oracle home for the database software and unzip the installer into the directory.
cd $ORACLE_BASE
mkdir -p product/18.x.0/dbhome_1
unzip LINUX.X64_180000_db_home.zip -d /opt/app/oracle/product/18.x.0/dbhome_1/
2. Run cluvfy to check the pre dbinst state. The failures related to shm could be ignored
cluvfy stage -pre dbinst -allnodes -r 18

Verifying Physical Memory ...PASSED
Verifying Available Physical Memory ...PASSED
Verifying Swap Size ...PASSED
Verifying Free Space: rhel72:/tmp ...PASSED
Verifying Free Space: rhel71:/tmp ...PASSED
Verifying User Existence: oracle ...
  Verifying Users With Same UID: 501 ...PASSED
Verifying User Existence: oracle ...PASSED
Verifying Group Existence: dba ...PASSED
Verifying Group Existence: oinstall ...PASSED
Verifying Group Membership: oinstall(Primary) ...PASSED
Verifying Group Membership: dba ...PASSED
Verifying Run Level ...PASSED
Verifying Hard Limit: maximum open file descriptors ...PASSED
Verifying Soft Limit: maximum open file descriptors ...PASSED
Verifying Hard Limit: maximum user processes ...PASSED
Verifying Soft Limit: maximum user processes ...PASSED
Verifying Soft Limit: maximum stack size ...PASSED
Verifying Architecture ...PASSED
Verifying OS Kernel Version ...PASSED
Verifying OS Kernel Parameter: semmsl ...PASSED
Verifying OS Kernel Parameter: semmns ...PASSED
Verifying OS Kernel Parameter: semopm ...PASSED
Verifying OS Kernel Parameter: semmni ...PASSED
Verifying OS Kernel Parameter: shmmax ...PASSED
Verifying OS Kernel Parameter: shmmni ...PASSED
Verifying OS Kernel Parameter: shmall ...PASSED
Verifying OS Kernel Parameter: file-max ...PASSED
Verifying OS Kernel Parameter: ip_local_port_range ...PASSED
Verifying OS Kernel Parameter: rmem_default ...PASSED
Verifying OS Kernel Parameter: rmem_max ...PASSED
Verifying OS Kernel Parameter: wmem_default ...PASSED
Verifying OS Kernel Parameter: wmem_max ...PASSED
Verifying OS Kernel Parameter: aio-max-nr ...PASSED
Verifying Package: binutils-2.23.52.0.1 ...PASSED
Verifying Package: compat-libcap1-1.10 ...PASSED
Verifying Package: libgcc-4.8.2 (x86_64) ...PASSED
Verifying Package: libstdc++-4.8.2 (x86_64) ...PASSED
Verifying Package: libstdc++-devel-4.8.2 (x86_64) ...PASSED
Verifying Package: sysstat-10.1.5 ...PASSED
Verifying Package: gcc-c++-4.8.2 ...PASSED
Verifying Package: ksh ...PASSED
Verifying Package: make-3.82 ...PASSED
Verifying Package: glibc-2.17 (x86_64) ...PASSED
Verifying Package: glibc-devel-2.17 (x86_64) ...PASSED
Verifying Package: libaio-0.3.109 (x86_64) ...PASSED
Verifying Package: libaio-devel-0.3.109 (x86_64) ...PASSED
Verifying Package: smartmontools-6.2-4 ...PASSED
Verifying Package: net-tools-2.0-0.17 ...PASSED
Verifying Package: compat-libstdc++-33-3.2.3 (x86_64) ...PASSED
Verifying Package: libxcb-1.11 (x86_64) ...PASSED
Verifying Package: libX11-1.6.3 (x86_64) ...PASSED
Verifying Package: libXau-1.0.8 (x86_64) ...PASSED
Verifying Package: libXi-1.7.4 (x86_64) ...PASSED
Verifying Package: libXtst-1.2.2 (x86_64) ...PASSED
Verifying Users With Same UID: 0 ...PASSED
Verifying Current Group ID ...PASSED
Verifying Root user consistency ...PASSED
Verifying Host name ...PASSED
Verifying Node Connectivity ...
  Verifying Hosts File ...PASSED
  Verifying Check that maximum (MTU) size packet goes through subnet ...PASSED
  Verifying subnet mask consistency for subnet "192.168.1.0" ...PASSED
  Verifying subnet mask consistency for subnet "192.168.0.0" ...PASSED
Verifying Node Connectivity ...PASSED
Verifying Multicast or broadcast check ...PASSED
Verifying User Mask ...PASSED
Verifying CRS Integrity ...
  Verifying Clusterware Version Consistency ...PASSED
Verifying CRS Integrity ...PASSED
Verifying Cluster Manager Integrity ...PASSED
Verifying Node Application Existence ...PASSED
Verifying Clock Synchronization ...PASSED
Verifying resolv.conf Integrity ...PASSED
Verifying Time zone consistency ...PASSED
Verifying Single Client Access Name (SCAN) ...
  Verifying DNS/NIS name service 'rac-scan.domain.net' ...
    Verifying Name Service Switch Configuration File Integrity ...PASSED
  Verifying DNS/NIS name service 'rac-scan.domain.net' ...PASSED
Verifying Single Client Access Name (SCAN) ...PASSED
Verifying VIP Subnet configuration check ...PASSED
Verifying Database Clusterware Version Compatibility ...PASSED
Verifying ASM storage privileges for the user: oracle ...
  Verifying Group Membership: asmdba ...PASSED
Verifying ASM storage privileges for the user: oracle ...PASSED
Verifying Daemon "proxyt" not configured and running ...PASSED
Verifying ACFS device special file ...PASSED
Verifying /dev/shm mounted as temporary file system ...FAILED (PRVE-0421)
Verifying Maximum locked memory check ...PASSED

Pre-check for database installation was unsuccessful on all the nodes.


Failures were encountered during execution of CVU verification request "stage -pre dbinst".

Verifying /dev/shm mounted as temporary file system ...FAILED
rhel72: PRVE-0421 : No entry exists in /etc/fstab for mounting /dev/shm

rhel71: PRVE-0421 : No entry exists in /etc/fstab for mounting /dev/shm
3. Execute runInstaller from the Oracle home and select software only install option.
Other steps are similar to 12.2. However, one noticeable change is that in 18c the space available for Oracle base is also checked as part of pre-reqs. Any warning raised for test system could be ignored.
4. Once the database software is installed proceed to creating the database. As this is a role separate setup, certain folders need write permission for the oinstall group so oracle user could write into them.
ls -l
total 0
drwxr-x---.  5 grid oinstall  45 Mar 20 11:50 admin
drwxr-x---.  2 grid oinstall   6 Mar 20 11:51 audit

[grid@rhel71 oracle]$ chmod 770 admin audit

[grid@rhel71 oracle]$ ls -l
total 0
drwxrwx---.  5 grid oinstall  45 Mar 20 11:50 admin
drwxrwx---.  2 grid oinstall   6 Mar 20 11:51 audit

chmod 770 $ORACLE_BASE/cfgtoollogs/dbca
5. Run cluvfy could to check pre-requisite for RAC DB (output is without verbose option).
cluvfy stage -pre dbcfg -allnodes -d /opt/app/oracle/product/18.x.0/dbhome_1

Verifying Physical Memory ...PASSED
Verifying Available Physical Memory ...PASSED
Verifying Swap Size ...PASSED
Verifying Free Space: rhel72:/tmp ...PASSED
Verifying Free Space: rhel71:/tmp ...PASSED
Verifying User Existence: oracle ...
  Verifying Users With Same UID: 501 ...PASSED
Verifying User Existence: oracle ...PASSED
Verifying Group Existence: dgdba ...PASSED
Verifying Group Existence: dba ...PASSED
Verifying Group Existence: racdba ...PASSED
Verifying Group Existence: backupdba ...PASSED
Verifying Group Existence: oper ...PASSED
Verifying Group Existence: oinstall ...PASSED
Verifying Group Membership: backupdba ...PASSED
Verifying Group Membership: racdba ...PASSED
Verifying Group Membership: dgdba ...PASSED
Verifying Group Membership: oinstall(Primary) ...PASSED
Verifying Group Membership: dba ...PASSED
Verifying Group Membership: oper ...PASSED
Verifying Run Level ...PASSED
Verifying Hard Limit: maximum open file descriptors ...PASSED
Verifying Soft Limit: maximum open file descriptors ...PASSED
Verifying Hard Limit: maximum user processes ...PASSED
Verifying Soft Limit: maximum user processes ...PASSED
Verifying Soft Limit: maximum stack size ...PASSED
Verifying Architecture ...PASSED
Verifying OS Kernel Version ...PASSED
Verifying OS Kernel Parameter: semmsl ...PASSED
Verifying OS Kernel Parameter: semmns ...PASSED
Verifying OS Kernel Parameter: semopm ...PASSED
Verifying OS Kernel Parameter: semmni ...PASSED
Verifying OS Kernel Parameter: shmmax ...PASSED
Verifying OS Kernel Parameter: shmmni ...PASSED
Verifying OS Kernel Parameter: shmall ...PASSED
Verifying OS Kernel Parameter: file-max ...PASSED
Verifying OS Kernel Parameter: ip_local_port_range ...PASSED
Verifying OS Kernel Parameter: rmem_default ...PASSED
Verifying OS Kernel Parameter: rmem_max ...PASSED
Verifying OS Kernel Parameter: wmem_default ...PASSED
Verifying OS Kernel Parameter: wmem_max ...PASSED
Verifying OS Kernel Parameter: aio-max-nr ...PASSED
Verifying Package: binutils-2.23.52.0.1 ...PASSED
Verifying Package: compat-libcap1-1.10 ...PASSED
Verifying Package: libgcc-4.8.2 (x86_64) ...PASSED
Verifying Package: libstdc++-4.8.2 (x86_64) ...PASSED
Verifying Package: libstdc++-devel-4.8.2 (x86_64) ...PASSED
Verifying Package: sysstat-10.1.5 ...PASSED
Verifying Package: gcc-c++-4.8.2 ...PASSED
Verifying Package: ksh ...PASSED
Verifying Package: make-3.82 ...PASSED
Verifying Package: glibc-2.17 (x86_64) ...PASSED
Verifying Package: glibc-devel-2.17 (x86_64) ...PASSED
Verifying Package: libaio-0.3.109 (x86_64) ...PASSED
Verifying Package: libaio-devel-0.3.109 (x86_64) ...PASSED
Verifying Package: smartmontools-6.2-4 ...PASSED
Verifying Package: net-tools-2.0-0.17 ...PASSED
Verifying Package: compat-libstdc++-33-3.2.3 (x86_64) ...PASSED
Verifying Package: libxcb-1.11 (x86_64) ...PASSED
Verifying Package: libX11-1.6.3 (x86_64) ...PASSED
Verifying Package: libXau-1.0.8 (x86_64) ...PASSED
Verifying Package: libXi-1.7.4 (x86_64) ...PASSED
Verifying Package: libXtst-1.2.2 (x86_64) ...PASSED
Verifying Current Group ID ...PASSED
Verifying CRS Integrity ...
  Verifying Clusterware Version Consistency ...PASSED
Verifying CRS Integrity ...PASSED
Verifying Node Application Existence ...PASSED
Verifying Time zone consistency ...PASSED
Verifying Single Client Access Name (SCAN) ...
  Verifying DNS/NIS name service 'rac-scan.domain.net' ...
    Verifying Name Service Switch Configuration File Integrity ...PASSED
  Verifying DNS/NIS name service 'rac-scan.domain.net' ...PASSED
Verifying Single Client Access Name (SCAN) ...PASSED
Verifying ASM Integrity ...
  Verifying Node Connectivity ...
    Verifying Hosts File ...PASSED
    Verifying Check that maximum (MTU) size packet goes through subnet ...PASSED
    Verifying subnet mask consistency for subnet "192.168.1.0" ...PASSED
    Verifying subnet mask consistency for subnet "192.168.0.0" ...PASSED
  Verifying Node Connectivity ...PASSED
Verifying ASM Integrity ...PASSED
Verifying Database Clusterware Version Compatibility ...PASSED
Verifying File system mount options for path ORACLE_HOME ...PASSED
Verifying /dev/shm mounted as temporary file system ...FAILED (PRVE-0421)

Pre-check for database configuration was unsuccessful on all the nodes.


Failures were encountered during execution of CVU verification request "stage -pre dbcfg".

Verifying /dev/shm mounted as temporary file system ...FAILED
rhel72: PRVE-0421 : No entry exists in /etc/fstab for mounting /dev/shm

rhel71: PRVE-0421 : No entry exists in /etc/fstab for mounting /dev/shm
6. Create the database using dbca.
7. Create PDBs are required and associate services with them so PDBs are auto started. Assuming PDB named pdb18c is created, add a database service associated with PDB with
srvctl add service -db ent18c -pdb pdb18c -service pdbsrv -preferred "ent18c1,ent18c2"
srvctl start service -db ent18c -s pdbsrv
8. Finally, the stopping of the ASM proxy instances has no relevance to the "flex" functioning of DB instances. If an ASM instances in one node goes down, these DB instances are able to use a remote ASM instance to access the disks. This could be verified with the following.
[grid@rhel71 ~]$  srvctl status asm -detail
ASM is running on rhel71,rhel72
ASM is enabled.
ASM instance +ASM1 is running on node rhel71
Number of connected clients: 2
Client names: ent18c1:ent18c:rhel7-cluster rhel71.domain.net:_OCR:rhel7-cluster
ASM instance +ASM2 is running on node rhel72
Number of connected clients: 3
Client names: -MGMTDB:_mgmtdb:rhel7-cluster ent18c2:ent18c:rhel7-cluster rhel72.domain.net:_OCR:rhel7-cluster
The above output shows the clients connected to each ASM instance on each node. Currently ASM1 has 2 clients while ASM2 has 3 clients, *OCR is available as client on both ASM instances.
Then an ASM instance is stopped.
srvctl stop asm  -node rhel72  -force
When one of the ASM instances is stopped forcefully those clients start using the service of the available ASM instance
srvctl status asm -detail
ASM is running on rhel71
ASM is enabled.
ASM instance +ASM1 is running on node rhel71
Number of connected clients: 4
Client names: -MGMTDB:_mgmtdb:rhel7-cluster ent18c1:ent18c:rhel7-cluster ent18c2:ent18c:rhel7-cluster rhel71.domain.net:_OCR:rhel7-cluster
As seen above, clients that previously clients of ASM2 instance are now using ASM1 instance to access ASM disks. (total client count is 4 not 5 due to not duplicating the OCR* client).
When ASM instance is restarted the clients automatically select the local ASM instance over remote ASM instances.
srvctl start asm  -node rhel72
srvctl status asm -detail
ASM is running on rhel71,rhel72
ASM is enabled.
ASM instance +ASM1 is running on node rhel71
Number of connected clients: 2
Client names: ent18c1:ent18c:rhel7-cluster rhel71.domain.net:_OCR:rhel7-cluster
ASM instance +ASM2 is running on node rhel72
Number of connected clients: 3
Client names: -MGMTDB:_mgmtdb:rhel7-cluster ent18c2:ent18c:rhel7-cluster rhel72.domain.net:_OCR:rhel7-cluster
This concludes the setting up RAC DB on 18c.

Related Posts
Installing 12cR2 (12.2.0.1) RAC on RHEL 6 with Role Separation - Database
Installing 12c (12.1.0.1) RAC on RHEL 6 with Role Separation - Database Software
Installing 12c (12.1.0.1) RAC on RHEL 6 with Role Separation - Creating CDB & PDB

Monday, April 1, 2019

Installing 18c (18.3) RAC on RHEL 7 with Role Separation - Clusterware

This post is not a step by step guide to installing 18c RAC but list the main differences in the installation process compared to previous versions and things to look out for. Similar to 12cR2 clusterware the 18c cluster installation is no longer done through runInstaller. The downloaded Oracle Grid Infrastructure image files are unzipped into the folder path where GI is to be installed. This only need to be done on the local node only. During installation, the software is copied and installed on all other nodes in the cluster. Unlike 12cR1, it is possible to specify a separate disk group for GIMR at the installation time. Oracle documentation says "You cannot migrate the GIMR from one disk group to another later". 12cR1 also mentioned the same but there were ways to move GIMR to different disk group after the installation. It has been not verified whether the same techniques could be used for 18c as well.
1. The OS version and kernel versions used for this RAC setup is shown below. Unlike the previous post on 12cR2 this post for 18c uses RHEL7
cat /etc/redhat-release
Red Hat Enterprise Linux Server release 7.4 (Maipo)

uname -r
3.10.0-693.el7.x86_64
2. Create the user groups and users similar to previous 12.1 installation. As said in an earlier post, 12.2 introduce a new user group for RAC management. Following from install guide "You must designate a group as the OSRACDBA group during database installation. Members of this group are granted the SYSRAC privileges to perform day–to–day administration of Oracle databases on an Oracle RAC cluster".
groupadd dba
groupadd oinstall
groupadd oper
groupadd asmoper
groupadd asmdba
groupadd asmadmin
groupadd backupdba
groupadd dgdba
groupadd kmdba
groupadd racdba

useradd -g oinstall -G dba,oper,asmdba,backupdba,dgdba,kmdba,racdba oracle
useradd -g oinstall -G asmadmin,asmdba,asmoper,dba grid
For user equivelancy ensure that user id and group id are same across the cluster nodes. The option -g could be used to ensure that same user/group id is used for creating users across all nodes. The nodes in this case are named rhel71 and rhel72.
[oracle@rhel71 ~]$ id oracle
uid=501(oracle) gid=501(oinstall) groups=501(oinstall),1000(dba),1002(oper),1004(asmdba),1006(backupdba),1007(dgdba),1008(kmdba),1009(racdba)
[oracle@rhel71 ~]$ ssh rhel72 id oracle
uid=501(oracle) gid=501(oinstall) groups=501(oinstall),1000(dba),1002(oper),1004(asmdba),1006(backupdba),1007(dgdba),1008(kmdba),1009(racdba)

[oracle@rhel71 ~]$ id grid
uid=502(grid) gid=501(oinstall) groups=501(oinstall),1000(dba),1003(asmoper),1004(asmdba),1005(asmadmin)
[oracle@rhel71 ~]$ ssh rhel72 id grid
uid=502(grid) gid=501(oinstall) groups=501(oinstall),1000(dba),1003(asmoper),1004(asmdba),1005(asmadmin)
3. If a separate disk group is used for GIMR then it needs to be at least 35GB in size. Following from installation doc "during installation of an Oracle Standalone Cluster, if you create the MGMT disk group for Grid Infrastructure Management Repository (GIMR), then the installer requires that you use a disk group with at least 35 GB of available space". A 40GB disk was created to store the GIMR and udev rules were created for granting necessary ownership and permission.
cat /etc/udev/rules.d/99-sd.rules
KERNEL=="sd*1", SUBSYSTEM=="block", PROGRAM=="/usr/lib/udev/scsi_id -g -u -d /dev/$parent",RESULT=="1ATA_VBOX_HARDDISK_VB1a5c8787-e082e7a7",SYMLINK+="oracleasm/ocr1",OWNER="grid",GROUP="asmadmin",MODE="0660"
KERNEL=="sd*1", SUBSYSTEM=="block", PROGRAM=="/usr/lib/udev/scsi_id -g -u -d /dev/$parent",RESULT=="1ATA_VBOX_HARDDISK_VB1a72679c-ab23e15f",SYMLINK+="oracleasm/ocr2",OWNER="grid",GROUP="asmadmin",MODE="0660"
KERNEL=="sd*1", SUBSYSTEM=="block", PROGRAM=="/usr/lib/udev/scsi_id -g -u -d /dev/$parent",RESULT=="1ATA_VBOX_HARDDISK_VB8f43e75c-3e0bec6d",SYMLINK+="oracleasm/ocr3",OWNER="grid",GROUP="asmadmin",MODE="0660"

KERNEL=="sd*1", SUBSYSTEM=="block", PROGRAM=="/usr/lib/udev/scsi_id -g -u -d /dev/$parent",RESULT=="1ATA_VBOX_HARDDISK_VBf8475ab7-d401ec8b",SYMLINK+="oracleasm/gimr",OWNER="grid",GROUP="asmadmin",MODE="0660"

KERNEL=="sd*1", SUBSYSTEM=="block", PROGRAM=="/usr/lib/udev/scsi_id -g -u -d /dev/$parent",RESULT=="1ATA_VBOX_HARDDISK_VB34650f33-5ecdc029",SYMLINK+="oracleasm/data1",OWNER="grid",GROUP="asmadmin",MODE="0660"

KERNEL=="sd*1", SUBSYSTEM=="block", PROGRAM=="/usr/lib/udev/scsi_id -g -u -d /dev/$parent",RESULT=="1ATA_VBOX_HARDDISK_VBdb057b08-1733da4b",SYMLINK+="oracleasm/fra1",OWNER="grid",GROUP="asmadmin",MODE="0660"
4. Create the GI installation directory and unzip the downloaded install media into the directory.
mkdir -p /opt/app/18.x.0/grid
chown grid:oinstall /opt/app/18.x.0/grid
unzip ./LINUX.X64_180000_grid_home.zip -d /opt/app/18.x.0/grid
5. Run cluvfy to check the pre-reqs checks. The failure related to shm shown at the end could be ignored as per 2065603.1 and 25907259.8
[grid@rhel71 grid]$ ./runcluvfy.sh stage -pre crsinst -n rhel71,rhel72 -r 18 -osdba asmdba -osoper asmoper -asm -asmgrp asmadmin -presence local -asmdev /dev/oracleasm/data1,/dev/oracleasm/fra1,/dev/oracleasm/ocr1,/dev/oracleasm/ocr2,/dev/oracleasm/ocr3,/dev/oracleasm/gimr  -crshome /opt/app/18.x.0/grid -fixup  -verbose

Verifying Physical Memory ...
  Node Name     Available                 Required                  Status
  ------------  ------------------------  ------------------------  ----------
  rhel72        2.7806GB (2915620.0KB)    8GB (8388608.0KB)         failed
  rhel71        2.7806GB (2915620.0KB)    8GB (8388608.0KB)         failed
Verifying Physical Memory ...FAILED (PRVF-7530)
Verifying Available Physical Memory ...
  Node Name     Available                 Required                  Status
  ------------  ------------------------  ------------------------  ----------
  rhel72        2.6736GB (2803492.0KB)    50MB (51200.0KB)          passed
  rhel71        2.4872GB (2608032.0KB)    50MB (51200.0KB)          passed
Verifying Available Physical Memory ...PASSED
Verifying Swap Size ...
  Node Name     Available                 Required                  Status
  ------------  ------------------------  ------------------------  ----------
  rhel72        3.7246GB (3905532.0KB)    2.7806GB (2915620.0KB)    passed
  rhel71        3.7246GB (3905532.0KB)    2.7806GB (2915620.0KB)    passed
Verifying Swap Size ...PASSED
Verifying Free Space: rhel72:/usr,rhel72:/var,rhel72:/etc,rhel72:/opt/app/18.x.0/grid,rhel72:/sbin,rhel72:/tmp ...
  Path              Node Name     Mount point   Available     Required      Status
  ----------------  ------------  ------------  ------------  ------------  ------------
  /usr              rhel72        /             28.9385GB     25MB          passed
  /var              rhel72        /             28.9385GB     5MB           passed
  /etc              rhel72        /             28.9385GB     25MB          passed
  /opt/app/18.x.0/grid  rhel72        /             28.9385GB     6.9GB         passed
  /sbin             rhel72        /             28.9385GB     10MB          passed
  /tmp              rhel72        /             28.9385GB     1GB           passed
Verifying Free Space: rhel72:/usr,rhel72:/var,rhel72:/etc,rhel72:/opt/app/18.x.0/grid,rhel72:/sbin,rhel72:/tmp ...PASSED
Verifying Free Space: rhel71:/usr,rhel71:/var,rhel71:/etc,rhel71:/opt/app/18.x.0/grid,rhel71:/sbin,rhel71:/tmp ...
  Path              Node Name     Mount point   Available     Required      Status
  ----------------  ------------  ------------  ------------  ------------  ------------
  /usr              rhel71        /             18.9439GB     25MB          passed
  /var              rhel71        /             18.9439GB     5MB           passed
  /etc              rhel71        /             18.9439GB     25MB          passed
  /opt/app/18.x.0/grid  rhel71        /             18.9439GB     6.9GB         passed
  /sbin             rhel71        /             18.9439GB     10MB          passed
  /tmp              rhel71        /             18.9439GB     1GB           passed
Verifying Free Space: rhel71:/usr,rhel71:/var,rhel71:/etc,rhel71:/opt/app/18.x.0/grid,rhel71:/sbin,rhel71:/tmp ...PASSED
Verifying User Existence: grid ...
  Node Name     Status                    Comment
  ------------  ------------------------  ------------------------
  rhel72        passed                    exists(502)
  rhel71        passed                    exists(502)

  Verifying Users With Same UID: 502 ...PASSED
Verifying User Existence: grid ...PASSED
Verifying Group Existence: asmadmin ...
  Node Name     Status                    Comment
  ------------  ------------------------  ------------------------
  rhel72        passed                    exists
  rhel71        passed                    exists
Verifying Group Existence: asmadmin ...PASSED
Verifying Group Existence: asmoper ...
  Node Name     Status                    Comment
  ------------  ------------------------  ------------------------
  rhel72        passed                    exists
  rhel71        passed                    exists
Verifying Group Existence: asmoper ...PASSED
Verifying Group Existence: asmdba ...
  Node Name     Status                    Comment
  ------------  ------------------------  ------------------------
  rhel72        passed                    exists
  rhel71        passed                    exists
Verifying Group Existence: asmdba ...PASSED
Verifying Group Existence: oinstall ...
  Node Name     Status                    Comment
  ------------  ------------------------  ------------------------
  rhel72        passed                    exists
  rhel71        passed                    exists
Verifying Group Existence: oinstall ...PASSED
Verifying Group Membership: asmdba ...
  Node Name         User Exists   Group Exists  User in Group  Status
  ----------------  ------------  ------------  ------------  ----------------
  rhel72            yes           yes           yes           passed
  rhel71            yes           yes           yes           passed
Verifying Group Membership: asmdba ...PASSED
Verifying Group Membership: asmadmin ...
  Node Name         User Exists   Group Exists  User in Group  Status
  ----------------  ------------  ------------  ------------  ----------------
  rhel72            yes           yes           yes           passed
  rhel71            yes           yes           yes           passed
Verifying Group Membership: asmadmin ...PASSED
Verifying Group Membership: oinstall(Primary) ...
  Node Name         User Exists   Group Exists  User in Group  Primary       Status
  ----------------  ------------  ------------  ------------  ------------  ------------
  rhel72            yes           yes           yes           yes           passed
  rhel71            yes           yes           yes           yes           passed
Verifying Group Membership: oinstall(Primary) ...PASSED
Verifying Group Membership: asmoper ...
  Node Name         User Exists   Group Exists  User in Group  Status
  ----------------  ------------  ------------  ------------  ----------------
  rhel72            yes           yes           yes           passed
  rhel71            yes           yes           yes           passed
Verifying Group Membership: asmoper ...PASSED
Verifying Run Level ...
  Node Name     run level                 Required                  Status
  ------------  ------------------------  ------------------------  ----------
  rhel72        3                         3,5                       passed
  rhel71        3                         3,5                       passed
Verifying Run Level ...PASSED
Verifying Hard Limit: maximum open file descriptors ...
  Node Name         Type          Available     Required      Status
  ----------------  ------------  ------------  ------------  ----------------
  rhel72            hard          65536         65536         passed
  rhel71            hard          65536         65536         passed
Verifying Hard Limit: maximum open file descriptors ...PASSED
Verifying Soft Limit: maximum open file descriptors ...
  Node Name         Type          Available     Required      Status
  ----------------  ------------  ------------  ------------  ----------------
  rhel72            soft          65536         1024          passed
  rhel71            soft          65536         1024          passed
Verifying Soft Limit: maximum open file descriptors ...PASSED
Verifying Hard Limit: maximum user processes ...
  Node Name         Type          Available     Required      Status
  ----------------  ------------  ------------  ------------  ----------------
  rhel72            hard          16384         16384         passed
  rhel71            hard          16384         16384         passed
Verifying Hard Limit: maximum user processes ...PASSED
Verifying Soft Limit: maximum user processes ...
  Node Name         Type          Available     Required      Status
  ----------------  ------------  ------------  ------------  ----------------
  rhel72            soft          16384         2047          passed
  rhel71            soft          16384         2047          passed
Verifying Soft Limit: maximum user processes ...PASSED
Verifying Soft Limit: maximum stack size ...
  Node Name         Type          Available     Required      Status
  ----------------  ------------  ------------  ------------  ----------------
  rhel72            soft          10240         10240         passed
  rhel71            soft          10240         10240         passed
Verifying Soft Limit: maximum stack size ...PASSED
Verifying Architecture ...
  Node Name     Available                 Required                  Status
  ------------  ------------------------  ------------------------  ----------
  rhel72        x86_64                    x86_64                    passed
  rhel71        x86_64                    x86_64                    passed
Verifying Architecture ...PASSED
Verifying OS Kernel Version ...
  Node Name     Available                 Required                  Status
  ------------  ------------------------  ------------------------  ----------
  rhel72        3.10.0-693.el7.x86_64     3.10.0                    passed
  rhel71        3.10.0-693.el7.x86_64     3.10.0                    passed
Verifying OS Kernel Version ...PASSED
Verifying OS Kernel Parameter: semmsl ...
  Node Name         Current       Configured    Required      Status        Comment
  ----------------  ------------  ------------  ------------  ------------  ------------
  rhel71            3010          3010          250           passed
  rhel72            3010          3010          250           passed
Verifying OS Kernel Parameter: semmsl ...PASSED
Verifying OS Kernel Parameter: semmns ...
  Node Name         Current       Configured    Required      Status        Comment
  ----------------  ------------  ------------  ------------  ------------  ------------
  rhel71            385280        385280        32000         passed
  rhel72            385280        385280        32000         passed
Verifying OS Kernel Parameter: semmns ...PASSED
Verifying OS Kernel Parameter: semopm ...
  Node Name         Current       Configured    Required      Status        Comment
  ----------------  ------------  ------------  ------------  ------------  ------------
  rhel71            3010          3010          100           passed
  rhel72            3010          3010          100           passed
Verifying OS Kernel Parameter: semopm ...PASSED
Verifying OS Kernel Parameter: semmni ...
  Node Name         Current       Configured    Required      Status        Comment
  ----------------  ------------  ------------  ------------  ------------  ------------
  rhel71            128           128           128           passed
  rhel72            128           128           128           passed
Verifying OS Kernel Parameter: semmni ...PASSED
Verifying OS Kernel Parameter: shmmax ...
  Node Name         Current       Configured    Required      Status        Comment
  ----------------  ------------  ------------  ------------  ------------  ------------
  rhel71            4398046511104  4398046511104  1492797440    passed
  rhel72            4398046511104  4398046511104  1492797440    passed
Verifying OS Kernel Parameter: shmmax ...PASSED
Verifying OS Kernel Parameter: shmmni ...
  Node Name         Current       Configured    Required      Status        Comment
  ----------------  ------------  ------------  ------------  ------------  ------------
  rhel71            4096          4096          4096          passed
  rhel72            4096          4096          4096          passed
Verifying OS Kernel Parameter: shmmni ...PASSED
Verifying OS Kernel Parameter: shmall ...
  Node Name         Current       Configured    Required      Status        Comment
  ----------------  ------------  ------------  ------------  ------------  ------------
  rhel71            1073741824    1073741824    1073741824    passed
  rhel72            1073741824    1073741824    1073741824    passed
Verifying OS Kernel Parameter: shmall ...PASSED
Verifying OS Kernel Parameter: file-max ...
  Node Name         Current       Configured    Required      Status        Comment
  ----------------  ------------  ------------  ------------  ------------  ------------
  rhel71            6815744       6815744       6815744       passed
  rhel72            6815744       6815744       6815744       passed
Verifying OS Kernel Parameter: file-max ...PASSED
Verifying OS Kernel Parameter: ip_local_port_range ...
  Node Name         Current       Configured    Required      Status        Comment
  ----------------  ------------  ------------  ------------  ------------  ------------
  rhel71            between 9000 & 65500  between 9000 & 65500  between 9000 & 65535  passed
  rhel72            between 9000 & 65500  between 9000 & 65500  between 9000 & 65535  passed
Verifying OS Kernel Parameter: ip_local_port_range ...PASSED
Verifying OS Kernel Parameter: rmem_default ...
  Node Name         Current       Configured    Required      Status        Comment
  ----------------  ------------  ------------  ------------  ------------  ------------
  rhel71            4194304       4194304       262144        passed
  rhel72            4194304       4194304       262144        passed
Verifying OS Kernel Parameter: rmem_default ...PASSED
Verifying OS Kernel Parameter: rmem_max ...
  Node Name         Current       Configured    Required      Status        Comment
  ----------------  ------------  ------------  ------------  ------------  ------------
  rhel71            4194304       4194304       4194304       passed
  rhel72            4194304       4194304       4194304       passed
Verifying OS Kernel Parameter: rmem_max ...PASSED
Verifying OS Kernel Parameter: wmem_default ...
  Node Name         Current       Configured    Required      Status        Comment
  ----------------  ------------  ------------  ------------  ------------  ------------
  rhel71            1048576       1048576       262144        passed
  rhel72            1048576       1048576       262144        passed
Verifying OS Kernel Parameter: wmem_default ...PASSED
Verifying OS Kernel Parameter: wmem_max ...
  Node Name         Current       Configured    Required      Status        Comment
  ----------------  ------------  ------------  ------------  ------------  ------------
  rhel71            2097152       2097152       1048576       passed
  rhel72            2097152       2097152       1048576       passed
Verifying OS Kernel Parameter: wmem_max ...PASSED
Verifying OS Kernel Parameter: aio-max-nr ...
  Node Name         Current       Configured    Required      Status        Comment
  ----------------  ------------  ------------  ------------  ------------  ------------
  rhel71            3145728       3145728       1048576       passed
  rhel72            3145728       3145728       1048576       passed
Verifying OS Kernel Parameter: aio-max-nr ...PASSED
Verifying Package: binutils-2.23.52.0.1 ...
  Node Name     Available                 Required                  Status
  ------------  ------------------------  ------------------------  ----------
  rhel72        binutils-2.25.1-31.base.el7  binutils-2.23.52.0.1      passed
  rhel71        binutils-2.25.1-31.base.el7  binutils-2.23.52.0.1      passed
Verifying Package: binutils-2.23.52.0.1 ...PASSED
Verifying Package: compat-libcap1-1.10 ...
  Node Name     Available                 Required                  Status
  ------------  ------------------------  ------------------------  ----------
  rhel72        compat-libcap1-1.10-7.el7  compat-libcap1-1.10       passed
  rhel71        compat-libcap1-1.10-7.el7  compat-libcap1-1.10       passed
Verifying Package: compat-libcap1-1.10 ...PASSED
Verifying Package: libgcc-4.8.2 (x86_64) ...
  Node Name     Available                 Required                  Status
  ------------  ------------------------  ------------------------  ----------
  rhel72        libgcc(x86_64)-4.8.5-16.el7  libgcc(x86_64)-4.8.2      passed
  rhel71        libgcc(x86_64)-4.8.5-16.el7  libgcc(x86_64)-4.8.2      passed
Verifying Package: libgcc-4.8.2 (x86_64) ...PASSED
Verifying Package: libstdc++-4.8.2 (x86_64) ...
  Node Name     Available                 Required                  Status
  ------------  ------------------------  ------------------------  ----------
  rhel72        libstdc++(x86_64)-4.8.5-16.el7  libstdc++(x86_64)-4.8.2   passed
  rhel71        libstdc++(x86_64)-4.8.5-16.el7  libstdc++(x86_64)-4.8.2   passed
Verifying Package: libstdc++-4.8.2 (x86_64) ...PASSED
Verifying Package: libstdc++-devel-4.8.2 (x86_64) ...
  Node Name     Available                 Required                  Status
  ------------  ------------------------  ------------------------  ----------
  rhel72        libstdc++-devel(x86_64)-4.8.5-16.el7  libstdc++-devel(x86_64)-4.8.2  passed
  rhel71        libstdc++-devel(x86_64)-4.8.5-16.el7  libstdc++-devel(x86_64)-4.8.2  passed
Verifying Package: libstdc++-devel-4.8.2 (x86_64) ...PASSED
Verifying Package: sysstat-10.1.5 ...
  Node Name     Available                 Required                  Status
  ------------  ------------------------  ------------------------  ----------
  rhel72        sysstat-10.1.5-12.el7     sysstat-10.1.5            passed
  rhel71        sysstat-10.1.5-12.el7     sysstat-10.1.5            passed
Verifying Package: sysstat-10.1.5 ...PASSED
Verifying Package: gcc-c++-4.8.2 ...
  Node Name     Available                 Required                  Status
  ------------  ------------------------  ------------------------  ----------
  rhel72        gcc-c++-4.8.5-16.el7      gcc-c++-4.8.2             passed
  rhel71        gcc-c++-4.8.5-16.el7      gcc-c++-4.8.2             passed
Verifying Package: gcc-c++-4.8.2 ...PASSED
Verifying Package: ksh ...
  Node Name     Available                 Required                  Status
  ------------  ------------------------  ------------------------  ----------
  rhel72        ksh                       ksh                       passed
  rhel71        ksh                       ksh                       passed
Verifying Package: ksh ...PASSED
Verifying Package: make-3.82 ...
  Node Name     Available                 Required                  Status
  ------------  ------------------------  ------------------------  ----------
  rhel72        make-3.82-23.el7          make-3.82                 passed
  rhel71        make-3.82-23.el7          make-3.82                 passed
Verifying Package: make-3.82 ...PASSED
Verifying Package: glibc-2.17 (x86_64) ...
  Node Name     Available                 Required                  Status
  ------------  ------------------------  ------------------------  ----------
  rhel72        glibc(x86_64)-2.17-196.el7  glibc(x86_64)-2.17        passed
  rhel71        glibc(x86_64)-2.17-196.el7  glibc(x86_64)-2.17        passed
Verifying Package: glibc-2.17 (x86_64) ...PASSED
Verifying Package: glibc-devel-2.17 (x86_64) ...
  Node Name     Available                 Required                  Status
  ------------  ------------------------  ------------------------  ----------
  rhel72        glibc-devel(x86_64)-2.17-196.el7  glibc-devel(x86_64)-2.17  passed
  rhel71        glibc-devel(x86_64)-2.17-196.el7  glibc-devel(x86_64)-2.17  passed
Verifying Package: glibc-devel-2.17 (x86_64) ...PASSED
Verifying Package: libaio-0.3.109 (x86_64) ...
  Node Name     Available                 Required                  Status
  ------------  ------------------------  ------------------------  ----------
  rhel72        libaio(x86_64)-0.3.109-13.el7  libaio(x86_64)-0.3.109    passed
  rhel71        libaio(x86_64)-0.3.109-13.el7  libaio(x86_64)-0.3.109    passed
Verifying Package: libaio-0.3.109 (x86_64) ...PASSED
Verifying Package: libaio-devel-0.3.109 (x86_64) ...
  Node Name     Available                 Required                  Status
  ------------  ------------------------  ------------------------  ----------
  rhel72        libaio-devel(x86_64)-0.3.109-13.el7  libaio-devel(x86_64)-0.3.109  passed
  rhel71        libaio-devel(x86_64)-0.3.109-13.el7  libaio-devel(x86_64)-0.3.109  passed
Verifying Package: libaio-devel-0.3.109 (x86_64) ...PASSED
Verifying Package: nfs-utils-1.2.3-15 ...
  Node Name     Available                 Required                  Status
  ------------  ------------------------  ------------------------  ----------
  rhel72        nfs-utils-1.3.0-0.48.el7  nfs-utils-1.2.3-15        passed
  rhel71        nfs-utils-1.3.0-0.48.el7  nfs-utils-1.2.3-15        passed
Verifying Package: nfs-utils-1.2.3-15 ...PASSED
Verifying Package: smartmontools-6.2-4 ...
  Node Name     Available                 Required                  Status
  ------------  ------------------------  ------------------------  ----------
  rhel72        smartmontools-6.2-8.el7   smartmontools-6.2-4       passed
  rhel71        smartmontools-6.2-8.el7   smartmontools-6.2-4       passed
Verifying Package: smartmontools-6.2-4 ...PASSED
Verifying Package: net-tools-2.0-0.17 ...
  Node Name     Available                 Required                  Status
  ------------  ------------------------  ------------------------  ----------
  rhel72        net-tools-2.0-0.22.20131004git.el7  net-tools-2.0-0.17        passed
  rhel71        net-tools-2.0-0.22.20131004git.el7  net-tools-2.0-0.17        passed
Verifying Package: net-tools-2.0-0.17 ...PASSED
Verifying Package: compat-libstdc++-33-3.2.3 (x86_64) ...
  Node Name     Available                 Required                  Status
  ------------  ------------------------  ------------------------  ----------
  rhel72        compat-libstdc++-33(x86_64)-3.2.3-72.el7  compat-libstdc++-33(x86_64)-3.2.3  passed
  rhel71        compat-libstdc++-33(x86_64)-3.2.3-72.el7  compat-libstdc++-33(x86_64)-3.2.3  passed
Verifying Package: compat-libstdc++-33-3.2.3 (x86_64) ...PASSED
Verifying Package: libxcb-1.11 (x86_64) ...
  Node Name     Available                 Required                  Status
  ------------  ------------------------  ------------------------  ----------
  rhel72        libxcb(x86_64)-1.12-1.el7  libxcb(x86_64)-1.11       passed
  rhel71        libxcb(x86_64)-1.12-1.el7  libxcb(x86_64)-1.11       passed
Verifying Package: libxcb-1.11 (x86_64) ...PASSED
Verifying Package: libX11-1.6.3 (x86_64) ...
  Node Name     Available                 Required                  Status
  ------------  ------------------------  ------------------------  ----------
  rhel72        libX11(x86_64)-1.6.5-1.el7  libX11(x86_64)-1.6.3      passed
  rhel71        libX11(x86_64)-1.6.5-1.el7  libX11(x86_64)-1.6.3      passed
Verifying Package: libX11-1.6.3 (x86_64) ...PASSED
Verifying Package: libXau-1.0.8 (x86_64) ...
  Node Name     Available                 Required                  Status
  ------------  ------------------------  ------------------------  ----------
  rhel72        libXau(x86_64)-1.0.8-2.1.el7  libXau(x86_64)-1.0.8      passed
  rhel71        libXau(x86_64)-1.0.8-2.1.el7  libXau(x86_64)-1.0.8      passed
Verifying Package: libXau-1.0.8 (x86_64) ...PASSED
Verifying Package: libXi-1.7.4 (x86_64) ...
  Node Name     Available                 Required                  Status
  ------------  ------------------------  ------------------------  ----------
  rhel72        libXi(x86_64)-1.7.9-1.el7  libXi(x86_64)-1.7.4       passed
  rhel71        libXi(x86_64)-1.7.9-1.el7  libXi(x86_64)-1.7.4       passed
Verifying Package: libXi-1.7.4 (x86_64) ...PASSED
Verifying Package: libXtst-1.2.2 (x86_64) ...
  Node Name     Available                 Required                  Status
  ------------  ------------------------  ------------------------  ----------
  rhel72        libXtst(x86_64)-1.2.3-1.el7  libXtst(x86_64)-1.2.2     passed
  rhel71        libXtst(x86_64)-1.2.3-1.el7  libXtst(x86_64)-1.2.2     passed
Verifying Package: libXtst-1.2.2 (x86_64) ...PASSED
Verifying Port Availability for component "Oracle Notification Service (ONS)" ...
  Node Name         Port Number   Protocol      Available     Status
  ----------------  ------------  ------------  ------------  ----------------
  rhel72            6200          TCP           yes           successful
  rhel71            6200          TCP           yes           successful
  rhel72            6100          TCP           yes           successful
  rhel71            6100          TCP           yes           successful
Verifying Port Availability for component "Oracle Notification Service (ONS)" ...PASSED
Verifying Port Availability for component "Oracle Cluster Synchronization Services (CSSD)" ...
  Node Name         Port Number   Protocol      Available     Status
  ----------------  ------------  ------------  ------------  ----------------
  rhel72            42424         TCP           yes           successful
  rhel71            42424         TCP           yes           successful
Verifying Port Availability for component "Oracle Cluster Synchronization Services (CSSD)" ...PASSED
Verifying Users With Same UID: 0 ...PASSED
Verifying Current Group ID ...PASSED
Verifying Root user consistency ...
  Node Name                             Status
  ------------------------------------  ------------------------
  rhel72                                passed
  rhel71                                passed
Verifying Root user consistency ...PASSED
Verifying Package: cvuqdisk-1.0.10-1 ...
  Node Name     Available                 Required                  Status
  ------------  ------------------------  ------------------------  ----------
  rhel72        cvuqdisk-1.0.10-1         cvuqdisk-1.0.10-1         passed
  rhel71        cvuqdisk-1.0.10-1         cvuqdisk-1.0.10-1         passed
Verifying Package: cvuqdisk-1.0.10-1 ...PASSED
Verifying Host name ...PASSED
Verifying Node Connectivity ...
  Verifying Hosts File ...
  Node Name                             Status
  ------------------------------------  ------------------------
  rhel71                                passed
  rhel72                                passed
  Verifying Hosts File ...PASSED

Interface information for node "rhel71"

 Name   IP Address      Subnet          Gateway         Def. Gateway    HW Address        MTU
 ------ --------------- --------------- --------------- --------------- ----------------- ------
 enp0s3 192.168.0.93    192.168.0.0     0.0.0.0         192.168.0.102   08:00:27:BD:32:39 1500
 enp0s8 192.168.1.93    192.168.1.0     0.0.0.0         192.168.0.102   08:00:27:BD:A9:0A 1500

Interface information for node "rhel72"

 Name   IP Address      Subnet          Gateway         Def. Gateway    HW Address        MTU
 ------ --------------- --------------- --------------- --------------- ----------------- ------
 enp0s3 192.168.0.94    192.168.0.0     0.0.0.0         192.168.0.102   08:00:27:31:0F:1F 1500
 enp0s8 192.168.1.94    192.168.1.0     0.0.0.0         192.168.0.102   08:00:27:98:22:A6 1500

Check: MTU consistency of the subnet "192.168.1.0".

  Node              Name          IP Address    Subnet        MTU
  ----------------  ------------  ------------  ------------  ----------------
  rhel71            enp0s8        192.168.1.93  192.168.1.0   1500
  rhel72            enp0s8        192.168.1.94  192.168.1.0   1500

Check: MTU consistency of the subnet "192.168.0.0".

  Node              Name          IP Address    Subnet        MTU
  ----------------  ------------  ------------  ------------  ----------------
  rhel71            enp0s3        192.168.0.93  192.168.0.0   1500
  rhel72            enp0s3        192.168.0.94  192.168.0.0   1500
  Verifying Check that maximum (MTU) size packet goes through subnet ...PASSED

  Source                          Destination                     Connected?
  ------------------------------  ------------------------------  ----------------
  rhel71[enp0s8:192.168.1.93]     rhel72[enp0s8:192.168.1.94]     yes

  Source                          Destination                     Connected?
  ------------------------------  ------------------------------  ----------------
  rhel71[enp0s3:192.168.0.93]     rhel72[enp0s3:192.168.0.94]     yes
  Verifying subnet mask consistency for subnet "192.168.1.0" ...PASSED
  Verifying subnet mask consistency for subnet "192.168.0.0" ...PASSED
Verifying Node Connectivity ...PASSED
Verifying Multicast or broadcast check ...
Checking subnet "192.168.1.0" for multicast communication with multicast group "224.0.0.251"
Verifying Multicast or broadcast check ...PASSED
Verifying Device Checks for ASM ...
  Verifying Package: cvuqdisk-1.0.10-1 ...PASSED
  Verifying ASM device sharedness check ...
    Verifying Shared Storage Accessibility:/dev/oracleasm/gimr,/dev/oracleasm/fra1,/dev/oracleasm/data1,/dev/oracleasm/ocr3,/dev/oracleasm/ocr1,/dev/oracleasm/ocr2 ...PASSED

  Device                                Device Type
  ------------------------------------  ------------------------
  /dev/oracleasm/ocr3                   Disk
  /dev/oracleasm/data1                  Disk
  /dev/oracleasm/ocr2                   Disk
  /dev/oracleasm/fra1                   Disk
  /dev/oracleasm/ocr1                   Disk
  /dev/oracleasm/gimr                   Disk
  Verifying ASM device sharedness check ...PASSED
  Verifying Access Control List check ...
Checking consistency of device owner across all nodes...
Consistency check of device owner for "/dev/oracleasm/gimr" PASSED
Consistency check of device owner for "/dev/oracleasm/fra1" PASSED
Consistency check of device owner for "/dev/oracleasm/ocr3" PASSED
Consistency check of device owner for "/dev/oracleasm/data1" PASSED
Consistency check of device owner for "/dev/oracleasm/ocr1" PASSED
Consistency check of device owner for "/dev/oracleasm/ocr2" PASSED

Checking consistency of device group across all nodes...
Consistency check of device group for "/dev/oracleasm/gimr" PASSED
Consistency check of device group for "/dev/oracleasm/fra1" PASSED
Consistency check of device group for "/dev/oracleasm/ocr3" PASSED
Consistency check of device group for "/dev/oracleasm/data1" PASSED
Consistency check of device group for "/dev/oracleasm/ocr1" PASSED
Consistency check of device group for "/dev/oracleasm/ocr2" PASSED

Checking consistency of device permissions across all nodes...
Consistency check of device permissions for "/dev/oracleasm/gimr" PASSED
Consistency check of device permissions for "/dev/oracleasm/fra1" PASSED
Consistency check of device permissions for "/dev/oracleasm/ocr3" PASSED
Consistency check of device permissions for "/dev/oracleasm/data1" PASSED
Consistency check of device permissions for "/dev/oracleasm/ocr1" PASSED
Consistency check of device permissions for "/dev/oracleasm/ocr2" PASSED
  Verifying Access Control List check ...PASSED
  Verifying I/O scheduler ...PASSED
Verifying Device Checks for ASM ...PASSED
Verifying Network Time Protocol (NTP) ...
  Verifying '/etc/ntp.conf' ...
  Node Name                             File exists?
  ------------------------------------  ------------------------
  rhel72                                no
  rhel71                                no

  Verifying '/etc/ntp.conf' ...PASSED
  Verifying '/etc/chrony.conf' ...
  Node Name                             File exists?
  ------------------------------------  ------------------------
  rhel72                                no
  rhel71                                no

  Verifying '/etc/chrony.conf' ...PASSED
  Verifying '/var/run/ntpd.pid' ...
  Node Name                             File exists?
  ------------------------------------  ------------------------
  rhel72                                no
  rhel71                                no

  Verifying '/var/run/ntpd.pid' ...PASSED
  Verifying '/var/run/chronyd.pid' ...
  Node Name                             File exists?
  ------------------------------------  ------------------------
  rhel72                                no
  rhel71                                no

  Verifying '/var/run/chronyd.pid' ...PASSED
Verifying Network Time Protocol (NTP) ...PASSED
Verifying Same core file name pattern ...PASSED
Verifying User Mask ...
  Node Name     Available                 Required                  Comment
  ------------  ------------------------  ------------------------  ----------
  rhel72        0022                      0022                      passed
  rhel71        0022                      0022                      passed
Verifying User Mask ...PASSED
Verifying User Not In Group "root": grid ...
  Node Name     Status                    Comment
  ------------  ------------------------  ------------------------
  rhel72        passed                    does not exist
  rhel71        passed                    does not exist
Verifying User Not In Group "root": grid ...PASSED
Verifying Time zone consistency ...PASSED
Verifying Time offset between nodes ...PASSED
Verifying resolv.conf Integrity ...
  Node Name                             Status
  ------------------------------------  ------------------------
  rhel71                                passed
  rhel72                                passed

checking response for name "rhel72" from each of the name servers specified in
"/etc/resolv.conf"

  Node Name     Source                    Comment                   Status
  ------------  ------------------------  ------------------------  ----------
  rhel72        192.168.0.66              IPv4                      passed

checking response for name "rhel71" from each of the name servers specified in
"/etc/resolv.conf"

  Node Name     Source                    Comment                   Status
  ------------  ------------------------  ------------------------  ----------
  rhel71        192.168.0.66              IPv4                      passed
Verifying resolv.conf Integrity ...PASSED
Verifying DNS/NIS name service ...PASSED
Verifying Domain Sockets ...PASSED
Verifying /boot mount ...PASSED
Verifying File system mount options for path GI_HOME ...PASSED
Verifying Daemon "avahi-daemon" not configured and running ...
  Node Name     Configured                Status
  ------------  ------------------------  ------------------------
  rhel72        no                        passed
  rhel71        no                        passed

  Node Name     Running?                  Status
  ------------  ------------------------  ------------------------
  rhel72        no                        passed
  rhel71        no                        passed
Verifying Daemon "avahi-daemon" not configured and running ...PASSED
Verifying Daemon "proxyt" not configured and running ...
  Node Name     Configured                Status
  ------------  ------------------------  ------------------------
  rhel72        no                        passed
  rhel71        no                        passed

  Node Name     Running?                  Status
  ------------  ------------------------  ------------------------
  rhel72        no                        passed
  rhel71        no                        passed
Verifying Daemon "proxyt" not configured and running ...PASSED
Verifying Grid Infrastructure home path: /opt/app/18.x.0/grid ...
  Verifying '/opt/app/18.x.0/grid' ...PASSED
Verifying Grid Infrastructure home path: /opt/app/18.x.0/grid ...PASSED
Verifying User Equivalence ...PASSED
Verifying /dev/shm mounted as temporary file system ...FAILED (PRVE-0421)
Verifying File system mount options for path /var ...PASSED
Verifying zeroconf check ...PASSED
Verifying ASM Filter Driver configuration ...PASSED

Pre-check for cluster services setup was unsuccessful on all the nodes.


Failures were encountered during execution of CVU verification request "stage -pre crsinst".

Verifying Physical Memory ...FAILED
rhel72: PRVF-7530 : Sufficient physical memory is not available on node
        "rhel72" [Required physical memory = 8GB (8388608.0KB)]

rhel71: PRVF-7530 : Sufficient physical memory is not available on node
        "rhel71" [Required physical memory = 8GB (8388608.0KB)]

Verifying /dev/shm mounted as temporary file system ...FAILED
rhel72: PRVE-0421 : No entry exists in /etc/fstab for mounting /dev/shm

rhel71: PRVE-0421 : No entry exists in /etc/fstab for mounting /dev/shm
6. From the GI_HOME directory run gridSetup.sh to begin the installation. Select new cluster configuration option.
Select standard cluster (this would create a flex cluster which is the default for all standard clusters).
Set cluster name and scan name. No GNS is setup or used for this installation.
Add node and node VIP names that will be used in the cluster configuration. Importantly all node roles are set as HUB nodes.
Select the network interfaces for public and ASM/Private interconnect.
The OCR and vote disks are stored in ASM but ASM could be configured either block devices or using NFS. This setup uses block devices.
A separate disk group is used to store GIMR.
Create the disk group for OCR files.
Create disk group for GIMR
Set the oracle base location. The GI install location is immutable.
The failed pre-reqs are ignorable for test setup.
Summary
7. When prompted run the root script. Following output from running root.sh on the first node.
./root.sh
Performing root user operation.

Creating /etc/oratab file...
Entries will be added to the /etc/oratab file as needed by
Database Configuration Assistant when a database is created
Finished running generic part of root script.
Now product-specific root actions will be performed.
Relinking oracle with rac_on option
Using configuration parameter file: /opt/app/18.x.0/grid/crs/install/crsconfig_params
The log of current session can be found at:
  /opt/app/oracle/crsdata/rhel71/crsconfig/rootcrs_rhel71_2019-03-19_02-59-46PM.log
2019/03/19 15:00:12 CLSRSC-594: Executing installation step 1 of 20: 'SetupTFA'.
2019/03/19 15:00:12 CLSRSC-4001: Installing Oracle Trace File Analyzer (TFA) Collector.
2019/03/19 15:00:56 CLSRSC-4002: Successfully installed Oracle Trace File Analyzer (TFA) Collector.
2019/03/19 15:00:56 CLSRSC-594: Executing installation step 2 of 20: 'ValidateEnv'.
2019/03/19 15:00:57 CLSRSC-363: User ignored prerequisites during installation
2019/03/19 15:00:57 CLSRSC-594: Executing installation step 3 of 20: 'CheckFirstNode'.
2019/03/19 15:01:01 CLSRSC-594: Executing installation step 4 of 20: 'GenSiteGUIDs'.
2019/03/19 15:01:07 CLSRSC-594: Executing installation step 5 of 20: 'SaveParamFile'.
2019/03/19 15:01:28 CLSRSC-594: Executing installation step 6 of 20: 'SetupOSD'.
2019/03/19 15:01:28 CLSRSC-594: Executing installation step 7 of 20: 'CheckCRSConfig'.
2019/03/19 15:01:28 CLSRSC-594: Executing installation step 8 of 20: 'SetupLocalGPNP'.
2019/03/19 15:02:17 CLSRSC-594: Executing installation step 9 of 20: 'CreateRootCert'.
2019/03/19 15:02:30 CLSRSC-594: Executing installation step 10 of 20: 'ConfigOLR'.
2019/03/19 15:02:58 CLSRSC-594: Executing installation step 11 of 20: 'ConfigCHMOS'.
2019/03/19 15:02:58 CLSRSC-594: Executing installation step 12 of 20: 'CreateOHASD'.
2019/03/19 15:03:17 CLSRSC-594: Executing installation step 13 of 20: 'ConfigOHASD'.
2019/03/19 15:03:17 CLSRSC-330: Adding Clusterware entries to file 'oracle-ohasd.service'
2019/03/19 15:04:47 CLSRSC-594: Executing installation step 14 of 20: 'InstallAFD'.
2019/03/19 15:05:05 CLSRSC-594: Executing installation step 15 of 20: 'InstallACFS'.
CRS-2791: Starting shutdown of Oracle High Availability Services-managed resources on 'rhel71'
CRS-2793: Shutdown of Oracle High Availability Services-managed resources on 'rhel71' has completed
CRS-4133: Oracle High Availability Services has been stopped.
CRS-4123: Oracle High Availability Services has been started.
2019/03/19 15:06:37 CLSRSC-594: Executing installation step 16 of 20: 'InstallKA'.
2019/03/19 15:06:55 CLSRSC-594: Executing installation step 17 of 20: 'InitConfig'.
CRS-2791: Starting shutdown of Oracle High Availability Services-managed resources on 'rhel71'
CRS-2793: Shutdown of Oracle High Availability Services-managed resources on 'rhel71' has completed
CRS-4133: Oracle High Availability Services has been stopped.
CRS-4123: Oracle High Availability Services has been started.
CRS-2672: Attempting to start 'ora.evmd' on 'rhel71'
CRS-2672: Attempting to start 'ora.mdnsd' on 'rhel71'
CRS-2676: Start of 'ora.mdnsd' on 'rhel71' succeeded
CRS-2676: Start of 'ora.evmd' on 'rhel71' succeeded
CRS-2672: Attempting to start 'ora.gpnpd' on 'rhel71'
CRS-2676: Start of 'ora.gpnpd' on 'rhel71' succeeded
CRS-2672: Attempting to start 'ora.cssdmonitor' on 'rhel71'
CRS-2672: Attempting to start 'ora.gipcd' on 'rhel71'
CRS-2676: Start of 'ora.cssdmonitor' on 'rhel71' succeeded
CRS-2676: Start of 'ora.gipcd' on 'rhel71' succeeded
CRS-2672: Attempting to start 'ora.cssd' on 'rhel71'
CRS-2672: Attempting to start 'ora.diskmon' on 'rhel71'
CRS-2676: Start of 'ora.diskmon' on 'rhel71' succeeded
CRS-2676: Start of 'ora.cssd' on 'rhel71' succeeded

[INFO] [DBT-30001] Disk groups created successfully. Check /opt/app/oracle/cfgtoollogs/asmca/asmca-190319PM030741.log for details.


2019/03/19 15:08:59 CLSRSC-482: Running command: '/opt/app/18.x.0/grid/bin/ocrconfig -upgrade grid oinstall'
CRS-2672: Attempting to start 'ora.crf' on 'rhel71'
CRS-2672: Attempting to start 'ora.storage' on 'rhel71'
CRS-2676: Start of 'ora.storage' on 'rhel71' succeeded
CRS-2676: Start of 'ora.crf' on 'rhel71' succeeded
CRS-2672: Attempting to start 'ora.crsd' on 'rhel71'
CRS-2676: Start of 'ora.crsd' on 'rhel71' succeeded
CRS-4256: Updating the profile
Successful addition of voting disk c500a19f07ec4f62bf8ac318166a95ae.
Successful addition of voting disk 9100c10cb7bb4fd1bffbe97b8dc91245.
Successful addition of voting disk 300c2bf8b7094fa6bfaf11e49f826c7e.
Successfully replaced voting disk group with +OCRDG.
CRS-4256: Updating the profile
CRS-4266: Voting file(s) successfully replaced
##  STATE    File Universal Id                File Name Disk group
--  -----    -----------------                --------- ---------
 1. ONLINE   c500a19f07ec4f62bf8ac318166a95ae (/dev/oracleasm/ocr1) [OCRDG]
 2. ONLINE   9100c10cb7bb4fd1bffbe97b8dc91245 (/dev/oracleasm/ocr2) [OCRDG]
 3. ONLINE   300c2bf8b7094fa6bfaf11e49f826c7e (/dev/oracleasm/ocr3) [OCRDG]
Located 3 voting disk(s).
CRS-2791: Starting shutdown of Oracle High Availability Services-managed resources on 'rhel71'
CRS-2673: Attempting to stop 'ora.crsd' on 'rhel71'
CRS-2677: Stop of 'ora.crsd' on 'rhel71' succeeded
CRS-2673: Attempting to stop 'ora.storage' on 'rhel71'
CRS-2673: Attempting to stop 'ora.crf' on 'rhel71'
CRS-2673: Attempting to stop 'ora.drivers.acfs' on 'rhel71'
CRS-2673: Attempting to stop 'ora.mdnsd' on 'rhel71'
CRS-2677: Stop of 'ora.crf' on 'rhel71' succeeded
CRS-2677: Stop of 'ora.storage' on 'rhel71' succeeded
CRS-2673: Attempting to stop 'ora.asm' on 'rhel71'
CRS-2677: Stop of 'ora.drivers.acfs' on 'rhel71' succeeded
CRS-2677: Stop of 'ora.mdnsd' on 'rhel71' succeeded
CRS-2677: Stop of 'ora.asm' on 'rhel71' succeeded
CRS-2673: Attempting to stop 'ora.cluster_interconnect.haip' on 'rhel71'
CRS-2677: Stop of 'ora.cluster_interconnect.haip' on 'rhel71' succeeded
CRS-2673: Attempting to stop 'ora.ctssd' on 'rhel71'
CRS-2673: Attempting to stop 'ora.evmd' on 'rhel71'
CRS-2677: Stop of 'ora.ctssd' on 'rhel71' succeeded
CRS-2677: Stop of 'ora.evmd' on 'rhel71' succeeded
CRS-2673: Attempting to stop 'ora.cssd' on 'rhel71'
CRS-2677: Stop of 'ora.cssd' on 'rhel71' succeeded
CRS-2673: Attempting to stop 'ora.gipcd' on 'rhel71'
CRS-2673: Attempting to stop 'ora.gpnpd' on 'rhel71'
CRS-2677: Stop of 'ora.gipcd' on 'rhel71' succeeded
CRS-2677: Stop of 'ora.gpnpd' on 'rhel71' succeeded
CRS-2793: Shutdown of Oracle High Availability Services-managed resources on 'rhel71' has completed
CRS-4133: Oracle High Availability Services has been stopped.
2019/03/19 15:13:36 CLSRSC-594: Executing installation step 18 of 20: 'StartCluster'.
CRS-4123: Starting Oracle High Availability Services-managed resources
CRS-2672: Attempting to start 'ora.mdnsd' on 'rhel71'
CRS-2672: Attempting to start 'ora.evmd' on 'rhel71'
CRS-2676: Start of 'ora.evmd' on 'rhel71' succeeded
CRS-2676: Start of 'ora.mdnsd' on 'rhel71' succeeded
CRS-2672: Attempting to start 'ora.gpnpd' on 'rhel71'
CRS-2676: Start of 'ora.gpnpd' on 'rhel71' succeeded
CRS-2672: Attempting to start 'ora.gipcd' on 'rhel71'
CRS-2676: Start of 'ora.gipcd' on 'rhel71' succeeded
CRS-2672: Attempting to start 'ora.crf' on 'rhel71'
CRS-2672: Attempting to start 'ora.cssdmonitor' on 'rhel71'
CRS-2676: Start of 'ora.cssdmonitor' on 'rhel71' succeeded
CRS-2672: Attempting to start 'ora.cssd' on 'rhel71'
CRS-2672: Attempting to start 'ora.diskmon' on 'rhel71'
CRS-2676: Start of 'ora.diskmon' on 'rhel71' succeeded
CRS-2676: Start of 'ora.crf' on 'rhel71' succeeded
CRS-2676: Start of 'ora.cssd' on 'rhel71' succeeded
CRS-2672: Attempting to start 'ora.cluster_interconnect.haip' on 'rhel71'
CRS-2672: Attempting to start 'ora.ctssd' on 'rhel71'
CRS-2676: Start of 'ora.ctssd' on 'rhel71' succeeded
CRS-2676: Start of 'ora.cluster_interconnect.haip' on 'rhel71' succeeded
CRS-2672: Attempting to start 'ora.asm' on 'rhel71'
CRS-2676: Start of 'ora.asm' on 'rhel71' succeeded
CRS-2672: Attempting to start 'ora.storage' on 'rhel71'
CRS-2676: Start of 'ora.storage' on 'rhel71' succeeded
CRS-2672: Attempting to start 'ora.crsd' on 'rhel71'
CRS-2676: Start of 'ora.crsd' on 'rhel71' succeeded
CRS-6023: Starting Oracle Cluster Ready Services-managed resources
CRS-6017: Processing resource auto-start for servers: rhel71
CRS-6016: Resource auto-start has completed for server rhel71
CRS-6024: Completed start of Oracle Cluster Ready Services-managed resources
CRS-4123: Oracle High Availability Services has been started.
2019/03/19 15:16:19 CLSRSC-343: Successfully started Oracle Clusterware stack
2019/03/19 15:16:19 CLSRSC-594: Executing installation step 19 of 20: 'ConfigNode'.
CRS-2672: Attempting to start 'ora.ASMNET1LSNR_ASM.lsnr' on 'rhel71'
CRS-2676: Start of 'ora.ASMNET1LSNR_ASM.lsnr' on 'rhel71' succeeded
CRS-2672: Attempting to start 'ora.asm' on 'rhel71'
CRS-2676: Start of 'ora.asm' on 'rhel71' succeeded
CRS-2672: Attempting to start 'ora.OCRDG.dg' on 'rhel71'
CRS-2676: Start of 'ora.OCRDG.dg' on 'rhel71' succeeded
2019/03/19 15:19:48 CLSRSC-594: Executing installation step 20 of 20: 'PostConfig'.

[INFO] [DBT-30001] Disk groups created successfully. Check /opt/app/oracle/cfgtoollogs/asmca/asmca-190319PM031955.log for details.


2019/03/19 15:23:06 CLSRSC-325: Configure Oracle Grid Infrastructure for a Cluster ... succeeded
Root script run on the second (last) node.
./root.sh
Performing root user operation.

Creating /etc/oratab file...
Entries will be added to the /etc/oratab file as needed by
Database Configuration Assistant when a database is created
Finished running generic part of root script.
Now product-specific root actions will be performed.
Relinking oracle with rac_on option
Using configuration parameter file: /opt/app/18.x.0/grid/crs/install/crsconfig_params
The log of current session can be found at:
  /opt/app/oracle/crsdata/rhel72/crsconfig/rootcrs_rhel72_2019-03-19_03-25-21PM.log
2019/03/19 15:25:38 CLSRSC-594: Executing installation step 1 of 20: 'SetupTFA'.
2019/03/19 15:25:38 CLSRSC-4001: Installing Oracle Trace File Analyzer (TFA) Collector.
2019/03/19 15:26:17 CLSRSC-4002: Successfully installed Oracle Trace File Analyzer (TFA) Collector.
2019/03/19 15:26:17 CLSRSC-594: Executing installation step 2 of 20: 'ValidateEnv'.
2019/03/19 15:26:17 CLSRSC-363: User ignored prerequisites during installation
2019/03/19 15:26:17 CLSRSC-594: Executing installation step 3 of 20: 'CheckFirstNode'.
2019/03/19 15:26:19 CLSRSC-594: Executing installation step 4 of 20: 'GenSiteGUIDs'.
2019/03/19 15:26:19 CLSRSC-594: Executing installation step 5 of 20: 'SaveParamFile'.
2019/03/19 15:26:23 CLSRSC-594: Executing installation step 6 of 20: 'SetupOSD'.
2019/03/19 15:26:23 CLSRSC-594: Executing installation step 7 of 20: 'CheckCRSConfig'.
2019/03/19 15:26:25 CLSRSC-594: Executing installation step 8 of 20: 'SetupLocalGPNP'.
2019/03/19 15:26:28 CLSRSC-594: Executing installation step 9 of 20: 'CreateRootCert'.
2019/03/19 15:26:28 CLSRSC-594: Executing installation step 10 of 20: 'ConfigOLR'.
2019/03/19 15:26:31 CLSRSC-594: Executing installation step 11 of 20: 'ConfigCHMOS'.
2019/03/19 15:26:31 CLSRSC-594: Executing installation step 12 of 20: 'CreateOHASD'.
2019/03/19 15:26:34 CLSRSC-594: Executing installation step 13 of 20: 'ConfigOHASD'.
2019/03/19 15:26:35 CLSRSC-330: Adding Clusterware entries to file 'oracle-ohasd.service'
2019/03/19 15:27:53 CLSRSC-594: Executing installation step 14 of 20: 'InstallAFD'.
2019/03/19 15:27:56 CLSRSC-594: Executing installation step 15 of 20: 'InstallACFS'.
CRS-2791: Starting shutdown of Oracle High Availability Services-managed resources on 'rhel72'
CRS-2793: Shutdown of Oracle High Availability Services-managed resources on 'rhel72' has completed
CRS-4133: Oracle High Availability Services has been stopped.
CRS-4123: Oracle High Availability Services has been started.
2019/03/19 15:29:13 CLSRSC-594: Executing installation step 16 of 20: 'InstallKA'.
2019/03/19 15:29:15 CLSRSC-594: Executing installation step 17 of 20: 'InitConfig'.
CRS-2791: Starting shutdown of Oracle High Availability Services-managed resources on 'rhel72'
CRS-2793: Shutdown of Oracle High Availability Services-managed resources on 'rhel72' has completed
CRS-4133: Oracle High Availability Services has been stopped.
CRS-4123: Oracle High Availability Services has been started.
CRS-2791: Starting shutdown of Oracle High Availability Services-managed resources on 'rhel72'
CRS-2673: Attempting to stop 'ora.drivers.acfs' on 'rhel72'
CRS-2677: Stop of 'ora.drivers.acfs' on 'rhel72' succeeded
CRS-2793: Shutdown of Oracle High Availability Services-managed resources on 'rhel72' has completed
CRS-4133: Oracle High Availability Services has been stopped.
2019/03/19 15:29:28 CLSRSC-594: Executing installation step 18 of 20: 'StartCluster'.
CRS-4123: Starting Oracle High Availability Services-managed resources
CRS-2672: Attempting to start 'ora.mdnsd' on 'rhel72'
CRS-2672: Attempting to start 'ora.evmd' on 'rhel72'
CRS-2676: Start of 'ora.mdnsd' on 'rhel72' succeeded
CRS-2676: Start of 'ora.evmd' on 'rhel72' succeeded
CRS-2672: Attempting to start 'ora.gpnpd' on 'rhel72'
CRS-2676: Start of 'ora.gpnpd' on 'rhel72' succeeded
CRS-2672: Attempting to start 'ora.gipcd' on 'rhel72'
CRS-2676: Start of 'ora.gipcd' on 'rhel72' succeeded
CRS-2672: Attempting to start 'ora.crf' on 'rhel72'
CRS-2672: Attempting to start 'ora.cssdmonitor' on 'rhel72'
CRS-2676: Start of 'ora.cssdmonitor' on 'rhel72' succeeded
CRS-2672: Attempting to start 'ora.cssd' on 'rhel72'
CRS-2672: Attempting to start 'ora.diskmon' on 'rhel72'
CRS-2676: Start of 'ora.diskmon' on 'rhel72' succeeded
CRS-2676: Start of 'ora.crf' on 'rhel72' succeeded
CRS-2676: Start of 'ora.cssd' on 'rhel72' succeeded
CRS-2672: Attempting to start 'ora.cluster_interconnect.haip' on 'rhel72'
CRS-2672: Attempting to start 'ora.ctssd' on 'rhel72'
CRS-2676: Start of 'ora.ctssd' on 'rhel72' succeeded
CRS-2672: Attempting to start 'ora.crsd' on 'rhel72'
CRS-2676: Start of 'ora.crsd' on 'rhel72' succeeded
CRS-2676: Start of 'ora.cluster_interconnect.haip' on 'rhel72' succeeded
CRS-2672: Attempting to start 'ora.asm' on 'rhel72'
CRS-2676: Start of 'ora.asm' on 'rhel72' succeeded
CRS-6017: Processing resource auto-start for servers: rhel72
CRS-2673: Attempting to stop 'ora.LISTENER_SCAN1.lsnr' on 'rhel71'
CRS-2672: Attempting to start 'ora.ons' on 'rhel72'
CRS-2672: Attempting to start 'ora.ASMNET1LSNR_ASM.lsnr' on 'rhel72'
CRS-2677: Stop of 'ora.LISTENER_SCAN1.lsnr' on 'rhel71' succeeded
CRS-2673: Attempting to stop 'ora.scan1.vip' on 'rhel71'
CRS-2677: Stop of 'ora.scan1.vip' on 'rhel71' succeeded
CRS-2672: Attempting to start 'ora.scan1.vip' on 'rhel72'
CRS-2676: Start of 'ora.ASMNET1LSNR_ASM.lsnr' on 'rhel72' succeeded
CRS-2672: Attempting to start 'ora.asm' on 'rhel72'
CRS-2676: Start of 'ora.scan1.vip' on 'rhel72' succeeded
CRS-2672: Attempting to start 'ora.LISTENER_SCAN1.lsnr' on 'rhel72'
CRS-2676: Start of 'ora.ons' on 'rhel72' succeeded
CRS-2676: Start of 'ora.LISTENER_SCAN1.lsnr' on 'rhel72' succeeded
CRS-2676: Start of 'ora.asm' on 'rhel72' succeeded
CRS-2672: Attempting to start 'ora.proxy_advm' on 'rhel71'
CRS-2672: Attempting to start 'ora.proxy_advm' on 'rhel72'
CRS-2676: Start of 'ora.proxy_advm' on 'rhel71' succeeded
CRS-2676: Start of 'ora.proxy_advm' on 'rhel72' succeeded
CRS-6016: Resource auto-start has completed for server rhel72
CRS-6024: Completed start of Oracle Cluster Ready Services-managed resources
CRS-4123: Oracle High Availability Services has been started.
2019/03/19 15:32:28 CLSRSC-343: Successfully started Oracle Clusterware stack
2019/03/19 15:32:29 CLSRSC-594: Executing installation step 19 of 20: 'ConfigNode'.
2019/03/19 15:33:03 CLSRSC-594: Executing installation step 20 of 20: 'PostConfig'.
2019/03/19 15:33:13 CLSRSC-325: Configure Oracle Grid Infrastructure for a Cluster ... succeeded
8. Once the root scripts are run click OK on the configuration dialogue box to complete the creation of GIMR and other configuration assistants. At the end this clusterware installation would be completed.



9. Run cluvfy with post crsinst to check the installation.
[grid@rhel71 ~]$ cluvfy stage -post crsinst -allnodes

Verifying Node Connectivity ...
  Verifying Hosts File ...PASSED
  Verifying Check that maximum (MTU) size packet goes through subnet ...PASSED
  Verifying subnet mask consistency for subnet "192.168.1.0" ...PASSED
  Verifying subnet mask consistency for subnet "192.168.0.0" ...PASSED
Verifying Node Connectivity ...PASSED
Verifying Multicast or broadcast check ...PASSED
Verifying ASM filter driver configuration consistency ...PASSED
Verifying Time zone consistency ...PASSED
Verifying Cluster Manager Integrity ...PASSED
Verifying User Mask ...PASSED
Verifying Cluster Integrity ...PASSED
Verifying OCR Integrity ...PASSED
Verifying CRS Integrity ...
  Verifying Clusterware Version Consistency ...PASSED
Verifying CRS Integrity ...PASSED
Verifying Node Application Existence ...PASSED
Verifying Single Client Access Name (SCAN) ...
  Verifying DNS/NIS name service 'rac-scan.domain.net' ...
    Verifying Name Service Switch Configuration File Integrity ...PASSED
  Verifying DNS/NIS name service 'rac-scan.domain.net' ...PASSED
Verifying Single Client Access Name (SCAN) ...PASSED
Verifying OLR Integrity ...PASSED
Verifying Voting Disk ...PASSED
Verifying ASM Integrity ...
  Verifying Node Connectivity ...
    Verifying Hosts File ...PASSED
    Verifying Check that maximum (MTU) size packet goes through subnet ...PASSED
    Verifying subnet mask consistency for subnet "192.168.1.0" ...PASSED
    Verifying subnet mask consistency for subnet "192.168.0.0" ...PASSED
  Verifying Node Connectivity ...PASSED
Verifying ASM Integrity ...PASSED
Verifying Device Checks for ASM ...
  Verifying Access Control List check ...PASSED
  Verifying I/O scheduler ...PASSED
Verifying Device Checks for ASM ...PASSED
Verifying ASM disk group free space ...PASSED
Verifying User Not In Group "root": grid ...PASSED
Verifying Clock Synchronization ...PASSED
Verifying VIP Subnet configuration check ...PASSED
Verifying Network configuration consistency checks ...PASSED
Verifying File system mount options for path GI_HOME ...PASSED
Verifying Access control attributes for /var/tmp/.oracle ...PASSED
Verifying Access control attributes for %OCRCONFIGDIR%/maps ...PASSED

Post-check for cluster services setup was successful.
10. There are few cluster and local resources that are new in 18c compared to 12.2 Most of them relate the rapid home provisioning feature and are in disable state.
 crsctl stat res -t
--------------------------------------------------------------------------------
Name           Target  State        Server                   State details
--------------------------------------------------------------------------------
Local Resources
--------------------------------------------------------------------------------
ora.GIMRDG.GHCHKPT.advm
               OFFLINE OFFLINE      rhel71                   STABLE
...
ora.gimrdg.ghchkpt.acfs
               OFFLINE OFFLINE      rhel71                   STABLE
               OFFLINE OFFLINE      rhel72                   STABLE
ora.helper
               OFFLINE OFFLINE      rhel71                   IDLE,STABLE
               OFFLINE OFFLINE      rhel72                   IDLE,STABLE
ora.proxy_advm
               ONLINE  ONLINE       rhel71                   STABLE
               ONLINE  ONLINE       rhel72                   STABLE
--------------------------------------------------------------------------------
Cluster Resources
--------------------------------------------------------------------------------
...
ora.rhpserver
      1        OFFLINE OFFLINE                               STABLE
--------------------------------------------------------------------------------
For example clusterware installation create a pre-configued ASM volume group call GHCHKPT which is to be used with rapid home provisioning framework.
ASMCMD> volinfo --all
Diskgroup Name: GIMRDG

         Volume Name: GHCHKPT
         Volume Device: /dev/asm/ghchkpt-493
         State: DISABLED
         Size (MB): 1536
         Resize Unit (MB): 512
         Redundancy: UNPROT
         Stripe Columns: 8
         Stripe Width (K): 1024
         Usage: ACFS
         Mountpath: /opt/oracle/rhp_images/chkbase
Because of this volume group the proxy ASM instance will have an active and online status. These would be visible as APX{node number} instances. Oracle docs states "an Oracle ASM proxy instance is an Oracle instance running on a Hub node with a direct Oracle ASM client. An Oracle ASM proxy instance provides support for Oracle Automatic Storage Management Cluster File System (Oracle ACFS) and Oracle ASM Dynamic Volume Manager (Oracle ADVM)". If no other ASM volume groups are to be created or online then ASM proxy instances could be stopped and disabled.
[grid@rhel71 ~]$ srvctl stop asm -proxy -node rhel71
[grid@rhel71 ~]$ srvctl stop asm -proxy -node rhel72
[grid@rhel71 ~]$ srvctl status asm -proxy
ADVM proxy is not running
The full list of resources after the installation is shown below (before proxy instances were disabled).
Resource Name                       Type                      Target             State              Host
-------------                       ------                    -------            --------           ----------
ora.ASMNET1LSNR_ASM.lsnr            ora.asm_listener.type     ONLINE             ONLINE             rhel71
ora.ASMNET1LSNR_ASM.lsnr            ora.asm_listener.type     ONLINE             ONLINE             rhel72
ora.GIMRDG.GHCHKPT.advm             ora.volume.type           OFFLINE            OFFLINE
ora.GIMRDG.GHCHKPT.advm             ora.volume.type           OFFLINE            OFFLINE
ora.GIMRDG.dg                       ora.diskgroup.type        ONLINE             ONLINE             rhel71
ora.GIMRDG.dg                       ora.diskgroup.type        ONLINE             ONLINE             rhel72
ora.LISTENER.lsnr                   ora.listener.type         ONLINE             ONLINE             rhel71
ora.LISTENER.lsnr                   ora.listener.type         ONLINE             ONLINE             rhel72
ora.LISTENER_SCAN1.lsnr             ora.scan_listener.type    ONLINE             ONLINE             rhel72
ora.LISTENER_SCAN2.lsnr             ora.scan_listener.type    ONLINE             ONLINE             rhel71
ora.LISTENER_SCAN3.lsnr             ora.scan_listener.type    ONLINE             ONLINE             rhel71
ora.MGMTLSNR                        ora.mgmtlsnr.type         ONLINE             ONLINE             rhel71
ora.OCRDG.dg                        ora.diskgroup.type        ONLINE             ONLINE             rhel71
ora.OCRDG.dg                        ora.diskgroup.type        ONLINE             ONLINE             rhel72
ora.asm                             ora.asm.type              ONLINE             ONLINE             rhel71
ora.asm                             ora.asm.type              ONLINE             ONLINE             rhel72
ora.asm                             ora.asm.type              OFFLINE            OFFLINE
ora.chad                            ora.chad.type             ONLINE             ONLINE             rhel71
ora.chad                            ora.chad.type             ONLINE             ONLINE             rhel72
ora.cvu                             ora.cvu.type              ONLINE             ONLINE             rhel71
ora.gimrdg.ghchkpt.acfs             ora.acfs.type             OFFLINE            OFFLINE
ora.gimrdg.ghchkpt.acfs             ora.acfs.type             OFFLINE            OFFLINE
ora.helper                          ora.helper.type           OFFLINE            OFFLINE
ora.helper                          ora.helper.type           OFFLINE            OFFLINE
ora.mgmtdb                          ora.mgmtdb.type           ONLINE             ONLINE             rhel71
ora.net1.network                    ora.network.type          ONLINE             ONLINE             rhel71
ora.net1.network                    ora.network.type          ONLINE             ONLINE             rhel72
ora.ons                             ora.ons.type              ONLINE             ONLINE             rhel71
ora.ons                             ora.ons.type              ONLINE             ONLINE             rhel72
ora.proxy_advm                      ora.proxy_advm.type       ONLINE             ONLINE             rhel71
ora.proxy_advm                      ora.proxy_advm.type       ONLINE             ONLINE             rhel72
ora.qosmserver                      ora.qosmserver.type       ONLINE             ONLINE             rhel71
ora.rhel71.vip                      ora.cluster_vip_net1.type ONLINE             ONLINE             rhel71
ora.rhel72.vip                      ora.cluster_vip_net1.type ONLINE             ONLINE             rhel72
ora.rhpserver                       ora.rhpserver.type        OFFLINE            OFFLINE
ora.scan1.vip                       ora.scan_vip.type         ONLINE             ONLINE             rhel72
ora.scan2.vip                       ora.scan_vip.type         ONLINE             ONLINE             rhel71
ora.scan3.vip                       ora.scan_vip.type         ONLINE             ONLINE             rhel71
11. The default backup location for OCR is the disk group used for GIMR
ocrconfig -showbackuploc
The Oracle Cluster Registry backup location is [+GIMRDG].
12. The clusermode and versions are as follows.
$ crsctl get cluster mode status
Cluster is running in "flex" mode

$crsctl get node role config -all
Node 'rhel71' configured role is 'hub'
Node 'rhel72' configured role is 'hub'

$ crsctl query crs activeversion
Oracle Clusterware active version on the cluster is [18.0.0.0.0]
The next step is to install the database software and craete the RAC DB.

Related Posts
Installing 12cR2 (12.2.0.1) RAC on RHEL 6 with Role Separation - Clusterware
Installing 12c (12.1.0.2) Flex Cluster on RHEL 6 with Role Separation
Installing 12c (12.1.0.1) RAC on RHEL 6 with Role Separation - Clusterware
Installing 11gR2 (11.2.0.3) GI with Role Separation on RHEL 6
Installing 11gR2 (11.2.0.3) GI with Role Separation on OEL 6
Installing 11gR2 Standalone Server with ASM and Role Separation on RHEL 6
11gR2 Standalone Data Guard (with ASM and Role Separation)


Update on 2020-10-01
Following error was given when intsalling a role separated Oracle restart on 19c. Error was thrown during the Oracle home installaton.
 ./runInstaller -silent -responseFile /home/oracle/db_install.rsp -ignorePrereqFailure
Launching Oracle Database Setup Wizard...

[FATAL] [INS-30215] Installer has detected that Oracle Grid Infrastructure owner "grid" does not belong to the OS group "racdba" on local node.
   CAUSE: The OS group "racdba" does not exist or Oracle Grid Infrastructure owner "grid" is not a member of the OS group "racdba" on local node.
   ACTION: Ensure that OS group "racdba" exists and add Oracle Grid Infrastructure owner "grid" as a member of the OS group "racdba" on local node or specify a different OS group in which Oracle Grid Infrastructure owner is a member.
*ADDITIONAL INFORMATION:*
 - PRVF-7566 : User "grid" does not belong to group "racdba" on node "ip-172-31-2-77"
 - Cause:  The specified user is not a member of the specified group on the specified node.
 - Action:  Make the user a member of the group on the specified node.

It seems with 19c grid user should also be a memeber of the racdba group.