Showing posts with label role separation. Show all posts
Showing posts with label role separation. Show all posts

Sunday, April 7, 2019

Installing 18c (18.3) RAC on RHEL 7 with Role Separation - Database

Following the 18c clusterware installation the next step is to install database software and create the database. The database software installation is not much different from that of 12.2 database software installation. Even though the oracle installation guides states gcc and gcc-c++ are not required for RHEL, the cluvfy still searches for gcc-c++.
1. Create the Oracle home for the database software and unzip the installer into the directory.
cd $ORACLE_BASE
mkdir -p product/18.x.0/dbhome_1
unzip LINUX.X64_180000_db_home.zip -d /opt/app/oracle/product/18.x.0/dbhome_1/
2. Run cluvfy to check the pre dbinst state. The failures related to shm could be ignored
cluvfy stage -pre dbinst -allnodes -r 18

Verifying Physical Memory ...PASSED
Verifying Available Physical Memory ...PASSED
Verifying Swap Size ...PASSED
Verifying Free Space: rhel72:/tmp ...PASSED
Verifying Free Space: rhel71:/tmp ...PASSED
Verifying User Existence: oracle ...
  Verifying Users With Same UID: 501 ...PASSED
Verifying User Existence: oracle ...PASSED
Verifying Group Existence: dba ...PASSED
Verifying Group Existence: oinstall ...PASSED
Verifying Group Membership: oinstall(Primary) ...PASSED
Verifying Group Membership: dba ...PASSED
Verifying Run Level ...PASSED
Verifying Hard Limit: maximum open file descriptors ...PASSED
Verifying Soft Limit: maximum open file descriptors ...PASSED
Verifying Hard Limit: maximum user processes ...PASSED
Verifying Soft Limit: maximum user processes ...PASSED
Verifying Soft Limit: maximum stack size ...PASSED
Verifying Architecture ...PASSED
Verifying OS Kernel Version ...PASSED
Verifying OS Kernel Parameter: semmsl ...PASSED
Verifying OS Kernel Parameter: semmns ...PASSED
Verifying OS Kernel Parameter: semopm ...PASSED
Verifying OS Kernel Parameter: semmni ...PASSED
Verifying OS Kernel Parameter: shmmax ...PASSED
Verifying OS Kernel Parameter: shmmni ...PASSED
Verifying OS Kernel Parameter: shmall ...PASSED
Verifying OS Kernel Parameter: file-max ...PASSED
Verifying OS Kernel Parameter: ip_local_port_range ...PASSED
Verifying OS Kernel Parameter: rmem_default ...PASSED
Verifying OS Kernel Parameter: rmem_max ...PASSED
Verifying OS Kernel Parameter: wmem_default ...PASSED
Verifying OS Kernel Parameter: wmem_max ...PASSED
Verifying OS Kernel Parameter: aio-max-nr ...PASSED
Verifying Package: binutils-2.23.52.0.1 ...PASSED
Verifying Package: compat-libcap1-1.10 ...PASSED
Verifying Package: libgcc-4.8.2 (x86_64) ...PASSED
Verifying Package: libstdc++-4.8.2 (x86_64) ...PASSED
Verifying Package: libstdc++-devel-4.8.2 (x86_64) ...PASSED
Verifying Package: sysstat-10.1.5 ...PASSED
Verifying Package: gcc-c++-4.8.2 ...PASSED
Verifying Package: ksh ...PASSED
Verifying Package: make-3.82 ...PASSED
Verifying Package: glibc-2.17 (x86_64) ...PASSED
Verifying Package: glibc-devel-2.17 (x86_64) ...PASSED
Verifying Package: libaio-0.3.109 (x86_64) ...PASSED
Verifying Package: libaio-devel-0.3.109 (x86_64) ...PASSED
Verifying Package: smartmontools-6.2-4 ...PASSED
Verifying Package: net-tools-2.0-0.17 ...PASSED
Verifying Package: compat-libstdc++-33-3.2.3 (x86_64) ...PASSED
Verifying Package: libxcb-1.11 (x86_64) ...PASSED
Verifying Package: libX11-1.6.3 (x86_64) ...PASSED
Verifying Package: libXau-1.0.8 (x86_64) ...PASSED
Verifying Package: libXi-1.7.4 (x86_64) ...PASSED
Verifying Package: libXtst-1.2.2 (x86_64) ...PASSED
Verifying Users With Same UID: 0 ...PASSED
Verifying Current Group ID ...PASSED
Verifying Root user consistency ...PASSED
Verifying Host name ...PASSED
Verifying Node Connectivity ...
  Verifying Hosts File ...PASSED
  Verifying Check that maximum (MTU) size packet goes through subnet ...PASSED
  Verifying subnet mask consistency for subnet "192.168.1.0" ...PASSED
  Verifying subnet mask consistency for subnet "192.168.0.0" ...PASSED
Verifying Node Connectivity ...PASSED
Verifying Multicast or broadcast check ...PASSED
Verifying User Mask ...PASSED
Verifying CRS Integrity ...
  Verifying Clusterware Version Consistency ...PASSED
Verifying CRS Integrity ...PASSED
Verifying Cluster Manager Integrity ...PASSED
Verifying Node Application Existence ...PASSED
Verifying Clock Synchronization ...PASSED
Verifying resolv.conf Integrity ...PASSED
Verifying Time zone consistency ...PASSED
Verifying Single Client Access Name (SCAN) ...
  Verifying DNS/NIS name service 'rac-scan.domain.net' ...
    Verifying Name Service Switch Configuration File Integrity ...PASSED
  Verifying DNS/NIS name service 'rac-scan.domain.net' ...PASSED
Verifying Single Client Access Name (SCAN) ...PASSED
Verifying VIP Subnet configuration check ...PASSED
Verifying Database Clusterware Version Compatibility ...PASSED
Verifying ASM storage privileges for the user: oracle ...
  Verifying Group Membership: asmdba ...PASSED
Verifying ASM storage privileges for the user: oracle ...PASSED
Verifying Daemon "proxyt" not configured and running ...PASSED
Verifying ACFS device special file ...PASSED
Verifying /dev/shm mounted as temporary file system ...FAILED (PRVE-0421)
Verifying Maximum locked memory check ...PASSED

Pre-check for database installation was unsuccessful on all the nodes.


Failures were encountered during execution of CVU verification request "stage -pre dbinst".

Verifying /dev/shm mounted as temporary file system ...FAILED
rhel72: PRVE-0421 : No entry exists in /etc/fstab for mounting /dev/shm

rhel71: PRVE-0421 : No entry exists in /etc/fstab for mounting /dev/shm
3. Execute runInstaller from the Oracle home and select software only install option.
Other steps are similar to 12.2. However, one noticeable change is that in 18c the space available for Oracle base is also checked as part of pre-reqs. Any warning raised for test system could be ignored.
4. Once the database software is installed proceed to creating the database. As this is a role separate setup, certain folders need write permission for the oinstall group so oracle user could write into them.
ls -l
total 0
drwxr-x---.  5 grid oinstall  45 Mar 20 11:50 admin
drwxr-x---.  2 grid oinstall   6 Mar 20 11:51 audit

[grid@rhel71 oracle]$ chmod 770 admin audit

[grid@rhel71 oracle]$ ls -l
total 0
drwxrwx---.  5 grid oinstall  45 Mar 20 11:50 admin
drwxrwx---.  2 grid oinstall   6 Mar 20 11:51 audit

chmod 770 $ORACLE_BASE/cfgtoollogs/dbca
5. Run cluvfy could to check pre-requisite for RAC DB (output is without verbose option).
cluvfy stage -pre dbcfg -allnodes -d /opt/app/oracle/product/18.x.0/dbhome_1

Verifying Physical Memory ...PASSED
Verifying Available Physical Memory ...PASSED
Verifying Swap Size ...PASSED
Verifying Free Space: rhel72:/tmp ...PASSED
Verifying Free Space: rhel71:/tmp ...PASSED
Verifying User Existence: oracle ...
  Verifying Users With Same UID: 501 ...PASSED
Verifying User Existence: oracle ...PASSED
Verifying Group Existence: dgdba ...PASSED
Verifying Group Existence: dba ...PASSED
Verifying Group Existence: racdba ...PASSED
Verifying Group Existence: backupdba ...PASSED
Verifying Group Existence: oper ...PASSED
Verifying Group Existence: oinstall ...PASSED
Verifying Group Membership: backupdba ...PASSED
Verifying Group Membership: racdba ...PASSED
Verifying Group Membership: dgdba ...PASSED
Verifying Group Membership: oinstall(Primary) ...PASSED
Verifying Group Membership: dba ...PASSED
Verifying Group Membership: oper ...PASSED
Verifying Run Level ...PASSED
Verifying Hard Limit: maximum open file descriptors ...PASSED
Verifying Soft Limit: maximum open file descriptors ...PASSED
Verifying Hard Limit: maximum user processes ...PASSED
Verifying Soft Limit: maximum user processes ...PASSED
Verifying Soft Limit: maximum stack size ...PASSED
Verifying Architecture ...PASSED
Verifying OS Kernel Version ...PASSED
Verifying OS Kernel Parameter: semmsl ...PASSED
Verifying OS Kernel Parameter: semmns ...PASSED
Verifying OS Kernel Parameter: semopm ...PASSED
Verifying OS Kernel Parameter: semmni ...PASSED
Verifying OS Kernel Parameter: shmmax ...PASSED
Verifying OS Kernel Parameter: shmmni ...PASSED
Verifying OS Kernel Parameter: shmall ...PASSED
Verifying OS Kernel Parameter: file-max ...PASSED
Verifying OS Kernel Parameter: ip_local_port_range ...PASSED
Verifying OS Kernel Parameter: rmem_default ...PASSED
Verifying OS Kernel Parameter: rmem_max ...PASSED
Verifying OS Kernel Parameter: wmem_default ...PASSED
Verifying OS Kernel Parameter: wmem_max ...PASSED
Verifying OS Kernel Parameter: aio-max-nr ...PASSED
Verifying Package: binutils-2.23.52.0.1 ...PASSED
Verifying Package: compat-libcap1-1.10 ...PASSED
Verifying Package: libgcc-4.8.2 (x86_64) ...PASSED
Verifying Package: libstdc++-4.8.2 (x86_64) ...PASSED
Verifying Package: libstdc++-devel-4.8.2 (x86_64) ...PASSED
Verifying Package: sysstat-10.1.5 ...PASSED
Verifying Package: gcc-c++-4.8.2 ...PASSED
Verifying Package: ksh ...PASSED
Verifying Package: make-3.82 ...PASSED
Verifying Package: glibc-2.17 (x86_64) ...PASSED
Verifying Package: glibc-devel-2.17 (x86_64) ...PASSED
Verifying Package: libaio-0.3.109 (x86_64) ...PASSED
Verifying Package: libaio-devel-0.3.109 (x86_64) ...PASSED
Verifying Package: smartmontools-6.2-4 ...PASSED
Verifying Package: net-tools-2.0-0.17 ...PASSED
Verifying Package: compat-libstdc++-33-3.2.3 (x86_64) ...PASSED
Verifying Package: libxcb-1.11 (x86_64) ...PASSED
Verifying Package: libX11-1.6.3 (x86_64) ...PASSED
Verifying Package: libXau-1.0.8 (x86_64) ...PASSED
Verifying Package: libXi-1.7.4 (x86_64) ...PASSED
Verifying Package: libXtst-1.2.2 (x86_64) ...PASSED
Verifying Current Group ID ...PASSED
Verifying CRS Integrity ...
  Verifying Clusterware Version Consistency ...PASSED
Verifying CRS Integrity ...PASSED
Verifying Node Application Existence ...PASSED
Verifying Time zone consistency ...PASSED
Verifying Single Client Access Name (SCAN) ...
  Verifying DNS/NIS name service 'rac-scan.domain.net' ...
    Verifying Name Service Switch Configuration File Integrity ...PASSED
  Verifying DNS/NIS name service 'rac-scan.domain.net' ...PASSED
Verifying Single Client Access Name (SCAN) ...PASSED
Verifying ASM Integrity ...
  Verifying Node Connectivity ...
    Verifying Hosts File ...PASSED
    Verifying Check that maximum (MTU) size packet goes through subnet ...PASSED
    Verifying subnet mask consistency for subnet "192.168.1.0" ...PASSED
    Verifying subnet mask consistency for subnet "192.168.0.0" ...PASSED
  Verifying Node Connectivity ...PASSED
Verifying ASM Integrity ...PASSED
Verifying Database Clusterware Version Compatibility ...PASSED
Verifying File system mount options for path ORACLE_HOME ...PASSED
Verifying /dev/shm mounted as temporary file system ...FAILED (PRVE-0421)

Pre-check for database configuration was unsuccessful on all the nodes.


Failures were encountered during execution of CVU verification request "stage -pre dbcfg".

Verifying /dev/shm mounted as temporary file system ...FAILED
rhel72: PRVE-0421 : No entry exists in /etc/fstab for mounting /dev/shm

rhel71: PRVE-0421 : No entry exists in /etc/fstab for mounting /dev/shm
6. Create the database using dbca.
7. Create PDBs are required and associate services with them so PDBs are auto started. Assuming PDB named pdb18c is created, add a database service associated with PDB with
srvctl add service -db ent18c -pdb pdb18c -service pdbsrv -preferred "ent18c1,ent18c2"
srvctl start service -db ent18c -s pdbsrv
8. Finally, the stopping of the ASM proxy instances has no relevance to the "flex" functioning of DB instances. If an ASM instances in one node goes down, these DB instances are able to use a remote ASM instance to access the disks. This could be verified with the following.
[grid@rhel71 ~]$  srvctl status asm -detail
ASM is running on rhel71,rhel72
ASM is enabled.
ASM instance +ASM1 is running on node rhel71
Number of connected clients: 2
Client names: ent18c1:ent18c:rhel7-cluster rhel71.domain.net:_OCR:rhel7-cluster
ASM instance +ASM2 is running on node rhel72
Number of connected clients: 3
Client names: -MGMTDB:_mgmtdb:rhel7-cluster ent18c2:ent18c:rhel7-cluster rhel72.domain.net:_OCR:rhel7-cluster
The above output shows the clients connected to each ASM instance on each node. Currently ASM1 has 2 clients while ASM2 has 3 clients, *OCR is available as client on both ASM instances.
Then an ASM instance is stopped.
srvctl stop asm  -node rhel72  -force
When one of the ASM instances is stopped forcefully those clients start using the service of the available ASM instance
srvctl status asm -detail
ASM is running on rhel71
ASM is enabled.
ASM instance +ASM1 is running on node rhel71
Number of connected clients: 4
Client names: -MGMTDB:_mgmtdb:rhel7-cluster ent18c1:ent18c:rhel7-cluster ent18c2:ent18c:rhel7-cluster rhel71.domain.net:_OCR:rhel7-cluster
As seen above, clients that previously clients of ASM2 instance are now using ASM1 instance to access ASM disks. (total client count is 4 not 5 due to not duplicating the OCR* client).
When ASM instance is restarted the clients automatically select the local ASM instance over remote ASM instances.
srvctl start asm  -node rhel72
srvctl status asm -detail
ASM is running on rhel71,rhel72
ASM is enabled.
ASM instance +ASM1 is running on node rhel71
Number of connected clients: 2
Client names: ent18c1:ent18c:rhel7-cluster rhel71.domain.net:_OCR:rhel7-cluster
ASM instance +ASM2 is running on node rhel72
Number of connected clients: 3
Client names: -MGMTDB:_mgmtdb:rhel7-cluster ent18c2:ent18c:rhel7-cluster rhel72.domain.net:_OCR:rhel7-cluster
This concludes the setting up RAC DB on 18c.

Related Posts
Installing 12cR2 (12.2.0.1) RAC on RHEL 6 with Role Separation - Database
Installing 12c (12.1.0.1) RAC on RHEL 6 with Role Separation - Database Software
Installing 12c (12.1.0.1) RAC on RHEL 6 with Role Separation - Creating CDB & PDB

Monday, April 1, 2019

Installing 18c (18.3) RAC on RHEL 7 with Role Separation - Clusterware

This post is not a step by step guide to installing 18c RAC but list the main differences in the installation process compared to previous versions and things to look out for. Similar to 12cR2 clusterware the 18c cluster installation is no longer done through runInstaller. The downloaded Oracle Grid Infrastructure image files are unzipped into the folder path where GI is to be installed. This only need to be done on the local node only. During installation, the software is copied and installed on all other nodes in the cluster. Unlike 12cR1, it is possible to specify a separate disk group for GIMR at the installation time. Oracle documentation says "You cannot migrate the GIMR from one disk group to another later". 12cR1 also mentioned the same but there were ways to move GIMR to different disk group after the installation. It has been not verified whether the same techniques could be used for 18c as well.
1. The OS version and kernel versions used for this RAC setup is shown below. Unlike the previous post on 12cR2 this post for 18c uses RHEL7
cat /etc/redhat-release
Red Hat Enterprise Linux Server release 7.4 (Maipo)

uname -r
3.10.0-693.el7.x86_64
2. Create the user groups and users similar to previous 12.1 installation. As said in an earlier post, 12.2 introduce a new user group for RAC management. Following from install guide "You must designate a group as the OSRACDBA group during database installation. Members of this group are granted the SYSRAC privileges to perform day–to–day administration of Oracle databases on an Oracle RAC cluster".
groupadd dba
groupadd oinstall
groupadd oper
groupadd asmoper
groupadd asmdba
groupadd asmadmin
groupadd backupdba
groupadd dgdba
groupadd kmdba
groupadd racdba

useradd -g oinstall -G dba,oper,asmdba,backupdba,dgdba,kmdba,racdba oracle
useradd -g oinstall -G asmadmin,asmdba,asmoper,dba grid
For user equivelancy ensure that user id and group id are same across the cluster nodes. The option -g could be used to ensure that same user/group id is used for creating users across all nodes. The nodes in this case are named rhel71 and rhel72.
[oracle@rhel71 ~]$ id oracle
uid=501(oracle) gid=501(oinstall) groups=501(oinstall),1000(dba),1002(oper),1004(asmdba),1006(backupdba),1007(dgdba),1008(kmdba),1009(racdba)
[oracle@rhel71 ~]$ ssh rhel72 id oracle
uid=501(oracle) gid=501(oinstall) groups=501(oinstall),1000(dba),1002(oper),1004(asmdba),1006(backupdba),1007(dgdba),1008(kmdba),1009(racdba)

[oracle@rhel71 ~]$ id grid
uid=502(grid) gid=501(oinstall) groups=501(oinstall),1000(dba),1003(asmoper),1004(asmdba),1005(asmadmin)
[oracle@rhel71 ~]$ ssh rhel72 id grid
uid=502(grid) gid=501(oinstall) groups=501(oinstall),1000(dba),1003(asmoper),1004(asmdba),1005(asmadmin)
3. If a separate disk group is used for GIMR then it needs to be at least 35GB in size. Following from installation doc "during installation of an Oracle Standalone Cluster, if you create the MGMT disk group for Grid Infrastructure Management Repository (GIMR), then the installer requires that you use a disk group with at least 35 GB of available space". A 40GB disk was created to store the GIMR and udev rules were created for granting necessary ownership and permission.
cat /etc/udev/rules.d/99-sd.rules
KERNEL=="sd*1", SUBSYSTEM=="block", PROGRAM=="/usr/lib/udev/scsi_id -g -u -d /dev/$parent",RESULT=="1ATA_VBOX_HARDDISK_VB1a5c8787-e082e7a7",SYMLINK+="oracleasm/ocr1",OWNER="grid",GROUP="asmadmin",MODE="0660"
KERNEL=="sd*1", SUBSYSTEM=="block", PROGRAM=="/usr/lib/udev/scsi_id -g -u -d /dev/$parent",RESULT=="1ATA_VBOX_HARDDISK_VB1a72679c-ab23e15f",SYMLINK+="oracleasm/ocr2",OWNER="grid",GROUP="asmadmin",MODE="0660"
KERNEL=="sd*1", SUBSYSTEM=="block", PROGRAM=="/usr/lib/udev/scsi_id -g -u -d /dev/$parent",RESULT=="1ATA_VBOX_HARDDISK_VB8f43e75c-3e0bec6d",SYMLINK+="oracleasm/ocr3",OWNER="grid",GROUP="asmadmin",MODE="0660"

KERNEL=="sd*1", SUBSYSTEM=="block", PROGRAM=="/usr/lib/udev/scsi_id -g -u -d /dev/$parent",RESULT=="1ATA_VBOX_HARDDISK_VBf8475ab7-d401ec8b",SYMLINK+="oracleasm/gimr",OWNER="grid",GROUP="asmadmin",MODE="0660"

KERNEL=="sd*1", SUBSYSTEM=="block", PROGRAM=="/usr/lib/udev/scsi_id -g -u -d /dev/$parent",RESULT=="1ATA_VBOX_HARDDISK_VB34650f33-5ecdc029",SYMLINK+="oracleasm/data1",OWNER="grid",GROUP="asmadmin",MODE="0660"

KERNEL=="sd*1", SUBSYSTEM=="block", PROGRAM=="/usr/lib/udev/scsi_id -g -u -d /dev/$parent",RESULT=="1ATA_VBOX_HARDDISK_VBdb057b08-1733da4b",SYMLINK+="oracleasm/fra1",OWNER="grid",GROUP="asmadmin",MODE="0660"
4. Create the GI installation directory and unzip the downloaded install media into the directory.
mkdir -p /opt/app/18.x.0/grid
chown grid:oinstall /opt/app/18.x.0/grid
unzip ./LINUX.X64_180000_grid_home.zip -d /opt/app/18.x.0/grid
5. Run cluvfy to check the pre-reqs checks. The failure related to shm shown at the end could be ignored as per 2065603.1 and 25907259.8
[grid@rhel71 grid]$ ./runcluvfy.sh stage -pre crsinst -n rhel71,rhel72 -r 18 -osdba asmdba -osoper asmoper -asm -asmgrp asmadmin -presence local -asmdev /dev/oracleasm/data1,/dev/oracleasm/fra1,/dev/oracleasm/ocr1,/dev/oracleasm/ocr2,/dev/oracleasm/ocr3,/dev/oracleasm/gimr  -crshome /opt/app/18.x.0/grid -fixup  -verbose

Verifying Physical Memory ...
  Node Name     Available                 Required                  Status
  ------------  ------------------------  ------------------------  ----------
  rhel72        2.7806GB (2915620.0KB)    8GB (8388608.0KB)         failed
  rhel71        2.7806GB (2915620.0KB)    8GB (8388608.0KB)         failed
Verifying Physical Memory ...FAILED (PRVF-7530)
Verifying Available Physical Memory ...
  Node Name     Available                 Required                  Status
  ------------  ------------------------  ------------------------  ----------
  rhel72        2.6736GB (2803492.0KB)    50MB (51200.0KB)          passed
  rhel71        2.4872GB (2608032.0KB)    50MB (51200.0KB)          passed
Verifying Available Physical Memory ...PASSED
Verifying Swap Size ...
  Node Name     Available                 Required                  Status
  ------------  ------------------------  ------------------------  ----------
  rhel72        3.7246GB (3905532.0KB)    2.7806GB (2915620.0KB)    passed
  rhel71        3.7246GB (3905532.0KB)    2.7806GB (2915620.0KB)    passed
Verifying Swap Size ...PASSED
Verifying Free Space: rhel72:/usr,rhel72:/var,rhel72:/etc,rhel72:/opt/app/18.x.0/grid,rhel72:/sbin,rhel72:/tmp ...
  Path              Node Name     Mount point   Available     Required      Status
  ----------------  ------------  ------------  ------------  ------------  ------------
  /usr              rhel72        /             28.9385GB     25MB          passed
  /var              rhel72        /             28.9385GB     5MB           passed
  /etc              rhel72        /             28.9385GB     25MB          passed
  /opt/app/18.x.0/grid  rhel72        /             28.9385GB     6.9GB         passed
  /sbin             rhel72        /             28.9385GB     10MB          passed
  /tmp              rhel72        /             28.9385GB     1GB           passed
Verifying Free Space: rhel72:/usr,rhel72:/var,rhel72:/etc,rhel72:/opt/app/18.x.0/grid,rhel72:/sbin,rhel72:/tmp ...PASSED
Verifying Free Space: rhel71:/usr,rhel71:/var,rhel71:/etc,rhel71:/opt/app/18.x.0/grid,rhel71:/sbin,rhel71:/tmp ...
  Path              Node Name     Mount point   Available     Required      Status
  ----------------  ------------  ------------  ------------  ------------  ------------
  /usr              rhel71        /             18.9439GB     25MB          passed
  /var              rhel71        /             18.9439GB     5MB           passed
  /etc              rhel71        /             18.9439GB     25MB          passed
  /opt/app/18.x.0/grid  rhel71        /             18.9439GB     6.9GB         passed
  /sbin             rhel71        /             18.9439GB     10MB          passed
  /tmp              rhel71        /             18.9439GB     1GB           passed
Verifying Free Space: rhel71:/usr,rhel71:/var,rhel71:/etc,rhel71:/opt/app/18.x.0/grid,rhel71:/sbin,rhel71:/tmp ...PASSED
Verifying User Existence: grid ...
  Node Name     Status                    Comment
  ------------  ------------------------  ------------------------
  rhel72        passed                    exists(502)
  rhel71        passed                    exists(502)

  Verifying Users With Same UID: 502 ...PASSED
Verifying User Existence: grid ...PASSED
Verifying Group Existence: asmadmin ...
  Node Name     Status                    Comment
  ------------  ------------------------  ------------------------
  rhel72        passed                    exists
  rhel71        passed                    exists
Verifying Group Existence: asmadmin ...PASSED
Verifying Group Existence: asmoper ...
  Node Name     Status                    Comment
  ------------  ------------------------  ------------------------
  rhel72        passed                    exists
  rhel71        passed                    exists
Verifying Group Existence: asmoper ...PASSED
Verifying Group Existence: asmdba ...
  Node Name     Status                    Comment
  ------------  ------------------------  ------------------------
  rhel72        passed                    exists
  rhel71        passed                    exists
Verifying Group Existence: asmdba ...PASSED
Verifying Group Existence: oinstall ...
  Node Name     Status                    Comment
  ------------  ------------------------  ------------------------
  rhel72        passed                    exists
  rhel71        passed                    exists
Verifying Group Existence: oinstall ...PASSED
Verifying Group Membership: asmdba ...
  Node Name         User Exists   Group Exists  User in Group  Status
  ----------------  ------------  ------------  ------------  ----------------
  rhel72            yes           yes           yes           passed
  rhel71            yes           yes           yes           passed
Verifying Group Membership: asmdba ...PASSED
Verifying Group Membership: asmadmin ...
  Node Name         User Exists   Group Exists  User in Group  Status
  ----------------  ------------  ------------  ------------  ----------------
  rhel72            yes           yes           yes           passed
  rhel71            yes           yes           yes           passed
Verifying Group Membership: asmadmin ...PASSED
Verifying Group Membership: oinstall(Primary) ...
  Node Name         User Exists   Group Exists  User in Group  Primary       Status
  ----------------  ------------  ------------  ------------  ------------  ------------
  rhel72            yes           yes           yes           yes           passed
  rhel71            yes           yes           yes           yes           passed
Verifying Group Membership: oinstall(Primary) ...PASSED
Verifying Group Membership: asmoper ...
  Node Name         User Exists   Group Exists  User in Group  Status
  ----------------  ------------  ------------  ------------  ----------------
  rhel72            yes           yes           yes           passed
  rhel71            yes           yes           yes           passed
Verifying Group Membership: asmoper ...PASSED
Verifying Run Level ...
  Node Name     run level                 Required                  Status
  ------------  ------------------------  ------------------------  ----------
  rhel72        3                         3,5                       passed
  rhel71        3                         3,5                       passed
Verifying Run Level ...PASSED
Verifying Hard Limit: maximum open file descriptors ...
  Node Name         Type          Available     Required      Status
  ----------------  ------------  ------------  ------------  ----------------
  rhel72            hard          65536         65536         passed
  rhel71            hard          65536         65536         passed
Verifying Hard Limit: maximum open file descriptors ...PASSED
Verifying Soft Limit: maximum open file descriptors ...
  Node Name         Type          Available     Required      Status
  ----------------  ------------  ------------  ------------  ----------------
  rhel72            soft          65536         1024          passed
  rhel71            soft          65536         1024          passed
Verifying Soft Limit: maximum open file descriptors ...PASSED
Verifying Hard Limit: maximum user processes ...
  Node Name         Type          Available     Required      Status
  ----------------  ------------  ------------  ------------  ----------------
  rhel72            hard          16384         16384         passed
  rhel71            hard          16384         16384         passed
Verifying Hard Limit: maximum user processes ...PASSED
Verifying Soft Limit: maximum user processes ...
  Node Name         Type          Available     Required      Status
  ----------------  ------------  ------------  ------------  ----------------
  rhel72            soft          16384         2047          passed
  rhel71            soft          16384         2047          passed
Verifying Soft Limit: maximum user processes ...PASSED
Verifying Soft Limit: maximum stack size ...
  Node Name         Type          Available     Required      Status
  ----------------  ------------  ------------  ------------  ----------------
  rhel72            soft          10240         10240         passed
  rhel71            soft          10240         10240         passed
Verifying Soft Limit: maximum stack size ...PASSED
Verifying Architecture ...
  Node Name     Available                 Required                  Status
  ------------  ------------------------  ------------------------  ----------
  rhel72        x86_64                    x86_64                    passed
  rhel71        x86_64                    x86_64                    passed
Verifying Architecture ...PASSED
Verifying OS Kernel Version ...
  Node Name     Available                 Required                  Status
  ------------  ------------------------  ------------------------  ----------
  rhel72        3.10.0-693.el7.x86_64     3.10.0                    passed
  rhel71        3.10.0-693.el7.x86_64     3.10.0                    passed
Verifying OS Kernel Version ...PASSED
Verifying OS Kernel Parameter: semmsl ...
  Node Name         Current       Configured    Required      Status        Comment
  ----------------  ------------  ------------  ------------  ------------  ------------
  rhel71            3010          3010          250           passed
  rhel72            3010          3010          250           passed
Verifying OS Kernel Parameter: semmsl ...PASSED
Verifying OS Kernel Parameter: semmns ...
  Node Name         Current       Configured    Required      Status        Comment
  ----------------  ------------  ------------  ------------  ------------  ------------
  rhel71            385280        385280        32000         passed
  rhel72            385280        385280        32000         passed
Verifying OS Kernel Parameter: semmns ...PASSED
Verifying OS Kernel Parameter: semopm ...
  Node Name         Current       Configured    Required      Status        Comment
  ----------------  ------------  ------------  ------------  ------------  ------------
  rhel71            3010          3010          100           passed
  rhel72            3010          3010          100           passed
Verifying OS Kernel Parameter: semopm ...PASSED
Verifying OS Kernel Parameter: semmni ...
  Node Name         Current       Configured    Required      Status        Comment
  ----------------  ------------  ------------  ------------  ------------  ------------
  rhel71            128           128           128           passed
  rhel72            128           128           128           passed
Verifying OS Kernel Parameter: semmni ...PASSED
Verifying OS Kernel Parameter: shmmax ...
  Node Name         Current       Configured    Required      Status        Comment
  ----------------  ------------  ------------  ------------  ------------  ------------
  rhel71            4398046511104  4398046511104  1492797440    passed
  rhel72            4398046511104  4398046511104  1492797440    passed
Verifying OS Kernel Parameter: shmmax ...PASSED
Verifying OS Kernel Parameter: shmmni ...
  Node Name         Current       Configured    Required      Status        Comment
  ----------------  ------------  ------------  ------------  ------------  ------------
  rhel71            4096          4096          4096          passed
  rhel72            4096          4096          4096          passed
Verifying OS Kernel Parameter: shmmni ...PASSED
Verifying OS Kernel Parameter: shmall ...
  Node Name         Current       Configured    Required      Status        Comment
  ----------------  ------------  ------------  ------------  ------------  ------------
  rhel71            1073741824    1073741824    1073741824    passed
  rhel72            1073741824    1073741824    1073741824    passed
Verifying OS Kernel Parameter: shmall ...PASSED
Verifying OS Kernel Parameter: file-max ...
  Node Name         Current       Configured    Required      Status        Comment
  ----------------  ------------  ------------  ------------  ------------  ------------
  rhel71            6815744       6815744       6815744       passed
  rhel72            6815744       6815744       6815744       passed
Verifying OS Kernel Parameter: file-max ...PASSED
Verifying OS Kernel Parameter: ip_local_port_range ...
  Node Name         Current       Configured    Required      Status        Comment
  ----------------  ------------  ------------  ------------  ------------  ------------
  rhel71            between 9000 & 65500  between 9000 & 65500  between 9000 & 65535  passed
  rhel72            between 9000 & 65500  between 9000 & 65500  between 9000 & 65535  passed
Verifying OS Kernel Parameter: ip_local_port_range ...PASSED
Verifying OS Kernel Parameter: rmem_default ...
  Node Name         Current       Configured    Required      Status        Comment
  ----------------  ------------  ------------  ------------  ------------  ------------
  rhel71            4194304       4194304       262144        passed
  rhel72            4194304       4194304       262144        passed
Verifying OS Kernel Parameter: rmem_default ...PASSED
Verifying OS Kernel Parameter: rmem_max ...
  Node Name         Current       Configured    Required      Status        Comment
  ----------------  ------------  ------------  ------------  ------------  ------------
  rhel71            4194304       4194304       4194304       passed
  rhel72            4194304       4194304       4194304       passed
Verifying OS Kernel Parameter: rmem_max ...PASSED
Verifying OS Kernel Parameter: wmem_default ...
  Node Name         Current       Configured    Required      Status        Comment
  ----------------  ------------  ------------  ------------  ------------  ------------
  rhel71            1048576       1048576       262144        passed
  rhel72            1048576       1048576       262144        passed
Verifying OS Kernel Parameter: wmem_default ...PASSED
Verifying OS Kernel Parameter: wmem_max ...
  Node Name         Current       Configured    Required      Status        Comment
  ----------------  ------------  ------------  ------------  ------------  ------------
  rhel71            2097152       2097152       1048576       passed
  rhel72            2097152       2097152       1048576       passed
Verifying OS Kernel Parameter: wmem_max ...PASSED
Verifying OS Kernel Parameter: aio-max-nr ...
  Node Name         Current       Configured    Required      Status        Comment
  ----------------  ------------  ------------  ------------  ------------  ------------
  rhel71            3145728       3145728       1048576       passed
  rhel72            3145728       3145728       1048576       passed
Verifying OS Kernel Parameter: aio-max-nr ...PASSED
Verifying Package: binutils-2.23.52.0.1 ...
  Node Name     Available                 Required                  Status
  ------------  ------------------------  ------------------------  ----------
  rhel72        binutils-2.25.1-31.base.el7  binutils-2.23.52.0.1      passed
  rhel71        binutils-2.25.1-31.base.el7  binutils-2.23.52.0.1      passed
Verifying Package: binutils-2.23.52.0.1 ...PASSED
Verifying Package: compat-libcap1-1.10 ...
  Node Name     Available                 Required                  Status
  ------------  ------------------------  ------------------------  ----------
  rhel72        compat-libcap1-1.10-7.el7  compat-libcap1-1.10       passed
  rhel71        compat-libcap1-1.10-7.el7  compat-libcap1-1.10       passed
Verifying Package: compat-libcap1-1.10 ...PASSED
Verifying Package: libgcc-4.8.2 (x86_64) ...
  Node Name     Available                 Required                  Status
  ------------  ------------------------  ------------------------  ----------
  rhel72        libgcc(x86_64)-4.8.5-16.el7  libgcc(x86_64)-4.8.2      passed
  rhel71        libgcc(x86_64)-4.8.5-16.el7  libgcc(x86_64)-4.8.2      passed
Verifying Package: libgcc-4.8.2 (x86_64) ...PASSED
Verifying Package: libstdc++-4.8.2 (x86_64) ...
  Node Name     Available                 Required                  Status
  ------------  ------------------------  ------------------------  ----------
  rhel72        libstdc++(x86_64)-4.8.5-16.el7  libstdc++(x86_64)-4.8.2   passed
  rhel71        libstdc++(x86_64)-4.8.5-16.el7  libstdc++(x86_64)-4.8.2   passed
Verifying Package: libstdc++-4.8.2 (x86_64) ...PASSED
Verifying Package: libstdc++-devel-4.8.2 (x86_64) ...
  Node Name     Available                 Required                  Status
  ------------  ------------------------  ------------------------  ----------
  rhel72        libstdc++-devel(x86_64)-4.8.5-16.el7  libstdc++-devel(x86_64)-4.8.2  passed
  rhel71        libstdc++-devel(x86_64)-4.8.5-16.el7  libstdc++-devel(x86_64)-4.8.2  passed
Verifying Package: libstdc++-devel-4.8.2 (x86_64) ...PASSED
Verifying Package: sysstat-10.1.5 ...
  Node Name     Available                 Required                  Status
  ------------  ------------------------  ------------------------  ----------
  rhel72        sysstat-10.1.5-12.el7     sysstat-10.1.5            passed
  rhel71        sysstat-10.1.5-12.el7     sysstat-10.1.5            passed
Verifying Package: sysstat-10.1.5 ...PASSED
Verifying Package: gcc-c++-4.8.2 ...
  Node Name     Available                 Required                  Status
  ------------  ------------------------  ------------------------  ----------
  rhel72        gcc-c++-4.8.5-16.el7      gcc-c++-4.8.2             passed
  rhel71        gcc-c++-4.8.5-16.el7      gcc-c++-4.8.2             passed
Verifying Package: gcc-c++-4.8.2 ...PASSED
Verifying Package: ksh ...
  Node Name     Available                 Required                  Status
  ------------  ------------------------  ------------------------  ----------
  rhel72        ksh                       ksh                       passed
  rhel71        ksh                       ksh                       passed
Verifying Package: ksh ...PASSED
Verifying Package: make-3.82 ...
  Node Name     Available                 Required                  Status
  ------------  ------------------------  ------------------------  ----------
  rhel72        make-3.82-23.el7          make-3.82                 passed
  rhel71        make-3.82-23.el7          make-3.82                 passed
Verifying Package: make-3.82 ...PASSED
Verifying Package: glibc-2.17 (x86_64) ...
  Node Name     Available                 Required                  Status
  ------------  ------------------------  ------------------------  ----------
  rhel72        glibc(x86_64)-2.17-196.el7  glibc(x86_64)-2.17        passed
  rhel71        glibc(x86_64)-2.17-196.el7  glibc(x86_64)-2.17        passed
Verifying Package: glibc-2.17 (x86_64) ...PASSED
Verifying Package: glibc-devel-2.17 (x86_64) ...
  Node Name     Available                 Required                  Status
  ------------  ------------------------  ------------------------  ----------
  rhel72        glibc-devel(x86_64)-2.17-196.el7  glibc-devel(x86_64)-2.17  passed
  rhel71        glibc-devel(x86_64)-2.17-196.el7  glibc-devel(x86_64)-2.17  passed
Verifying Package: glibc-devel-2.17 (x86_64) ...PASSED
Verifying Package: libaio-0.3.109 (x86_64) ...
  Node Name     Available                 Required                  Status
  ------------  ------------------------  ------------------------  ----------
  rhel72        libaio(x86_64)-0.3.109-13.el7  libaio(x86_64)-0.3.109    passed
  rhel71        libaio(x86_64)-0.3.109-13.el7  libaio(x86_64)-0.3.109    passed
Verifying Package: libaio-0.3.109 (x86_64) ...PASSED
Verifying Package: libaio-devel-0.3.109 (x86_64) ...
  Node Name     Available                 Required                  Status
  ------------  ------------------------  ------------------------  ----------
  rhel72        libaio-devel(x86_64)-0.3.109-13.el7  libaio-devel(x86_64)-0.3.109  passed
  rhel71        libaio-devel(x86_64)-0.3.109-13.el7  libaio-devel(x86_64)-0.3.109  passed
Verifying Package: libaio-devel-0.3.109 (x86_64) ...PASSED
Verifying Package: nfs-utils-1.2.3-15 ...
  Node Name     Available                 Required                  Status
  ------------  ------------------------  ------------------------  ----------
  rhel72        nfs-utils-1.3.0-0.48.el7  nfs-utils-1.2.3-15        passed
  rhel71        nfs-utils-1.3.0-0.48.el7  nfs-utils-1.2.3-15        passed
Verifying Package: nfs-utils-1.2.3-15 ...PASSED
Verifying Package: smartmontools-6.2-4 ...
  Node Name     Available                 Required                  Status
  ------------  ------------------------  ------------------------  ----------
  rhel72        smartmontools-6.2-8.el7   smartmontools-6.2-4       passed
  rhel71        smartmontools-6.2-8.el7   smartmontools-6.2-4       passed
Verifying Package: smartmontools-6.2-4 ...PASSED
Verifying Package: net-tools-2.0-0.17 ...
  Node Name     Available                 Required                  Status
  ------------  ------------------------  ------------------------  ----------
  rhel72        net-tools-2.0-0.22.20131004git.el7  net-tools-2.0-0.17        passed
  rhel71        net-tools-2.0-0.22.20131004git.el7  net-tools-2.0-0.17        passed
Verifying Package: net-tools-2.0-0.17 ...PASSED
Verifying Package: compat-libstdc++-33-3.2.3 (x86_64) ...
  Node Name     Available                 Required                  Status
  ------------  ------------------------  ------------------------  ----------
  rhel72        compat-libstdc++-33(x86_64)-3.2.3-72.el7  compat-libstdc++-33(x86_64)-3.2.3  passed
  rhel71        compat-libstdc++-33(x86_64)-3.2.3-72.el7  compat-libstdc++-33(x86_64)-3.2.3  passed
Verifying Package: compat-libstdc++-33-3.2.3 (x86_64) ...PASSED
Verifying Package: libxcb-1.11 (x86_64) ...
  Node Name     Available                 Required                  Status
  ------------  ------------------------  ------------------------  ----------
  rhel72        libxcb(x86_64)-1.12-1.el7  libxcb(x86_64)-1.11       passed
  rhel71        libxcb(x86_64)-1.12-1.el7  libxcb(x86_64)-1.11       passed
Verifying Package: libxcb-1.11 (x86_64) ...PASSED
Verifying Package: libX11-1.6.3 (x86_64) ...
  Node Name     Available                 Required                  Status
  ------------  ------------------------  ------------------------  ----------
  rhel72        libX11(x86_64)-1.6.5-1.el7  libX11(x86_64)-1.6.3      passed
  rhel71        libX11(x86_64)-1.6.5-1.el7  libX11(x86_64)-1.6.3      passed
Verifying Package: libX11-1.6.3 (x86_64) ...PASSED
Verifying Package: libXau-1.0.8 (x86_64) ...
  Node Name     Available                 Required                  Status
  ------------  ------------------------  ------------------------  ----------
  rhel72        libXau(x86_64)-1.0.8-2.1.el7  libXau(x86_64)-1.0.8      passed
  rhel71        libXau(x86_64)-1.0.8-2.1.el7  libXau(x86_64)-1.0.8      passed
Verifying Package: libXau-1.0.8 (x86_64) ...PASSED
Verifying Package: libXi-1.7.4 (x86_64) ...
  Node Name     Available                 Required                  Status
  ------------  ------------------------  ------------------------  ----------
  rhel72        libXi(x86_64)-1.7.9-1.el7  libXi(x86_64)-1.7.4       passed
  rhel71        libXi(x86_64)-1.7.9-1.el7  libXi(x86_64)-1.7.4       passed
Verifying Package: libXi-1.7.4 (x86_64) ...PASSED
Verifying Package: libXtst-1.2.2 (x86_64) ...
  Node Name     Available                 Required                  Status
  ------------  ------------------------  ------------------------  ----------
  rhel72        libXtst(x86_64)-1.2.3-1.el7  libXtst(x86_64)-1.2.2     passed
  rhel71        libXtst(x86_64)-1.2.3-1.el7  libXtst(x86_64)-1.2.2     passed
Verifying Package: libXtst-1.2.2 (x86_64) ...PASSED
Verifying Port Availability for component "Oracle Notification Service (ONS)" ...
  Node Name         Port Number   Protocol      Available     Status
  ----------------  ------------  ------------  ------------  ----------------
  rhel72            6200          TCP           yes           successful
  rhel71            6200          TCP           yes           successful
  rhel72            6100          TCP           yes           successful
  rhel71            6100          TCP           yes           successful
Verifying Port Availability for component "Oracle Notification Service (ONS)" ...PASSED
Verifying Port Availability for component "Oracle Cluster Synchronization Services (CSSD)" ...
  Node Name         Port Number   Protocol      Available     Status
  ----------------  ------------  ------------  ------------  ----------------
  rhel72            42424         TCP           yes           successful
  rhel71            42424         TCP           yes           successful
Verifying Port Availability for component "Oracle Cluster Synchronization Services (CSSD)" ...PASSED
Verifying Users With Same UID: 0 ...PASSED
Verifying Current Group ID ...PASSED
Verifying Root user consistency ...
  Node Name                             Status
  ------------------------------------  ------------------------
  rhel72                                passed
  rhel71                                passed
Verifying Root user consistency ...PASSED
Verifying Package: cvuqdisk-1.0.10-1 ...
  Node Name     Available                 Required                  Status
  ------------  ------------------------  ------------------------  ----------
  rhel72        cvuqdisk-1.0.10-1         cvuqdisk-1.0.10-1         passed
  rhel71        cvuqdisk-1.0.10-1         cvuqdisk-1.0.10-1         passed
Verifying Package: cvuqdisk-1.0.10-1 ...PASSED
Verifying Host name ...PASSED
Verifying Node Connectivity ...
  Verifying Hosts File ...
  Node Name                             Status
  ------------------------------------  ------------------------
  rhel71                                passed
  rhel72                                passed
  Verifying Hosts File ...PASSED

Interface information for node "rhel71"

 Name   IP Address      Subnet          Gateway         Def. Gateway    HW Address        MTU
 ------ --------------- --------------- --------------- --------------- ----------------- ------
 enp0s3 192.168.0.93    192.168.0.0     0.0.0.0         192.168.0.102   08:00:27:BD:32:39 1500
 enp0s8 192.168.1.93    192.168.1.0     0.0.0.0         192.168.0.102   08:00:27:BD:A9:0A 1500

Interface information for node "rhel72"

 Name   IP Address      Subnet          Gateway         Def. Gateway    HW Address        MTU
 ------ --------------- --------------- --------------- --------------- ----------------- ------
 enp0s3 192.168.0.94    192.168.0.0     0.0.0.0         192.168.0.102   08:00:27:31:0F:1F 1500
 enp0s8 192.168.1.94    192.168.1.0     0.0.0.0         192.168.0.102   08:00:27:98:22:A6 1500

Check: MTU consistency of the subnet "192.168.1.0".

  Node              Name          IP Address    Subnet        MTU
  ----------------  ------------  ------------  ------------  ----------------
  rhel71            enp0s8        192.168.1.93  192.168.1.0   1500
  rhel72            enp0s8        192.168.1.94  192.168.1.0   1500

Check: MTU consistency of the subnet "192.168.0.0".

  Node              Name          IP Address    Subnet        MTU
  ----------------  ------------  ------------  ------------  ----------------
  rhel71            enp0s3        192.168.0.93  192.168.0.0   1500
  rhel72            enp0s3        192.168.0.94  192.168.0.0   1500
  Verifying Check that maximum (MTU) size packet goes through subnet ...PASSED

  Source                          Destination                     Connected?
  ------------------------------  ------------------------------  ----------------
  rhel71[enp0s8:192.168.1.93]     rhel72[enp0s8:192.168.1.94]     yes

  Source                          Destination                     Connected?
  ------------------------------  ------------------------------  ----------------
  rhel71[enp0s3:192.168.0.93]     rhel72[enp0s3:192.168.0.94]     yes
  Verifying subnet mask consistency for subnet "192.168.1.0" ...PASSED
  Verifying subnet mask consistency for subnet "192.168.0.0" ...PASSED
Verifying Node Connectivity ...PASSED
Verifying Multicast or broadcast check ...
Checking subnet "192.168.1.0" for multicast communication with multicast group "224.0.0.251"
Verifying Multicast or broadcast check ...PASSED
Verifying Device Checks for ASM ...
  Verifying Package: cvuqdisk-1.0.10-1 ...PASSED
  Verifying ASM device sharedness check ...
    Verifying Shared Storage Accessibility:/dev/oracleasm/gimr,/dev/oracleasm/fra1,/dev/oracleasm/data1,/dev/oracleasm/ocr3,/dev/oracleasm/ocr1,/dev/oracleasm/ocr2 ...PASSED

  Device                                Device Type
  ------------------------------------  ------------------------
  /dev/oracleasm/ocr3                   Disk
  /dev/oracleasm/data1                  Disk
  /dev/oracleasm/ocr2                   Disk
  /dev/oracleasm/fra1                   Disk
  /dev/oracleasm/ocr1                   Disk
  /dev/oracleasm/gimr                   Disk
  Verifying ASM device sharedness check ...PASSED
  Verifying Access Control List check ...
Checking consistency of device owner across all nodes...
Consistency check of device owner for "/dev/oracleasm/gimr" PASSED
Consistency check of device owner for "/dev/oracleasm/fra1" PASSED
Consistency check of device owner for "/dev/oracleasm/ocr3" PASSED
Consistency check of device owner for "/dev/oracleasm/data1" PASSED
Consistency check of device owner for "/dev/oracleasm/ocr1" PASSED
Consistency check of device owner for "/dev/oracleasm/ocr2" PASSED

Checking consistency of device group across all nodes...
Consistency check of device group for "/dev/oracleasm/gimr" PASSED
Consistency check of device group for "/dev/oracleasm/fra1" PASSED
Consistency check of device group for "/dev/oracleasm/ocr3" PASSED
Consistency check of device group for "/dev/oracleasm/data1" PASSED
Consistency check of device group for "/dev/oracleasm/ocr1" PASSED
Consistency check of device group for "/dev/oracleasm/ocr2" PASSED

Checking consistency of device permissions across all nodes...
Consistency check of device permissions for "/dev/oracleasm/gimr" PASSED
Consistency check of device permissions for "/dev/oracleasm/fra1" PASSED
Consistency check of device permissions for "/dev/oracleasm/ocr3" PASSED
Consistency check of device permissions for "/dev/oracleasm/data1" PASSED
Consistency check of device permissions for "/dev/oracleasm/ocr1" PASSED
Consistency check of device permissions for "/dev/oracleasm/ocr2" PASSED
  Verifying Access Control List check ...PASSED
  Verifying I/O scheduler ...PASSED
Verifying Device Checks for ASM ...PASSED
Verifying Network Time Protocol (NTP) ...
  Verifying '/etc/ntp.conf' ...
  Node Name                             File exists?
  ------------------------------------  ------------------------
  rhel72                                no
  rhel71                                no

  Verifying '/etc/ntp.conf' ...PASSED
  Verifying '/etc/chrony.conf' ...
  Node Name                             File exists?
  ------------------------------------  ------------------------
  rhel72                                no
  rhel71                                no

  Verifying '/etc/chrony.conf' ...PASSED
  Verifying '/var/run/ntpd.pid' ...
  Node Name                             File exists?
  ------------------------------------  ------------------------
  rhel72                                no
  rhel71                                no

  Verifying '/var/run/ntpd.pid' ...PASSED
  Verifying '/var/run/chronyd.pid' ...
  Node Name                             File exists?
  ------------------------------------  ------------------------
  rhel72                                no
  rhel71                                no

  Verifying '/var/run/chronyd.pid' ...PASSED
Verifying Network Time Protocol (NTP) ...PASSED
Verifying Same core file name pattern ...PASSED
Verifying User Mask ...
  Node Name     Available                 Required                  Comment
  ------------  ------------------------  ------------------------  ----------
  rhel72        0022                      0022                      passed
  rhel71        0022                      0022                      passed
Verifying User Mask ...PASSED
Verifying User Not In Group "root": grid ...
  Node Name     Status                    Comment
  ------------  ------------------------  ------------------------
  rhel72        passed                    does not exist
  rhel71        passed                    does not exist
Verifying User Not In Group "root": grid ...PASSED
Verifying Time zone consistency ...PASSED
Verifying Time offset between nodes ...PASSED
Verifying resolv.conf Integrity ...
  Node Name                             Status
  ------------------------------------  ------------------------
  rhel71                                passed
  rhel72                                passed

checking response for name "rhel72" from each of the name servers specified in
"/etc/resolv.conf"

  Node Name     Source                    Comment                   Status
  ------------  ------------------------  ------------------------  ----------
  rhel72        192.168.0.66              IPv4                      passed

checking response for name "rhel71" from each of the name servers specified in
"/etc/resolv.conf"

  Node Name     Source                    Comment                   Status
  ------------  ------------------------  ------------------------  ----------
  rhel71        192.168.0.66              IPv4                      passed
Verifying resolv.conf Integrity ...PASSED
Verifying DNS/NIS name service ...PASSED
Verifying Domain Sockets ...PASSED
Verifying /boot mount ...PASSED
Verifying File system mount options for path GI_HOME ...PASSED
Verifying Daemon "avahi-daemon" not configured and running ...
  Node Name     Configured                Status
  ------------  ------------------------  ------------------------
  rhel72        no                        passed
  rhel71        no                        passed

  Node Name     Running?                  Status
  ------------  ------------------------  ------------------------
  rhel72        no                        passed
  rhel71        no                        passed
Verifying Daemon "avahi-daemon" not configured and running ...PASSED
Verifying Daemon "proxyt" not configured and running ...
  Node Name     Configured                Status
  ------------  ------------------------  ------------------------
  rhel72        no                        passed
  rhel71        no                        passed

  Node Name     Running?                  Status
  ------------  ------------------------  ------------------------
  rhel72        no                        passed
  rhel71        no                        passed
Verifying Daemon "proxyt" not configured and running ...PASSED
Verifying Grid Infrastructure home path: /opt/app/18.x.0/grid ...
  Verifying '/opt/app/18.x.0/grid' ...PASSED
Verifying Grid Infrastructure home path: /opt/app/18.x.0/grid ...PASSED
Verifying User Equivalence ...PASSED
Verifying /dev/shm mounted as temporary file system ...FAILED (PRVE-0421)
Verifying File system mount options for path /var ...PASSED
Verifying zeroconf check ...PASSED
Verifying ASM Filter Driver configuration ...PASSED

Pre-check for cluster services setup was unsuccessful on all the nodes.


Failures were encountered during execution of CVU verification request "stage -pre crsinst".

Verifying Physical Memory ...FAILED
rhel72: PRVF-7530 : Sufficient physical memory is not available on node
        "rhel72" [Required physical memory = 8GB (8388608.0KB)]

rhel71: PRVF-7530 : Sufficient physical memory is not available on node
        "rhel71" [Required physical memory = 8GB (8388608.0KB)]

Verifying /dev/shm mounted as temporary file system ...FAILED
rhel72: PRVE-0421 : No entry exists in /etc/fstab for mounting /dev/shm

rhel71: PRVE-0421 : No entry exists in /etc/fstab for mounting /dev/shm
6. From the GI_HOME directory run gridSetup.sh to begin the installation. Select new cluster configuration option.
Select standard cluster (this would create a flex cluster which is the default for all standard clusters).
Set cluster name and scan name. No GNS is setup or used for this installation.
Add node and node VIP names that will be used in the cluster configuration. Importantly all node roles are set as HUB nodes.
Select the network interfaces for public and ASM/Private interconnect.
The OCR and vote disks are stored in ASM but ASM could be configured either block devices or using NFS. This setup uses block devices.
A separate disk group is used to store GIMR.
Create the disk group for OCR files.
Create disk group for GIMR
Set the oracle base location. The GI install location is immutable.
The failed pre-reqs are ignorable for test setup.
Summary
7. When prompted run the root script. Following output from running root.sh on the first node.
./root.sh
Performing root user operation.

Creating /etc/oratab file...
Entries will be added to the /etc/oratab file as needed by
Database Configuration Assistant when a database is created
Finished running generic part of root script.
Now product-specific root actions will be performed.
Relinking oracle with rac_on option
Using configuration parameter file: /opt/app/18.x.0/grid/crs/install/crsconfig_params
The log of current session can be found at:
  /opt/app/oracle/crsdata/rhel71/crsconfig/rootcrs_rhel71_2019-03-19_02-59-46PM.log
2019/03/19 15:00:12 CLSRSC-594: Executing installation step 1 of 20: 'SetupTFA'.
2019/03/19 15:00:12 CLSRSC-4001: Installing Oracle Trace File Analyzer (TFA) Collector.
2019/03/19 15:00:56 CLSRSC-4002: Successfully installed Oracle Trace File Analyzer (TFA) Collector.
2019/03/19 15:00:56 CLSRSC-594: Executing installation step 2 of 20: 'ValidateEnv'.
2019/03/19 15:00:57 CLSRSC-363: User ignored prerequisites during installation
2019/03/19 15:00:57 CLSRSC-594: Executing installation step 3 of 20: 'CheckFirstNode'.
2019/03/19 15:01:01 CLSRSC-594: Executing installation step 4 of 20: 'GenSiteGUIDs'.
2019/03/19 15:01:07 CLSRSC-594: Executing installation step 5 of 20: 'SaveParamFile'.
2019/03/19 15:01:28 CLSRSC-594: Executing installation step 6 of 20: 'SetupOSD'.
2019/03/19 15:01:28 CLSRSC-594: Executing installation step 7 of 20: 'CheckCRSConfig'.
2019/03/19 15:01:28 CLSRSC-594: Executing installation step 8 of 20: 'SetupLocalGPNP'.
2019/03/19 15:02:17 CLSRSC-594: Executing installation step 9 of 20: 'CreateRootCert'.
2019/03/19 15:02:30 CLSRSC-594: Executing installation step 10 of 20: 'ConfigOLR'.
2019/03/19 15:02:58 CLSRSC-594: Executing installation step 11 of 20: 'ConfigCHMOS'.
2019/03/19 15:02:58 CLSRSC-594: Executing installation step 12 of 20: 'CreateOHASD'.
2019/03/19 15:03:17 CLSRSC-594: Executing installation step 13 of 20: 'ConfigOHASD'.
2019/03/19 15:03:17 CLSRSC-330: Adding Clusterware entries to file 'oracle-ohasd.service'
2019/03/19 15:04:47 CLSRSC-594: Executing installation step 14 of 20: 'InstallAFD'.
2019/03/19 15:05:05 CLSRSC-594: Executing installation step 15 of 20: 'InstallACFS'.
CRS-2791: Starting shutdown of Oracle High Availability Services-managed resources on 'rhel71'
CRS-2793: Shutdown of Oracle High Availability Services-managed resources on 'rhel71' has completed
CRS-4133: Oracle High Availability Services has been stopped.
CRS-4123: Oracle High Availability Services has been started.
2019/03/19 15:06:37 CLSRSC-594: Executing installation step 16 of 20: 'InstallKA'.
2019/03/19 15:06:55 CLSRSC-594: Executing installation step 17 of 20: 'InitConfig'.
CRS-2791: Starting shutdown of Oracle High Availability Services-managed resources on 'rhel71'
CRS-2793: Shutdown of Oracle High Availability Services-managed resources on 'rhel71' has completed
CRS-4133: Oracle High Availability Services has been stopped.
CRS-4123: Oracle High Availability Services has been started.
CRS-2672: Attempting to start 'ora.evmd' on 'rhel71'
CRS-2672: Attempting to start 'ora.mdnsd' on 'rhel71'
CRS-2676: Start of 'ora.mdnsd' on 'rhel71' succeeded
CRS-2676: Start of 'ora.evmd' on 'rhel71' succeeded
CRS-2672: Attempting to start 'ora.gpnpd' on 'rhel71'
CRS-2676: Start of 'ora.gpnpd' on 'rhel71' succeeded
CRS-2672: Attempting to start 'ora.cssdmonitor' on 'rhel71'
CRS-2672: Attempting to start 'ora.gipcd' on 'rhel71'
CRS-2676: Start of 'ora.cssdmonitor' on 'rhel71' succeeded
CRS-2676: Start of 'ora.gipcd' on 'rhel71' succeeded
CRS-2672: Attempting to start 'ora.cssd' on 'rhel71'
CRS-2672: Attempting to start 'ora.diskmon' on 'rhel71'
CRS-2676: Start of 'ora.diskmon' on 'rhel71' succeeded
CRS-2676: Start of 'ora.cssd' on 'rhel71' succeeded

[INFO] [DBT-30001] Disk groups created successfully. Check /opt/app/oracle/cfgtoollogs/asmca/asmca-190319PM030741.log for details.


2019/03/19 15:08:59 CLSRSC-482: Running command: '/opt/app/18.x.0/grid/bin/ocrconfig -upgrade grid oinstall'
CRS-2672: Attempting to start 'ora.crf' on 'rhel71'
CRS-2672: Attempting to start 'ora.storage' on 'rhel71'
CRS-2676: Start of 'ora.storage' on 'rhel71' succeeded
CRS-2676: Start of 'ora.crf' on 'rhel71' succeeded
CRS-2672: Attempting to start 'ora.crsd' on 'rhel71'
CRS-2676: Start of 'ora.crsd' on 'rhel71' succeeded
CRS-4256: Updating the profile
Successful addition of voting disk c500a19f07ec4f62bf8ac318166a95ae.
Successful addition of voting disk 9100c10cb7bb4fd1bffbe97b8dc91245.
Successful addition of voting disk 300c2bf8b7094fa6bfaf11e49f826c7e.
Successfully replaced voting disk group with +OCRDG.
CRS-4256: Updating the profile
CRS-4266: Voting file(s) successfully replaced
##  STATE    File Universal Id                File Name Disk group
--  -----    -----------------                --------- ---------
 1. ONLINE   c500a19f07ec4f62bf8ac318166a95ae (/dev/oracleasm/ocr1) [OCRDG]
 2. ONLINE   9100c10cb7bb4fd1bffbe97b8dc91245 (/dev/oracleasm/ocr2) [OCRDG]
 3. ONLINE   300c2bf8b7094fa6bfaf11e49f826c7e (/dev/oracleasm/ocr3) [OCRDG]
Located 3 voting disk(s).
CRS-2791: Starting shutdown of Oracle High Availability Services-managed resources on 'rhel71'
CRS-2673: Attempting to stop 'ora.crsd' on 'rhel71'
CRS-2677: Stop of 'ora.crsd' on 'rhel71' succeeded
CRS-2673: Attempting to stop 'ora.storage' on 'rhel71'
CRS-2673: Attempting to stop 'ora.crf' on 'rhel71'
CRS-2673: Attempting to stop 'ora.drivers.acfs' on 'rhel71'
CRS-2673: Attempting to stop 'ora.mdnsd' on 'rhel71'
CRS-2677: Stop of 'ora.crf' on 'rhel71' succeeded
CRS-2677: Stop of 'ora.storage' on 'rhel71' succeeded
CRS-2673: Attempting to stop 'ora.asm' on 'rhel71'
CRS-2677: Stop of 'ora.drivers.acfs' on 'rhel71' succeeded
CRS-2677: Stop of 'ora.mdnsd' on 'rhel71' succeeded
CRS-2677: Stop of 'ora.asm' on 'rhel71' succeeded
CRS-2673: Attempting to stop 'ora.cluster_interconnect.haip' on 'rhel71'
CRS-2677: Stop of 'ora.cluster_interconnect.haip' on 'rhel71' succeeded
CRS-2673: Attempting to stop 'ora.ctssd' on 'rhel71'
CRS-2673: Attempting to stop 'ora.evmd' on 'rhel71'
CRS-2677: Stop of 'ora.ctssd' on 'rhel71' succeeded
CRS-2677: Stop of 'ora.evmd' on 'rhel71' succeeded
CRS-2673: Attempting to stop 'ora.cssd' on 'rhel71'
CRS-2677: Stop of 'ora.cssd' on 'rhel71' succeeded
CRS-2673: Attempting to stop 'ora.gipcd' on 'rhel71'
CRS-2673: Attempting to stop 'ora.gpnpd' on 'rhel71'
CRS-2677: Stop of 'ora.gipcd' on 'rhel71' succeeded
CRS-2677: Stop of 'ora.gpnpd' on 'rhel71' succeeded
CRS-2793: Shutdown of Oracle High Availability Services-managed resources on 'rhel71' has completed
CRS-4133: Oracle High Availability Services has been stopped.
2019/03/19 15:13:36 CLSRSC-594: Executing installation step 18 of 20: 'StartCluster'.
CRS-4123: Starting Oracle High Availability Services-managed resources
CRS-2672: Attempting to start 'ora.mdnsd' on 'rhel71'
CRS-2672: Attempting to start 'ora.evmd' on 'rhel71'
CRS-2676: Start of 'ora.evmd' on 'rhel71' succeeded
CRS-2676: Start of 'ora.mdnsd' on 'rhel71' succeeded
CRS-2672: Attempting to start 'ora.gpnpd' on 'rhel71'
CRS-2676: Start of 'ora.gpnpd' on 'rhel71' succeeded
CRS-2672: Attempting to start 'ora.gipcd' on 'rhel71'
CRS-2676: Start of 'ora.gipcd' on 'rhel71' succeeded
CRS-2672: Attempting to start 'ora.crf' on 'rhel71'
CRS-2672: Attempting to start 'ora.cssdmonitor' on 'rhel71'
CRS-2676: Start of 'ora.cssdmonitor' on 'rhel71' succeeded
CRS-2672: Attempting to start 'ora.cssd' on 'rhel71'
CRS-2672: Attempting to start 'ora.diskmon' on 'rhel71'
CRS-2676: Start of 'ora.diskmon' on 'rhel71' succeeded
CRS-2676: Start of 'ora.crf' on 'rhel71' succeeded
CRS-2676: Start of 'ora.cssd' on 'rhel71' succeeded
CRS-2672: Attempting to start 'ora.cluster_interconnect.haip' on 'rhel71'
CRS-2672: Attempting to start 'ora.ctssd' on 'rhel71'
CRS-2676: Start of 'ora.ctssd' on 'rhel71' succeeded
CRS-2676: Start of 'ora.cluster_interconnect.haip' on 'rhel71' succeeded
CRS-2672: Attempting to start 'ora.asm' on 'rhel71'
CRS-2676: Start of 'ora.asm' on 'rhel71' succeeded
CRS-2672: Attempting to start 'ora.storage' on 'rhel71'
CRS-2676: Start of 'ora.storage' on 'rhel71' succeeded
CRS-2672: Attempting to start 'ora.crsd' on 'rhel71'
CRS-2676: Start of 'ora.crsd' on 'rhel71' succeeded
CRS-6023: Starting Oracle Cluster Ready Services-managed resources
CRS-6017: Processing resource auto-start for servers: rhel71
CRS-6016: Resource auto-start has completed for server rhel71
CRS-6024: Completed start of Oracle Cluster Ready Services-managed resources
CRS-4123: Oracle High Availability Services has been started.
2019/03/19 15:16:19 CLSRSC-343: Successfully started Oracle Clusterware stack
2019/03/19 15:16:19 CLSRSC-594: Executing installation step 19 of 20: 'ConfigNode'.
CRS-2672: Attempting to start 'ora.ASMNET1LSNR_ASM.lsnr' on 'rhel71'
CRS-2676: Start of 'ora.ASMNET1LSNR_ASM.lsnr' on 'rhel71' succeeded
CRS-2672: Attempting to start 'ora.asm' on 'rhel71'
CRS-2676: Start of 'ora.asm' on 'rhel71' succeeded
CRS-2672: Attempting to start 'ora.OCRDG.dg' on 'rhel71'
CRS-2676: Start of 'ora.OCRDG.dg' on 'rhel71' succeeded
2019/03/19 15:19:48 CLSRSC-594: Executing installation step 20 of 20: 'PostConfig'.

[INFO] [DBT-30001] Disk groups created successfully. Check /opt/app/oracle/cfgtoollogs/asmca/asmca-190319PM031955.log for details.


2019/03/19 15:23:06 CLSRSC-325: Configure Oracle Grid Infrastructure for a Cluster ... succeeded
Root script run on the second (last) node.
./root.sh
Performing root user operation.

Creating /etc/oratab file...
Entries will be added to the /etc/oratab file as needed by
Database Configuration Assistant when a database is created
Finished running generic part of root script.
Now product-specific root actions will be performed.
Relinking oracle with rac_on option
Using configuration parameter file: /opt/app/18.x.0/grid/crs/install/crsconfig_params
The log of current session can be found at:
  /opt/app/oracle/crsdata/rhel72/crsconfig/rootcrs_rhel72_2019-03-19_03-25-21PM.log
2019/03/19 15:25:38 CLSRSC-594: Executing installation step 1 of 20: 'SetupTFA'.
2019/03/19 15:25:38 CLSRSC-4001: Installing Oracle Trace File Analyzer (TFA) Collector.
2019/03/19 15:26:17 CLSRSC-4002: Successfully installed Oracle Trace File Analyzer (TFA) Collector.
2019/03/19 15:26:17 CLSRSC-594: Executing installation step 2 of 20: 'ValidateEnv'.
2019/03/19 15:26:17 CLSRSC-363: User ignored prerequisites during installation
2019/03/19 15:26:17 CLSRSC-594: Executing installation step 3 of 20: 'CheckFirstNode'.
2019/03/19 15:26:19 CLSRSC-594: Executing installation step 4 of 20: 'GenSiteGUIDs'.
2019/03/19 15:26:19 CLSRSC-594: Executing installation step 5 of 20: 'SaveParamFile'.
2019/03/19 15:26:23 CLSRSC-594: Executing installation step 6 of 20: 'SetupOSD'.
2019/03/19 15:26:23 CLSRSC-594: Executing installation step 7 of 20: 'CheckCRSConfig'.
2019/03/19 15:26:25 CLSRSC-594: Executing installation step 8 of 20: 'SetupLocalGPNP'.
2019/03/19 15:26:28 CLSRSC-594: Executing installation step 9 of 20: 'CreateRootCert'.
2019/03/19 15:26:28 CLSRSC-594: Executing installation step 10 of 20: 'ConfigOLR'.
2019/03/19 15:26:31 CLSRSC-594: Executing installation step 11 of 20: 'ConfigCHMOS'.
2019/03/19 15:26:31 CLSRSC-594: Executing installation step 12 of 20: 'CreateOHASD'.
2019/03/19 15:26:34 CLSRSC-594: Executing installation step 13 of 20: 'ConfigOHASD'.
2019/03/19 15:26:35 CLSRSC-330: Adding Clusterware entries to file 'oracle-ohasd.service'
2019/03/19 15:27:53 CLSRSC-594: Executing installation step 14 of 20: 'InstallAFD'.
2019/03/19 15:27:56 CLSRSC-594: Executing installation step 15 of 20: 'InstallACFS'.
CRS-2791: Starting shutdown of Oracle High Availability Services-managed resources on 'rhel72'
CRS-2793: Shutdown of Oracle High Availability Services-managed resources on 'rhel72' has completed
CRS-4133: Oracle High Availability Services has been stopped.
CRS-4123: Oracle High Availability Services has been started.
2019/03/19 15:29:13 CLSRSC-594: Executing installation step 16 of 20: 'InstallKA'.
2019/03/19 15:29:15 CLSRSC-594: Executing installation step 17 of 20: 'InitConfig'.
CRS-2791: Starting shutdown of Oracle High Availability Services-managed resources on 'rhel72'
CRS-2793: Shutdown of Oracle High Availability Services-managed resources on 'rhel72' has completed
CRS-4133: Oracle High Availability Services has been stopped.
CRS-4123: Oracle High Availability Services has been started.
CRS-2791: Starting shutdown of Oracle High Availability Services-managed resources on 'rhel72'
CRS-2673: Attempting to stop 'ora.drivers.acfs' on 'rhel72'
CRS-2677: Stop of 'ora.drivers.acfs' on 'rhel72' succeeded
CRS-2793: Shutdown of Oracle High Availability Services-managed resources on 'rhel72' has completed
CRS-4133: Oracle High Availability Services has been stopped.
2019/03/19 15:29:28 CLSRSC-594: Executing installation step 18 of 20: 'StartCluster'.
CRS-4123: Starting Oracle High Availability Services-managed resources
CRS-2672: Attempting to start 'ora.mdnsd' on 'rhel72'
CRS-2672: Attempting to start 'ora.evmd' on 'rhel72'
CRS-2676: Start of 'ora.mdnsd' on 'rhel72' succeeded
CRS-2676: Start of 'ora.evmd' on 'rhel72' succeeded
CRS-2672: Attempting to start 'ora.gpnpd' on 'rhel72'
CRS-2676: Start of 'ora.gpnpd' on 'rhel72' succeeded
CRS-2672: Attempting to start 'ora.gipcd' on 'rhel72'
CRS-2676: Start of 'ora.gipcd' on 'rhel72' succeeded
CRS-2672: Attempting to start 'ora.crf' on 'rhel72'
CRS-2672: Attempting to start 'ora.cssdmonitor' on 'rhel72'
CRS-2676: Start of 'ora.cssdmonitor' on 'rhel72' succeeded
CRS-2672: Attempting to start 'ora.cssd' on 'rhel72'
CRS-2672: Attempting to start 'ora.diskmon' on 'rhel72'
CRS-2676: Start of 'ora.diskmon' on 'rhel72' succeeded
CRS-2676: Start of 'ora.crf' on 'rhel72' succeeded
CRS-2676: Start of 'ora.cssd' on 'rhel72' succeeded
CRS-2672: Attempting to start 'ora.cluster_interconnect.haip' on 'rhel72'
CRS-2672: Attempting to start 'ora.ctssd' on 'rhel72'
CRS-2676: Start of 'ora.ctssd' on 'rhel72' succeeded
CRS-2672: Attempting to start 'ora.crsd' on 'rhel72'
CRS-2676: Start of 'ora.crsd' on 'rhel72' succeeded
CRS-2676: Start of 'ora.cluster_interconnect.haip' on 'rhel72' succeeded
CRS-2672: Attempting to start 'ora.asm' on 'rhel72'
CRS-2676: Start of 'ora.asm' on 'rhel72' succeeded
CRS-6017: Processing resource auto-start for servers: rhel72
CRS-2673: Attempting to stop 'ora.LISTENER_SCAN1.lsnr' on 'rhel71'
CRS-2672: Attempting to start 'ora.ons' on 'rhel72'
CRS-2672: Attempting to start 'ora.ASMNET1LSNR_ASM.lsnr' on 'rhel72'
CRS-2677: Stop of 'ora.LISTENER_SCAN1.lsnr' on 'rhel71' succeeded
CRS-2673: Attempting to stop 'ora.scan1.vip' on 'rhel71'
CRS-2677: Stop of 'ora.scan1.vip' on 'rhel71' succeeded
CRS-2672: Attempting to start 'ora.scan1.vip' on 'rhel72'
CRS-2676: Start of 'ora.ASMNET1LSNR_ASM.lsnr' on 'rhel72' succeeded
CRS-2672: Attempting to start 'ora.asm' on 'rhel72'
CRS-2676: Start of 'ora.scan1.vip' on 'rhel72' succeeded
CRS-2672: Attempting to start 'ora.LISTENER_SCAN1.lsnr' on 'rhel72'
CRS-2676: Start of 'ora.ons' on 'rhel72' succeeded
CRS-2676: Start of 'ora.LISTENER_SCAN1.lsnr' on 'rhel72' succeeded
CRS-2676: Start of 'ora.asm' on 'rhel72' succeeded
CRS-2672: Attempting to start 'ora.proxy_advm' on 'rhel71'
CRS-2672: Attempting to start 'ora.proxy_advm' on 'rhel72'
CRS-2676: Start of 'ora.proxy_advm' on 'rhel71' succeeded
CRS-2676: Start of 'ora.proxy_advm' on 'rhel72' succeeded
CRS-6016: Resource auto-start has completed for server rhel72
CRS-6024: Completed start of Oracle Cluster Ready Services-managed resources
CRS-4123: Oracle High Availability Services has been started.
2019/03/19 15:32:28 CLSRSC-343: Successfully started Oracle Clusterware stack
2019/03/19 15:32:29 CLSRSC-594: Executing installation step 19 of 20: 'ConfigNode'.
2019/03/19 15:33:03 CLSRSC-594: Executing installation step 20 of 20: 'PostConfig'.
2019/03/19 15:33:13 CLSRSC-325: Configure Oracle Grid Infrastructure for a Cluster ... succeeded
8. Once the root scripts are run click OK on the configuration dialogue box to complete the creation of GIMR and other configuration assistants. At the end this clusterware installation would be completed.



9. Run cluvfy with post crsinst to check the installation.
[grid@rhel71 ~]$ cluvfy stage -post crsinst -allnodes

Verifying Node Connectivity ...
  Verifying Hosts File ...PASSED
  Verifying Check that maximum (MTU) size packet goes through subnet ...PASSED
  Verifying subnet mask consistency for subnet "192.168.1.0" ...PASSED
  Verifying subnet mask consistency for subnet "192.168.0.0" ...PASSED
Verifying Node Connectivity ...PASSED
Verifying Multicast or broadcast check ...PASSED
Verifying ASM filter driver configuration consistency ...PASSED
Verifying Time zone consistency ...PASSED
Verifying Cluster Manager Integrity ...PASSED
Verifying User Mask ...PASSED
Verifying Cluster Integrity ...PASSED
Verifying OCR Integrity ...PASSED
Verifying CRS Integrity ...
  Verifying Clusterware Version Consistency ...PASSED
Verifying CRS Integrity ...PASSED
Verifying Node Application Existence ...PASSED
Verifying Single Client Access Name (SCAN) ...
  Verifying DNS/NIS name service 'rac-scan.domain.net' ...
    Verifying Name Service Switch Configuration File Integrity ...PASSED
  Verifying DNS/NIS name service 'rac-scan.domain.net' ...PASSED
Verifying Single Client Access Name (SCAN) ...PASSED
Verifying OLR Integrity ...PASSED
Verifying Voting Disk ...PASSED
Verifying ASM Integrity ...
  Verifying Node Connectivity ...
    Verifying Hosts File ...PASSED
    Verifying Check that maximum (MTU) size packet goes through subnet ...PASSED
    Verifying subnet mask consistency for subnet "192.168.1.0" ...PASSED
    Verifying subnet mask consistency for subnet "192.168.0.0" ...PASSED
  Verifying Node Connectivity ...PASSED
Verifying ASM Integrity ...PASSED
Verifying Device Checks for ASM ...
  Verifying Access Control List check ...PASSED
  Verifying I/O scheduler ...PASSED
Verifying Device Checks for ASM ...PASSED
Verifying ASM disk group free space ...PASSED
Verifying User Not In Group "root": grid ...PASSED
Verifying Clock Synchronization ...PASSED
Verifying VIP Subnet configuration check ...PASSED
Verifying Network configuration consistency checks ...PASSED
Verifying File system mount options for path GI_HOME ...PASSED
Verifying Access control attributes for /var/tmp/.oracle ...PASSED
Verifying Access control attributes for %OCRCONFIGDIR%/maps ...PASSED

Post-check for cluster services setup was successful.
10. There are few cluster and local resources that are new in 18c compared to 12.2 Most of them relate the rapid home provisioning feature and are in disable state.
 crsctl stat res -t
--------------------------------------------------------------------------------
Name           Target  State        Server                   State details
--------------------------------------------------------------------------------
Local Resources
--------------------------------------------------------------------------------
ora.GIMRDG.GHCHKPT.advm
               OFFLINE OFFLINE      rhel71                   STABLE
...
ora.gimrdg.ghchkpt.acfs
               OFFLINE OFFLINE      rhel71                   STABLE
               OFFLINE OFFLINE      rhel72                   STABLE
ora.helper
               OFFLINE OFFLINE      rhel71                   IDLE,STABLE
               OFFLINE OFFLINE      rhel72                   IDLE,STABLE
ora.proxy_advm
               ONLINE  ONLINE       rhel71                   STABLE
               ONLINE  ONLINE       rhel72                   STABLE
--------------------------------------------------------------------------------
Cluster Resources
--------------------------------------------------------------------------------
...
ora.rhpserver
      1        OFFLINE OFFLINE                               STABLE
--------------------------------------------------------------------------------
For example clusterware installation create a pre-configued ASM volume group call GHCHKPT which is to be used with rapid home provisioning framework.
ASMCMD> volinfo --all
Diskgroup Name: GIMRDG

         Volume Name: GHCHKPT
         Volume Device: /dev/asm/ghchkpt-493
         State: DISABLED
         Size (MB): 1536
         Resize Unit (MB): 512
         Redundancy: UNPROT
         Stripe Columns: 8
         Stripe Width (K): 1024
         Usage: ACFS
         Mountpath: /opt/oracle/rhp_images/chkbase
Because of this volume group the proxy ASM instance will have an active and online status. These would be visible as APX{node number} instances. Oracle docs states "an Oracle ASM proxy instance is an Oracle instance running on a Hub node with a direct Oracle ASM client. An Oracle ASM proxy instance provides support for Oracle Automatic Storage Management Cluster File System (Oracle ACFS) and Oracle ASM Dynamic Volume Manager (Oracle ADVM)". If no other ASM volume groups are to be created or online then ASM proxy instances could be stopped and disabled.
[grid@rhel71 ~]$ srvctl stop asm -proxy -node rhel71
[grid@rhel71 ~]$ srvctl stop asm -proxy -node rhel72
[grid@rhel71 ~]$ srvctl status asm -proxy
ADVM proxy is not running
The full list of resources after the installation is shown below (before proxy instances were disabled).
Resource Name                       Type                      Target             State              Host
-------------                       ------                    -------            --------           ----------
ora.ASMNET1LSNR_ASM.lsnr            ora.asm_listener.type     ONLINE             ONLINE             rhel71
ora.ASMNET1LSNR_ASM.lsnr            ora.asm_listener.type     ONLINE             ONLINE             rhel72
ora.GIMRDG.GHCHKPT.advm             ora.volume.type           OFFLINE            OFFLINE
ora.GIMRDG.GHCHKPT.advm             ora.volume.type           OFFLINE            OFFLINE
ora.GIMRDG.dg                       ora.diskgroup.type        ONLINE             ONLINE             rhel71
ora.GIMRDG.dg                       ora.diskgroup.type        ONLINE             ONLINE             rhel72
ora.LISTENER.lsnr                   ora.listener.type         ONLINE             ONLINE             rhel71
ora.LISTENER.lsnr                   ora.listener.type         ONLINE             ONLINE             rhel72
ora.LISTENER_SCAN1.lsnr             ora.scan_listener.type    ONLINE             ONLINE             rhel72
ora.LISTENER_SCAN2.lsnr             ora.scan_listener.type    ONLINE             ONLINE             rhel71
ora.LISTENER_SCAN3.lsnr             ora.scan_listener.type    ONLINE             ONLINE             rhel71
ora.MGMTLSNR                        ora.mgmtlsnr.type         ONLINE             ONLINE             rhel71
ora.OCRDG.dg                        ora.diskgroup.type        ONLINE             ONLINE             rhel71
ora.OCRDG.dg                        ora.diskgroup.type        ONLINE             ONLINE             rhel72
ora.asm                             ora.asm.type              ONLINE             ONLINE             rhel71
ora.asm                             ora.asm.type              ONLINE             ONLINE             rhel72
ora.asm                             ora.asm.type              OFFLINE            OFFLINE
ora.chad                            ora.chad.type             ONLINE             ONLINE             rhel71
ora.chad                            ora.chad.type             ONLINE             ONLINE             rhel72
ora.cvu                             ora.cvu.type              ONLINE             ONLINE             rhel71
ora.gimrdg.ghchkpt.acfs             ora.acfs.type             OFFLINE            OFFLINE
ora.gimrdg.ghchkpt.acfs             ora.acfs.type             OFFLINE            OFFLINE
ora.helper                          ora.helper.type           OFFLINE            OFFLINE
ora.helper                          ora.helper.type           OFFLINE            OFFLINE
ora.mgmtdb                          ora.mgmtdb.type           ONLINE             ONLINE             rhel71
ora.net1.network                    ora.network.type          ONLINE             ONLINE             rhel71
ora.net1.network                    ora.network.type          ONLINE             ONLINE             rhel72
ora.ons                             ora.ons.type              ONLINE             ONLINE             rhel71
ora.ons                             ora.ons.type              ONLINE             ONLINE             rhel72
ora.proxy_advm                      ora.proxy_advm.type       ONLINE             ONLINE             rhel71
ora.proxy_advm                      ora.proxy_advm.type       ONLINE             ONLINE             rhel72
ora.qosmserver                      ora.qosmserver.type       ONLINE             ONLINE             rhel71
ora.rhel71.vip                      ora.cluster_vip_net1.type ONLINE             ONLINE             rhel71
ora.rhel72.vip                      ora.cluster_vip_net1.type ONLINE             ONLINE             rhel72
ora.rhpserver                       ora.rhpserver.type        OFFLINE            OFFLINE
ora.scan1.vip                       ora.scan_vip.type         ONLINE             ONLINE             rhel72
ora.scan2.vip                       ora.scan_vip.type         ONLINE             ONLINE             rhel71
ora.scan3.vip                       ora.scan_vip.type         ONLINE             ONLINE             rhel71
11. The default backup location for OCR is the disk group used for GIMR
ocrconfig -showbackuploc
The Oracle Cluster Registry backup location is [+GIMRDG].
12. The clusermode and versions are as follows.
$ crsctl get cluster mode status
Cluster is running in "flex" mode

$crsctl get node role config -all
Node 'rhel71' configured role is 'hub'
Node 'rhel72' configured role is 'hub'

$ crsctl query crs activeversion
Oracle Clusterware active version on the cluster is [18.0.0.0.0]
The next step is to install the database software and craete the RAC DB.

Related Posts
Installing 12cR2 (12.2.0.1) RAC on RHEL 6 with Role Separation - Clusterware
Installing 12c (12.1.0.2) Flex Cluster on RHEL 6 with Role Separation
Installing 12c (12.1.0.1) RAC on RHEL 6 with Role Separation - Clusterware
Installing 11gR2 (11.2.0.3) GI with Role Separation on RHEL 6
Installing 11gR2 (11.2.0.3) GI with Role Separation on OEL 6
Installing 11gR2 Standalone Server with ASM and Role Separation on RHEL 6
11gR2 Standalone Data Guard (with ASM and Role Separation)


Update on 2020-10-01
Following error was given when intsalling a role separated Oracle restart on 19c. Error was thrown during the Oracle home installaton.
 ./runInstaller -silent -responseFile /home/oracle/db_install.rsp -ignorePrereqFailure
Launching Oracle Database Setup Wizard...

[FATAL] [INS-30215] Installer has detected that Oracle Grid Infrastructure owner "grid" does not belong to the OS group "racdba" on local node.
   CAUSE: The OS group "racdba" does not exist or Oracle Grid Infrastructure owner "grid" is not a member of the OS group "racdba" on local node.
   ACTION: Ensure that OS group "racdba" exists and add Oracle Grid Infrastructure owner "grid" as a member of the OS group "racdba" on local node or specify a different OS group in which Oracle Grid Infrastructure owner is a member.
*ADDITIONAL INFORMATION:*
 - PRVF-7566 : User "grid" does not belong to group "racdba" on node "ip-172-31-2-77"
 - Cause:  The specified user is not a member of the specified group on the specified node.
 - Action:  Make the user a member of the group on the specified node.

It seems with 19c grid user should also be a memeber of the racdba group.

Tuesday, March 6, 2018

Oracle Data Guard on 12.2 CDB with Oracle Restart

This post lists the steps for setting up an Oracle data guard on 12.2 with Oracle restart. Both primary and standby uses ASM for data file storage. The primary database is a CDB with two PDBs. The primary and standby Oracle restart has been setup with role separation. There's a similar post for setting up data guard on 11.2 with Oracle restart.
Below are few of the new features introduced in 12.2.0.1, some of which is relevant to setting up DG with CDB and PDB.

As of Oracle Database 12c Release 2 (12.2.0.1) password file changes done on a primary database are automatically propagated to standby databases. The only exception to this is far sync instances.
A new INSTANCES [ ALL | integer] clause is available on the SQL ALTER RECOVER MANAGED STANDBY DATABASE command which enables you to control the number of instances on a physical standby that Redo Apply uses.
You can now detect lost writes and also inconsistencies between a primary database and physical standby databases by using the new PL/SQL procedure, DBMS_DBCOMP.DBCOMP.
The new ENABLED_PDBS_ON_STANDBY initialization parameter enables you to specify a subset of pluggable databases (PDBs) for replication on a physical standby of a multitenant container database (CDB). In releases prior to Oracle Database 12c Release 2 (12.2.0.1), you had to specify either all PDBs or none.
As of Oracle Database 12c Release 2 (12.2.0.1), when a physical standby database is converted into a primary you have the option to keep any sessions connected to the physical standby connected, without disruption, during the switchover/failover. To enable this feature, set the STANDBY_DB_PRESERVE_STATES initialization parameter in your init.ora file before the standby instance is started.
This parameter applies to physical standby databases only. The allowed values are:
• NONE — No sessions on the standby are retained during a switchover/failover. This is the default value.
• ALL — User sessions are retained during switchover/failover.
• SESSION — User sessions are retained during switchover/failover.
You can now use the Oracle Diagnostic Pack with an Oracle Active Data Guard standby database that is open read-only.


The table below list the relevant names used in the DG setup.
Item On Primary On Standby
Host namecity7city7s
Database Nameprodcdbstbycdb
Diskgroup NamesDATA
FRA
DATA
FRA
TNS Entry NamePRODCDBTNSSTBYTNS

1. Create the relevant *dump directories in the standby server
cd $ORACLE_BASE/admin
mkdir stbycdb
cd stbycdb
mkdir adump  dpdump  hdump  pfile
2.If not done already, put the primary database into archive log mode.
shutdown immediate;
startup mount;
alter database archivelog;
alter database open;
3. Enable force logging on the primary database
alter database force logging;
4. Create standby log files for each thread on primary db. These should be same size as the online redo log. There should be at least one more redo log group per thread than the online redo log groups.
alter database add standby logfile thread 1 size 209715200;
5. Add static listener entries on standby and primary $GI_HOME's listener.ora file. This must be done as grid user. On primary add the following static listener entry to listener.ora
SID_LIST_LISTENER =
(SID_LIST =
        (SID_DESC =
                (GLOBAL_DBNAME = prodcdb)
                (SID_NAME = prodcdb)
                (ORACLE_HOME = /opt/app/oracle/product/12.2.0/dbhome_1)
        )
)
On standby
SID_LIST_LISTENER =
(SID_LIST =
        (SID_DESC =
                (GLOBAL_DBNAME = stbycdb)
                (SID_NAME = stbycdb)
                (ORACLE_HOME = /opt/app/oracle/product/12.2.0/dbhome_1)
        )
)
stop and start the listener and verify service for static listener entry is shown (below only standby entry is shown).
srvctl stop listener
srvctl start listener

lsnrctl status

Service "stbycdb" has 1 instance(s).
  Instance "stbycdb", status UNKNOWN, has 1 handler(s) for this service...
6. As oracle user create tns entries in $ORACLE_HOME/network/admin/tnsnames.ora. The listeners on standby and primary runs on non-default port of 1581. On primary server's oracle home create a TNS entry pointing to the standby instance
STBYTNS =
  (DESCRIPTION =
    (ADDRESS = (PROTOCOL = TCP)(HOST = city7s)(PORT = 1581))
    (CONNECT_DATA =
      (SERVER = DEDICATED)
      (SERVICE_NAME = stbycdb)
    )
  )
On standby's tnsnames.ora file add an entry pointing to the primary instance
PRODCDBTNS =
  (DESCRIPTION =
    (ADDRESS = (PROTOCOL = TCP)(HOST = city7)(PORT = 1581))
    (CONNECT_DATA =
      (SERVER = DEDICATED)
      (SERVICE_NAME = prodcdb)
    )
  )
7. Create a backup pfile from the primary spfile
create pfile='/home/oracle/pfile.ora' from spfile;
8. Modify the initialization parameters on the primary db. Remote archive dest state is set to defer, enable it before running the standby duplication. LGWR with ASYNC and NOAFFIRM is used as the redo transport this should be changed to suite the desired protection mode.
alter system set log_archive_config='dg_config=(prodcdb,stbycdb)' scope=both ;
alter system set log_archive_dest_1='location=use_db_recovery_file_dest valid_for=(all_logfiles,all_roles) db_unique_name=prodcdb' scope=both;
alter system set log_archive_dest_2='service=STBYTNS LGWR ASYNC NOAFFIRM max_failure=10 max_connections=5 reopen=180 valid_for=(online_logfiles,primary_role) db_unique_name=stbycdb' scope=both;
alter system set log_archive_dest_state_2='defer' scope=both;
alter system set log_archive_dest_state_1='enable' scope=both;
alter system set fal_server='STBYTNS' scope=both;
alter system set log_archive_max_processes=10 scope=both;
alter system set db_file_name_convert='/stbycdb/','/prodcdb/' scope=spfile;
alter system set log_file_name_convert='/stbycdb/','/prodcdb/' scope=spfile;
alter system set standby_file_management='AUTO' scope=both;
Stop and start the database for changes to take effect. The initialization parameter enabled_PDBs_on_standby which is related to PDBs is left at the default value. Which means all PDBs on primary will be created on all the standbys.
SQL> show parameter enabled_PDBs_on_standby

NAME                     TYPE        VALUE
------------------------ ----------- ------
enabled_PDBs_on_standby  string      *
This only sets the minimum parameters needed for data guard setup. Optionally other data guard and performance related parameters could be set at this time or once the data guard setup is completed. One of the common parameters in this category is the lost write protection. In this setup, this parameter was set once the data guard setup was complete.
 alter system set db_lost_write_protect='typical' scope=both;
9. Copy password to standby and rename with the instance name
scp $ORACLE_HOME/dbs/orapwprodcdb city7s:$ORACLE_HOME/dbs/orapwstbycdb
10. Create pfile on standby with db_name parameter only and start the standby instance in nomount mode.
export ORACLE_SID=stbycdb
cat initstbycdb.ora
db_name=stbycdb

SQL> startup nomount;
11. Enable the remote log archive destination on primary (this was set to defer when settign the initialization parameters on step 7).
alter system set log_archive_dest_state_2='enable' SCOPE=both sid='*';
12. Unlike previous post, this DG creation does not use active data database option to create the standby DB. (At the end of the post, commands to create standby DB using active data base option is shown). Instead it uses backups from primary to create the standby database (similar to duplicating with backups). Below are the necessary RMAN commands to create the required backup files for standby duplication.
backup spfile format '/home/oracle/backup/spbackup.bkp';
backup current controlfile for standby format '/home/oracle/backup/stdbycontro.ctl';
backup database format '/home/oracle/backup/dbbackup%U' plus archivelog format '/home/oracle/backup/archbackup%U' delete all input;
switch few log files and backup the archive logs
sql 'alter system switch logfile';
sql 'alter system switch logfile';
sql 'alter system switch logfile';
backup archivelog all format '/home/oracle/backup/archbkp%U' delete all input;
13. Copy backup files to standby host. Backups are on a different location on standby than primary.
cd /home/oracle/backup
scp * city7s:/home/oracle/backup_dir
14. Login to standby server and connect to the standby instance using an auxiliary connection and run duplicate database for standby command. Since the command is run from standby omit the target keyword (duplicating while connected to primary is shown at the end of the post). Since ASM disk group names are same on both primary and standby lot of the db_*_dest* parameters could be kept same as primary. If not these would require file name conversions with disk groups prefixed as shown in previous post.
rman auxiliary sys/prodcdb
  
duplicate database for standby
spfile
parameter_value_convert 'prodcdb','stbycdb','PRODCDB','STBYCDB'
set db_name='prodcdb'
set db_unique_name='stbycdb'
set db_file_name_convert='/prodcdb/','/stbycdb/'
set log_file_name_convert='/prodcdb/','/stbycdb/'
set log_archive_max_processes='10'
set fal_server='PRODCDBTNS'
reset local_listener
set log_archive_dest_2='service=PRODCDBTNS LGWR ASYNC NOAFFIRM max_failure=10 max_connections=5 reopen=180 valid_for=(online_logfiles,primary_role) db_unique_name=prodcdb'
set log_archive_dest_1='location=use_db_recovery_file_dest valid_for=(all_logfiles,all_roles) db_unique_name=stbycdb'
BACKUP LOCATION '/home/oracle/backup_dir' dorecover nofilenamecheck;


Following output shows the full duplication process.
rman auxiliary sys/prodcdbdb

Recovery Manager: Release 12.2.0.1.0 - Production on Fri Feb 16 14:23:14 2018
Copyright (c) 1982, 2017, Oracle and/or its affiliates.  All rights reserved.
connected to auxiliary database: STBYCDB (not mounted)

RMAN> duplicate database for standby
2> spfile
parameter_value_convert 'prodcdb','stbycdb','PRODCDB','STBYCDB'
3> 4> set db_name='prodcdb'
5> set db_unique_name='stbycdb'
6> set db_file_name_convert='/prodcdb/','/stbycdb/'
set log_file_name_convert='/prodcdb/','/stbycdb/'
7> 8> set log_archive_max_processes='10'
set fal_server='PRODCDBTNS'
reset local_listener
9> 10> 11> set log_archive_dest_2='service=PRODCDBTNS LGWR ASYNC NOAFFIRM max_failure=10 max_connections=5 reopen=180 valid_for=(online_logfiles,primary_role) db_unique_name=prodcdb'
set log_archive_dest_1='location=use_db_recovery_file_dest valid_for=(all_logfiles,all_roles) db_unique_name=stbycdb'
12> 13> BACKUP LOCATION '/home/oracle/backup_dir' dorecover nofilenamecheck;

Starting Duplicate Db at 16-FEB-18

contents of Memory Script:
{
   restore clone spfile to  '/opt/app/oracle/product/12.2.0/dbhome_1/dbs/spfilestbycdb.ora' from
 '/home/oracle/backup_dir/spbackup.bkp';
   sql clone "alter system set spfile= ''/opt/app/oracle/product/12.2.0/dbhome_1/dbs/spfilestbycdb.ora''";
}
executing Memory Script

Starting restore at 16-FEB-18
allocated channel: ORA_AUX_DISK_1
channel ORA_AUX_DISK_1: SID=10 device type=DISK

channel ORA_AUX_DISK_1: restoring spfile from AUTOBACKUP /home/oracle/backup_dir/spbackup.bkp
channel ORA_AUX_DISK_1: SPFILE restore from AUTOBACKUP complete
Finished restore at 16-FEB-18

sql statement: alter system set spfile= ''/opt/app/oracle/product/12.2.0/dbhome_1/dbs/spfilestbycdb.ora''

contents of Memory Script:
{
   sql clone "alter system set  audit_file_dest =
 ''/opt/app/oracle/admin/stbycdb/adump'' comment=
 '''' scope=spfile";
   sql clone "alter system set  control_files =
 ''+DATA/STBYCDB/CONTROLFILE/current.266.965841019'', ''+FRA/STBYCDB/CONTROLFILE/current.256.965841019'' comment=
 '''' scope=spfile";
   sql clone "alter system set  dispatchers =
 ''(PROTOCOL=TCP) (SERVICE=stbycdbXDB)'' comment=
 '''' scope=spfile";
   sql clone "alter system set  db_name =
 ''prodcdb'' comment=
 '''' scope=spfile";
   sql clone "alter system set  db_unique_name =
 ''stbycdb'' comment=
 '''' scope=spfile";
   sql clone "alter system set  db_file_name_convert =
 ''/prodcdb/'', ''/stbycdb/'' comment=
 '''' scope=spfile";
   sql clone "alter system set  log_file_name_convert =
 ''/prodcdb/'', ''/stbycdb/'' comment=
 '''' scope=spfile";
   sql clone "alter system set  log_archive_max_processes =
 10 comment=
 '''' scope=spfile";
   sql clone "alter system set  fal_server =
 ''PRODCDBTNS'' comment=
 '''' scope=spfile";
   sql clone "alter system reset  local_listener scope=spfile";
   sql clone "alter system set  log_archive_dest_2 =
 ''service=PRODCDBTNS LGWR ASYNC NOAFFIRM max_failure=10 max_connections=5 reopen=180 valid_for=(online_logfiles,primary_role) db_unique_name=prodcdb'' comment=
 '''' scope=spfile";
   sql clone "alter system set  log_archive_dest_1 =
 ''location=use_db_recovery_file_dest valid_for=(all_logfiles,all_roles) db_unique_name=stbycdb'' comment=
 '''' scope=spfile";
   shutdown clone immediate;
   startup clone nomount;
}
executing Memory Script

sql statement: alter system set  audit_file_dest =  ''/opt/app/oracle/admin/stbycdb/adump'' comment= '''' scope=spfile

sql statement: alter system set  control_files =  ''+DATA/STBYCDB/CONTROLFILE/current.266.965841019'', ''+FRA/STBYCDB/CONTROLFILE/current.256.965841019'' comment= '''' scope=spfile

sql statement: alter system set  dispatchers =  ''(PROTOCOL=TCP) (SERVICE=stbycdbXDB)'' comment= '''' scope=spfile

sql statement: alter system set  db_name =  ''prodcdb'' comment= '''' scope=spfile

sql statement: alter system set  db_unique_name =  ''stbycdb'' comment= '''' scope=spfile

sql statement: alter system set  db_file_name_convert =  ''/prodcdb/'', ''/stbycdb/'' comment= '''' scope=spfile

sql statement: alter system set  log_file_name_convert =  ''/prodcdb/'', ''/stbycdb/'' comment= '''' scope=spfile

sql statement: alter system set  log_archive_max_processes =  10 comment= '''' scope=spfile

sql statement: alter system set  fal_server =  ''PRODCDBTNS'' comment= '''' scope=spfile

sql statement: alter system reset  local_listener scope=spfile

sql statement: alter system set  log_archive_dest_2 =  ''service=PRODCDBTNS LGWR ASYNC NOAFFIRM max_failure=10 max_connections=5 reopen=180 valid_for=(online_logfiles,primary_role) db_unique_name=prodcdb'' comment= '''' scope=spfile

sql statement: alter system set  log_archive_dest_1 =  ''location=use_db_recovery_file_dest valid_for=(all_logfiles,all_roles) db_unique_name=stbycdb'' comment= '''' scope=spfile

Oracle instance shut down

connected to auxiliary database (not started)
Oracle instance started

Total System Global Area    1191182336 bytes

Fixed Size                     8792104 bytes
Variable Size                452986840 bytes
Database Buffers             721420288 bytes
Redo Buffers                   7983104 bytes

contents of Memory Script:
{
   sql clone "alter system set  control_files =
  ''+DATA/STBYCDB/CONTROLFILE/current.263.968250233'', ''+FRA/STBYCDB/CONTROLFILE/current.263.968250233'' comment=
 ''Set by RMAN'' scope=spfile";
   restore clone standby controlfile from  '/home/oracle/backup_dir/stdbycontro.ctl';
}
executing Memory Script

sql statement: alter system set  control_files =   ''+DATA/STBYCDB/CONTROLFILE/current.263.968250233'', ''+FRA/STBYCDB/CONTROLFILE/current.263.968250233'' comment= ''Set by RMAN'' scope=spfile

Starting restore at 16-FEB-18
allocated channel: ORA_AUX_DISK_1
channel ORA_AUX_DISK_1: SID=10 device type=DISK

channel ORA_AUX_DISK_1: restoring control file
channel ORA_AUX_DISK_1: restore complete, elapsed time: 00:00:01
output file name=+DATA/STBYCDB/CONTROLFILE/current.262.968250233
output file name=+FRA/STBYCDB/CONTROLFILE/current.264.968250235
Finished restore at 16-FEB-18

contents of Memory Script:
{
   sql clone 'alter database mount standby database';
}
executing Memory Script

sql statement: alter database mount standby database
released channel: ORA_AUX_DISK_1
allocated channel: ORA_AUX_DISK_1
channel ORA_AUX_DISK_1: SID=10 device type=DISK
RMAN-05529: warning: DB_FILE_NAME_CONVERT resulted in invalid ASM names; names changed to disk group only.

contents of Memory Script:
{
   set until scn  1495617;
   set newname for tempfile  1 to
 "+DATA";
   set newname for tempfile  2 to
 "+DATA";
   set newname for tempfile  3 to
 "+DATA";
   set newname for tempfile  4 to
 "+DATA";
   switch clone tempfile all;
   set newname for datafile  1 to
 "+DATA";
   set newname for datafile  2 to
 "+DATA";
   set newname for datafile  3 to
 "+DATA";
   set newname for datafile  4 to
 "+DATA";
   set newname for datafile  5 to
 "+DATA";
   set newname for datafile  6 to
 "+DATA";
   set newname for datafile  7 to
 "+DATA";
   set newname for datafile  11 to
 "+DATA";
   set newname for datafile  12 to
 "+DATA";
   set newname for datafile  13 to
 "+DATA";
   set newname for datafile  14 to
 "+DATA";
   set newname for datafile  15 to
 "+DATA";
   set newname for datafile  16 to
 "+DATA";
   set newname for datafile  17 to
 "+DATA";
   restore
   clone database
   ;
}
executing Memory Script

executing command: SET until clause

executing command: SET NEWNAME

executing command: SET NEWNAME

executing command: SET NEWNAME

executing command: SET NEWNAME

renamed tempfile 1 to +DATA in control file
renamed tempfile 2 to +DATA in control file
renamed tempfile 3 to +DATA in control file
renamed tempfile 4 to +DATA in control file

executing command: SET NEWNAME

executing command: SET NEWNAME

executing command: SET NEWNAME

executing command: SET NEWNAME

executing command: SET NEWNAME

executing command: SET NEWNAME

executing command: SET NEWNAME

executing command: SET NEWNAME

executing command: SET NEWNAME

executing command: SET NEWNAME

executing command: SET NEWNAME

executing command: SET NEWNAME

executing command: SET NEWNAME

executing command: SET NEWNAME

Starting restore at 16-FEB-18
using channel ORA_AUX_DISK_1

channel ORA_AUX_DISK_1: starting datafile backup set restore
channel ORA_AUX_DISK_1: specifying datafile(s) to restore from backup set
channel ORA_AUX_DISK_1: restoring datafile 00001 to +DATA
channel ORA_AUX_DISK_1: restoring datafile 00003 to +DATA
channel ORA_AUX_DISK_1: restoring datafile 00005 to +DATA
channel ORA_AUX_DISK_1: restoring datafile 00007 to +DATA
channel ORA_AUX_DISK_1: reading from backup piece /home/oracle/backup_dir/dbbackup2isrce37_1_1
channel ORA_AUX_DISK_1: piece handle=/home/oracle/backup_dir/dbbackup2isrce37_1_1 tag=TAG20180216T122822
channel ORA_AUX_DISK_1: restored backup piece 1
channel ORA_AUX_DISK_1: restore complete, elapsed time: 00:00:36
channel ORA_AUX_DISK_1: starting datafile backup set restore
channel ORA_AUX_DISK_1: specifying datafile(s) to restore from backup set
channel ORA_AUX_DISK_1: restoring datafile 00002 to +DATA
channel ORA_AUX_DISK_1: restoring datafile 00004 to +DATA
channel ORA_AUX_DISK_1: restoring datafile 00006 to +DATA
channel ORA_AUX_DISK_1: reading from backup piece /home/oracle/backup_dir/dbbackup2ksrce3e_1_1
channel ORA_AUX_DISK_1: piece handle=/home/oracle/backup_dir/dbbackup2ksrce3e_1_1 tag=TAG20180216T122822
channel ORA_AUX_DISK_1: restored backup piece 1
channel ORA_AUX_DISK_1: restore complete, elapsed time: 00:00:25
channel ORA_AUX_DISK_1: starting datafile backup set restore
channel ORA_AUX_DISK_1: specifying datafile(s) to restore from backup set
channel ORA_AUX_DISK_1: restoring datafile 00011 to +DATA
channel ORA_AUX_DISK_1: restoring datafile 00012 to +DATA
channel ORA_AUX_DISK_1: restoring datafile 00013 to +DATA
channel ORA_AUX_DISK_1: reading from backup piece /home/oracle/backup_dir/dbbackup2lsrce3h_1_1
channel ORA_AUX_DISK_1: piece handle=/home/oracle/backup_dir/dbbackup2lsrce3h_1_1 tag=TAG20180216T122822
channel ORA_AUX_DISK_1: restored backup piece 1
channel ORA_AUX_DISK_1: restore complete, elapsed time: 00:00:15
channel ORA_AUX_DISK_1: starting datafile backup set restore
channel ORA_AUX_DISK_1: specifying datafile(s) to restore from backup set
channel ORA_AUX_DISK_1: restoring datafile 00014 to +DATA
channel ORA_AUX_DISK_1: restoring datafile 00015 to +DATA
channel ORA_AUX_DISK_1: restoring datafile 00016 to +DATA
channel ORA_AUX_DISK_1: restoring datafile 00017 to +DATA
channel ORA_AUX_DISK_1: reading from backup piece /home/oracle/backup_dir/dbbackup2jsrce3a_1_1
channel ORA_AUX_DISK_1: piece handle=/home/oracle/backup_dir/dbbackup2jsrce3a_1_1 tag=TAG20180216T122822
channel ORA_AUX_DISK_1: restored backup piece 1
channel ORA_AUX_DISK_1: restore complete, elapsed time: 00:00:15
Finished restore at 16-FEB-18

contents of Memory Script:
{
   switch clone datafile all;
}
executing Memory Script

datafile 1 switched to datafile copy
input datafile copy RECID=15 STAMP=968250346 file name=+DATA/STBYCDB/DATAFILE/system.261.968250253
datafile 2 switched to datafile copy
input datafile copy RECID=16 STAMP=968250346 file name=+DATA/STBYCDB/6325282AC695380EE0535500A8C0D89D/DATAFILE/system.271.968250291
datafile 3 switched to datafile copy
input datafile copy RECID=17 STAMP=968250346 file name=+DATA/STBYCDB/DATAFILE/sysaux.267.968250255
datafile 4 switched to datafile copy
input datafile copy RECID=18 STAMP=968250346 file name=+DATA/STBYCDB/6325282AC695380EE0535500A8C0D89D/DATAFILE/sysaux.270.968250291
datafile 5 switched to datafile copy
input datafile copy RECID=19 STAMP=968250347 file name=+DATA/STBYCDB/DATAFILE/undotbs1.268.968250255
datafile 6 switched to datafile copy
input datafile copy RECID=20 STAMP=968250347 file name=+DATA/STBYCDB/6325282AC695380EE0535500A8C0D89D/DATAFILE/undotbs1.272.968250291
datafile 7 switched to datafile copy
input datafile copy RECID=21 STAMP=968250347 file name=+DATA/STBYCDB/DATAFILE/users.269.968250255
datafile 11 switched to datafile copy
input datafile copy RECID=22 STAMP=968250347 file name=+DATA/STBYCDB/6360B564DAA26345E0535500A8C04976/DATAFILE/system.274.968250315
datafile 12 switched to datafile copy
input datafile copy RECID=23 STAMP=968250347 file name=+DATA/STBYCDB/6360B564DAA26345E0535500A8C04976/DATAFILE/sysaux.273.968250315
datafile 13 switched to datafile copy
input datafile copy RECID=24 STAMP=968250347 file name=+DATA/STBYCDB/6360B564DAA26345E0535500A8C04976/DATAFILE/undotbs1.275.968250315
datafile 14 switched to datafile copy
input datafile copy RECID=25 STAMP=968250347 file name=+DATA/STBYCDB/6360CF1837BF7EBBE0535500A8C0DE5B/DATAFILE/system.277.968250331
datafile 15 switched to datafile copy
input datafile copy RECID=26 STAMP=968250347 file name=+DATA/STBYCDB/6360CF1837BF7EBBE0535500A8C0DE5B/DATAFILE/sysaux.276.968250331
datafile 16 switched to datafile copy
input datafile copy RECID=27 STAMP=968250347 file name=+DATA/STBYCDB/6360CF1837BF7EBBE0535500A8C0DE5B/DATAFILE/undotbs1.278.968250331
datafile 17 switched to datafile copy
input datafile copy RECID=28 STAMP=968250347 file name=+DATA/STBYCDB/6360CF1837BF7EBBE0535500A8C0DE5B/DATAFILE/users.279.968250331

contents of Memory Script:
{
   set until scn  1495617;
   recover
   standby
   clone database
    delete archivelog
   ;
}
executing Memory Script

executing command: SET until clause

Starting recover at 16-FEB-18
using channel ORA_AUX_DISK_1

starting media recovery

channel ORA_AUX_DISK_1: starting archived log restore to default destination
channel ORA_AUX_DISK_1: restoring archived log
archived log thread=1 sequence=42
channel ORA_AUX_DISK_1: reading from backup piece /home/oracle/backup_dir/archbackup2hsrce34_1_1
channel ORA_AUX_DISK_1: piece handle=/home/oracle/backup_dir/archbackup2hsrce34_1_1 tag=TAG20180216T122820
channel ORA_AUX_DISK_1: restored backup piece 1
channel ORA_AUX_DISK_1: restore complete, elapsed time: 00:00:01
archived log file name=+FRA/STBYCDB/ARCHIVELOG/2018_02_16/thread_1_seq_42.265.968250369 thread=1 sequence=0
channel clone_default: deleting archived log(s)
archived log file name=+FRA/STBYCDB/ARCHIVELOG/2018_02_16/thread_1_seq_42.265.968250369 RECID=1 STAMP=968250369
channel ORA_AUX_DISK_1: starting archived log restore to default destination
channel ORA_AUX_DISK_1: restoring archived log
archived log thread=1 sequence=43
channel ORA_AUX_DISK_1: reading from backup piece /home/oracle/backup_dir/archbackup2msrce3l_1_1
channel ORA_AUX_DISK_1: piece handle=/home/oracle/backup_dir/archbackup2msrce3l_1_1 tag=TAG20180216T122837
channel ORA_AUX_DISK_1: restored backup piece 1
channel ORA_AUX_DISK_1: restore complete, elapsed time: 00:00:01
archived log file name=+FRA/STBYCDB/ARCHIVELOG/2018_02_16/thread_1_seq_43.265.968250371 thread=1 sequence=43
channel clone_default: deleting archived log(s)
archived log file name=+FRA/STBYCDB/ARCHIVELOG/2018_02_16/thread_1_seq_43.265.968250371 RECID=2 STAMP=968250371
channel ORA_AUX_DISK_1: starting archived log restore to default destination
channel ORA_AUX_DISK_1: restoring archived log
archived log thread=1 sequence=44
channel ORA_AUX_DISK_1: restoring archived log
archived log thread=1 sequence=45
channel ORA_AUX_DISK_1: restoring archived log
archived log thread=1 sequence=46
channel ORA_AUX_DISK_1: restoring archived log
archived log thread=1 sequence=47
channel ORA_AUX_DISK_1: restoring archived log
archived log thread=1 sequence=48
channel ORA_AUX_DISK_1: reading from backup piece /home/oracle/backup_dir/archbkp2osrce4b_1_1
channel ORA_AUX_DISK_1: piece handle=/home/oracle/backup_dir/archbkp2osrce4b_1_1 tag=TAG20180216T122859
channel ORA_AUX_DISK_1: restored backup piece 1
channel ORA_AUX_DISK_1: restore complete, elapsed time: 00:00:01
archived log file name=+FRA/STBYCDB/ARCHIVELOG/2018_02_16/thread_1_seq_44.265.968250373 thread=1 sequence=44
channel clone_default: deleting archived log(s)
archived log file name=+FRA/STBYCDB/ARCHIVELOG/2018_02_16/thread_1_seq_44.265.968250373 RECID=4 STAMP=968250373
archived log file name=+FRA/STBYCDB/ARCHIVELOG/2018_02_16/thread_1_seq_45.266.968250373 thread=1 sequence=45
channel clone_default: deleting archived log(s)
archived log file name=+FRA/STBYCDB/ARCHIVELOG/2018_02_16/thread_1_seq_45.266.968250373 RECID=5 STAMP=968250373
archived log file name=+FRA/STBYCDB/ARCHIVELOG/2018_02_16/thread_1_seq_46.269.968250373 thread=1 sequence=46
channel clone_default: deleting archived log(s)
archived log file name=+FRA/STBYCDB/ARCHIVELOG/2018_02_16/thread_1_seq_46.269.968250373 RECID=3 STAMP=968250373
archived log file name=+FRA/STBYCDB/ARCHIVELOG/2018_02_16/thread_1_seq_47.268.968250373 thread=1 sequence=47
channel clone_default: deleting archived log(s)
archived log file name=+FRA/STBYCDB/ARCHIVELOG/2018_02_16/thread_1_seq_47.268.968250373 RECID=7 STAMP=968250373
archived log file name=+FRA/STBYCDB/ARCHIVELOG/2018_02_16/thread_1_seq_48.267.968250373 thread=1 sequence=48
channel clone_default: deleting archived log(s)
archived log file name=+FRA/STBYCDB/ARCHIVELOG/2018_02_16/thread_1_seq_48.267.968250373 RECID=6 STAMP=968250373
media recovery complete, elapsed time: 00:00:03
Finished recover at 16-FEB-18

Finished Duplicate Db at 16-FEB-18
15. Once the duplication for standby is completed start the log apply process on standby. Monitor the standby instance's alert log for MRP starting and query the archived log view for log apply progress.
alter database recover managed standby database using current logfile disconnect;

select sequence#,applied from v$archived_log;
 SEQUENCE# APPLIED
---------- ---------
       273 YES
       274 YES
       275 YES
       276 YES
       277 YES
       278 YES
       279 YES
       280 IN-MEMORY
16. Final step is to bring the database under the control of oracle restart. To do this add the database to the oracle restart configuration with srvctl add database. This must be run as oracle user not as grid user. Running the command as grid user would fail as oracle home is owned by a different user
[grid@city7s ~]$ srvctl add database -db stbycdb -oraclehome /opt/app/oracle/product/12.2.0/dbhome_1  -spfile "/opt/app/oracle/product/12.2.0/dbhome_1/dbs/spfilestbycdb.ora" -role physical_standby -startoption mount -diskgroup "DATA,FRA"
PRCD-1025 : Failed to create database stbycdb
PRKH-1014 : Current user "grid" is not the oracle owner user "oracle" of oracle home "/opt/app/oracle/product/12.2.0/dbhome_1"
As oracle user run
$GI_HOME/bin/srvctl add database -db stbycdb -oraclehome $ORACLE_HOME  -spfile "/opt/app/oracle/product/12.2.0/dbhome_1/dbs/spfilestbycdb.ora" -role physical_standby -startoption mount -diskgroup "DATA,FRA"
Verify the database configuration
srvctl config database -db stbycdb
Database unique name: stbycdb
Database name:
Oracle home: /opt/app/oracle/product/12.2.0/dbhome_1
Oracle user: oracle
Spfile: /opt/app/oracle/product/12.2.0/dbhome_1/dbs/spfilestbycdb.ora
Password file:
Domain:
Start options: mount
Stop options: immediate
Database role: PHYSICAL_STANDBY
Management policy: AUTOMATIC
Disk Groups: DATA,FRA
Services:
OSDBA group:
OSOPER group:
Database instance: stbycdb
At this stage the oracle binary in standby oracle home will not have the correct ownership with regard to when role separation is used.
[oracle@city7s bin]$ cd $ORACLE_HOME/bin
[oracle@city7s bin]$ ls -l oracle
-rwsr-s--x. 1 oracle oinstall 408115488 Jan 19 16:27 oracle
This could lead to the issue highlighted here. In fact if the standby database is to be started without correcting the permission it will fail as shown below
srvctl  start database -db stbycdb
PRCR-1079 : Failed to start resource ora.stbycdb.db
CRS-5017: The resource action "ora.stbycdb.db start" encountered the following error:
ORA-27140: attach to post/wait facility failed
ORA-27300: OS system dependent operation:invalid_egid failed with status: 1
ORA-27301: OS failure message: Operation not permitted
ORA-27302: failure occurred at: skgpwinit6
ORA-27303: additional information: startup egid = 1001 (oinstall), current egid = 1005 (asmadmin)
. For details refer to "(:CLSN00107:)" in "/opt/app/oracle/diag/crs/city7s/crs/trace/ohasd_oraagent_grid.trc".

CRS-2674: Start of 'ora.stbycdb.db' on 'city7s' failed
Interestingly after the failure first time the permission get applied automatically to the oracle binary (without any manual intervention). So, if attempted the standby database start will succeed second time around.
This conclude the setting up of DG with CDB.

Standby DB creation with target database
In this case the location to which backup files are copied to in step 13 above must match the backup file location on primary. (i.e. if on primary backup files are on /home/oracle/backup then on standby also it must be on /home/oracle/backup).
While in primary server, connect to both target and standby (with an auxiliary connection) and run the duplicate command
rman target sys/prodcdbdb auxiliary sys/prodcdbdb@stbytns
run {
duplicate target database for standby
spfile
parameter_value_convert 'prodcdb','stbycdb','PRODCDB','STBYCDB'
set db_name='prodcdb'
set db_unique_name='stbycdb'
set db_file_name_convert='/prodcdb/','/stbycdb/'
set log_file_name_convert='/prodcdb/','/stbycdb/'
set log_archive_max_processes='10'
set fal_server='PRODCDBTNS'
reset local_listener
set log_archive_dest_2='service=PRODCDBTNS LGWR ASYNC NOAFFIRM max_failure=10 max_connections=5 reopen=180 valid_for=(online_logfiles,primary_role) db_unique_name=prodcdb'
set log_archive_dest_1='location=use_db_recovery_file_dest valid_for=(all_logfiles,all_roles) db_unique_name=stbycdb'
dorecover nofilenamecheck;
}
Standby DB creation with active database
With active database option there's no need to create any backups. Connect to both target and standby and run the duplicate command with active database.
rman target sys/prodcdbdb auxiliary sys/prodcdbdb@stbytns
duplicate target database for standby from active database
spfile
parameter_value_convert 'prodcdb','stbycdb','PRODCDB','STBYCDB'
set db_name='prodcdb'
set db_unique_name='stbycdb'
set db_file_name_convert='/prodcdb/','/stbycdb/'
set log_file_name_convert='/prodcdb/','/stbycdb/'
set log_archive_max_processes='10'
set fal_server='PRODCDBTNS'
reset local_listener
set log_archive_dest_2='service=PRODCDBTNS LGWR ASYNC NOAFFIRM max_failure=10 max_connections=5 reopen=180 valid_for=(online_logfiles,primary_role) db_unique_name=prodcdb'
set log_archive_dest_1='location=use_db_recovery_file_dest valid_for=(all_logfiles,all_roles) db_unique_name=stbycdb';
If the active data guard license is available then standby database could be open for read only access. To do this cancel the managed recovery, open the CDB root in read only mode and then the PDBs.
SQL> alter database recover managed standby database cancel;
SQL> alter database open read only;
SQL> alter pluggable database all open read only;
SQL> show pdbs;

    CON_ID CON_NAME                       OPEN MODE  RESTRICTED
---------- ------------------------------ ---------- ----------
         2 PDB$SEED                       READ ONLY  NO
         3 PDBAPP1                        READ ONLY  NO
         4 PDBAPP2                        READ ONLY  NO

Related Posts
Creating Data Guard Broker on 12c
Adding Far Sync Instances to Existing Data Guard Configuration
11gR2 RAC to RAC Data Guard
11gR2 Standalone Data Guard (with ASM and Role Separation)
Setting Up Data Guard Broker for an Existing Data Guard Configuration with Far Sync
Enable Active Dataguard on 11gR2 RAC Standby