nslookup rac-scan.londbsubnet.lonvcn.oraclevcn.com Server: 169.254.169.254 Address: 169.254.169.254#53 Non-authoritative answer: Name: rac-scan.londbsubnet.lonvcn.oraclevcn.com Address: 172.21.0.7 Name: rac-scan.londbsubnet.lonvcn.oraclevcn.com Address: 172.21.0.8 Name: rac-scan.londbsubnet.lonvcn.oraclevcn.com Address: 172.21.0.6The hostnames will be suffixed with node number when the cluster is created.
oifcfg getif eth0 172.21.0.0 global public eth1 192.168.16.0 global cluster_interconnect,asmThe /etc/oratab files on all nodes are empty of any ASM or DB instance informations. These need to be added manually if needed.
The RAC configuration is role separated. Following roles exists for oracle and grid user. There are no separate groups for backupdba, kmdba, dgdba or racdba.
[opc@rac1 ~]$ id oracle uid=101(oracle) gid=1001(oinstall) groups=1001(oinstall),1006(asmdba),1003(dba),1002(dbaoper) [opc@rac1 ~]$ id grid uid=102(grid) gid=1001(oinstall) groups=1001(oinstall),1006(asmdba),1005(asmoper),1004(asmadmin),1002(dbaoper)There's no user equivalence between nodes for oracle or grid user.
cluvfy stage -post crsinst -n all Verifying User Equivalence ...FAILED (PRVG-2019, PRKC-1191) PRVF-4009 : User equivalence is not set for nodes: rac2 [oracle@rac1 ~]$ ssh rac2 Permission denied (publickey,gssapi-keyex,gssapi-with-mic). [oracle@rac2 ~]$ ssh rac1 Permission denied (publickey,gssapi-keyex,gssapi-with-mic). [grid@rac1 ~]$ ssh rac2 Permission denied (publickey,gssapi-keyex,gssapi-with-mic). [grid@rac2 ~]$ ssh rac1 Permission denied (publickey,gssapi-keyex,gssapi-with-mic).Oracle documentation mentions (this documentation may not be relevant any more) that user equivalence exists for opc user. But that too didn't exists.
[opc@rac1 ~]$ ssh rac2 Permission denied (publickey,gssapi-keyex,gssapi-with-mic). [opc@rac2 ~]$ ssh rac1 Permission denied (publickey,gssapi-keyex,gssapi-with-mic).
The cluster does not use GNS. Below is the full list of resources at the end of the cluster creation.
Resource Name Type Target State Host ------------- ------ ------- -------- ---------- ora.ASMNET1LSNR_ASM.lsnr ora.asm_listener.type ONLINE ONLINE rac1 ora.ASMNET1LSNR_ASM.lsnr ora.asm_listener.type ONLINE ONLINE rac2 ora.DATA.COMMONSTORE.advm ora.volume.type ONLINE ONLINE rac1 ora.DATA.COMMONSTORE.advm ora.volume.type ONLINE ONLINE rac2 ora.DATA.dg ora.diskgroup.type ONLINE ONLINE rac1 ora.DATA.dg ora.diskgroup.type ONLINE ONLINE rac2 ora.LISTENER.lsnr ora.listener.type ONLINE ONLINE rac1 ora.LISTENER.lsnr ora.listener.type ONLINE ONLINE rac2 ora.LISTENER_SCAN1.lsnr ora.scan_listener.type ONLINE ONLINE rac1 ora.LISTENER_SCAN2.lsnr ora.scan_listener.type ONLINE ONLINE rac2 ora.LISTENER_SCAN3.lsnr ora.scan_listener.type ONLINE ONLINE rac2 ora.RECO.dg ora.diskgroup.type ONLINE ONLINE rac1 ora.RECO.dg ora.diskgroup.type ONLINE ONLINE rac2 ora.asm ora.asm.type ONLINE ONLINE rac1 ora.asm ora.asm.type ONLINE ONLINE rac2 ora.cdbrac_lhr1fq.db ora.database.type ONLINE ONLINE rac1 ora.cdbrac_lhr1fq.db ora.database.type ONLINE ONLINE rac2 ora.cvu ora.cvu.type ONLINE ONLINE rac2 ora.data.commonstore.acfs ora.acfs.type ONLINE ONLINE rac1 ora.data.commonstore.acfs ora.acfs.type ONLINE ONLINE rac2 ora.net1.network ora.network.type ONLINE ONLINE rac1 ora.net1.network ora.network.type ONLINE ONLINE rac2 ora.ons ora.ons.type ONLINE ONLINE rac1 ora.ons ora.ons.type ONLINE ONLINE rac2 ora.proxy_advm ora.proxy_advm.type ONLINE ONLINE rac1 ora.proxy_advm ora.proxy_advm.type ONLINE ONLINE rac2 ora.qosmserver ora.qosmserver.type OFFLINE OFFLINE ora.rac1.vip ora.cluster_vip_net1.type ONLINE ONLINE rac1 ora.rac2.vip ora.cluster_vip_net1.type ONLINE ONLINE rac2 ora.scan1.vip ora.scan_vip.type ONLINE ONLINE rac1 ora.scan2.vip ora.scan_vip.type ONLINE ONLINE rac2 ora.scan3.vip ora.scan_vip.type ONLINE ONLINE rac2Similar to single instance DB RAC DB too consists of two ASM disk groups. OCR and vote is stored in the DATA disk group. The backup location for OCR is also the DATA disk group.
SQL> select name,total_mb,free_Mb from v$asm_diskgroup; NAME TOTAL_MB FREE_MB ------------------------------ ---------- ---------- DATA 262144 253408 RECO 262144 257380 cat /etc/oracle/ocr.loc #Device/file +DATA getting replaced by device +DATA/dbSyslw4gbgsa/OCRFILE/registry.255.1002113311 ocrconfig_loc=+DATA/dbSyslw4gbgsa/OCRFILE/registry.255.1002113311 [grid@rac1 ~]$ crsctl query css votedisk ## STATE File Universal Id File Name Disk group -- ----- ----------------- --------- --------- 1. ONLINE 1e4fd3b7b9364fdfbf4dbf41ffde36d9 (/dev/DATADISK3) [DATA] ocrconfig -showbackuploc The Oracle Cluster Registry backup location is [+DATA].ASM disks are SCSI attached and permissions are set with udev rules.
SQL> select name,path from v$asm_disk;
NAME PATH
------------------------------ ------------------------------
RECODISK4 /dev/RECODISK4
RECODISK3 /dev/RECODISK3
RECODISK2 /dev/RECODISK2
RECODISK1 /dev/RECODISK1
DATA_0003 /dev/DATADISK4
DATA_0000 /dev/DATADISK3
DATA_0001 /dev/DATADISK2
DATA_0002 /dev/DATADISK1
[grid@rac1 admin]$ ls -l /dev/DATA*
lrwxrwxrwx 1 root root 3 Mar 5 14:58 /dev/DATADISK1 -> sdb
lrwxrwxrwx 1 root root 3 Mar 5 14:58 /dev/DATADISK2 -> sdc
lrwxrwxrwx 1 root root 3 Mar 5 14:58 /dev/DATADISK3 -> sdd
lrwxrwxrwx 1 root root 3 Mar 5 14:58 /dev/DATADISK4 -> sde
[grid@rac1 admin]$ ls -l /dev/disk/by-id/ | egrep "sdb|sdc|sdd|sde"
lrwxrwxrwx 1 root root 9 Mar 5 14:59 scsi-36012159faf2d4794be5ab57d8a5b7498 -> ../../sdc
lrwxrwxrwx 1 root root 10 Mar 5 12:32 scsi-36012159faf2d4794be5ab57d8a5b7498-part1 -> ../../sdc1
lrwxrwxrwx 1 root root 9 Mar 5 14:59 scsi-36022a3f8dffa46b4a57049af3a2dd275 -> ../../sdb
lrwxrwxrwx 1 root root 10 Mar 5 12:32 scsi-36022a3f8dffa46b4a57049af3a2dd275-part1 -> ../../sdb1
lrwxrwxrwx 1 root root 9 Mar 5 14:59 scsi-360748b62328c4ff6bb8fe3e9e4f04826 -> ../../sdd
lrwxrwxrwx 1 root root 10 Mar 5 12:32 scsi-360748b62328c4ff6bb8fe3e9e4f04826-part1 -> ../../sdd1
lrwxrwxrwx 1 root root 9 Mar 5 14:59 scsi-360923cd17c134686829d791e2b4ff82f -> ../../sde
lrwxrwxrwx 1 root root 10 Mar 5 12:32 scsi-360923cd17c134686829d791e2b4ff82f-part1 -> ../../sde1
cat /etc/udev/rules.d/70-names.rules
KERNEL=="sd*", ENV{DEVTYPE}=="disk", ENV{ID_SERIAL}=="36022a3f8dffa46b4a57049af3a2dd275", SYMLINK+="DATADISK1", OWNER="grid", GROUP="asmadmin", MODE="0660"
KERNEL=="sd*", ENV{DEVTYPE}=="disk", ENV{ID_SERIAL}=="36012159faf2d4794be5ab57d8a5b7498", SYMLINK+="DATADISK2", OWNER="grid", GROUP="asmadmin", MODE="0660"
KERNEL=="sd*", ENV{DEVTYPE}=="disk", ENV{ID_SERIAL}=="360748b62328c4ff6bb8fe3e9e4f04826", SYMLINK+="DATADISK3", OWNER="grid", GROUP="asmadmin", MODE="0660"
KERNEL=="sd*", ENV{DEVTYPE}=="disk", ENV{ID_SERIAL}=="360923cd17c134686829d791e2b4ff82f", SYMLINK+="DATADISK4", OWNER="grid", GROUP="asmadmin", MODE="0660"
KERNEL=="sd*", ENV{DEVTYPE}=="disk", ENV{ID_SERIAL}=="360bc2c20ddc145ceafa98bc1bb1532d3", SYMLINK+="RECODISK1", OWNER="grid", GROUP="asmadmin", MODE="0660"
KERNEL=="sd*", ENV{DEVTYPE}=="disk", ENV{ID_SERIAL}=="3606feb49caf045598f66042c2831ad38", SYMLINK+="RECODISK2", OWNER="grid", GROUP="asmadmin", MODE="0660"
KERNEL=="sd*", ENV{DEVTYPE}=="disk", ENV{ID_SERIAL}=="360ad6aca673e4d0a8d2d52aded3ee466", SYMLINK+="RECODISK3", OWNER="grid", GROUP="asmadmin", MODE="0660"
KERNEL=="sd*", ENV{DEVTYPE}=="disk", ENV{ID_SERIAL}=="360f2aa11901e4736b65d54f96f000dcf", SYMLINK+="RECODISK4", OWNER="grid", GROUP="asmadmin", MODE="0660"
KERNEL=="sd*", ENV{DEVTYPE}=="disk", ENV{ID_SERIAL}=="3606e328f14b14ddf9042d0eca9038593", SYMLINK+="localdisk", OWNER="grid", GROUP="asmadmin", MODE="0660"Manual intervention is needed to create the time synchronization between the two nodes. Although NTP service is running, it is not synchronized with an external time source.Verifying Clock Synchronization ...FAILED
Verifying Network Time Protocol (NTP) ...FAILED
Verifying NTP daemon is synchronized with at least one external time source
...FAILED
rac1: PRVG-13602 : NTP daemon is not synchronized with any external time
source on node "rac1".As the ntp.conf file is present, the ctss runs in observe mode.[grid@rac1 ~]$ crsctl check ctss CRS-4700: The Cluster Time Synchronization Service is in Observer mode.This also mentioned in oracle documentation where it is recommended "that you configure NTP on both nodes in a 2-node RAC DB System to synchronize time across the nodes. If you do not configure NTP, then Oracle Clusterware configures and uses the Cluster Time Synchronization Service (CTSS), and the cluster time might be out-of-sync with applications that use NTP for time synchronization".
The ntpd file is missing the -x option which prevents setting the clock backwards.
cat /etc/sysconfig/ntpd # Drop root to id 'ntp:ntp' by default. OPTIONS="-u ntp:ntp -p /var/run/ntpd.pid -g"Once the time synchronization is established it could be verified with
cluvfy comp clocksyncThough the cluster was 18c, the oracle home is a read/write home. Not a read only home which was introduced in 18c.
echo $ORACLE_HOME /u01/app/oracle/product/18.0.0.0/dbhome_1 [oracle@rac2 bin]$ ./orabasehome /u01/app/oracle/product/18.0.0.0/dbhome_1The RAC DB created has two log groups per thread.
SQL> select * from v$log;
GROUP# THREAD# SEQUENCE# BYTES BLOCKSIZE MEMBERS ARC STATUS FIRST_CHANGE# FIRST_TIM NEXT_CHANGE# NEXT_TIME CON_ID
---------- ---------- ---------- ---------- ---------- ---------- --- ---------------- ------------- --------- ------------ --------- ----------
1 1 3 1073741824 512 1 YES INACTIVE 1262776 05-MAR-19 1264208 05-MAR-19 0
2 1 4 1073741824 512 1 NO CURRENT 1264244 05-MAR-19 1.8447E+19 05-MAR-19 0
3 2 5 1073741824 512 1 NO CURRENT 1262779 05-MAR-19 1.8447E+19 0
4 2 4 1073741824 512 1 YES INACTIVE 1242884 05-MAR-19 1262779 05-MAR-19 0The RAC DB is a administrator managed DB.srvctl config database -db cdbrac_lhr1fq -a Database unique name: cdbrac_lhr1fq Database name: cdbrac Oracle home: /u01/app/oracle/product/18.0.0.0/dbhome_1 Oracle user: oracle Spfile: +DATA/CDBRAC_LHR1FQ/PARAMETERFILE/spfile.269.1002115587 Password file: +DATA/CDBRAC_LHR1FQ/PASSWORD/pwdcdbrac_lhr1fq.259.1002114845 Domain: londbsubnet.lonvcn.oraclevcn.com Start options: open Stop options: immediate Database role: PRIMARY Management policy: AUTOMATIC Server pools: Disk Groups: RECO,DATA Mount point paths: Services: Type: RAC Start concurrency: Stop concurrency: Database is enabled Database is individually enabled on nodes: Database is individually disabled on nodes: OSDBA group: dba OSOPER group: dbaoper Database instances: cdbrac1,cdbrac2 Configured nodes: rac1,rac2 CSS critical: no CPU count: 0 Memory target: 0 Maximum memory: 0 Default network number for database services: Database is administrator managedBoth DB System patching and DB patching happens in a rolling fashion. Other DB versions where OJVM patch is needed (non-rolling patch) may require total outage. But at least in this case with 18c the patch applying happened rolling fashion.
Similar to single instance DB, RAC DB can also be associated with a data guard configuration. The London region where the initial RAC DB was created had an issue creating a data guard association. Oracle support also confirmed that there are intermittent issues when creating a data guard for RAC DB. However, it was possible to create a RAC DB with data guard association in the Frankfurt region. Following screenshot is of this new RAC DB (as such some names are different).
DGMGRL> show configuration
Configuration - raccdb_fra2bg_raccdb_fra359
Protection Mode: MaxPerformance
Members:
raccdb_fra2bg - Primary database
raccdb_fra359 - Physical standby database
Fast-Start Failover: DISABLED
Configuration Status:
SUCCESS (status updated 19 seconds ago)Primary DatabaseDGMGRL> show database raccdb_fra2bg
Database - raccdb_fra2bg
Role: PRIMARY
Intended State: TRANSPORT-ON
Instance(s):
raccdb1
raccdb2
Database Status:
SUCCESSStandby Database has both instance up and running, with one being the apply instance.DGMGRL> show database raccdb_fra359
Database - raccdb_fra359
Role: PHYSICAL STANDBY
Intended State: APPLY-ON
Transport Lag: 0 seconds (computed 1 second ago)
Apply Lag: 0 seconds (computed 1 second ago)
Average Apply Rate: 1.00 KByte/s
Real Time Query: ON
Instance(s):
raccdb1 (apply instance)
raccdb2
Database Status:
SUCCESSSwitchover could be performed from the console.
DGMGRL> show configuration
Configuration - raccdb_fra2bg_raccdb_fra359
Protection Mode: MaxPerformance
Members:
raccdb_fra359 - Primary database
raccdb_fra2bg - Physical standby database
Fast-Start Failover: DISABLED
Configuration Status:
SUCCESS (status updated 58 seconds ago)Related Posts
Data Guard on OCI PaaS DB
