Saturday, May 29, 2021

Database Resident Connection Pool (DRCP) and Data Guard

There are few restrictions that must be considered when using DRCP in a data guard configuration. This post shows these restrictions in action in a physical data guard configuration.

Staring DRCP
The DRCP in standby cannot be started unless the DRCP in the primary is up. Trying to do so will result in an error similar to below.
SQL>  exec dbms_connection_pool.start_pool();
BEGIN dbms_connection_pool.start_pool(); END;

*
ERROR at line 1:
ORA-56501: DRCP: Pool startup failed
ORA-56501: DRCP: Pool startup failed
ORA-06512: at "SYS.DBMS_CONNECTION_POOL", line 4
ORA-06512: at line 1
Starting the DRCP in primary doesn't automatically start the DRCP in the standby. However, if dba_cpool_info is viewed it will show that pool is active. This is misleading. What this shows is the primary's DRCP status due to DG configuration. DRCP pool must be manually started on standby and one way to find out is to make a connection and see if connection. Another is to view on lsnrctl services for N000 and CMON processes.
Instance "testcdb3", status READY, has 2 handler(s) for this service...
    Handler(s):
      "DEDICATED" established:6 refused:0 state:ready
         LOCAL SERVER
      "N000" established:1 refused:0 current:0 max:40000 state:ready
         CMON <machine: ip-172-31-11-54, pid: 6935>
         (ADDRESS=(PROTOCOL=tcp)(HOST=127.0.0.1)(PORT=33362))
Stopping DRCP
Similar to starting, DRCP in the standby could only be stopped if the primary DRCP is stopped first. Trying to stop standby DRCP while the primary DRCP is up will result in an error similar to below.
SQL>  exec dbms_connection_pool.stop_pool();
BEGIN dbms_connection_pool.stop_pool(); END;

*
ERROR at line 1:
ORA-56506: DRCP: Pool shutdown failed
ORA-56506: DRCP: Pool shutdown failed
ORA-06512: at "SYS.DBMS_CONNECTION_POOL", line 16
ORA-06512: at line 1
Stopping the DRCP in primary doesn't stop the DRCP in standby. However, the dba_cpool_info will show pool inactive, simialr to described in starting DRCP section.
ONNECTION_POOL                STATUS
------------------------------ ---------
SYS_DEFAULT_CONNECTION_POOL    INACTIVE
This is misleading. The DRCP will remain open in the standby and new connections could be established. While primary DRCP is down , pool on standby could be stopped. However, once stopped it cannot be started again until primary is started as described earlier.

Auto Starting DRCP
If DRCP is active on the primary DB, then restaritng the primary DB will automatically will bring up the DRCP as well.
If DRCP is active on standby DB then restarting the standby DB will not automatically will bring up the pool even if pool on primary is also up. It has to be manually started after restart of the standby DB. This is due to bug 18116889.
One way to ensure DRCP always start is to use startup trigger similar to below.
create or replace trigger start_drcp after startup on database
begin
	sys.dbms_system.ksdwrt(2, 'Starting DRCP ...');
	sys.dbms_connection_pool.start_pool();
	sys.dbms_system.ksdwrt(2, 'DRCP started.');

end;
/
Startup trigger is also governed by earlier mentioned restrictions. For example starting standby while primary is down will not start the pool on the standny. If startup trigger is there it will give an error as below.
ORA-04088: error during execution of trigger 'SYS.START_DRCP'
ORA-56501: DRCP: Pool startup failed
ORA-56501: DRCP: Pool startup failed


Changing DRCP Configuration
DRCP related configuration values could only be changed in the primary. The new values take effect immediately on the primary but on the standby then DRCP would need a restart for changes to take effect. This means stopping and starting the primary pool as well since standby pool cannot be stopped while primary pool is up.
Below is an example where number of brokers is changed from 1 to 2.
ONNECTION_POOL                NUM_CBROK
------------------------------ ---------
SYS_DEFAULT_CONNECTION_POOL    1

EXECUTE DBMS_CONNECTION_POOL.ALTER_PARAM ('','NUM_CBROK',2);

ONNECTION_POOL                NUM_CBROK
------------------------------ ---------
SYS_DEFAULT_CONNECTION_POOL    2
Changes are immediately visible on the primary and could see two broker processes as well.
Instance "testcdb", status READY, has 3 handler(s) for this service...
    Handler(s):
      "DEDICATED" established:2 refused:0 state:ready
         LOCAL SERVER
      "N000" established:0 refused:0 current:0 max:40000 state:ready
         CMON <machine: ip-172-31-15-132, pid: 11969>
         (ADDRESS=(PROTOCOL=tcp)(HOST=127.0.0.1)(PORT=15748))
      "N001" established:0 refused:0 current:0 max:40000 state:ready
         CMON <machine: ip-172-31-15-132, pid: 12532>
         (ADDRESS=(PROTOCOL=tcp)(HOST=127.0.0.1)(PORT=31368))
However, on the standby instance there will be only a single broker.
Instance "testcdb3", status READY, has 2 handler(s) for this service...
    Handler(s):
      "DEDICATED" established:12 refused:0 state:ready
         LOCAL SERVER
      "N000" established:3 refused:0 current:1 max:40000 state:ready
         CMON <machine: ip-172-31-11-54, pid: 9874>
         (ADDRESS=(PROTOCOL=tcp)(HOST=127.0.0.1)(PORT=33761))
Restarting the DRCP on the standby result in additional broker being created.
Instance "testcdb3", status READY, has 3 handler(s) for this service...
    Handler(s):
      "DEDICATED" established:12 refused:0 state:ready
         LOCAL SERVER
      "N000" established:0 refused:0 current:0 max:40000 state:ready
         CMON <machine: ip-172-31-11-54, pid: 11119>
         (ADDRESS=(PROTOCOL=tcp)(HOST=127.0.0.1)(PORT=26996))
      "N001" established:0 refused:0 current:0 max:40000 state:ready
         CMON <machine: ip-172-31-11-54, pid: 11121>
         (ADDRESS=(PROTOCOL=tcp)(HOST=127.0.0.1)(PORT=14744))
Switchover and DRCP
In a multi-standby configuration, switchover between a primary and standby doesn't effect the availabilty of the DRCP on other standby instances. For example if there are multiple terminal standbys working as part of a reader farm then primary switching roles will not have any effect on the DRCP in the reader farm.

Saturday, May 22, 2021

Importing Unified Audit Data From Standby

In a primary database importing audit data involves a export job with INCLUDE=AUDIT_TRAILS parameter. However, situation is different on standby as audit records are written to file system location. In a CDB the root container audit data on standby is available in a folder path similar to $ORACLE_BASE/audit/$ORACLE_SID while the audit data for the PDB is available in a path similar to $ORACLE_BASE/audit/$ORACLE_SID/GUID where GUID is the PDB's generic unique identifier.
The audit files are not text file but binary files. The strings command could be used to get glance at the content of the audit file.
strings ora_audit_0629.bin
ANG Spillover Audit File
ORAAUDNG
oracle
ip-172-31-11-54.eu-west-1.compute.internal
pts/0
(TYPE=(DATABASE));(CLIENT ADDRESS=((ADDRESS=(PROTOCOL=tcp)(HOST=172.31.11.54)(PORT=23959))));
LOCKTEST
sqlplus@ip-172-31-11-54.eu-west-1.compute.intern
9837
ORA_LOGON_FAILURES
LOCKTEST
However, this doesn't give all the information as seen from above there's no date information.
Also same audit file would be appended with audit data for multiple sessions. For example above audit record generated for login failure for session originating in sever 172.31.11.54. However, after a while more records could be seen on the same file such as below. Now there exists another login failure for client originating from server 172.31.15.132.
strings ora_audit_0629.bin
ANG Spillover Audit File
ORAAUDNG
oracle
ip-172-31-11-54.eu-west-1.compute.internal
pts/0
(TYPE=(DATABASE));(CLIENT ADDRESS=((ADDRESS=(PROTOCOL=tcp)(HOST=172.31.11.54)(PORT=23983))));
LOCKTEST
sqlplus@ip-172-31-11-54.eu-west-1.compute.intern
10309
ORA_LOGON_FAILURES
LOCKTESTq
ORAAUDNG
oracle
ip-172-31-15-132.eu-west-1.compute.internal
pts/0
(TYPE=(DATABASE));(CLIENT ADDRESS=((ADDRESS=(PROTOCOL=tcp)(HOST=172.31.15.132)(PORT=30643))));
LOCKTEST
sqlplus@ip-172-31-15-132.eu-west-1.compute.inter
10478
ORA_LOGON_FAILURES
LOCKTEST
Therefore audit file must be queried via unified_audit_trail to get fine details of audit records.



If the unified audit table is queried on the standby DB (if it is open in read only mode) this will show records from both primary DB (available via redo apply) and from standby DB (earlier post on this causing high file waits).
To view the audit data of just the standby DB, the easiest and simplest way is to copy the required audit files (either from the CDB root or PDB location) to a different (test or temporary) database that doesn't have any audit data of its own and query the unified_audit_trail. The location where files are copied to must be similar to $ORACLE_BASE/audit/$ORACLE_SID or $ORACLE_BASE/audit/$ORACLE_SID/GUID. With audit files in above the pre-defined location, they are picked up automatically from their external location and exposed via the unified_audit_trail.
However, if the audit records must be kept in the DB then to improt the audit records into the database use the following.
EXEC DBMS_AUDIT_MGMT.LOAD_UNIFIED_AUDIT_FILES;
This will import all the audit files records in the pre-defined audit file location into the database. At the end of this the audit files are removed from the file system.

Friday, May 14, 2021

Master Key Not Set for the Database Shown After Applying DBRU 19.11 (19.11.0.0.210420)

After applying the DBRU 19.11 following message was seen on the alert log on a database which has TDE setup.
2021-04-26T21:14:47.432344+05:30
DEVPDB(3):ALTER PLUGGABLE DATABASE OPEN detects that an encrypted tablespace has been restored but the master key has not been set for the database, or the database has been flashback'ed prior to first set key of the master key (pdb 3) .
DEVPDB(3): Resetting the master key is required. Please execute ADMINISTER KEY MANAGEMENT SET ENCRYPTION KEY command, or select the latest master key from V$ENCRYPTION_KEYS and execute ADMINISTER KEY MANAGEMENT USE KEY <key_id> command if the SET ENCRYPTION KEY command cannot find and decide the master key to use.

If TDE is not used then no such message is shown.
This error is shown only when an auto login wallet is in place. Doesn't matter if the wallet is local auto login or not message still appears.
Message is not shown when auto login wallet is removed and key store is opened manually after CDB start. Message is shown with respect to the user created PDB.
Even when the above message is shown, there's no issue in accessing the encrypted tablespaces and querying data.
Setting the encryption key or recreating the auto login wallet has no effect on it. Below steps were carried out but still the message remains.
alter session set container=devpdb;
Session altered.

ADMINISTER KEY MANAGEMENT SET ENCRYPTION KEY IDENTIFIED BY asanga321 WITH BACKUP;
keystore altered.

SELECT con_id,MASTERKEY_ACTIVATED FROM V$DATABASE_KEY_INFO;
CON_ID MAS
---------- ---
3 YES

SQL> conn / as sysdba
Connected.
SQL> ADMINISTER KEY MANAGEMENT CREATE local AUTO_LOGIN KEYSTORE FROM KEYSTORE IDENTIFIED BY asanga321;
keystore altered.
There was no tablespace restore or flashback on it. It seems when auto login wallet is involved for some reason the correct master key is not identified.
Rolling back the DBRU 19.11 (DB went back to 19.10) resolved the issue. So it appears this is something introduced with the DBRU 19.11 patch. To confirm this a new database was created using 19.3 base release and 19.11 DBRU. Setup TDE and auto login wallet (below output from alert log).
TESTPDB(3):Creating new database key for new master key and wallet
TESTPDB(3):Creating new database key with the new master key
TESTPDB(3):New database key and new master key created successfully
TESTPDB(3):create tablespace enctest datafile size 10m ENCRYPTION USING 'AES256' ENCRYPT
TESTPDB(3):Completed: create tablespace enctest datafile size 10m ENCRYPTION USING 'AES256' ENCRYPT
Then restarted the database and could see the following on the alert log.
TESTPDB(3):ALTER PLUGGABLE DATABASE OPEN detects that an encrypted tablespace has been restored but the master key has not been set for the database, or the database has been flashback'ed prior to first set key of the master key (pdb 3).
TESTPDB(3): Resetting the master key is required. Please execute ADMINISTER KEY MANAGEMENT SET ENCRYPTION KEY command, or select the latest master key from V$ENCRYPTION_KEYS and execute ADMINISTER KEY MANAGEMENT USE KEY <key_id> command if the SET ENCRYPTION KEY command cannot find and decide the master key to use.
To further isolate the issue, created a third database was craeted. This time only TDE was setup, no encrypted tablespaces.
2021-05-11T11:23:48.690428+00:00
Creating new database key for new master key and wallet
Creating new database key with the new master key
Switching out all online logs for the new master key
2021-05-11T11:23:48.746184+00:00
Thread 1 advanced to log sequence 16 (LGWR switch),  current SCN: 1288117
  Current log# 1 seq# 16 mem# 0: +DATA/TDETEST/ONLINELOG/group_1.297.1072260997
  Current log# 1 seq# 16 mem# 1: +FRA/TDETEST/ONLINELOG/group_1.530.1072260999
2021-05-11T11:23:48.748707+00:00
Logfile switch for new master key complete
New database key and new master key created successfully
TDETESTPDB(3):Creating new database key for new master key and wallet
TDETESTPDB(3):Creating new database key with the new master key
TDETESTPDB(3):New database key and new master key created successfully
In this case too the alert log showed the same message as before.
2021-05-11T11:24:58.081692+00:00
TDETESTPDB(3):ALTER PLUGGABLE DATABASE OPEN detects that an encrypted tablespace has been restored but the master key has not been set for the database, or the database has been flashback'ed prior to first set key of the master key (pdb 3).
TDETESTPDB(3): Resetting the master key is required. Please execute ADMINISTER KEY MANAGEMENT SET ENCRYPTION KEY command, or select the latest master key from V$ENCRYPTION_KEYS and execute ADMINISTER KEY MANAGEMENT USE KEY <key_id> command if the SET ENCRYPTION KEY command cannot find and decide the master key to use.




The auto login wallet does contain the same key ids which could be checked with
SQL> show pdbs

CON_ID CON_NAME OPEN MODE RESTRICTED
---------- ------------------------------ ---------- ----------
2 PDB$SEED READ ONLY NO
3 TESTPDB READ WRITE NO

SQL> select * from gv$encryption_wallet;

INST_ID WRL_TYPE WRL_PARAMETER STATUS WALLET_TYPE WALLET_OR KEYSTORE FULLY_BAC CON_ID
---------- -------------------- ------------------------------ ------------------------------ -------------------- --------- -------- --------- ----------
1 FILE /opt/app/oracle/wallet/tde/ OPEN AUTOLOGIN SINGLE NONE NO 1
1 FILE OPEN AUTOLOGIN SINGLE UNITED NO 2
1 FILE OPEN AUTOLOGIN SINGLE UNITED NO 3 

SQL> select con_id,KEY_ID,CREATION_TIME,activation_time FROM V$ENCRYPTION_KEYS;

CON_ID KEY_ID CREATION_TIME ACTIVATION_TIME
---------- ------------------------------------------------------------ ----------------------------------- --------------------
3 AVQm2JpHDk+jv59DdGa3lz4AAAAAAAAAAAAAAAAAAAAAAAAAAAAA 28-APR-21 04.25.22.932575 PM +05:30 28-APR-21 04.25.23.243620 PM +05:30
1 ATJSoZoecE+jv6iBZKWk2s0AAAAAAAAAAAAAAAAAAAAAAAAAAAAA 28-APR-21 04.25.22.830950 PM +05:30 28-APR-21 04.25.23.064019 PM +05:30 


orapki wallet display -wallet /opt/app/oracle/wallet/tde
Oracle PKI Tool Release 19.0.0.0.0 - Production
Version 19.4.0.0.0
Copyright (c) 2004, 2021, Oracle and/or its affiliates. All rights reserved.

Requested Certificates:
Subject: CN=oracle
User Certificates:
Oracle Secret Store entries:
ORACLE.SECURITY.DB.ENCRYPTION.ATJSoZoecE+jv6iBZKWk2s0AAAAAAAAAAAAAAAAAAAAAAAAAAAAA
ORACLE.SECURITY.DB.ENCRYPTION.AVQm2JpHDk+jv59DdGa3lz4AAAAAAAAAAAAAAAAAAAAAAAAAAAAA
ORACLE.SECURITY.DB.ENCRYPTION.MASTERKEY
ORACLE.SECURITY.DB.ENCRYPTION.MASTERKEY.C107201B5EA38B36E053FE04A8C049CF
ORACLE.SECURITY.ID.ENCRYPTION.
ORACLE.SECURITY.KB.ENCRYPTION.
ORACLE.SECURITY.KM.ENCRYPTION.ATJSoZoecE+jv6iBZKWk2s0AAAAAAAAAAAAAAAAAAAAAAAAAAAAA
ORACLE.SECURITY.KM.ENCRYPTION.AVQm2JpHDk+jv59DdGa3lz4AAAAAAAAAAAAAAAAAAAAAAAAAAAAA
ORACLE.SECURITY.KT.ENCRYPTION.ATJSoZoecE+jv6iBZKWk2s0AAAAAAAAAAAAAAAAAAAAAAAAAAAAA
ORACLE.SECURITY.KT.ENCRYPTION.AVQm2JpHDk+jv59DdGa3lz4AAAAAAAAAAAAAAAAAAAAAAAAAAAAA
Trusted Certificates: 
The issue was happening on OCI DBCS VM databases after applying the DBRU 19.11.
Raised a P1 24x7 SR and 48 hours later Oracle confirmed this is internal bug 29528184 filed as happening only on exadata. Reason for issue is given as "The lookup of MKID for PDB was done in PDB's keystore which was not OPEN". It seems the bug is now packaged with 19.11 thus seeing it on OCI DBCS and on-prem DBs.
No fix available yet. Oracle is working on a backport for non exadata environments.

Update on 2021-07-02

Patch 29528184: X5-2: OPEN PDB HAD "MASTER KEY NOT SET" MESSAGE IN ALERT.LOG AFTER WE ENABLED TDE is now available to address this issue.

Saturday, May 8, 2021

ORA-12754 When Setting ADG_ACCOUNT_INFO_TRACKING to Global

ADG_ACCOUNT_INFO_TRACKING was first introduced in 18.1 enchance the security posture in a data guard configuration. Setting this parameter to global result in login failures accross the data guard configuraiton being counted towards FAILED_LOGIN_ATTEMPTS value set in the user account profile. Setting it to local result in the default behavior which is only login failures in the primary is counted towards the FAILED_LOGIN_ATTEMPTS values.
In a recent deployment which was on AWS EC2 VMs (this point is key to this post and it is not using AWS RDS) following error was observed when testing the ADG_ACCOUNT_INFO_TRACKING works as expected, by using a wrong password and trying to login from a standby instance and to see if it locks after FAILED_LOGIN_ATTEMPTS number of login failures.
sqlplus locktest/locktest321@testro

SQL*Plus: Release 19.0.0.0.0 - Production on Tue Jan 26 13:59:45 2021
Version 19.10.0.0.0

Copyright (c) 1982, 2020, Oracle. All rights reserved.

ERROR:
ORA-12754: Feature 'ADG Statement Redirection' is disabled due to missing capability 'Runtime Environment'.
ORA-01017: invalid username/password; logon denied 
Error is strange as ADG statement redirection wasn't being used. This feature was introduced in 19.1 and controlled by ADG_REDIRECT_DML. Secondly the due to the error ADG_ACCOUNT_INFO_TRACKING wasn't working as expected and one could have unlimited number of login failures.
The two parameters in concern had the following values.
adg_account_info_tracking string GLOBAL
adg_redirect_dml boolean FALSE 
Setting adg_redirect_dml to true didn't make any difference either.
Further investigations showed that dml redirecation is not avaiable on "Authorized Cloud Environments". This is mentioned in the license documentation. But nowhere on it says anything about ADG_ACCOUNT_INFO_TRACKING.



Priror to testing on AWS EC2 a similar setup was created and operational in Azure (using Azure VM). If ADG_ACCOUNT_INFO_TRACKING is also not supported in authorized cloud environment similar to dml redirection then it should fail on Azure VM setup as well.
It seems that ADG_ACCOUNT_INFO_TRACKING is using DML redirection behind the scene to update the failed login atttempt count in the primary. However, use of this is flagged as "missing capability" only when databases are craeted on AWS EC2 VMs and not on Azure VM even though license doc says dml redirection is not supported on authorized cloud environments. It seems issue is localized only to AWS EC2.
SR raised and Oracle has created internal bug 32838564 - SETTING ADG_ACCOUNT_INFO_TRACKING=GLOBAL RESULTS IN ORA-12754 IN AWS DG ENVIRONMENT and investigating.

Update 07 July 2022
Oracle confirmed that ADG_ACCOUNT_INFO_TRACKING is using dml redirection behind the scene. Since dml redirection is not allowed on cloud environment the ADG_ACCOUNT_INFO_TRACKING also fails. Bug 32838564 was closed as not a bug based on above explanation.

Saturday, May 1, 2021

Far Sync Instance Not Detected For Template Apply

A data guard configuration (same configuration mentioned in a previous post) is promoted to EM 13.4 and has serveral far sync instances.

The instance type is identified as far sync in the instance page and in data guard administer page.

A monitoring template was created with far sync as the instance type.

However, when the template is to be applied to far sync instances non is detected and following error is shown on the EM console.




Seems even though far sync is identified in all other places when it comes to template it as there are no far sync instances to monitor.
SR raised and status is "development working".

Related Post
Far Sync Instance Stats Not Shown on Data Guard Performance Page on Cloud Control 13.4