Showing posts with label dataguard. Show all posts
Showing posts with label dataguard. Show all posts

Thursday, September 12, 2024

Restore Standby Database from Standby Backups

This post shows the steps for full restoring (controlfile + data files) for a standby database using standby database backups. The same could be achieved using restore from service (2283978.1). However, this method is useful when the database size is large and high network latencies are invovled.

1. As the first step disable log apply and transport.
DGMGRL> edit database fsfodr set state='apply-off';
Succeeded.
DGMGRL> edit database fsfopr set state='transport-off';
Succeeded.
DGMGRL>
2. Start the standby database in nomount mode and restore the standby controlfile.
$ rman target /
RMAN > startup nomount

RMAN> restore standby controlfile from '/opt/backup/fsfodr/full_c-1245564449-20230913-02.ctl';

Starting restore at 13-SEP-23
using target database control file instead of recovery catalog
allocated channel: ORA_DISK_1
channel ORA_DISK_1: SID=2836 device type=DISK

channel ORA_DISK_1: restoring control file
channel ORA_DISK_1: restore complete, elapsed time: 00:00:02
output file name=+DATA/FSFODR/CONTROLFILE/current.341.1147435431
output file name=+FRA/FSFODR/CONTROLFILE/current.573.1147435431
Finished restore at 13-SEP-23
3. Mount the database.
RMAN> alter database mount;
4. Since the standby controfile was restored from a backup taken on standby database no need to catalog backup file location. The controlfile is aware of the backup locations. Run a restore and recover statements.
RMAN> run {
2> restore database;
3> recover database;
4> }
5. Clear the online logfiles on the standby database.
SQL> begin
for log_cur in ( select group# group_no from v$log )
loop
execute immediate 'alter database clear logfile group '||log_cur.group_no;
end loop;
end;
/

PL/SQL procedure successfully completed.

SQL> select * from v$log;

    GROUP#    THREAD#  SEQUENCE#      BYTES  BLOCKSIZE    MEMBERS ARC STATUS           FIRST_CHANGE# FIRST_TIM NEXT_CHANGE# NEXT_TIME     CON_ID
---------- ---------- ---------- ---------- ---------- ---------- --- ---------------- ------------- --------- ------------ --------- ----------
         1          1          0  104857600        512          2 YES UNUSED                 2635375 13-SEP-23      2635378 13-SEP-23          0
         2          1          0  104857600        512          2 YES UNUSED                 2635378 13-SEP-23      2636371 13-SEP-23          0
         5          1          0  104857600        512          2 YES UNUSED                 2636825 13-SEP-23   9.2954E+18                    0
         4          1          0  104857600        512          2 YES UNUSED                 2636470 13-SEP-23      2636825 13-SEP-23          0
         3          1          0  104857600        512          2 YES UNUSED                 2636371 13-SEP-23      2636470 13-SEP-23          0
6. Clear the standby logfiles
begin
for log_cur in ( select group# group_no from v$standby_log )
loop
execute immediate 'alter database clear logfile group '||log_cur.group_no;
end loop;
end;
/


PL/SQL procedure successfully completed.

SQL> select * from v$standby_log;

    GROUP# DBID                                        THREAD#  SEQUENCE#      BYTES  BLOCKSIZE       USED ARC STATUS     FIRST_CHANGE# FIRST_TIM NEXT_CHANGE# NEXT_TIME LAST_CHANGE# LAST_TIME     CON_ID
---------- ---------------------------------------- ---------- ---------- ---------- ---------- ---------- --- ---------- ------------- --------- ------------ --------- ------------ --------- ----------
         6 UNASSIGNED                                        1          0  104857600        512          0 NO  UNASSIGNED                                                                                0
         7 UNASSIGNED                                        1          0  104857600        512          0 YES UNASSIGNED                                                                                0
         8 UNASSIGNED                                        1          0  104857600        512          0 YES UNASSIGNED                                                                                0
         9 UNASSIGNED                                        1          0  104857600        512          0 YES UNASSIGNED                                                                                0
        10 UNASSIGNED                                        1          0  104857600        512          0 YES UNASSIGNED                                                                                0
        11 UNASSIGNED                                        1          0  104857600        512          0 YES UNASSIGNED                                                                                0

6 rows selected.



7. Enable log transport and redo apply
DGMGRL>  edit database fsfopr set state='transport-on';
Succeeded.
DGMGRL>  edit database fsfodr set state='apply-on';
Succeeded.
8. Check data guard configuration status and valdiate the standby database
DGMGRL> show configuration

Configuration - fsfo_dg

  Protection Mode: MaxAvailability
  Members:
  fsfopr - Primary database
    fsfodr - Physical standby database

Fast-Start Failover:  Disabled

Configuration Status:
SUCCESS   (status updated 25 seconds ago)

DGMGRL> validate database fsfodr;

  Database Role:     Physical standby database
  Primary Database:  fsfopr

  Ready for Switchover:  Yes
  Ready for Failover:    Yes (Primary Running)

  Managed by Clusterware:
    fsfopr:  YES
    fsfodr:  YES

Useful metalink notes
Creating a Physical Standby database using RMAN restore database from service [ID 2283978.1]

Saturday, March 26, 2022

Initiating Switchover on a Physical Standby

It seems the best instance to intiaiate the switchover changes based on whether data guard broker is used or not. Oracle documentation states when performing switchover using SQL, the switchover process is initiated on the primary database.
Initiate the switchover on the primary database, BOSTON, by issuing the following SQL statement
On the data guard broker documentation it doesn't explicitly states where to initiate the switchover. In practice switchover command could be initiated while logged into any instance. However, executing switchover on the primary instance when data gaurd broker is used will output the following in the data guard broker log. The DG configuration is
DGMGRL> show configuration

Configuration - test_dg

  Protection Mode: MaxAvailability
  Members:
  dgtest  - Primary database
    dgtest2 - Physical standby database
    dgtest3 - Physical standby database

Fast-Start Failover:  Disabled

Configuration Status:
SUCCESS   (status updated 9 seconds ago)
Switchover command is issued while connected to the current primary, dgtest.
2022-03-21T10:41:33.122+00:00
Forwarding MON_PROPERTY operation to member dgtest2 for processing
2022-03-21T10:41:42.319+00:00
Initiating a healthcheck...
SWITCHOVER TO dgtest2
Switchover to physical standby database cannot be initiated from the primary database
redirecting connection to switchover target database dgtest2...
...using connect identifier: dgtest2tns
SWITCHOVER TO dgtest2
Notifying Oracle Clusterware to prepare primary database for switchover
2022-03-21T10:41:44.319+00:00
Executing SQL: [ALTER DATABASE SWITCHOVER TO 'dgtest2']
2022-03-21T10:42:02.573+00:00
SQL [ALTER DATABASE SWITCHOVER TO 'dgtest2'] executed successfully
2022-03-21T10:42:03.652+00:00
Switchover successful
From the output it seems that DG broker is creating a remote connection to the standby (new primary) using the TNS entry used when the standby was added to the data guard broker.



On the otherhand if connected to the standby and switchover command is issued no such message appears.
DGMGRL>  show configuration

Configuration - test_dg

  Protection Mode: MaxAvailability
  Members:
  dgtest2 - Primary database
    dgtest  - Physical standby database
    dgtest3 - Physical standby database

Fast-Start Failover:  Disabled

Configuration Status:
SUCCESS   (status updated 5 seconds ago)
New primary is dgtest2 and switchover command is issued from standby, dgtest. The output on DG broker log of the primary (dgtest2) shows the below
2022-03-21T10:50:27.266+00:00
Initiating a healthcheck...
SWITCHOVER TO dgtest
2022-03-21T10:50:28.682+00:00
Notifying Oracle Clusterware to prepare primary database for switchover
Executing SQL: [ALTER DATABASE SWITCHOVER TO 'dgtest']
2022-03-21T10:50:49.451+00:00
SQL [ALTER DATABASE SWITCHOVER TO 'dgtest'] executed successfully
Similary if the switchvoer is issued from another standby that is not the future primary (in this confiugration dgtest3) then again the primary's DG broker log output will have the below output
2022-03-21T11:31:51.242+00:00
Initiating a healthcheck...
SWITCHOVER TO dgtest2
2022-03-21T11:31:52.601+00:00
Notifying Oracle Clusterware to prepare primary database for switchover
Executing SQL: [ALTER DATABASE SWITCHOVER TO 'dgtest2']

Saturday, November 20, 2021

Using UR=A to Connect to Databases in Nomount Mode

Time to time there are situation where connecting to a database in nomount mode is needed. Most noteably is the data guard standby creation. When a database is started in nomount mode
startup nomount;
the service on the listener would have a blocke status.
Service "devcdb" has 1 instance(s).
  Instance "devcdb", status BLOCKED, has 1 handler(s) for this service...
Any attempt to connect to the instance using service name will fail.
sqlplus sys@devcdb as sysdba

SQL*Plus: Release 19.0.0.0.0 - Production on Thu Oct 14 20:27:16 2021
Version 19.12.0.0.0

Copyright (c) 1982, 2021, Oracle.  All rights reserved.

ERROR:
ORA-12528: TNS:listener: all appropriate instances are blocking new connections
The TNS entry used is shown below.
DEVCDB =
  (DESCRIPTION =
    (ADDRESS = (PROTOCOL = TCP)(HOST = test-4)(PORT = 1521))
    (CONNECT_DATA =
      (SERVER = DEDICATED)
      (SERVICE_NAME = devcdb)
    )
  )
One solution is to create a static listener entries as done when creating data guard configurations.



Other options is to use UR=A in the TNS entry. With the use of UR=A in TNS entry there's no need to create a static listener entry. Dynamic listener would allow connection without any issue.
DEVCDBUR =
  (DESCRIPTION =
    (ADDRESS = (PROTOCOL = TCP)(HOST = test-4-254.company.net)(PORT = 1521))
    (CONNECT_DATA =
      (SERVER = DEDICATED)
      (SERVICE_NAME = devcdb)
      (UR = A)
    )
  )
  
sqlplus sys@devcdbur as sysdba

SQL*Plus: Release 19.0.0.0.0 - Production on Thu Oct 14 20:47:13 2021
Version 19.12.0.0.0

Copyright (c) 1982, 2021, Oracle.  All rights reserved.

Enter password:

Connected to:
Oracle Database 19c Enterprise Edition Release 19.0.0.0.0 - Production
Version 19.12.0.0.0

SQL>
Even during normal DG operation use of UR=A in TNS entries used for DG traffic could be useful. During switchover primary could attmept to connect to standby which is in the nomount state (progressing to mount or read only). Without UR=A the primary connection would fail (and succeed later after reattempt, once standby has transitioned out of nomount). To reduce the false alerts occuring in these situation UR=A could be used.
For example without UR=A entry it took nearly 15s for DG broker to connect to the new standby during a switchover (output on a 19.13 system).
DGMGRL> switchover to dgtest
Performing switchover NOW, please wait...
Operation requires a connection to database "dgtest"
Connecting ...
Connected to "dgtest"
Connected as SYSDBA.
New primary database "dgtest" is opening...
Oracle Clusterware is restarting database "dgtest3" ...
[W000 2021-10-25T13:49:46.227+00:00] Failed to attach to dgtest3tns.
[W000 2021-10-25T13:49:47.231+00:00] Failed to attach to dgtest3tns.
[W000 2021-10-25T13:49:48.234+00:00] Failed to attach to dgtest3tns.
[W000 2021-10-25T13:49:49.237+00:00] Failed to attach to dgtest3tns.
[W000 2021-10-25T13:49:50.240+00:00] Failed to attach to dgtest3tns.
[W000 2021-10-25T13:49:51.242+00:00] Failed to attach to dgtest3tns.
[W000 2021-10-25T13:49:52.245+00:00] Failed to attach to dgtest3tns.
[W000 2021-10-25T13:49:53.248+00:00] Failed to attach to dgtest3tns.
[W000 2021-10-25T13:49:54.250+00:00] Failed to attach to dgtest3tns.
[W000 2021-10-25T13:49:55.252+00:00] Failed to attach to dgtest3tns.
[W000 2021-10-25T13:49:56.255+00:00] Failed to attach to dgtest3tns.
[W000 2021-10-25T13:49:57.257+00:00] Failed to attach to dgtest3tns.
[W000 2021-10-25T13:49:58.260+00:00] Failed to attach to dgtest3tns.
[W000 2021-10-25T13:49:59.262+00:00] Failed to attach to dgtest3tns.
[W000 2021-10-25T13:50:00.265+00:00] Failed to attach to dgtest3tns.
[W000 2021-10-25T13:50:01.274+00:00] Failed to attach to dgtest3tns.
Connected to "dgtest3"
Connected to "dgtest3"
Switchover succeeded, new primary is "dgtest"
However, with UR=A in the TNS entry it was able to connect sooner.
DGMGRL> switchover to dgtest
Performing switchover NOW, please wait...
Operation requires a connection to database "dgtest"
Connecting ...
Connected to "dgtest"
Connected as SYSDBA.
New primary database "dgtest" is opening...
Oracle Clusterware is restarting database "dgtest3" ...
[W000 2021-10-25T13:55:24.705+00:00] Failed to attach to dgtest3tns.
[W000 2021-10-25T13:55:25.708+00:00] Failed to attach to dgtest3tns.
[W000 2021-10-25T13:55:26.710+00:00] Failed to attach to dgtest3tns.
[W000 2021-10-25T13:55:27.714+00:00] Failed to attach to dgtest3tns.
[W000 2021-10-25T13:55:28.716+00:00] Failed to attach to dgtest3tns.
[W000 2021-10-25T13:55:29.718+00:00] Failed to attach to dgtest3tns.
[W000 2021-10-25T13:55:30.722+00:00] Failed to attach to dgtest3tns.
Connected to "dgtest3"
Connected to "dgtest3"
Switchover succeeded, new primary is "dgtest"
UR=A is used in the TNS entries created in DBCS VM DBs that also has data guard enabled.

Related Metalink note
Connections to NOMOUNT/MOUNTED or RESTRICTED Databases Fail [ID 362656.1]

Update 28 January 2022
Entries similar to "[W000 2021-10-25T13:55:24.705+00:00] Failed to attach to dgtest3tns" are due to bug 30870248. Apply the patch if available for the RU.

Saturday, November 6, 2021

Encrypting / Decrypting Tablespaces Online / Offline in a Data Guard Configuration

An earlier post showed the steps for encrypting/decrypting tablespaces both online and offline in a single database.
This post shows the steps for doing the same when there's a data guard configuration.
The data guard configuration consists of three databases.
DGMGRL> show configuration

Configuration - drcp_dg

  Protection Mode: MaxAvailability
  Members:
  testcdb  - Primary database
    testcdb2 - Physical standby database
    testcdb3 - Physical standby database

Fast-Start Failover:  Disabled

Configuration Status:
SUCCESS   (status updated 51 seconds ago)
Each standby has a different open mode. testcdb2 is open in read only mode with apply
SQL> select open_mode,db_unique_name from v$database;

OPEN_MODE            DB_UNIQUE_NAME
-------------------- ------------------------------
READ ONLY WITH APPLY testcdb2
while testcdb3 is in mount mode.
SQL> select open_mode,db_unique_name from v$database;

OPEN_MODE            DB_UNIQUE_NAME
-------------------- ------------------------------
MOUNTED              testcdb3
It's assumed TDE is setup for all the databases in the DG configuration. For the testing a unencrypted tablespace called "enctest" is created. DB version is 19.12

Online Encryption
Online encryption is pretty straigth fowrad. Encrypt the tablespace on the primary.
select t.name,e.ENCRYPTIONALG,e.ENCRYPTEDTS,e.STATUS from V$ENCRYPTED_TABLESPACES e, v$tablespace t where t.ts#=e.ts#(+) and NAME='ENCTEST';

NAME                           ENCRYPT ENC STATUS
------------------------------ ------- --- ----------
ENCTEST

SQL> ALTER TABLESPACE ENCTEST ENCRYPTION ONLINE USING 'AES256' ENCRYPT;

SQL>  select t.name,e.ENCRYPTIONALG,e.ENCRYPTEDTS,e.STATUS from V$ENCRYPTED_TABLESPACES e, v$tablespace t where t.ts#=e.ts#(+) and NAME='ENCTEST';

NAME                           ENCRYPT ENC STATUS
------------------------------ ------- --- ----------
ENCTEST                        AES256  YES NORMAL
Tablespaces on the standby databases (both in mount and active data guard) are automatically encrypted. On the standby alerts log following log entries, related to encryption are shown. On testcdb2 (open read only)
2021-10-12T13:22:41.197253+00:00
(3):Rekeying datafile +DATA/TESTCDB2/9D48FF44CE8A7D32E053F4071FACDFB3/DATAFILE/enctest.296.1085749247 (16) to +DATA
(3):Rekey operation committed for file +DATA/TESTCDB2/9D48FF44CE8A7D32E053F4071FACDFB3/DATAFILE/enctest.297.1085750561
2021-10-12T13:22:44.005273+00:00
(3):About to zero out original file "+DATA/TESTCDB2/9D48FF44CE8A7D32E053F4071FACDFB3/DATAFILE/enctest.296.1085749247"
(3):Successfully zero'ed out original file "+DATA/TESTCDB2/9D48FF44CE8A7D32E053F4071FACDFB3/DATAFILE/enctest.296.1085749247"
(3):Successfully deleted original file "+DATA/TESTCDB2/9D48FF44CE8A7D32E053F4071FACDFB3/DATAFILE/enctest.296.1085749247"
On testcdb3 (open mount)
2021-10-12T13:22:41.470038+00:00
(3):Rekeying datafile +DATA/TESTCDB3/9D48FF44CE8A7D32E053F4071FACDFB3/DATAFILE/enctest.317.1085749247 (16) to +DATA
2021-10-12T13:22:42.781816+00:00
(3):Rekey operation committed for file +DATA/TESTCDB3/9D48FF44CE8A7D32E053F4071FACDFB3/DATAFILE/enctest.315.1085750561
2021-10-12T13:22:44.806025+00:00
(3):About to zero out original file "+DATA/TESTCDB3/9D48FF44CE8A7D32E053F4071FACDFB3/DATAFILE/enctest.317.1085749247"
(3):Successfully zero'ed out original file "+DATA/TESTCDB3/9D48FF44CE8A7D32E053F4071FACDFB3/DATAFILE/enctest.317.1085749247"
(3):Successfully deleted original file "+DATA/TESTCDB3/9D48FF44CE8A7D32E053F4071FACDFB3/DATAFILE/enctest.317.1085749247"

Online Decryption
Online decryption is also similar to encryption. Execute the decryption on primary
ALTER TABLESPACE ENCTEST ENCRYPTION online DECRYPT;

select t.name,e.ENCRYPTIONALG,e.ENCRYPTEDTS,e.STATUS from V$ENCRYPTED_TABLESPACES e, v$tablespace t where t.ts#=e.ts#(+) and NAME='ENCTEST';

NAME                           ENCRYPT ENC STATUS
------------------------------ ------- --- ----------
ENCTEST
Changes are automatically propergated to standby databases. As before alert log shows the decrption related activities. On testcdb2
2021-10-12T13:27:32.614421+00:00
(3):Rekeying datafile +DATA/TESTCDB2/9D48FF44CE8A7D32E053F4071FACDFB3/DATAFILE/enctest.297.1085750561 (16) to +DATA
(3):Rekey operation committed for file +DATA/TESTCDB2/9D48FF44CE8A7D32E053F4071FACDFB3/DATAFILE/enctest.296.1085750853
2021-10-12T13:27:35.351234+00:00
(3):About to zero out original file "+DATA/TESTCDB2/9D48FF44CE8A7D32E053F4071FACDFB3/DATAFILE/enctest.297.1085750561"
(3):Successfully zero'ed out original file "+DATA/TESTCDB2/9D48FF44CE8A7D32E053F4071FACDFB3/DATAFILE/enctest.297.1085750561"
(3):Successfully deleted original file "+DATA/TESTCDB2/9D48FF44CE8A7D32E053F4071FACDFB3/DATAFILE/enctest.297.1085750561"
On testcdb3
2021-10-12T13:27:32.869097+00:00
(3):Rekeying datafile +DATA/TESTCDB3/9D48FF44CE8A7D32E053F4071FACDFB3/DATAFILE/enctest.315.1085750561 (16) to +DATA
2021-10-12T13:27:34.461534+00:00
(3):Rekey operation committed for file +DATA/TESTCDB3/9D48FF44CE8A7D32E053F4071FACDFB3/DATAFILE/enctest.317.1085750853
2021-10-12T13:27:36.477042+00:00
(3):About to zero out original file "+DATA/TESTCDB3/9D48FF44CE8A7D32E053F4071FACDFB3/DATAFILE/enctest.315.1085750561"
(3):Successfully zero'ed out original file "+DATA/TESTCDB3/9D48FF44CE8A7D32E053F4071FACDFB3/DATAFILE/enctest.315.1085750561"
(3):Successfully deleted original file "+DATA/TESTCDB3/9D48FF44CE8A7D32E053F4071FACDFB3/DATAFILE/enctest.315.1085750561"

Offline Encryption
Offline encryption (and decryption) can happen in any order, meaning primary first or standby first. There's a difference to this in lower versions of 11.2 and 12.1. Read asymmetrical configurations in this white paper.
There are no special consideration for encryption tablespace in offline mode in primary. Same as before offline the tablespace and encrypt.
alter tablespace ENCTEST offline normal;

ALTER TABLESPACE ENCTEST ENCRYPTION OFFLINE  USING 'AES256' ENCRYPT;

alter tablespace ENCTEST online;
 
select t.name,e.ENCRYPTIONALG,e.ENCRYPTEDTS,e.STATUS from V$ENCRYPTED_TABLESPACES e, v$tablespace t where t.ts#=e.ts#(+) and NAME='ENCTEST';

NAME                           ENCRYPT ENC STATUS
------------------------------ ------- --- ----------
ENCTEST                        AES256  YES NORMAL
There's no entries in the standby alert logs. The tablespace on standby databases will remain unencrypted.
select t.name,e.ENCRYPTIONALG,e.ENCRYPTEDTS,e.STATUS from V$ENCRYPTED_TABLESPACES e, v$tablespace t where t.ts#=e.ts#(+) and NAME='ENCTEST';

NAME                           ENCRYPT ENC STATUS
------------------------------ ------- --- ----------
ENCTEST

Offline Encryption On Standby Open in Mount mode
Once the tablespace is encrypted on primary same could be done standby as well. This step shows how this is done on standby in mount mode. As it's not open (in mount mode) there's no need to offline tablespaces.
SQL> alter tablespace ENCTEST offline normal;
alter tablespace ENCTEST offline normal
*
ERROR at line 1:
ORA-01109: database not open
However, encryption cannot happen while recovery is active.
SQL> ALTER TABLESPACE ENCTEST ENCRYPTION OFFLINE  USING 'AES256' ENCRYPT;
ALTER TABLESPACE ENCTEST ENCRYPTION OFFLINE  USING 'AES256' ENCRYPT
*
ERROR at line 1:
ORA-01153: an incompatible media recovery is active
As the first step for encryption on standby, stop the log apply.
DGMGRL> edit database testcdb3 set state='apply-off';
Succeeded.

DGMGRL> show database testcdb3

Database - testcdb3

  Role:               PHYSICAL STANDBY
  Intended State:     APPLY-OFF
  Transport Lag:      0 seconds (computed 1 second ago)
  Apply Lag:          0 seconds (computed 1 second ago)
  Average Apply Rate: (unknown)
  Real Time Query:    OFF
  Instance(s):
    testcdb3

Database Status:
SUCCESS
If DG broker is not used then verify apply is off with select * from gv$managed_standby where process='MRP0'. This should not any rows if apply is off.
Once the log apply is off run the encrption.
SQL>  ALTER TABLESPACE ENCTEST ENCRYPTION OFFLINE  USING 'AES256' ENCRYPT;

Tablespace altered.

SQL>  select t.name,e.ENCRYPTIONALG,e.ENCRYPTEDTS,e.STATUS from V$ENCRYPTED_TABLESPACES e, v$tablespace t where t.ts#=e.ts#(+) and NAME='ENCTEST';

NAME                           ENCRYPT ENC STATUS
------------------------------ ------- --- ----------
ENCTEST
The status of the tablespace will not be updated to encrypted=yes until the log apply is enabled. Once enabled the query will show tablespace is encrypted.
DGMGRL> edit database testcdb3 set state='apply-on';
Succeeded.

NAME                           ENCRYPT ENC STATUS
------------------------------ ------- --- ----------
ENCTEST                        AES256  YES NORMAL
Alert log will have entries related to the encryption.
Managed Standby Recovery Canceled (testcdb3)
Completed: ALTER DATABASE RECOVER MANAGED STANDBY DATABASE CANCEL
2021-10-12T14:28:47.715122+00:00
TESTPDB(3): ALTER TABLESPACE ENCTEST ENCRYPTION OFFLINE  USING 'AES256' ENCRYPT
2021-10-12T14:28:47.761081+00:00
TESTPDB(3):About to offline encrypt tablespace 8 ..
2021-10-12T14:28:49.179873+00:00
TESTPDB(3):Successfully encrypted tablespace 8 with database key.
TESTPDB(3):Completed:  ALTER TABLESPACE ENCTEST ENCRYPTION OFFLINE  USING 'AES256' ENCRYPT
2021-10-12T14:29:25.473015+00:00
ALTER DATABASE RECOVER MANAGED STANDBY DATABASE DISCONNECT  NODELAY
2021-10-12T14:29:25.477314+00:00
Attempt to start background Managed Standby Recovery process (testcdb3)
Starting background process MRP0
2021-10-12T14:29:25.494580+00:00
MRP0 started with pid=25, OS id=25152
2021-10-12T14:29:25.496232+00:00
Background Managed Standby Recovery process started (testcdb3)
Offline Decryption On Standby Open in Mount mode
Similar to encrypt, to decrypt turn off apply and run the decryption command.
DGMGRL> edit database testcdb3 set state='apply-off';
Succeeded.

SQL> ALTER TABLESPACE ENCTEST ENCRYPTION OFFLINE DECRYPT;

Tablespace altered.

SQL> select t.name,e.ENCRYPTIONALG,e.ENCRYPTEDTS,e.STATUS from V$ENCRYPTED_TABLESPACES e, v$tablespace t where t.ts#=e.ts#(+) and NAME='ENCTEST';

NAME                           ENCRYPT ENC STATUS
------------------------------ ------- --- ----------
ENCTEST                        AES256  YES NORMAL
Once the apply is enabled the tablespace encryption status will be updated to un-encrypted.
DGMGRL> edit database testcdb3 set state='apply-on';
Succeeded.


SQL>  select t.name,e.ENCRYPTIONALG,e.ENCRYPTEDTS,e.STATUS from V$ENCRYPTED_TABLESPACES e, v$tablespace t where t.ts#=e.ts#(+) and NAME='ENCTEST';

NAME                           ENCRYPT ENC STATUS
------------------------------ ------- --- ----------
ENCTEST



Offline Encryption/Decryption On Standby Open in Read Only mode
Things are bit different for encrypting/decrypting tablespaces on a standby open in read only mode. The tablespace cannot be made offline as DB is open for read only access
SQL>  alter tablespace ENCTEST offline normal;
 alter tablespace ENCTEST offline normal
                                       *
ERROR at line 1:
ORA-16000: database or pluggable database open for read-only access
Offline encryption fails as DB is open for read only access
SQL> ALTER TABLESPACE ENCTEST ENCRYPTION OFFLINE  USING 'AES256' ENCRYPT;
ALTER TABLESPACE ENCTEST ENCRYPTION OFFLINE  USING 'AES256' ENCRYPT
                                                                  *
ERROR at line 1:
ORA-16000: database or pluggable database open for read-only access
Stopping the apply process has no effect in above.

Stopping the PDB also has no effect.
QL> show pdbs

    CON_ID CON_NAME                       OPEN MODE  RESTRICTED
---------- ------------------------------ ---------- ----------
         3 TESTPDB                        READ ONLY  NO
         
SQL> shutdown immediate;
Pluggable Database closed.

SQL> show pdbs

    CON_ID CON_NAME                       OPEN MODE  RESTRICTED
---------- ------------------------------ ---------- ----------
         3 TESTPDB                        MOUNTED
         
SQL>  ALTER TABLESPACE ENCTEST ENCRYPTION OFFLINE  USING 'AES256' ENCRYPT;
 ALTER TABLESPACE ENCTEST ENCRYPTION OFFLINE  USING 'AES256' ENCRYPT
*
ERROR at line 1:
ORA-28429: cannot encrypt or decrypt a data file on standby when it is open read-only
Only solution is to stop the standby CDB and open it in mount mode and do the encryption/decryption as shown in standby in mount mode case.

Standby First Encryption and Switchover
Online encryption requires additional space compared to offline encryption. As offline encryption requires tablespace to be taken offline (no read/write) which result in application down time. To avoid these limitiations the encryption could be done first on standby in offline mode. Once completed a switchover could be done to encrypt the tablespace on old primary.
Current DG configuration.
DGMGRL> show configuration

Configuration - drcp_dg

  Protection Mode: MaxAvailability
  Members:
  testcdb  - Primary database
    testcdb2 - Physical standby database
    testcdb3 - Physical standby database

Fast-Start Failover:  Disabled

Configuration Status:
SUCCESS   (status updated 51 seconds ago)
Encrypt the tablespace on the standby.
ALTER TABLESPACE ENCTEST ENCRYPTION OFFLINE  USING 'AES256' ENCRYPT;

NAME                           ENCRYPT ENC STATUS
------------------------------ ------- --- ----------
ENCTEST                        AES256  YES NORMAL
Tablespace encryption status on primary
NAME                           ENCRYPT ENC STATUS
------------------------------ ------- --- ----------
ENCTEST
Do a switchover to standby with encrypted tablespace
switchover to testcdb3

DGMGRL> show configuration

Configuration - drcp_dg

  Protection Mode: MaxAvailability
  Members:
  testcdb3 - Primary database
    testcdb  - Physical standby database
    testcdb2 - Physical standby database

Fast-Start Failover:  Disabled

Configuration Status:
SUCCESS   (status updated 59 seconds ago)
Offline encrypt the tablespace on old primary.
ALTER TABLESPACE ENCTEST ENCRYPTION OFFLINE  USING 'AES256' ENCRYPT;

NAME                           ENCRYPT ENC STATUS
------------------------------ ------- --- ----------
ENCTEST                        AES256  YES NORMAL

Online Encryption When Tablespace Encryption Status on Each Standby Differ
A tablespace that is already encrypted cannot be encrypted again. Doesn't matter if it is online or offline the encryption operation will fail. Below shows trying to re-encrypt an already encrypted tablespace
SQL>  select t.name,e.ENCRYPTIONALG,e.ENCRYPTEDTS,e.STATUS from V$ENCRYPTED_TABLESPACES e, v$tablespace t where t.ts#=e.ts#(+);

NAME                           ENCRYPT ENC STATUS
------------------------------ ------- --- ----------
ENCTEST

SQL> ALTER TABLESPACE ENCTEST ENCRYPTION ONLINE USING 'AES256' ENCRYPT;

Tablespace altered.

SQL> select t.name,e.ENCRYPTIONALG,e.ENCRYPTEDTS,e.STATUS from V$ENCRYPTED_TABLESPACES e, v$tablespace t where t.ts#=e.ts#(+) and NAME='ENCTEST';

NAME                           ENCRYPT ENC STATUS
------------------------------ ------- --- ----------
ENCTEST                        AES256  YES NORMAL

SQL> ALTER TABLESPACE ENCTEST ENCRYPTION ONLINE USING 'AES256' ENCRYPT;
ALTER TABLESPACE ENCTEST ENCRYPTION ONLINE USING 'AES256' ENCRYPT
*
ERROR at line 1:
ORA-28370: ENCRYPT, DECRYPT or REKEY option not allowed


alter tablespace ENCTEST offline normal;
ALTER TABLESPACE ENCTEST ENCRYPTION OFFLINE  USING 'AES256' ENCRYPT;


SQL>  ALTER TABLESPACE ENCTEST ENCRYPTION OFFLINE  USING 'AES256' ENCRYPT
*
ERROR at line 1:
ORA-28431: cannot encrypt an already encrypted data file ENCTEST
Same happens for decryption as well. A tablespace already decrypted (unencrypted) cannot be decrypted again.
SQL> ALTER TABLESPACE ENCTEST ENCRYPTION online DECRYPT;

Tablespace altered.

SQL>  select t.name,e.ENCRYPTIONALG,e.ENCRYPTEDTS,e.STATUS from V$ENCRYPTED_TABLESPACES e, v$tablespace t where t.ts#=e.ts#(+) and NAME='ENCTEST';

NAME                           ENCRYPT ENC STATUS
------------------------------ ------- --- ----------
ENCTEST

SQL>  ALTER TABLESPACE ENCTEST ENCRYPTION online DECRYPT;
 ALTER TABLESPACE ENCTEST ENCRYPTION online DECRYPT
*
ERROR at line 1:
ORA-28370: ENCRYPT, DECRYPT or REKEY option not allowed


SQL> alter tablespace ENCTEST offline normal;
 ALTER TABLESPACE ENCTEST ENCRYPTION OFFLINE DECRYPT;
 alter tablespace ENCTEST online;
Tablespace altered.

SQL>  ALTER TABLESPACE ENCTEST ENCRYPTION OFFLINE DECRYPT
*
ERROR at line 1:
ORA-28434: cannot decrypt an unencrypted data file
+DATA/TDETEST/C20D0B518CB34375E053360B1FACB37C/DATAFILE/enctest.317.1085912207

However, it appears, tablespace that are already encrypted or un-encrypted will get rekeyed in a DG configuration if online encryption/decryption is done on primary. At the start the primary (testcdb) has un-encrypted tablespace.
SQL> select t.name,e.ENCRYPTIONALG,e.ENCRYPTEDTS,e.STATUS from V$ENCRYPTED_TABLESPACES e, v$tablespace t where t.ts#=e.ts#(+) and NAME='ENCTEST';

NAME                           ENCRYPT ENC STATUS
------------------------------ ------- --- ----------
ENCTEST
One of the standby (testcdb2) databases also has un-encrypted tablespace
SQL> select t.name,e.ENCRYPTIONALG,e.ENCRYPTEDTS,e.STATUS from V$ENCRYPTED_TABLESPACES e, v$tablespace t where t.ts#=e.ts#(+) and NAME='ENCTEST';

NAME                           ENCRYPT ENC STATUS
------------------------------ ------- --- ----------
ENCTEST
Another standby (testcdb3) has an encrypted tablespace
NAME                           ENCRYPT ENC STATUS
------------------------------ ------- --- ----------
ENCTEST                        AES256  YES NORMAL
On the primary the the tablespace is encrypted online.
SQL> ALTER TABLESPACE ENCTEST ENCRYPTION ONLINE USING 'AES256' ENCRYPT;

Tablespace altered.

SQL>  select t.name,e.ENCRYPTIONALG,e.ENCRYPTEDTS,e.STATUS from V$ENCRYPTED_TABLESPACES e, v$tablespace t where t.ts#=e.ts#(+) and NAME='ENCTEST';

NAME                           ENCRYPT ENC STATUS
------------------------------ ------- --- ----------
ENCTEST                        AES256  YES NORMAL
As a result the tablespace that was un-encrypted on a standby (testcdb2) gets encrypted.
SQL> select t.name,e.ENCRYPTIONALG,e.ENCRYPTEDTS,e.STATUS from V$ENCRYPTED_TABLESPACES e, v$tablespace t where t.ts#=e.ts#(+) and NAME='ENCTEST';

NAME                           ENCRYPT ENC STATUS
------------------------------ ------- --- ----------
ENCTEST                        AES256  YES NORMAL
Alert log shows following
2021-10-14T10:33:18.996464+00:00
(3):Rekeying datafile +DATA/TESTCDB2/9D48FF44CE8A7D32E053F4071FACDFB3/DATAFILE/enctest.297.1085912727 (21) to +DATA
(3):Rekey operation committed for file +DATA/TESTCDB2/9D48FF44CE8A7D32E053F4071FACDFB3/DATAFILE/enctest.298.1085913199
2021-10-14T10:33:22.026323+00:00
(3):About to zero out original file "+DATA/TESTCDB2/9D48FF44CE8A7D32E053F4071FACDFB3/DATAFILE/enctest.297.1085912727"
(3):Successfully zero'ed out original file "+DATA/TESTCDB2/9D48FF44CE8A7D32E053F4071FACDFB3/DATAFILE/enctest.297.1085912727"
(3):Successfully deleted original file "+DATA/TESTCDB2/9D48FF44CE8A7D32E053F4071FACDFB3/DATAFILE/enctest.297.1085912727"
The tablespace that was already encrypted on a standby (testcdb3) also get rekeyed
2021-10-14T10:33:19.519352+00:00
(3):Rekeying datafile +DATA/TESTCDB3/9D48FF44CE8A7D32E053F4071FACDFB3/DATAFILE/enctest.318.1085912727 (21) to +DATA
2021-10-14T10:33:20.968269+00:00
(3):Rekey operation committed for file +DATA/TESTCDB3/9D48FF44CE8A7D32E053F4071FACDFB3/DATAFILE/enctest.319.1085913199
2021-10-14T10:33:22.987772+00:00
(3):About to zero out original file "+DATA/TESTCDB3/9D48FF44CE8A7D32E053F4071FACDFB3/DATAFILE/enctest.318.1085912727"
(3):Successfully zero'ed out original file "+DATA/TESTCDB3/9D48FF44CE8A7D32E053F4071FACDFB3/DATAFILE/enctest.318.1085912727"
(3):Successfully deleted original file "+DATA/TESTCDB3/9D48FF44CE8A7D32E053F4071FACDFB3/DATAFILE/enctest.318.1085912727"
The same happens for decryption as well. For example primary (testcdb) and one more stnadby (testcdb2) could have encrypted tablespaces while other standby (testcdb3) has an un-encrypted tablespace. Running online decryption results in tablespace in both primary and standby with encrypted tablespace getting decrypted. No error is shown on the standby that had the un-encrypted tablespace and alert log output rekeying of datafiles similar to above.
2021-10-14T10:49:06.058622+00:00
(3):Rekeying datafile +DATA/TESTCDB3/9D48FF44CE8A7D32E053F4071FACDFB3/DATAFILE/enctest.319.1085913199 (21) to +DATA
2021-10-14T10:49:07.523466+00:00
(3):Rekey operation committed for file +DATA/TESTCDB3/9D48FF44CE8A7D32E053F4071FACDFB3/DATAFILE/enctest.318.1085914147
2021-10-14T10:49:09.539791+00:00
(3):About to zero out original file "+DATA/TESTCDB3/9D48FF44CE8A7D32E053F4071FACDFB3/DATAFILE/enctest.319.1085913199"
(3):Successfully zero'ed out original file "+DATA/TESTCDB3/9D48FF44CE8A7D32E053F4071FACDFB3/DATAFILE/enctest.319.1085913199"
(3):Successfully deleted original file "+DATA/TESTCDB3/9D48FF44CE8A7D32E053F4071FACDFB3/DATAFILE/enctest.319.1085913199"

Related Posts
Encrypting / Decrypting Tablespaces Online / Offline
19c Encryption
12c Encryption
Rekeying Master Key in a Data Guard Setup

Wednesday, October 27, 2021

Backup Configuraiton in DBCS VM DB With Data Guard

There is an earlier post on how to setup data guard association on DBCS VM DB.
This post looks at the backup configuraiton of such a setup.
Primary has automatic backup enabled.

The new data guard configuration is created for database between two regions.
DGMGRL> show configuration

Configuration - londb_lhr1cq_londb_cwl1zf

  Protection Mode: MaxPerformance
  Members:
  londb_lhr1cq - Primary database
    londb_cwl1zf - Physical standby database

Fast-Start Failover:  Disabled

Configuration Status:
SUCCESS   (status updated 48 seconds ago)
Once the DG association is created the archive log deletion policy on primary gets changed from
CONFIGURE ARCHIVELOG DELETION POLICY TO BACKED UP 1 TIMES TO 'SBT_TAPE';
to
CONFIGURE ARCHIVELOG DELETION POLICY TO BACKED UP 1 TIMES TO 'SBT_TAPE' SHIPPED TO ALL STANDBY;
Even though backed up time appears before the shipped clause this has not prevented archvie logs getting deleted.
On the standby the archive log deletion policy is
CONFIGURE ARCHIVELOG DELETION POLICY TO SHIPPED TO ALL STANDBY;
Even though primary had automatic backup enabled, the standby is created with automatic backup disabled.

There is no simple way to enable automatic backups on standby. There's no "configure automatic backup" button on the standby database detail page. Even with terraform there's no way to enable automatic backup on the standby database (this could change in the future).




Only way to enable automatic backup on the standby is to make a switchover to the standby. After the switchover the "configure automatic backup" button will appear on the new primary (old standby) database detail page. However, automatic backup will still be disabled.

Enable automatic backup on the new primary (old standby).

Afterwards automatic backup enabled on the new primary (old standby).

The archive log deletion policy on the new primary (old standby) gets changed from
CONFIGURE ARCHIVELOG DELETION POLICY TO SHIPPED TO ALL STANDBY;
to
CONFIGURE ARCHIVELOG DELETION POLICY TO BACKED UP 1 TIMES TO 'SBT_TAPE' SHIPPED TO ALL STANDBY;
after the first automatic backup has taken place.
The old primary (new standby) will have automatic backup enabled status.

However, it doesn't take full or incremental backups. Only the hourly archive log backups run with automatic backup enabled on the standby.
Furthermore with standby in mount no manual backups are not allowed using dbcli.
dbcli create-backup -bt ArchiveLog -i dd2a13fb-46cf-4a51-9cce-485b7ff4ace1
DCS-10089:Database londb is in an invalid state '{ nwpdbvm: Mounted (Closed) }'.
There's also no way to disable automatic backup on a standby that already has it enabled. Same as enabling, DB must be made a primary DB to disable the automatic backups.

Related Posts
Data Guard on OCI PaaS DB
Data Guard With OCI PaaS DB (DBaaS) Primary and Standby on IaaS

Tuesday, October 19, 2021

Remote Log Archive Destinations are Not Mentioned in Alert Log After DBRU 19.12 Applied

A data guard configuration consists of several standby and far sync instances.
DGMGRL> show configuration

Configuration - db_dg

  Protection Mode: MaxAvailability
  Members:
  db2  - Primary database
    db1  - Physical standby database
    db3  - Physical standby database
      db4  - Physical standby database (receiving current redo)
    fs1  - Far sync instance
      db5  - Physical standby database
      db6  - Physical standby database
      db7  - Physical standby database
      db8  - Physical standby database
      db9  - Physical standby database
      db10 - Physical standby database
      db11 - Physical standby database
      db12 - Physical standby database
      db13 - Physical standby database
      db14 - Physical standby database

  Members Not Receiving Redo:
  fs2  - Far sync instance (alternate of fs1)

Fast-Start Failover:  Disabled

Configuration Status:
SUCCESS   (status updated 57 seconds ago)
When ever there is a log switch the alert log would have entries corresponding to all remote log archive destinations (denoted LAD). For example in the above configuration primary write sto db1,db3 and fs1. On the primary alert log entries like below would be visible.
LGWR (PID:107336): SRL selected to archive T-1.S-3023
LGWR (PID:107336): SRL selected for T-1.S-3023 for LAD:6
LGWR (PID:107336): SRL selected to archive T-1.S-3023
LGWR (PID:107336): SRL selected for T-1.S-3023 for LAD:4
LGWR (PID:107336): SRL selected to archive T-1.S-3023
LGWR (PID:107336): SRL selected for T-1.S-3023 for LAD:3
The LAD:n where n corresponds to log archive detination parameter number (log_archive_dest_n).
Similarly on the far sync alert log also there are entries corresponding to each log archive destination.
TT02 (PID:5781): SRL selected for T-1.S-36125 for LAD:12
TT02 (PID:5781): SRL selected for T-1.S-36125 for LAD:11
TT02 (PID:5781): SRL selected for T-1.S-36125 for LAD:10
TT02 (PID:5781): SRL selected for T-1.S-36125 for LAD:9
TT02 (PID:5781): SRL selected for T-1.S-36125 for LAD:8
TT02 (PID:5781): SRL selected for T-1.S-36125 for LAD:7
TT02 (PID:5781): SRL selected for T-1.S-36125 for LAD:6
TT02 (PID:5781): SRL selected for T-1.S-36125 for LAD:5
TT02 (PID:5781): SRL selected for T-1.S-36125 for LAD:4
TT02 (PID:5781): SRL selected for T-1.S-36125 for LAD:3



However, after applying DBRU 19.12 only one LAD is shown and that corresponds to local archive log destination. On primary
NET  (PID:27469): Archived Log entry 236855 added for T-1.S-77841 ID 0xfd561a70 LAD:1
One far sync
ARC3 (PID:3874): Archived Log entry 712212 added for T-1.S-77842 ID 0xfd561a70 LAD:1
LAD:1 corresponds to log_archive_dest_1 and has location=use_db_recovery_file_dest.

LAD entries related remote destinations are not visible in the alert logs of primary, cascade standby and far sync instasnces.

Further testing has shown that this change (missing alert log entries for remote LAD) started with DBRU 19.11 and could be seen on both 19.12 and 19.13.

Friday, October 15, 2021

ORA-16856: transport lag could not be determined on Far Sync After Archive Logs Delete With DBRU 19.12 Applied

The behaviour mentioned in this post only started to appear after applying the DBRU 19.12. Prior to applying the DBRU there was no issue of ORA-16856 with the combination of conditions mentioned in the post. System has been in operation for sometime.
After applying DBRU 19.12 noticed increase number of alerts related to ORA-16856 coming from a data guard configuration that has far sync instance in it. Behaviour was observed not one but several such environments. On the data guard broker log of the primary DB entries similar to below could be seen.
2021-08-19T14:08:17.271+00:00
Data Guard Broker Status Summary:
Type Name Severity Status
Configuration tst_dg Warning ORA-16608: one or more members have warnings
Primary Database tstdb1 Success ORA-0: normal, successful completion
Physical Standby Database tstdb2 Success ORA-0: normal, successful completion
Far Sync Instance tstfs1 Warning ORA-16856: transport lag could not be determined
Physical Standby Database tstdb6 Success ORA-0: normal, successful completion
Physical Standby Database tstdb5 Success ORA-0: normal, successful completion
No ORA-16856 related entreis were shown on the far sync instance logs. The data guard broker goes to a warning state for several minutes and rectify itself aftwards.
Analysis of the data guard broker logs showed that ORA-16856 was starting at the same minute of the hour and every hour. Further investigation showed that that first occurence of ORA-16856 corresponds to archive log deletion on the far sync instance. Far sync had archive deletion policy set to applied on all standby (later it was found that archive log deletion policy has no relavence to this issue) and deleting all the archive logs.
There were no overlapping rman activities between far sync and primary or other standby instances. Neverthless the archive log deletion on far sync was shifted to a different time. After that the ORA-16856 error started to appear at the newly scheduled time for the archive log deletion on far sync. This confirmed that issue is caused by the archive log deletion on far sync.
At this stage, looked at when did the new pattern of ORA-16856 started. It was confirmed current pattern of (every hour at the same minute) ORA-16856 startd after the DBRU 19.12 was applied. As the system has been operational for number of years there was enough historic log entries to compare against to see if pattern was there before as well. It was not.
To further isolate the possible root cause, archive log deletion was stopped on far sync in one of the non critical environments that exhibit the behaviour. Once the archive log deletion was stopped there was no recurring ORA-16856 as before. This indicated the log archive deletion could be the possible root cause.
As the next step the log archive deletion command was changed from
delete noprompt archivelog all;
to
delete noprompt archivelog all completed before 'sysdate - 1/24';
With this not all the archive logs were getting deleted. Some were omitted from the deletion as they fell out side the completed before time period. After the rman comamnd change the occurence of periodic ORA-16856 also stopped.



After this it was possible to manually recreate the issue, at will, by running a full archive log deletion on the far sync. Issue doesn't happen when all archive logs are deleted on standby databases. It only happens when all archive logs are deleted on the far sync instance.
As a workaround to the problem, in production systems, on far sync instances the archive log deletion command was changed so that some archive logs would always remain.
Hypothesis was that to calculate the transport lag far sync instnace uses archive log entries. Removing all of them makes it unable to calculate the transport lag hence the ORA-16856.
Raised a SR and Oracle support asked to recreate the issue with 'TraceLevel' = 'SUPPORT' in the DG broker. With this trace level when the issue was recreated far sync DG broker log also output the line related to ORA-16856 as shown below.
2021-10-01T11:10:21.900+00:00 DMON: Checking critical status of this database.
2021-10-01T11:10:21.900+00:00 00000000 1128079456 DMON: Broker configuration file successfully loaded for EDIT CONFIGURATION RE/SET PROPERTY
2021-10-01T11:10:21.900+00:00 INSV: Reply received for message with
2021-10-01T11:10:21.901+00:00 req ID 1.1.1128079456, opcode EDIT_DRC_PROP, phase RESYNCH
2021-10-01T11:10:21.901+00:00 00000000 1128079456 DMON: rfm_release_chief_lock() called for EDIT_DRC_PROP, called from rfmspd
2021-10-01T11:11:50.183+00:00 RSM0: HEALTH CHECK WARNING: ORA-16856: transport lag could not be determined
2021-10-01T11:11:50.234+00:00 03001000 1447213588 Operation HEALTH_CHECK continuing with warning, status = ORA-16856
2021-10-01T11:11:50.235+00:00 03001000 1447213588 Operation HEALTH_CHECK continuing with warning, status = ORA-16856
SR has resulted in an internal bug 33427973. No permanent solution yet. SR continues.

Friday, October 8, 2021

Flashback PDB in a Multi Standby Data Guard Configuraiton

There is an earlier post which looks flashback use with various configuration such as PDB, non-CDB and CDB on 12.2. This post looks flashback use when multiple standbys are invovlved and when each are open in different modes. The 19c release introduced a change which flashback the standby when primary is flashback. Therefore unlike the previous versions, there's no need to flashback the standby PDB after the primary is flashback. However, this transparent flashback of standby PDB doesn't work if any of them are open in read only mode at the time of primary flashback.
The DG configuration consits of 3 instances.
DGMGRL> show configuration

Configuration - test_dg

  Protection Mode: MaxAvailability
  Members:
  dgtest  - Primary database
    dgtest2 - Physical standby database
    dgtest3 - Physical standby database

Fast-Start Failover:  Disabled

Configuration Status:
SUCCESS   (status updated 34 seconds ago)
The dgtest2 instance is open in read only mode.
SQL> select open_mode,db_unique_name from v$database;

OPEN_MODE            DB_UNIQUE_NAME
-------------------- --------------
READ ONLY WITH APPLY dgtest2

DGMGRL> validate database dgtest2

  Database Role:     Physical standby database
  Primary Database:  dgtest

  Ready for Switchover:  Yes
  Ready for Failover:    Yes (Primary Running)

  Managed by Clusterware:
    dgtest :  YES
    dgtest2:  YES
The dgtest3 is open in mount mode.
SQL>  select open_mode,db_unique_name from v$database;

OPEN_MODE            DB_UNIQUE_NAME
-------------------- --------------
MOUNTED              dgtest3

DGMGRL> validtae database dgtest3
Unrecognized command "validtae", try "help"
DGMGRL> validate database dgtest3

  Database Role:     Physical standby database
  Primary Database:  dgtest

  Ready for Switchover:  Yes
  Ready for Failover:    Yes (Primary Running)

  Managed by Clusterware:
    dgtest :  YES
    dgtest3:  YES
To verify the flashback works correctly, create a guarantee restore point, truncate a table and flashback the PDB in primary.
SQL> alter session set container=dgpdb;

Session altered.

SQL> create restore point pdb_restore_point guarantee flashback database;

Restore point created.

truncate table x;

SQL> alter session set container=dgpdb;

Session altered.

SQL> show con_name

CON_NAME
------------------------------
DGPDB

SQL> alter pluggable database close;

Pluggable database altered.


SQL> flashback pluggable database DGPDB to restore point PDB_RESTORE_POINT;

Flashback complete.


SQL> alter pluggable database open resetlogs;

Pluggable database altered.
On the primary DB alert log could see the flashback activity on the PDB
2021-09-23T13:27:24.756572+00:00
DGPDB(3):alter pluggable database close
2021-09-23T13:27:24.780090+00:00
DGPDB(3):Pluggable database DGPDB closing
DGPDB(3):JIT: pid 14510 requesting stop
DGPDB(3):Closing sequence subsystem (10152431219).
DGPDB(3):Buffer Cache flush started: 3
DGPDB(3):Buffer Cache flush finished: 3
Pluggable database DGPDB closed
DGPDB(3):Completed: alter pluggable database close
2021-09-23T13:27:52.164215+00:00
DGPDB(3):flashback database to restore point PDB_RESTORE_POINT
DGPDB(3):ORA-65040 signalled during: flashback database to restore point PDB_RESTORE_POINT...
2021-09-23T13:28:16.884252+00:00
DGPDB(3):flashback pluggable database DGPDB to restore point PDB_RESTORE_POINT
2021-09-23T13:28:17.044790+00:00
DGPDB(3):Flashback Restore Start
DGPDB(3):Restore Flashback Pluggable Database DGPDB (3) until change 4361208
DGPDB(3):Flashback Restore Complete
DGPDB(3):Flashback Media Recovery Start
2021-09-23T13:28:17.169821+00:00
DGPDB(3):Serial Media Recovery started
DGPDB(3):max_pdb is 3
2021-09-23T13:28:17.262801+00:00
DGPDB(3):Recovery of Online Redo Log: Thread 1 Group 1 Seq 22 Reading mem 0
DGPDB(3):  Mem# 0: +DATA/DGTEST/ONLINELOG/group_1.264.1067878075
DGPDB(3):  Mem# 1: +FRA/DGTEST/ONLINELOG/group_1.418.1067878077
2021-09-23T13:28:17.285159+00:00
DGPDB(3):Incomplete Recovery applied until change 4361208 time 09/23/2021 13:25:36
DGPDB(3):Flashback Media Recovery Complete
DGPDB(3):Flashback Pluggable Database DGPDB (3) recovered until change 4361208
DGPDB(3):Completed: flashback pluggable database DGPDB to restore point PDB_RESTORE_POINT
2021-09-23T13:29:06.616625+00:00
Thread 1 advanced to log sequence 23 (LGWR switch),  current SCN: 4362053
  Current log# 2 seq# 23 mem# 0: +DATA/DGTEST/ONLINELOG/group_2.262.1067878077
  Current log# 2 seq# 23 mem# 1: +FRA/DGTEST/ONLINELOG/group_2.419.1067878079
2021-09-23T13:29:06.733706+00:00
ARC8 (PID:13201): Archived Log entry 800 added for T-1.S-22 ID 0xf0d12d4d LAD:1
2021-09-23T13:29:22.740922+00:00
Control autobackup written to DISK device

handle '+FRA/DGTEST/AUTOBACKUP/2021_09_23/s_1084022962.365.1084022963'

2021-09-23T13:29:56.709000+00:00
DGPDB(3):alter pluggable database open resetlogs
2021-09-23T13:29:56.810137+00:00
Online datafile 11
Online datafile 10
Online datafile 9
Online datafile 8
DGPDB(3):Pluggable database DGPDB pseudo opening
DGPDB(3):SUPLOG: Initialize PDB SUPLOG SGA, old value 0x0, new value 0x18
DGPDB(3):Autotune of undo retention is turned on.
DGPDB(3):Endian type of dictionary set to little
DGPDB(3):Undo initialization recovery: Parallel FPTR failed: start:10304853 end:10304857 diff:4 ms (0.0 seconds)
DGPDB(3):Undo initialization recovery: err:0 start: 10304850 end: 10304912 diff: 62 ms (0.1 seconds)
DGPDB(3):[14628] Successfully onlined Undo Tablespace 2.
DGPDB(3):Undo initialization online undo segments: err:0 start: 10304912 end: 10305010 diff: 98 ms (0.1 seconds)
DGPDB(3):Undo initialization finished serial:0 start:10304850 end:10305014 diff:164 ms (0.2 seconds)
DGPDB(3):Database Characterset for DGPDB is AL32UTF8
DGPDB(3):Pluggable database DGPDB pseudo closing
DGPDB(3):JIT: pid 14628 requesting stop
DGPDB(3):Closing sequence subsystem (10305197570).
2021-09-23T13:29:57.761732+00:00
DGPDB(3):Buffer Cache flush started: 3
2021-09-23T13:29:57.831904+00:00
DGPDB(3):Buffer Cache flush finished: 3
DGPDB(3):Pluggable database DGPDB opening in read write
DGPDB(3):SUPLOG: Initialize PDB SUPLOG SGA, old value 0x0, new value 0x18
DGPDB(3):Autotune of undo retention is turned on.
DGPDB(3):Endian type of dictionary set to little
DGPDB(3):Undo initialization recovery: Parallel FPTR complete: start:10305988 end:10305999 diff:11 ms (0.0 seconds)
DGPDB(3):Undo initialization recovery: err:0 start: 10305987 end: 10305999 diff: 12 ms (0.0 seconds)
DGPDB(3):[14628] Successfully onlined Undo Tablespace 2.
DGPDB(3):Undo initialization online undo segments: err:0 start: 10305999 end: 10306236 diff: 237 ms (0.2 seconds)
DGPDB(3):Undo initialization finished serial:0 start:10305987 end:10306247 diff:260 ms (0.3 seconds)
DGPDB(3):Pluggable database DGPDB dictionary check beginning
2021-09-23T13:29:58.820047+00:00
DGPDB(3):Pluggable Database DGPDB Dictionary check complete
2021-09-23T13:29:58.831530+00:00
DGPDB(3):Database Characterset for DGPDB is AL32UTF8
DGPDB(3):SUPLOG: Set PDB SUPLOG SGA at PDB OPEN, old 0x18, new 0x0 (no suplog)
2021-09-23T13:30:00.195291+00:00
DGPDB(3):Opening pdb with no Resource Manager plan active
DGPDB(3):joxcsys_required_dirobj_exists: directory object exists with required path /opt/app/oracle/product/19.x.0/dbhome_2/javavm/admin/, pid 14628 cid 3
Control autobackup written to DISK device

handle '+FRA/DGTEST/AUTOBACKUP/2021_09_23/s_1084023000.364.1084023001'

Pluggable database DGPDB closed
DGPDB(3):Completed: alter pluggable database open resetlogs




On the alter log of the standby instance open in mount mode (dgtest3) entries show flashback happening transparaently.
2021-09-23T13:29:56.669475+00:00
(3):Recovery of pluggable database DGPDB aborted due to pluggable database open resetlog marker.
(3):To continue recovery, restore all data files for this PDB to checkpoint SCN lower than 4361208, or timestamp before 09/23/2021 13:25:36, and restart recovery
PR00 (PID:13309): MRP0: Detected orphaned datafiles!
PR00 (PID:13309): Recovery will possibly be retried after pluggable database flashback...
2021-09-23T13:29:56.670731+00:00
Errors in file /opt/app/oracle/diag/rdbms/dgtest3/dgtest3/trace/dgtest3_pr00_13309.trc:
ORA-39874: Pluggable Database DGPDB recovery halted
ORA-39873: Restore all data files to a checkpoint SCN lower than 4361208.
PR00 (PID:13309): Managed Standby Recovery not using Real Time Apply
Recovery interrupted!
Recovery stopped due to failure in applying recovery marker (opcode 17.46).
Datafiles are recovered to a consistent state at change 4363117 but controlfile could be ahead of datafiles.
Stopping change tracking
2021-09-23T13:29:56.878879+00:00
Errors in file /opt/app/oracle/diag/rdbms/dgtest3/dgtest3/trace/dgtest3_pr00_13309.trc:
ORA-39874: Pluggable Database DGPDB recovery halted
ORA-39873: Restore all data files to a checkpoint SCN lower than 4361208.
2021-09-23T13:30:16.929698+00:00
MRP0 (PID:13302): Recovery coordinator performing automatic flashback of pluggable database 3 to SCN:0x0000000000428bf7 (4361207)
Flashback Restore Start
Restore Flashback Pluggable Database DGPDB (3) until change 4356167
Flashback Restore Complete
 Started logmerger process
2021-09-23T13:30:17.324525+00:00
PR00 (PID:14265): Managed Standby Recovery starting Real Time Apply
max_pdb is 3
2021-09-23T13:30:17.694838+00:00
Parallel Media Recovery started with 8 slaves
2021-09-23T13:30:17.739944+00:00
Stopping change tracking
2021-09-23T13:30:17.802774+00:00
PR00 (PID:14265): Media Recovery Log +FRA/DGTEST3/ARCHIVELOG/2021_09_23/thread_1_seq_21.285.1084022085
2021-09-23T13:30:17.975446+00:00
PR00 (PID:14265): Media Recovery Log +FRA/DGTEST3/ARCHIVELOG/2021_09_23/thread_1_seq_22.282.1084022947
2021-09-23T13:30:18.373527+00:00
PR00 (PID:14265): Media Recovery Waiting for T-1.S-23 (in transit)
2021-09-23T13:30:18.378327+00:00
Recovery of Online Redo Log: Thread 1 Group 4 Seq 23 Reading mem 0
  Mem# 0: +FRA/DGTEST3/ONLINELOG/group_4.461.1069533181
(3):Applying tablespace dictionary check redo for tablespace #0
(3):Applying tablespace dictionary check redo for tablespace #1
(3):Applying tablespace dictionary check redo for tablespace #2
(3):Applying tablespace dictionary check redo for tablespace #3
(3):Dropping offline tempfile '+DATA'
(3):Applying tablespace dictionary check redo for tablespace #4
2021-09-23T13:30:19.374861+00:00
(3):Applying datafile dictionary check redo for datafile #8
2021-09-23T13:30:19.431584+00:00
(3):Applying datafile dictionary check redo for datafile #9
(3):Applying datafile dictionary check redo for datafile #10
(3):Applying datafile dictionary check redo for datafile #11

However, on the alert log of the standby instance open in read only mode (dgtest2) shows recovery process shutting down.
2021-09-23T13:29:56.441259+00:00
(3):Recovery of pluggable database DGPDB aborted due to pluggable database open resetlog marker.
(3):To continue recovery, restore all data files for this PDB to checkpoint SCN lower than 4361208, or timestamp before 09/23/2021 13:25:36, and restart recovery
PR00 (PID:12442): MRP0: Detected orphaned datafiles!
PR00 (PID:12442): Recovery will possibly be retried after pluggable database flashback...
2021-09-23T13:29:56.442191+00:00
Errors in file /opt/app/oracle/diag/rdbms/dgtest2/dgtest2/trace/dgtest2_pr00_12442.trc:
ORA-39874: Pluggable Database DGPDB recovery halted
ORA-39873: Restore all data files to a checkpoint SCN lower than 4361208.
PR00 (PID:12442): Managed Standby Recovery not using Real Time Apply
Recovery interrupted!
Recovered data files to a consistent state at change 4363117
Stopping change tracking
2021-09-23T13:29:56.585631+00:00
Errors in file /opt/app/oracle/diag/rdbms/dgtest2/dgtest2/trace/dgtest2_pr00_12442.trc:
ORA-39874: Pluggable Database DGPDB recovery halted
ORA-39873: Restore all data files to a checkpoint SCN lower than 4361208.
2021-09-23T13:30:17.728791+00:00
MRP0 (PID:12431): Recovery coordinator encountered one or more errors during automatic flashback on standby
2021-09-23T13:30:17.728863+00:00
Background Media Recovery process shutdown (dgtest2)

This is expected as flashback cannot happen when database is in open mode even if it is read only. The DG broker shows an error state.
DGMGRL> show configuration

Configuration - test_dg

  Protection Mode: MaxAvailability
  Members:
  dgtest  - Primary database
    dgtest2 - Physical standby database
      Error: ORA-16810: multiple errors or warnings detected for the member

    dgtest3 - Physical standby database

Fast-Start Failover:  Disabled

Configuration Status:
ERROR   (status updated 48 seconds ago)

DGMGRL> show database dgtest2

Database - dgtest2

  Role:               PHYSICAL STANDBY
  Intended State:     APPLY-ON
  Transport Lag:      0 seconds (computed 1 second ago)
  Apply Lag:          2 minutes 15 seconds (computed 1 second ago)
  Average Apply Rate: 20.00 KByte/s
  Real Time Query:    OFF
  Instance(s):
    dgtest2

  Database Error(s):
    ORA-16766: Redo Apply is stopped

  Database Warning(s):
    ORA-16853: apply lag has exceeded specified threshold

Database Status:
ERROR
Stopping and starting the PDB on the standby instance does not start the recovery process.

Stopping and starting the apply process (apply-off/on) on the standby instance also does not start the recovery process.

Keeping the PDB in mount mode and stopping and starting the apply service also doesn't start the recovery process.

Trying to flashback the PDB while PDB is in mount also fails.
SQL> flashback pluggable database to scn 4361208;
flashback pluggable database to scn 4361208
*
ERROR at line 1:
ORA-01126: database must be mounted in this instance and not open in any instance
Intresting the error says DB must be in mount mode even while it's mounted. What it actually refers here is that the CDB must also be in a mounted state not just the PDB.

To resolve the error and start the recovery process stop the CDB and start it in mount mode.
srvctl stop database -db $ORACLE_SID
srvctl start database -db $ORACLE_SID -startoption mount
This will result in flashback happening transparently on the PDB similar to instance that was in mount mode and apply process will continue without any issue.
After recovery on the standby has caught up with primary, it could be open in read only mode again.

Related Post
Flashback Primary DB in a Data Guard - PDB vs non-CDB vs CDB

Tuesday, September 28, 2021

Failover and Reinstate With Multiple Physical Standbys

The data guard configuration consists of three databases.
 DGMGRL> show configuration

Configuration - test_dg

  Protection Mode: MaxAvailability
  Members:
  dgtest3 - Primary database
    dgtest  - Physical standby database
    dgtest2 - Physical standby database

Fast-Start Failover:  Disabled

Configuration Status:
SUCCESS   (status updated 67 seconds ago)
At the moment dgtest3 is the primary. One of the standby (dgtest2) databases is open in mount mode.
SQL> select db_unique_name,open_mode from v$database;

DB_UNIQUE_NAME                 OPEN_MODE
------------------------------ --------------------
dgtest2                        MOUNTED
The other standby (dgtest) is an active data guard and is open in read only mode with log apply.
SQL> select db_unique_name,open_mode from v$database;

DB_UNIQUE_NAME                 OPEN_MODE
------------------------------ --------------------
dgtest                         READ ONLY WITH APPLY
To simulate primary DB failure it is shutdown with abort.
SQL> shutdown abort;
ORACLE instance shut down.
Checking the data guard configuration from dgtest2 shows an error state.
DGMGRL> show configuration

Configuration - test_dg

  Protection Mode: MaxAvailability
  Members:
  dgtest3 - Primary database
    Error: ORA-12514: TNS:listener does not currently know of service requested in connect descriptor

    dgtest  - Physical standby database
    dgtest2 - Physical standby database

Fast-Start Failover:  Disabled

Configuration Status:
ERROR   (status updated 0 seconds ago)
One of the existing standbys could be choosen for failover. In this case dgtest2( instance in mount mode) was chosen. Validating the dgtest2 show that it is ready for failover.
DGMGRL> validate database dgtest2

  Database Role:     Physical standby database
  Primary Database:  dgtest3
    Warning: primary database was not reachable

  Ready for Switchover:  No
  Ready for Failover:    Yes (Primary Not Running)

  Flashback Database Status:
    dgtest3:  Unknown
    dgtest2:  On

  Managed by Clusterware:
    dgtest3:  Unknown
    dgtest2:  YES
    Validating static connect identifier for the primary database dgtest3...
Unable to connect to database using (DESCRIPTION=(ADDRESS=(PROTOCOL=TCP)(HOST=ip-132)(PORT=1521))(CONNECT                                                                                                                          _DATA=(SERVICE_NAME=dgtest3_DGMGRL)(INSTANCE_NAME=dgtest3)(SERVER=DEDICATED)(STATIC_SERVICE=TRUE)))
ORA-12514: TNS:listener does not currently know of service requested in connect descriptor

Failed.
    Warning: Ensure primary database's StaticConnectIdentifier property
    is configured properly so that the primary database can be restarted
    by DGMGRL after switchover

  Temporary Tablespace File Information:
    dgtest3 TEMP Files:  Unknown
    dgtest2 TEMP Files:  3

  Data file Online Move in Progress:
    dgtest3:  Unknown
    dgtest2:  No

  Transport-Related Information:
    Transport On:  No
    Gap Status:    Unknown
    Transport Lag:  0 seconds (computed 24 seconds ago)
    Transport Status:  Success

  Log Files Cleared:
    dgtest3 Standby Redo Log Files:  Unknown
    dgtest2 Online Redo Log Files:   Unknown
    dgtest2 Standby Redo Log Files:  Unknown

Initiate failover to dgtest2
DGMGRL> failover to dgtest2;
Performing failover NOW, please wait...
Failover succeeded, new primary is "dgtest2"
During the failover process the active data guard instance continue to function. It will cancel the current redo apply service and start a new one with the incarnation that results from the failover. At the same time any references to the old primary will be removed (such as fal_server).
2021-09-20T09:53:29.718199+00:00
ALTER DATABASE RECOVER MANAGED STANDBY DATABASE CANCEL
2021-09-20T09:53:29.720039+00:00
PR00 (PID:5312): MRP0: Background Media Recovery cancelled with status 16037
2021-09-20T09:53:29.720747+00:00
Errors in file /opt/app/oracle/diag/rdbms/dgtest/dgtest/trace/dgtest_pr00_5312.trc:
ORA-16037: user requested cancel of managed recovery operation
PR00 (PID:5312): Managed Standby Recovery not using Real Time Apply
Recovery interrupted!
Recovered data files to a consistent state at change 4236455
Stopping change tracking
2021-09-20T09:53:29.943216+00:00
Errors in file /opt/app/oracle/diag/rdbms/dgtest/dgtest/trace/dgtest_pr00_5312.trc:
ORA-16037: user requested cancel of managed recovery operation
2021-09-20T09:53:30.067266+00:00
Background Media Recovery process shutdown (dgtest)
2021-09-20T09:53:30.720982+00:00
Managed Standby Recovery Canceled (dgtest)
Completed: ALTER DATABASE RECOVER MANAGED STANDBY DATABASE CANCEL
2021-09-20T09:53:32.641452+00:00
ALTER SYSTEM SET fal_server='dgtest2tns' SCOPE=BOTH;
ALTER DATABASE RECOVER MANAGED STANDBY DATABASE DISCONNECT  NODELAY
2021-09-20T09:53:32.658894+00:00
Attempt to start background Managed Standby Recovery process (dgtest)
Starting background process MRP0
2021-09-20T09:53:32.673122+00:00
MRP0 started with pid=63, OS id=5816
2021-09-20T09:53:32.674424+00:00
Background Managed Standby Recovery process started (dgtest)
2021-09-20T09:53:36.270863+00:00
 rfs (PID:5820): krsr_rfs_atc: Identified database type as 'PHYSICAL STANDBY': Client is Foreground (PID:11152)
Deleted Oracle managed file +FRA/DGTEST/ARCHIVELOG/2021_09_20/thread_0_seq_0.371.1083750817
2021-09-20T09:53:37.697852+00:00
 Started logmerger process
2021-09-20T09:53:37.712394+00:00

IM on ADG: Start of Empty Journal

IM on ADG: End of Empty Journal
PR00 (PID:5826): Managed Standby Recovery starting Real Time Apply
max_pdb is 3
2021-09-20T09:53:37.888440+00:00
 rfs (PID:5824): krsr_rfs_atc: Identified database type as 'PHYSICAL STANDBY': Client is SYNC (PID:10754)
 rfs (PID:5824): New archival redo branch: 1083750811 current: 1067878072
 rfs (PID:5824): Primary database is in MAXIMUM AVAILABILITY mode
 rfs (PID:5824): Changing standby controlfile to RESYNCHRONIZATION level
 rfs (PID:5824): Standby controlfile consistent with primary
2021-09-20T09:53:37.951928+00:00
 rfs (PID:5824): Selected LNO:6 for T-1.S-3 dbid 4024401720 branch 1083750811
2021-09-20T09:53:37.974747+00:00
Parallel Media Recovery started with 8 slaves
2021-09-20T09:53:38.018389+00:00
Stopping change tracking
PR00 (PID:5826): Media Recovery Waiting for T-1.S-382
2021-09-20T09:53:38.440514+00:00
 rfs (PID:5855): krsr_rfs_atc: Identified database type as 'PHYSICAL STANDBY': Client is FAL (PID:11166)
2021-09-20T09:53:38.505442+00:00
 rfs (PID:5855): Selected LNO:7 for T-1.S-2 dbid 4024401720 branch 1083750811
 rfs (PID:5855): A new recovery destination branch has been registered
 rfs (PID:5855): New Archival REDO Branch(resetlogs_id): 1083750811  Prior: 1067878072
 rfs (PID:5855): Archival Activation ID: 0xf0d1268f Current: 0xf0d199cf
 rfs (PID:5855): Effect of primary database OPEN RESETLOGS
 rfs (PID:5855): Managed Standby Recovery process is active
2021-09-20T09:53:38.536066+00:00
Incarnation entry added for Branch(resetlogs_id): 1083750811 (dgtest)
2021-09-20T09:53:38.548638+00:00
Setting recovery target incarnation to 2
2021-09-20T09:53:38.678029+00:00
Completed: ALTER DATABASE RECOVER MANAGED STANDBY DATABASE DISCONNECT  NODELAY
2021-09-20T09:53:39.126620+00:00
PR00 (PID:5826): MRP0: Incarnation has changed! Retry recovery...
2021-09-20T09:53:39.127329+00:00
Errors in file /opt/app/oracle/diag/rdbms/dgtest/dgtest/trace/dgtest_pr00_5826.trc:
ORA-19906: recovery target incarnation changed during recovery
PR00 (PID:5826): Managed Standby Recovery not using Real Time Apply
Recovery interrupted!
Stopping change tracking
2021-09-20T09:53:39.233028+00:00
Errors in file /opt/app/oracle/diag/rdbms/dgtest/dgtest/trace/dgtest_pr00_5826.trc:
ORA-19906: recovery target incarnation changed during recovery
2021-09-20T09:53:39.432821+00:00
 rfs (PID:5858): krsr_rfs_atc: Identified database type as 'PHYSICAL STANDBY': Client is FAL (PID:11172)
2021-09-20T09:53:39.480429+00:00
 rfs (PID:5858): Opened log for T-1.S-382 dbid 4024401720 branch 1067878072
2021-09-20T09:53:39.509873+00:00
 rfs (PID:5858): Archived Log entry 746 added for B-1067878072.T-1.S-382 ID 0xf0d199cf LAD:2
2021-09-20T09:53:39.573601+00:00
 Started logmerger process
2021-09-20T09:53:39.614023+00:00

IM on ADG: Start of Empty Journal

IM on ADG: End of Empty Journal
PR00 (PID:5861): Managed Standby Recovery starting Real Time Apply
2021-09-20T09:53:39.622286+00:00
ARC8 (PID:5169): Archived Log entry 747 added for T-1.S-2 ID 0xf0d1268f LAD:1
2021-09-20T09:53:39.678880+00:00
max_pdb is 3
2021-09-20T09:53:39.878314+00:00
Parallel Media Recovery started with 8 slaves
2021-09-20T09:53:39.894206+00:00
Media Recovery start incarnation depth : 1, target inc# : 2, irscn : 4236456
Stopping change tracking
2021-09-20T09:53:39.917344+00:00
 rfs (PID:5863): krsr_rfs_atc: Identified database type as 'PHYSICAL STANDBY': Client is FAL (PID:11158)
2021-09-20T09:53:39.974576+00:00
 rfs (PID:5863): Opened log for T-1.S-1 dbid 4024401720 branch 1083750811
2021-09-20T09:53:40.001952+00:00
 rfs (PID:5863): Archived Log entry 748 added for B-1083750811.T-1.S-1 ID 0xf0d1268f LAD:2
2021-09-20T09:53:40.011908+00:00
PR00 (PID:5861): Media Recovery Log +FRA/DGTEST/ARCHIVELOG/2021_09_20/thread_1_seq_382.371.1083750819
2021-09-20T09:53:40.134398+00:00
PR00 (PID:5861): Resetting standby activation ID 4040268239 (0xf0d199cf)
2021-09-20T09:53:40.138397+00:00
Media Recovery End-Of-Redo indicator encountered
2021-09-20T09:53:40.138471+00:00
Media Recovery Continuing
2021-09-20T09:53:40.401844+00:00
PR00 (PID:5861): Media Recovery Log +FRA/DGTEST/ARCHIVELOG/2021_09_20/thread_1_seq_1.454.1083750819
2021-09-20T09:53:40.499037+00:00
PR00 (PID:5861): Media Recovery Log +FRA/DGTEST/ARCHIVELOG/2021_09_20/thread_1_seq_2.372.1083750819
2021-09-20T09:53:40.692602+00:00
PR00 (PID:5861): Media Recovery Waiting for T-1.S-3 (in transit)
2021-09-20T09:53:40.698462+00:00
Recovery of Online Redo Log: Thread 1 Group 6 Seq 3 Reading mem 0
  Mem# 0: +FRA/DGTEST/ONLINELOG/group_6.423.1067949163
2021-09-20T09:53:41.024853+00:00
 rfs (PID:5824): Changing standby controlfile to MAXIMUM AVAILABILITY level
2021-09-20T09:53:41.041788+00:00
 rfs (PID:5824): Selected LNO:7 for T-1.S-4 dbid 4024401720 branch 1083750811
2021-09-20T09:53:41.045351+00:00
ARC9 (PID:5171): Archived Log entry 749 added for T-1.S-3 ID 0xf0d1268f LAD:1
2021-09-20T09:53:41.118868+00:00
PR00 (PID:5861): Media Recovery Waiting for T-1.S-4 (in transit)
2021-09-20T09:53:41.124519+00:00
Recovery of Online Redo Log: Thread 1 Group 7 Seq 4 Reading mem 0
  Mem# 0: +FRA/DGTEST/ONLINELOG/group_7.424.1067949165

The instance failed over to will stop it's current redo apply service and convert to a read/write primary. It is intresting to see some of the lines in the log output "switchover" instead of failover.
2021-09-20T09:53:24.419649+00:00
Beginning failover to database dgtest2.
Starting background process NSV1
2021-09-20T09:53:24.492112+00:00
NSV1 started with pid=16, OS id=12324
2021-09-20T09:53:29.976461+00:00
ALTER DATABASE FAILOVER TO dgtest2
2021-09-20T09:53:29.976612+00:00
RSM0 (PID:11547): The Time Management Interface (TMI) is being enabled for role transition
RSM0 (PID:11547): information.  This will result in messages beingoutput to the alert log
RSM0 (PID:11547): file with the prefix 'TMI: '.  This is being enabled to make the timing of
RSM0 (PID:11547): the various stages of the role transition available for diagnostic purposes.
RSM0 (PID:11547): This output will end when the role transition is complete.
TMI: dbsdrv failover to target BEGIN 2021-09-20 09:53:29.977021
Terminal Recovery requested in process 11547
TMI: adbdrv termRecovery BEGIN 2021-09-20 09:53:29.978960
RSM0 (PID:11547): Terminal Recovery: Stopping real time apply
2021-09-20T09:53:29.980343+00:00
PR00 (PID:11627): MRP0: Background Media Recovery cancelled with status 16037
2021-09-20T09:53:29.980556+00:00
Errors in file /opt/app/oracle/diag/rdbms/dgtest2/dgtest2/trace/dgtest2_pr00_11627.trc:
ORA-16037: user requested cancel of managed recovery operation
PR00 (PID:11627): Managed Standby Recovery not using Real Time Apply
Recovery interrupted!
Recovered data files to a consistent state at change 4236455
Stopping change tracking
2021-09-20T09:53:30.221440+00:00
Errors in file /opt/app/oracle/diag/rdbms/dgtest2/dgtest2/trace/dgtest2_pr00_11627.trc:
ORA-16037: user requested cancel of managed recovery operation
2021-09-20T09:53:30.319390+00:00
Background Media Recovery process shutdown (dgtest2)
2021-09-20T09:53:30.981370+00:00
RSM0 (PID:11547): Terminal Recovery: Stopped real time apply
2021-09-20T09:53:30.986937+00:00
Attempt to do a Terminal Recovery (dgtest2)
TMI: adbdrv termRecovery END 2021-09-20 09:53:30.986957
2021-09-20T09:53:30.987291+00:00
Media Recovery Start: Managed Standby Recovery (dgtest2)
2021-09-20T09:53:30.992470+00:00
Serial Media Recovery started
RSM0 (PID:11547): Managed Standby Recovery not using Real Time Apply
max_pdb is 3
Stopping change tracking
RSM0 (PID:11547): Begin: SRL archival
RSM0 (PID:11547): End: SRL archival
RSM0 (PID:11547): Terminal Recovery timestamp is '09/20/2021 09:53:31'
RSM0 (PID:11547): Terminal Recovery: applying standby redo logs.
RSM0 (PID:11547): Terminal Recovery: thread 1 seq# 382 redo required
2021-09-20T09:53:31.259258+00:00
RSM0 (PID:11547): Terminal Recovery:
2021-09-20T09:53:31.264259+00:00
Recovery of Online Redo Log: Thread 1 Group 5 Seq 382 Reading mem 0
  Mem# 0: +FRA/DGTEST2/ONLINELOG/group_5.423.1067949477
Terminal Recovery finished with No-Data-Loss
2021-09-20T09:53:31.297911+00:00
Incomplete Recovery applied until change 4236456 time 09/20/2021 09:52:41
2021-09-20T09:53:31.312164+00:00
Media Recovery Complete (dgtest2)
Terminal Recovery: successful completion
RSM0 (PID:11547): Forcing ARSCN to IRSCN for TR SCN:0x000000000040a4a8
RSM0 (PID:11547): Attempt to set limbo arscn SCN:0x000000000040a4a8 irscn SCN:0x000000000040a4a8
RSM0 (PID:11547): Resetting standby activation ID 4040268239 (0xf0d199cf)
Stopping change tracking
2021-09-20T09:53:31.464571+00:00
ALTER DATABASE SWITCHOVER TO PRIMARY (dgtest2)
Maximum wait for role transition is 15 minutes.
TMI: kcv_commit_to_so_to_primary wait for MRP to finish BEGIN 2021-09-20 09:53:31.466893
TMI: kcv_commit_to_so_to_primary wait for MRP to finish END 2021-09-20 09:53:31.467051
TMI: kcv_commit_to_so_to_primary Switchover from physical BEGIN 2021-09-20 09:53:31.468362
Backup controlfile written to trace file /opt/app/oracle/diag/rdbms/dgtest2/dgtest2/trace/dgtest2_rsm0_11547.trc
Standby terminal recovery start SCN: 4236455
RESETLOGS after incomplete recovery UNTIL CHANGE 4236456 time 09/20/2021 09:52:41
RSM0 (PID:11547): ORL pre-clearing operation disabled by switchover
Online log +DATA/DGTEST2/ONLINELOG/group_1.289.1067949469: Thread 1 Group 1 was previously cleared
Online log +FRA/DGTEST2/ONLINELOG/group_1.419.1067949471: Thread 1 Group 1 was previously cleared
Online log +DATA/DGTEST2/ONLINELOG/group_2.290.1067949471: Thread 1 Group 2 was previously cleared
Online log +FRA/DGTEST2/ONLINELOG/group_2.420.1067949473: Thread 1 Group 2 was previously cleared
Online log +DATA/DGTEST2/ONLINELOG/group_3.291.1067949473: Thread 1 Group 3 was previously cleared
Online log +FRA/DGTEST2/ONLINELOG/group_3.421.1067949473: Thread 1 Group 3 was previously cleared
Standby became primary SCN: 4236454
2021-09-20T09:53:31.635945+00:00
Setting recovery target incarnation to 2
2021-09-20T09:53:31.673002+00:00
RSM0 (PID:11547): RT: Role transition work is not done
RSM0 (PID:11547): The Time Management Interface (TMI) is being enabled for role transition
RSM0 (PID:11547): information.  This will result in messages beingoutput to the alert log
RSM0 (PID:11547): file with the prefix 'TMI: '.  This is being enabled to make the timing of
RSM0 (PID:11547): the various stages of the role transition available for diagnostic purposes.
RSM0 (PID:11547): This output will end when the role transition is complete.
RSM0 (PID:11547): Redo network throttle feature is disabled at mount time
2021-09-20T09:53:31.710578+00:00
RSM0 (PID:11547): Database role cleared from PHYSICAL STANDBY [kcvs.c:1099]
Switchover: Complete - Database mounted as primary
TMI: kcv_commit_to_so_to_primary Switchover from physical END 2021-09-20 09:53:31.712243
TMI: dbsdrv failover to target END 2021-09-20 09:53:31.712367
Failover completed with No-Data-Loss.
Completed: ALTER DATABASE FAILOVER TO dgtest2
2021-09-20T09:53:31.975876+00:00
RSM0 (PID:11547): Using STANDBY_ARCHIVE_DEST parameter default value as USE_DB_RECOVERY_FILE_DEST [krsd.c:18222]
2021-09-20T09:53:31.976280+00:00
ARC9 (PID:11172): Becoming the 'no SRL' ARCH
2021-09-20T09:53:31.983848+00:00
ALTER SYSTEM SET log_archive_dest_2='service="dgtesttns"','SYNC AFFIRM delay=0 optional compression=disable max_fai                                                                                                                          lure=0 reopen=300 db_unique_name="dgtest" net_timeout=30','valid_for=(online_logfile,all_roles)' SCOPE=BOTH;
2021-09-20T09:53:32.015296+00:00
ALTER SYSTEM SET log_archive_dest_state_2='ENABLE' SCOPE=BOTH;
ALTER DATABASE SET STANDBY DATABASE TO MAXIMIZE AVAILABILITY
Completed: ALTER DATABASE SET STANDBY DATABASE TO MAXIMIZE AVAILABILITY
ALTER DATABASE OPEN
2021-09-20T09:53:32.030768+00:00
TMI: adbdrv open database BEGIN 2021-09-20 09:53:32.030695
Data Guard Broker initializing...
Ping without log force is disabled:
  instance mounted in exclusive mode.
Buffer Cache Full DB Caching mode changing from FULL CACHING DISABLED to FULL CACHING ENABLED
2021-09-20T09:53:32.116297+00:00
Crash Recovery excluding pdb 2 which was cleanly closed.
2021-09-20T09:53:32.116442+00:00
Crash Recovery excluding pdb 3 which was cleanly closed.
Endian type of dictionary set to little
2021-09-20T09:53:32.136779+00:00
Assigning activation ID 4040238735 (0xf0d1268f)
LGWR (PID:10754): Primary database is in MAXIMUM AVAILABILITY mode
2021-09-20T09:53:32.140956+00:00
LGWR (PID:10754): LAD:2 is UNSYNCHRONIZED
LGWR (PID:10754): LAD:1 is not serviced by LGWR
2021-09-20T09:53:33.177813+00:00
Thread 1 advanced to log sequence 2 (thread open)
Redo log for group 2, sequence 2 is not located on DAX storage
Thread 1 opened at log sequence 2
  Current log# 2 seq# 2 mem# 0: +DATA/DGTEST2/ONLINELOG/group_2.290.1067949471
  Current log# 2 seq# 2 mem# 1: +FRA/DGTEST2/ONLINELOG/group_2.420.1067949473
Successful open of redo thread 1
2021-09-20T09:53:33.246706+00:00
MTTR advisory is disabled because FAST_START_MTTR_TARGET is not set
Stopping change tracking
2021-09-20T09:53:33.287479+00:00
ARC1 (PID:11156): Archived Log entry 520 added for T-1.S-1 ID 0xf0d1268f LAD:1
2021-09-20T09:53:33.290677+00:00
TT03 (PID:12462): Sleep 5 seconds and then try to clear SRLs in 2 time(s)
2021-09-20T09:53:33.549541+00:00
Undo initialization recovery: Parallel FPTR failed: start:740398 end:740405 diff:7 ms (0.0 seconds)
Undo initialization recovery: err:0 start: 740397 end: 740435 diff: 38 ms (0.0 seconds)
[11547] Successfully onlined Undo Tablespace 2.
Undo initialization online undo segments: err:0 start: 740435 end: 740959 diff: 524 ms (0.5 seconds)
Undo initialization finished serial:0 start:740397 end:740967 diff:570 ms (0.6 seconds)
Dictionary check beginning
Dictionary check complete
2021-09-20T09:53:34.245425+00:00
Database Characterset is AL32UTF8
No Resource Manager plan active
2021-09-20T09:53:35.468394+00:00
joxcsys_required_dirobj_exists: directory object exists with required path /opt/app/oracle/product/19.x.0/dbhome_2/                                                                                                                          javavm/admin/, pid 11547 cid 1
replication_dependency_tracking turned off (no async multimaster replication found)
LOGSTDBY: Validating controlfile with logical metadata
LOGSTDBY: Validation complete
Starting background process AQPC
2021-09-20T09:53:36.133706+00:00
AQPC started with pid=70, OS id=12472
PDB$SEED(2):Pluggable database PDB$SEED opening in read only
2021-09-20T09:53:36.942191+00:00
PDB$SEED(2):Autotune of undo retention is turned on.
2021-09-20T09:53:37.055480+00:00
TT03 (PID:12462): Enabling archival of deferred physical standby SRLs
2021-09-20T09:53:37.152868+00:00
TT03 (PID:12462): Archived Log entry 521 added for T-1.S-382 ID 0xf0d199cf LAD:1
2021-09-20T09:53:37.246644+00:00
PDB$SEED(2):Endian type of dictionary set to little
PDB$SEED(2):Undo initialization finished serial:0 start:744276 end:744276 diff:0 ms (0.0 seconds)
PDB$SEED(2):Pluggable database PDB$SEED dictionary check beginning
2021-09-20T09:53:38.232036+00:00
Thread 1 advanced to log sequence 3 (LGWR switch),  current SCN: 4236634
  Current log# 3 seq# 3 mem# 0: +DATA/DGTEST2/ONLINELOG/group_3.291.1067949473
  Current log# 3 seq# 3 mem# 1: +FRA/DGTEST2/ONLINELOG/group_3.421.1067949473
2021-09-20T09:53:38.316860+00:00
PDB$SEED(2):Pluggable Database PDB$SEED Dictionary check complete
2021-09-20T09:53:38.327245+00:00
ARC5 (PID:11164): Archived Log entry 522 added for T-1.S-2 ID 0xf0d1268f LAD:1
2021-09-20T09:53:38.328930+00:00
PDB$SEED(2):Database Characterset for PDB$SEED is AL32UTF8
2021-09-20T09:53:38.503847+00:00
QPI: opatch file present, opatch
QPI: qopiprep.bat file present
2021-09-20T09:53:38.642191+00:00
ALTER SYSTEM SET log_archive_dest_state_2='ENABLE' SCOPE=MEMORY SID='*';
2021-09-20T09:53:38.835626+00:00
PDB$SEED(2):SUPLOG: Set PDB SUPLOG SGA at PDB OPEN, old 0x0, new 0x0 (no suplog)
PDB$SEED(2):Opening pdb with no Resource Manager plan active
2021-09-20T09:53:39.622239+00:00
Starting background process CJQ0
Completed: ALTER DATABASE OPEN




After the failover completes the data guard broker status shows old primary is disabled and need reinstating.
DGMGRL> show configuration

Configuration - test_dg

  Protection Mode: MaxAvailability
  Members:
  dgtest2 - Primary database
    dgtest  - Physical standby database
    dgtest3 - Physical standby database (disabled)
      ORA-16661: the standby database needs to be reinstated

Fast-Start Failover:  Disabled

Configuration Status:
SUCCESS   (status updated 40 seconds ago)
If the old primary issue is rectified it would be reinstated as a standby (instead of adding as a new standby). When the old primary is started the DG broker will detect failover has happened and will prevent the opening the opening of the DB in read/write mode.
Completed: ALTER DATABASE MOUNT /* db agent *//* {0:0:312} */
ALTER DATABASE OPEN /* db agent *//* {0:0:312} */
Data Guard Broker initializing...
2021-09-20T09:58:19.408297+00:00
Starting Data Guard Broker (DMON)
Starting background process INSV
2021-09-20T09:58:19.425650+00:00
INSV started with pid=49, OS id=5982
2021-09-20T09:58:23.448529+00:00
Data Guard Broker initialization complete
Data Guard: verifying database primary role...
2021-09-20T09:58:23.467893+00:00
Starting background process NSV1
2021-09-20T09:58:23.484569+00:00
NSV1 started with pid=50, OS id=5987
2021-09-20T09:58:27.484306+00:00
Starting background process NSV2
2021-09-20T09:58:27.501947+00:00
NSV2 started with pid=51, OS id=5992
2021-09-20T09:58:30.605237+00:00
Starting background process RSM0
2021-09-20T09:58:30.620725+00:00
RSM0 started with pid=52, OS id=5997
2021-09-20T09:58:31.448386+00:00
Data Guard: broker startup completed
Data Guard determines a failover has occurred - instance will not be opened
ORA-16649 signalled during: ALTER DATABASE OPEN /* db agent *//* {0:0:312} */...
The database will be left at mount mode.
SQL> select db_unique_name,open_mode from v$database;

DB_UNIQUE_NAME                 OPEN_MODE
------------------------------ --------------------
dgtest3                        MOUNTED
From the current primary DG broker run the reinstate command.
DGMGRL> reinstate database dgtest3
Reinstating database "dgtest3", please wait...
Reinstatement of database "dgtest3" succeeded
During the reinstate process the old primary will be flashback to the time where failover occured and converted to a physical standby.
2021-09-20T09:59:51.612704+00:00
FLASHBACK DATABASE TO SCN 4236454
2021-09-20T09:59:51.774069+00:00
Flashback Restore Start
Flashback Restore Complete
Flashback Media Recovery Start
 Started logmerger process
2021-09-20T09:59:52.058152+00:00
max_pdb is 3
2021-09-20T09:59:52.297907+00:00
Parallel Media Recovery started with 8 slaves
Flashback Media Recovery Log +FRA/DGTEST3/ARCHIVELOG/2021_09_20/thread_1_seq_373.332.1083750351
Flashback Media Recovery Log +FRA/DGTEST3/ARCHIVELOG/2021_09_20/thread_1_seq_374.264.1083750365
2021-09-20T09:59:52.812785+00:00
Flashback Media Recovery Log +FRA/DGTEST3/ARCHIVELOG/2021_09_20/thread_1_seq_375.265.1083750375
Flashback Media Recovery Log +FRA/DGTEST3/ARCHIVELOG/2021_09_20/thread_1_seq_376.267.1083750649
2021-09-20T09:59:53.157016+00:00
Media Recovery End-Of-Redo indicator encountered
2021-09-20T09:59:53.157085+00:00
Media Recovery Continuing
2021-09-20T09:59:53.202162+00:00
Flashback Media Recovery Log +FRA/DGTEST3/ARCHIVELOG/2021_09_20/thread_1_seq_377.271.1083750659
Flashback Media Recovery Log +FRA/DGTEST3/ARCHIVELOG/2021_09_20/thread_1_seq_378.269.1083750671
Flashback Media Recovery Log +FRA/DGTEST3/ARCHIVELOG/2021_09_20/thread_1_seq_379.268.1083750683
2021-09-20T09:59:53.834774+00:00
Recovery of Online Redo Log: Thread 1 Group 1 Seq 380 Reading mem 0
  Mem# 0: +DATA/DGTEST3/ONLINELOG/group_1.299.1069533175
  Mem# 1: +FRA/DGTEST3/ONLINELOG/group_1.460.1069533177
2021-09-20T09:59:53.955443+00:00
Recovery of Online Redo Log: Thread 1 Group 2 Seq 381 Reading mem 0
  Mem# 0: +DATA/DGTEST3/ONLINELOG/group_2.300.1069533177
  Mem# 1: +FRA/DGTEST3/ONLINELOG/group_2.459.1069533177
2021-09-20T09:59:54.017466+00:00
Recovery of Online Redo Log: Thread 1 Group 3 Seq 382 Reading mem 0
  Mem# 0: +DATA/DGTEST3/ONLINELOG/group_3.301.1069533179
  Mem# 1: +FRA/DGTEST3/ONLINELOG/group_3.458.1069533179
2021-09-20T09:59:54.086085+00:00
Incomplete Recovery applied until change 4236455 time 09/20/2021 09:52:41
2021-09-20T09:59:54.094097+00:00
Flashback Media Recovery Complete
Completed: FLASHBACK DATABASE TO SCN 4236454
alter database convert to physical standby
2021-09-20T09:59:54.263414+00:00
ALTER DATABASE CONVERT TO PHYSICAL STANDBY (dgtest3)
Clearing standby activation ID 4040262352 (0xf0d182d0)
The primary database controlfile was created using the
'MAXLOGFILES 16' clause.
There is space for up to 13 standby redo logfiles
Use the following SQL commands on the standby database to create
standby redo logfiles that match the primary database:
ALTER DATABASE ADD STANDBY LOGFILE 'srl1.f' SIZE 209715200;
ALTER DATABASE ADD STANDBY LOGFILE 'srl2.f' SIZE 209715200;
ALTER DATABASE ADD STANDBY LOGFILE 'srl3.f' SIZE 209715200;
ALTER DATABASE ADD STANDBY LOGFILE 'srl4.f' SIZE 209715200;
Offline data file 2 marked as online during convert to standby or switchover to standby.
Restore of backup may be required if the file is not physically accessible.
Offline data file 4 marked as online during convert to standby or switchover to standby.
Restore of backup may be required if the file is not physically accessible.
Offline data file 6 marked as online during convert to standby or switchover to standby.
Restore of backup may be required if the file is not physically accessible.
Offline data file 8 marked as online during convert to standby or switchover to standby.
Restore of backup may be required if the file is not physically accessible.
Offline data file 9 marked as online during convert to standby or switchover to standby.
Restore of backup may be required if the file is not physically accessible.
Offline data file 10 marked as online during convert to standby or switchover to standby.
Restore of backup may be required if the file is not physically accessible.
RSM0 (PID:5997): Database role changed from PRIMARY to PHYSICAL STANDBY [kcvs.c:8936]
2021-09-20T09:59:54.325231+00:00
RSM0 (PID:5997): Waiting for all non-current ORLs to be archived
2021-09-20T09:59:54.325324+00:00
RSM0 (PID:5997): All non-current ORLs have been archived
RSM0 (PID:5997): Clearing online redo logfile 1 +DATA/DGTEST3/ONLINELOG/group_1.299.1069533175
RSM0 (PID:5997): Clearing online redo logfile 2 +DATA/DGTEST3/ONLINELOG/group_2.300.1069533177
RSM0 (PID:5997): Clearing online redo logfile 3 +DATA/DGTEST3/ONLINELOG/group_3.301.1069533179
Clearing online log 1 of thread 1 sequence number 380
Clearing online log 2 of thread 1 sequence number 381
Clearing online log 3 of thread 1 sequence number 382
2021-09-20T10:00:03.959609+00:00
RSM0 (PID:5997): Clearing online redo logfile 1 complete
RSM0 (PID:5997): Clearing online redo logfile 2 complete
RSM0 (PID:5997): Clearing online redo logfile 3 complete
RSM0 (PID:5997): RT: Role transition work is not done
RSM0 (PID:5997): Redo network throttle feature is disabled at mount time
Physical Standby Database mounted.
2021-09-20T10:00:03.970236+00:00
In-memory operation on ADG is currently only supported on Engineered systems and PaaS.
inmemory_adg_enabled is turned off automatically.
Please contact our support team for EXADATA solutions
CONVERT TO PHYSICAL STANDBY: Complete - Database mounted as physical standby

At the same time the DG broker will add log archive destination on the primary for the reinstated physical standby.
ALTER SYSTEM SET log_archive_dest_3='service="dgtest3tns"','SYNC AFFIRM delay=0 optional compression=disable max_failure=0 reopen=300 db_unique_name="dgtest3" net_timeout=30','valid_for=(online_logfile,all_roles)' SCOPE=BOTH;
The other standby will get an update on its fal_server with reference to the reinstated standby.
ALTER SYSTEM SET fal_server='dgtest2tns','dgtest3tns' SCOPE=BOTH;
At the end the data guard configuration will have same number of standbys as before but with a different primary.
DGMGRL> show configuration

Configuration - test_dg

  Protection Mode: MaxAvailability
  Members:
  dgtest2 - Primary database
    dgtest  - Physical standby database
    dgtest3 - Physical standby database

Fast-Start Failover:  Disabled

Configuration Status:
SUCCESS   (status updated 58 seconds ago)