Active Data Guard instances are not supported with DBMS_ROLLING

There is an earlier post on how to use dbms_rolling for database upgrade. The configuration used in that post consists of two instances. The data guard configuration this post is based on had three instances. One of them was is an active data guard instnace, open read only. This instance is to simulate a reader farm (Example 14-9 Rolling Upgrade on a Reader Farm). Below is the database names, their role and current open mode.

DB_UNIQUE_NAME   OPEN_MODE              DATABASE_ROLE
---------------- ----------             -------------
dgtest           READ WRITE             PRIMARY
dgtest2          MOUNTED                PHYSICAL STANDBY
dgtest3          READ ONLY WITH APPLY   PHYSICAL STANDBY

dgtest2 is designated as the future primary. This standby DB, as shown above is already in mount mode. MOS doc 2832235.1 states standby must be in mount mode.

SQL> exec DBMS_ROLLING.INIT_PLAN(future_primary=>'dgtest2');

PL/SQL procedure successfully completed.

However, when the build plan is called it gives the below error.

SQL> exec DBMS_ROLLING.BUILD_PLAN;
BEGIN DBMS_ROLLING.BUILD_PLAN; END;

*
ERROR at line 1:
ORA-45438: database is not in mounted mode
ORA-06512: at "SYS.DBMS_ROLLING", line 16
ORA-06512: at line 1

This is strange as the future primary, the standby that will be invovled in the switchover is already in mount mode but build plan is giving an error.
To see if the error is due to the active data guard instance (dgtest3), that instance was closed and started in mount mode. The build plan executes without error. So it seems all instances must be in mount mode for build plan to work. This means any application that rely on reader farm will incur down time.



After raising a SR Oracle confirmed that this is a bug and gave the following "Bug 25882840 : ACTIVE DATA GUARD NOT PERMITTED AS DBMS_ROLLING MEMBER". Unfortunately this is an internal bug so couldn't get much details on it. There was also no ETA on thie fix for the bug nor confirmation that ADG will be supported in the future.

Related MOS Docs
Oracle Exadata Database Service 19c Rolling Upgrade with DBMS_ROLLING from Oracle Database 12.1.0.2 [ID 2889464.1]
Exadata Cloud Database 19c Rolling Upgrade With DBMS_ROLLING [ID 2832235.1]

Related Posts
Upgrading Oracle Restart Databases From 12.2 to 18c Using DBMS_ROLLING
DG Broker Continue With Status "ROLLING DATABASE MAINTENANCE IN PROGRESS" Even After ROLLBACK_PLAN is Called

Active Data Guard and PDB Parameters

In an earlier post it was shown how undo_retention value in a PDB that is part of an ADG standby could only be changed by updating the value on the primary PDB. In the case of undo_retention parameter Oracle introduced change 30577591 which prevented it from inheriting the value from the standby CDB.
This post looks at the effects of setting parameter values at PDB level in a ADG enviornment. It seems that the PDB in a standby CDB will inherit the init values from cdb$root spfile as long as the parameter is not set at PDB level in the primary.
open_cursors parameter is used as the test case here. The initial values at primary and standby are same and no value has been set at PDB level.

SQL> show parameter open_cursor

NAME                                 TYPE        VALUE
------------------------------------ ----------- ----------
open_cursors                         integer     3000

On the primary the value is changed at CDB level.

SQL> alter system set open_cursors=3500 scope=both;

System altered.

SQL> show parameter open_cursor

NAME                                 TYPE        VALUE
------------------------------------ ----------- ----------
open_cursors                         integer     3500

As no value has been set at PDB level it will inherit the value from CDB$root.

SQL> alter session set container=dgpdb;

Session altered.

SQL> show parameter cursor

NAME                                 TYPE        VALUE
------------------------------------ ----------- -----------
open_cursors                         integer     3500

Next the value is changed at PDB level.

SQL> alter session set container=dgpdb;

SQL>  alter system set open_cursors=2500;

System altered.

SQL> show parameter cursor

NAME                                 TYPE        VALUE
------------------------------------ ----------- -----------
open_cursors                         integer     2500

At this stage primary cdb$root has value 3500 and PDB has 2500. No changes has been made to standby and value there remains 3000.



On the standby checking the value in the PDB shows the initial value.

SQL>  alter session set container=dgpdb;

Session altered.

SQL> show parameter cursor

NAME                                 TYPE        VALUE
------------------------------------ ----------- -------------
open_cursors                         integer     3000

Changing the parameter value at CDB level and check value in PDB.

SQL>  alter system set open_cursors=4500 scope=both;

System altered.

SQL>  show parameter open_cursor

NAME                                 TYPE        VALUE
------------------------------------ ----------- ---------
open_cursors                         integer     4500

Value checked at PDB level.

SQL> alter session set container=dgpdb;

Session altered.

SQL> show parameter open_cursor

NAME                                 TYPE        VALUE
------------------------------------ ----------- -------
open_cursors                         integer     4500

This shows value being inherited from standby cdb$root.
Next the PDB in the standby instance is restarted and parameter value checked.

SQL> shutdown ;
Pluggable Database closed.

SQL> startup;
Pluggable Database opened.

SQL> show parameter open_cursor

NAME                                 TYPE        VALUE
------------------------------------ ----------- -----------
open_cursors                         integer     2500

It not longer shows the 4500 value inherited from standby but the value set at primary PDB. It appears during PDB start the table PDB_SPFILE$ is read for PDB level parameter values. Since PDB_SPFILE$ is a database table, any writes to it will generate redo. As such changes made in primary PDB will cascade to all standby instances and to all relevant PDBs. When values are read from this table they are set for PDB overwriting values inherited from cdb$root spfile.
Any subsequent change to the init parameter at cdb$root doesn't override the value set at PDB level.

SQL> show con_name

CON_NAME
---------
CDB$ROOT

SQL> alter system set open_cursors=4600 scope=both;

System altered.

SQL> alter session set container=dgpdb;

Session altered.

SQL>  show parameter open_cursor

NAME                                 TYPE        VALUE
------------------------------------ ----------- ------
open_cursors                         integer     2500

This behaviour, inheriting from local cdb$root spfile vs primary PDB could have consequences for asymmetric data guard deployments where primary and standy differ in the amount of resources (memory,cpu) and user requirments (BI vs OLTP, user counts, etc). If for whatever an init value is set on PDB level at primary, remember that will get applied on all standby PDBs next time those are restarted.

Failover and Reinstate With Multiple Physical Standbys

The data guard configuration consists of three databases.

 DGMGRL> show configuration

Configuration - test_dg

  Protection Mode: MaxAvailability
  Members:
  dgtest3 - Primary database
    dgtest  - Physical standby database
    dgtest2 - Physical standby database

Fast-Start Failover:  Disabled

Configuration Status:
SUCCESS   (status updated 67 seconds ago)

At the moment dgtest3 is the primary. One of the standby (dgtest2) databases is open in mount mode.

SQL> select db_unique_name,open_mode from v$database;

DB_UNIQUE_NAME                 OPEN_MODE
------------------------------ --------------------
dgtest2                        MOUNTED

The other standby (dgtest) is an active data guard and is open in read only mode with log apply.

SQL> select db_unique_name,open_mode from v$database;

DB_UNIQUE_NAME                 OPEN_MODE
------------------------------ --------------------
dgtest                         READ ONLY WITH APPLY

To simulate primary DB failure it is shutdown with abort.

SQL> shutdown abort;
ORACLE instance shut down.

Checking the data guard configuration from dgtest2 shows an error state.

DGMGRL> show configuration

Configuration - test_dg

  Protection Mode: MaxAvailability
  Members:
  dgtest3 - Primary database
    Error: ORA-12514: TNS:listener does not currently know of service requested in connect descriptor

    dgtest  - Physical standby database
    dgtest2 - Physical standby database

Fast-Start Failover:  Disabled

Configuration Status:
ERROR   (status updated 0 seconds ago)

One of the existing standbys could be choosen for failover. In this case dgtest2( instance in mount mode) was chosen. Validating the dgtest2 show that it is ready for failover.

DGMGRL> validate database dgtest2

  Database Role:     Physical standby database
  Primary Database:  dgtest3
    Warning: primary database was not reachable

  Ready for Switchover:  No
  Ready for Failover:    Yes (Primary Not Running)

  Flashback Database Status:
    dgtest3:  Unknown
    dgtest2:  On

  Managed by Clusterware:
    dgtest3:  Unknown
    dgtest2:  YES
    Validating static connect identifier for the primary database dgtest3...
Unable to connect to database using (DESCRIPTION=(ADDRESS=(PROTOCOL=TCP)(HOST=ip-132)(PORT=1521))(CONNECT                                                                                                                          _DATA=(SERVICE_NAME=dgtest3_DGMGRL)(INSTANCE_NAME=dgtest3)(SERVER=DEDICATED)(STATIC_SERVICE=TRUE)))
ORA-12514: TNS:listener does not currently know of service requested in connect descriptor

Failed.
    Warning: Ensure primary database's StaticConnectIdentifier property
    is configured properly so that the primary database can be restarted
    by DGMGRL after switchover

  Temporary Tablespace File Information:
    dgtest3 TEMP Files:  Unknown
    dgtest2 TEMP Files:  3

  Data file Online Move in Progress:
    dgtest3:  Unknown
    dgtest2:  No

  Transport-Related Information:
    Transport On:  No
    Gap Status:    Unknown
    Transport Lag:  0 seconds (computed 24 seconds ago)
    Transport Status:  Success

  Log Files Cleared:
    dgtest3 Standby Redo Log Files:  Unknown
    dgtest2 Online Redo Log Files:   Unknown
    dgtest2 Standby Redo Log Files:  Unknown

Initiate failover to dgtest2

DGMGRL> failover to dgtest2;
Performing failover NOW, please wait...
Failover succeeded, new primary is "dgtest2"

During the failover process the active data guard instance continue to function. It will cancel the current redo apply service and start a new one with the incarnation that results from the failover. At the same time any references to the old primary will be removed (such as fal_server).

2021-09-20T09:53:29.718199+00:00
ALTER DATABASE RECOVER MANAGED STANDBY DATABASE CANCEL
2021-09-20T09:53:29.720039+00:00
PR00 (PID:5312): MRP0: Background Media Recovery cancelled with status 16037
2021-09-20T09:53:29.720747+00:00
Errors in file /opt/app/oracle/diag/rdbms/dgtest/dgtest/trace/dgtest_pr00_5312.trc:
ORA-16037: user requested cancel of managed recovery operation
PR00 (PID:5312): Managed Standby Recovery not using Real Time Apply
Recovery interrupted!
Recovered data files to a consistent state at change 4236455
Stopping change tracking
2021-09-20T09:53:29.943216+00:00
Errors in file /opt/app/oracle/diag/rdbms/dgtest/dgtest/trace/dgtest_pr00_5312.trc:
ORA-16037: user requested cancel of managed recovery operation
2021-09-20T09:53:30.067266+00:00
Background Media Recovery process shutdown (dgtest)
2021-09-20T09:53:30.720982+00:00
Managed Standby Recovery Canceled (dgtest)
Completed: ALTER DATABASE RECOVER MANAGED STANDBY DATABASE CANCEL
2021-09-20T09:53:32.641452+00:00
ALTER SYSTEM SET fal_server='dgtest2tns' SCOPE=BOTH;
ALTER DATABASE RECOVER MANAGED STANDBY DATABASE DISCONNECT  NODELAY
2021-09-20T09:53:32.658894+00:00
Attempt to start background Managed Standby Recovery process (dgtest)
Starting background process MRP0
2021-09-20T09:53:32.673122+00:00
MRP0 started with pid=63, OS id=5816
2021-09-20T09:53:32.674424+00:00
Background Managed Standby Recovery process started (dgtest)
2021-09-20T09:53:36.270863+00:00
 rfs (PID:5820): krsr_rfs_atc: Identified database type as 'PHYSICAL STANDBY': Client is Foreground (PID:11152)
Deleted Oracle managed file +FRA/DGTEST/ARCHIVELOG/2021_09_20/thread_0_seq_0.371.1083750817
2021-09-20T09:53:37.697852+00:00
 Started logmerger process
2021-09-20T09:53:37.712394+00:00

IM on ADG: Start of Empty Journal

IM on ADG: End of Empty Journal
PR00 (PID:5826): Managed Standby Recovery starting Real Time Apply
max_pdb is 3
2021-09-20T09:53:37.888440+00:00
 rfs (PID:5824): krsr_rfs_atc: Identified database type as 'PHYSICAL STANDBY': Client is SYNC (PID:10754)
 rfs (PID:5824): New archival redo branch: 1083750811 current: 1067878072
 rfs (PID:5824): Primary database is in MAXIMUM AVAILABILITY mode
 rfs (PID:5824): Changing standby controlfile to RESYNCHRONIZATION level
 rfs (PID:5824): Standby controlfile consistent with primary
2021-09-20T09:53:37.951928+00:00
 rfs (PID:5824): Selected LNO:6 for T-1.S-3 dbid 4024401720 branch 1083750811
2021-09-20T09:53:37.974747+00:00
Parallel Media Recovery started with 8 slaves
2021-09-20T09:53:38.018389+00:00
Stopping change tracking
PR00 (PID:5826): Media Recovery Waiting for T-1.S-382
2021-09-20T09:53:38.440514+00:00
 rfs (PID:5855): krsr_rfs_atc: Identified database type as 'PHYSICAL STANDBY': Client is FAL (PID:11166)
2021-09-20T09:53:38.505442+00:00
 rfs (PID:5855): Selected LNO:7 for T-1.S-2 dbid 4024401720 branch 1083750811
 rfs (PID:5855): A new recovery destination branch has been registered
 rfs (PID:5855): New Archival REDO Branch(resetlogs_id): 1083750811  Prior: 1067878072
 rfs (PID:5855): Archival Activation ID: 0xf0d1268f Current: 0xf0d199cf
 rfs (PID:5855): Effect of primary database OPEN RESETLOGS
 rfs (PID:5855): Managed Standby Recovery process is active
2021-09-20T09:53:38.536066+00:00
Incarnation entry added for Branch(resetlogs_id): 1083750811 (dgtest)
2021-09-20T09:53:38.548638+00:00
Setting recovery target incarnation to 2
2021-09-20T09:53:38.678029+00:00
Completed: ALTER DATABASE RECOVER MANAGED STANDBY DATABASE DISCONNECT  NODELAY
2021-09-20T09:53:39.126620+00:00
PR00 (PID:5826): MRP0: Incarnation has changed! Retry recovery...
2021-09-20T09:53:39.127329+00:00
Errors in file /opt/app/oracle/diag/rdbms/dgtest/dgtest/trace/dgtest_pr00_5826.trc:
ORA-19906: recovery target incarnation changed during recovery
PR00 (PID:5826): Managed Standby Recovery not using Real Time Apply
Recovery interrupted!
Stopping change tracking
2021-09-20T09:53:39.233028+00:00
Errors in file /opt/app/oracle/diag/rdbms/dgtest/dgtest/trace/dgtest_pr00_5826.trc:
ORA-19906: recovery target incarnation changed during recovery
2021-09-20T09:53:39.432821+00:00
 rfs (PID:5858): krsr_rfs_atc: Identified database type as 'PHYSICAL STANDBY': Client is FAL (PID:11172)
2021-09-20T09:53:39.480429+00:00
 rfs (PID:5858): Opened log for T-1.S-382 dbid 4024401720 branch 1067878072
2021-09-20T09:53:39.509873+00:00
 rfs (PID:5858): Archived Log entry 746 added for B-1067878072.T-1.S-382 ID 0xf0d199cf LAD:2
2021-09-20T09:53:39.573601+00:00
 Started logmerger process
2021-09-20T09:53:39.614023+00:00

IM on ADG: Start of Empty Journal

IM on ADG: End of Empty Journal
PR00 (PID:5861): Managed Standby Recovery starting Real Time Apply
2021-09-20T09:53:39.622286+00:00
ARC8 (PID:5169): Archived Log entry 747 added for T-1.S-2 ID 0xf0d1268f LAD:1
2021-09-20T09:53:39.678880+00:00
max_pdb is 3
2021-09-20T09:53:39.878314+00:00
Parallel Media Recovery started with 8 slaves
2021-09-20T09:53:39.894206+00:00
Media Recovery start incarnation depth : 1, target inc# : 2, irscn : 4236456
Stopping change tracking
2021-09-20T09:53:39.917344+00:00
 rfs (PID:5863): krsr_rfs_atc: Identified database type as 'PHYSICAL STANDBY': Client is FAL (PID:11158)
2021-09-20T09:53:39.974576+00:00
 rfs (PID:5863): Opened log for T-1.S-1 dbid 4024401720 branch 1083750811
2021-09-20T09:53:40.001952+00:00
 rfs (PID:5863): Archived Log entry 748 added for B-1083750811.T-1.S-1 ID 0xf0d1268f LAD:2
2021-09-20T09:53:40.011908+00:00
PR00 (PID:5861): Media Recovery Log +FRA/DGTEST/ARCHIVELOG/2021_09_20/thread_1_seq_382.371.1083750819
2021-09-20T09:53:40.134398+00:00
PR00 (PID:5861): Resetting standby activation ID 4040268239 (0xf0d199cf)
2021-09-20T09:53:40.138397+00:00
Media Recovery End-Of-Redo indicator encountered
2021-09-20T09:53:40.138471+00:00
Media Recovery Continuing
2021-09-20T09:53:40.401844+00:00
PR00 (PID:5861): Media Recovery Log +FRA/DGTEST/ARCHIVELOG/2021_09_20/thread_1_seq_1.454.1083750819
2021-09-20T09:53:40.499037+00:00
PR00 (PID:5861): Media Recovery Log +FRA/DGTEST/ARCHIVELOG/2021_09_20/thread_1_seq_2.372.1083750819
2021-09-20T09:53:40.692602+00:00
PR00 (PID:5861): Media Recovery Waiting for T-1.S-3 (in transit)
2021-09-20T09:53:40.698462+00:00
Recovery of Online Redo Log: Thread 1 Group 6 Seq 3 Reading mem 0
  Mem# 0: +FRA/DGTEST/ONLINELOG/group_6.423.1067949163
2021-09-20T09:53:41.024853+00:00
 rfs (PID:5824): Changing standby controlfile to MAXIMUM AVAILABILITY level
2021-09-20T09:53:41.041788+00:00
 rfs (PID:5824): Selected LNO:7 for T-1.S-4 dbid 4024401720 branch 1083750811
2021-09-20T09:53:41.045351+00:00
ARC9 (PID:5171): Archived Log entry 749 added for T-1.S-3 ID 0xf0d1268f LAD:1
2021-09-20T09:53:41.118868+00:00
PR00 (PID:5861): Media Recovery Waiting for T-1.S-4 (in transit)
2021-09-20T09:53:41.124519+00:00
Recovery of Online Redo Log: Thread 1 Group 7 Seq 4 Reading mem 0
  Mem# 0: +FRA/DGTEST/ONLINELOG/group_7.424.1067949165

The instance failed over to will stop it's current redo apply service and convert to a read/write primary. It is intresting to see some of the lines in the log output "switchover" instead of failover.

2021-09-20T09:53:24.419649+00:00
Beginning failover to database dgtest2.
Starting background process NSV1
2021-09-20T09:53:24.492112+00:00
NSV1 started with pid=16, OS id=12324
2021-09-20T09:53:29.976461+00:00
ALTER DATABASE FAILOVER TO dgtest2
2021-09-20T09:53:29.976612+00:00
RSM0 (PID:11547): The Time Management Interface (TMI) is being enabled for role transition
RSM0 (PID:11547): information.  This will result in messages beingoutput to the alert log
RSM0 (PID:11547): file with the prefix 'TMI: '.  This is being enabled to make the timing of
RSM0 (PID:11547): the various stages of the role transition available for diagnostic purposes.
RSM0 (PID:11547): This output will end when the role transition is complete.
TMI: dbsdrv failover to target BEGIN 2021-09-20 09:53:29.977021
Terminal Recovery requested in process 11547
TMI: adbdrv termRecovery BEGIN 2021-09-20 09:53:29.978960
RSM0 (PID:11547): Terminal Recovery: Stopping real time apply
2021-09-20T09:53:29.980343+00:00
PR00 (PID:11627): MRP0: Background Media Recovery cancelled with status 16037
2021-09-20T09:53:29.980556+00:00
Errors in file /opt/app/oracle/diag/rdbms/dgtest2/dgtest2/trace/dgtest2_pr00_11627.trc:
ORA-16037: user requested cancel of managed recovery operation
PR00 (PID:11627): Managed Standby Recovery not using Real Time Apply
Recovery interrupted!
Recovered data files to a consistent state at change 4236455
Stopping change tracking
2021-09-20T09:53:30.221440+00:00
Errors in file /opt/app/oracle/diag/rdbms/dgtest2/dgtest2/trace/dgtest2_pr00_11627.trc:
ORA-16037: user requested cancel of managed recovery operation
2021-09-20T09:53:30.319390+00:00
Background Media Recovery process shutdown (dgtest2)
2021-09-20T09:53:30.981370+00:00
RSM0 (PID:11547): Terminal Recovery: Stopped real time apply
2021-09-20T09:53:30.986937+00:00
Attempt to do a Terminal Recovery (dgtest2)
TMI: adbdrv termRecovery END 2021-09-20 09:53:30.986957
2021-09-20T09:53:30.987291+00:00
Media Recovery Start: Managed Standby Recovery (dgtest2)
2021-09-20T09:53:30.992470+00:00
Serial Media Recovery started
RSM0 (PID:11547): Managed Standby Recovery not using Real Time Apply
max_pdb is 3
Stopping change tracking
RSM0 (PID:11547): Begin: SRL archival
RSM0 (PID:11547): End: SRL archival
RSM0 (PID:11547): Terminal Recovery timestamp is '09/20/2021 09:53:31'
RSM0 (PID:11547): Terminal Recovery: applying standby redo logs.
RSM0 (PID:11547): Terminal Recovery: thread 1 seq# 382 redo required
2021-09-20T09:53:31.259258+00:00
RSM0 (PID:11547): Terminal Recovery:
2021-09-20T09:53:31.264259+00:00
Recovery of Online Redo Log: Thread 1 Group 5 Seq 382 Reading mem 0
  Mem# 0: +FRA/DGTEST2/ONLINELOG/group_5.423.1067949477
Terminal Recovery finished with No-Data-Loss
2021-09-20T09:53:31.297911+00:00
Incomplete Recovery applied until change 4236456 time 09/20/2021 09:52:41
2021-09-20T09:53:31.312164+00:00
Media Recovery Complete (dgtest2)
Terminal Recovery: successful completion
RSM0 (PID:11547): Forcing ARSCN to IRSCN for TR SCN:0x000000000040a4a8
RSM0 (PID:11547): Attempt to set limbo arscn SCN:0x000000000040a4a8 irscn SCN:0x000000000040a4a8
RSM0 (PID:11547): Resetting standby activation ID 4040268239 (0xf0d199cf)
Stopping change tracking
2021-09-20T09:53:31.464571+00:00
ALTER DATABASE SWITCHOVER TO PRIMARY (dgtest2)
Maximum wait for role transition is 15 minutes.
TMI: kcv_commit_to_so_to_primary wait for MRP to finish BEGIN 2021-09-20 09:53:31.466893
TMI: kcv_commit_to_so_to_primary wait for MRP to finish END 2021-09-20 09:53:31.467051
TMI: kcv_commit_to_so_to_primary Switchover from physical BEGIN 2021-09-20 09:53:31.468362
Backup controlfile written to trace file /opt/app/oracle/diag/rdbms/dgtest2/dgtest2/trace/dgtest2_rsm0_11547.trc
Standby terminal recovery start SCN: 4236455
RESETLOGS after incomplete recovery UNTIL CHANGE 4236456 time 09/20/2021 09:52:41
RSM0 (PID:11547): ORL pre-clearing operation disabled by switchover
Online log +DATA/DGTEST2/ONLINELOG/group_1.289.1067949469: Thread 1 Group 1 was previously cleared
Online log +FRA/DGTEST2/ONLINELOG/group_1.419.1067949471: Thread 1 Group 1 was previously cleared
Online log +DATA/DGTEST2/ONLINELOG/group_2.290.1067949471: Thread 1 Group 2 was previously cleared
Online log +FRA/DGTEST2/ONLINELOG/group_2.420.1067949473: Thread 1 Group 2 was previously cleared
Online log +DATA/DGTEST2/ONLINELOG/group_3.291.1067949473: Thread 1 Group 3 was previously cleared
Online log +FRA/DGTEST2/ONLINELOG/group_3.421.1067949473: Thread 1 Group 3 was previously cleared
Standby became primary SCN: 4236454
2021-09-20T09:53:31.635945+00:00
Setting recovery target incarnation to 2
2021-09-20T09:53:31.673002+00:00
RSM0 (PID:11547): RT: Role transition work is not done
RSM0 (PID:11547): The Time Management Interface (TMI) is being enabled for role transition
RSM0 (PID:11547): information.  This will result in messages beingoutput to the alert log
RSM0 (PID:11547): file with the prefix 'TMI: '.  This is being enabled to make the timing of
RSM0 (PID:11547): the various stages of the role transition available for diagnostic purposes.
RSM0 (PID:11547): This output will end when the role transition is complete.
RSM0 (PID:11547): Redo network throttle feature is disabled at mount time
2021-09-20T09:53:31.710578+00:00
RSM0 (PID:11547): Database role cleared from PHYSICAL STANDBY [kcvs.c:1099]
Switchover: Complete - Database mounted as primary
TMI: kcv_commit_to_so_to_primary Switchover from physical END 2021-09-20 09:53:31.712243
TMI: dbsdrv failover to target END 2021-09-20 09:53:31.712367
Failover completed with No-Data-Loss.
Completed: ALTER DATABASE FAILOVER TO dgtest2
2021-09-20T09:53:31.975876+00:00
RSM0 (PID:11547): Using STANDBY_ARCHIVE_DEST parameter default value as USE_DB_RECOVERY_FILE_DEST [krsd.c:18222]
2021-09-20T09:53:31.976280+00:00
ARC9 (PID:11172): Becoming the 'no SRL' ARCH
2021-09-20T09:53:31.983848+00:00
ALTER SYSTEM SET log_archive_dest_2='service="dgtesttns"','SYNC AFFIRM delay=0 optional compression=disable max_fai                                                                                                                          lure=0 reopen=300 db_unique_name="dgtest" net_timeout=30','valid_for=(online_logfile,all_roles)' SCOPE=BOTH;
2021-09-20T09:53:32.015296+00:00
ALTER SYSTEM SET log_archive_dest_state_2='ENABLE' SCOPE=BOTH;
ALTER DATABASE SET STANDBY DATABASE TO MAXIMIZE AVAILABILITY
Completed: ALTER DATABASE SET STANDBY DATABASE TO MAXIMIZE AVAILABILITY
ALTER DATABASE OPEN
2021-09-20T09:53:32.030768+00:00
TMI: adbdrv open database BEGIN 2021-09-20 09:53:32.030695
Data Guard Broker initializing...
Ping without log force is disabled:
  instance mounted in exclusive mode.
Buffer Cache Full DB Caching mode changing from FULL CACHING DISABLED to FULL CACHING ENABLED
2021-09-20T09:53:32.116297+00:00
Crash Recovery excluding pdb 2 which was cleanly closed.
2021-09-20T09:53:32.116442+00:00
Crash Recovery excluding pdb 3 which was cleanly closed.
Endian type of dictionary set to little
2021-09-20T09:53:32.136779+00:00
Assigning activation ID 4040238735 (0xf0d1268f)
LGWR (PID:10754): Primary database is in MAXIMUM AVAILABILITY mode
2021-09-20T09:53:32.140956+00:00
LGWR (PID:10754): LAD:2 is UNSYNCHRONIZED
LGWR (PID:10754): LAD:1 is not serviced by LGWR
2021-09-20T09:53:33.177813+00:00
Thread 1 advanced to log sequence 2 (thread open)
Redo log for group 2, sequence 2 is not located on DAX storage
Thread 1 opened at log sequence 2
  Current log# 2 seq# 2 mem# 0: +DATA/DGTEST2/ONLINELOG/group_2.290.1067949471
  Current log# 2 seq# 2 mem# 1: +FRA/DGTEST2/ONLINELOG/group_2.420.1067949473
Successful open of redo thread 1
2021-09-20T09:53:33.246706+00:00
MTTR advisory is disabled because FAST_START_MTTR_TARGET is not set
Stopping change tracking
2021-09-20T09:53:33.287479+00:00
ARC1 (PID:11156): Archived Log entry 520 added for T-1.S-1 ID 0xf0d1268f LAD:1
2021-09-20T09:53:33.290677+00:00
TT03 (PID:12462): Sleep 5 seconds and then try to clear SRLs in 2 time(s)
2021-09-20T09:53:33.549541+00:00
Undo initialization recovery: Parallel FPTR failed: start:740398 end:740405 diff:7 ms (0.0 seconds)
Undo initialization recovery: err:0 start: 740397 end: 740435 diff: 38 ms (0.0 seconds)
[11547] Successfully onlined Undo Tablespace 2.
Undo initialization online undo segments: err:0 start: 740435 end: 740959 diff: 524 ms (0.5 seconds)
Undo initialization finished serial:0 start:740397 end:740967 diff:570 ms (0.6 seconds)
Dictionary check beginning
Dictionary check complete
2021-09-20T09:53:34.245425+00:00
Database Characterset is AL32UTF8
No Resource Manager plan active
2021-09-20T09:53:35.468394+00:00
joxcsys_required_dirobj_exists: directory object exists with required path /opt/app/oracle/product/19.x.0/dbhome_2/                                                                                                                          javavm/admin/, pid 11547 cid 1
replication_dependency_tracking turned off (no async multimaster replication found)
LOGSTDBY: Validating controlfile with logical metadata
LOGSTDBY: Validation complete
Starting background process AQPC
2021-09-20T09:53:36.133706+00:00
AQPC started with pid=70, OS id=12472
PDB$SEED(2):Pluggable database PDB$SEED opening in read only
2021-09-20T09:53:36.942191+00:00
PDB$SEED(2):Autotune of undo retention is turned on.
2021-09-20T09:53:37.055480+00:00
TT03 (PID:12462): Enabling archival of deferred physical standby SRLs
2021-09-20T09:53:37.152868+00:00
TT03 (PID:12462): Archived Log entry 521 added for T-1.S-382 ID 0xf0d199cf LAD:1
2021-09-20T09:53:37.246644+00:00
PDB$SEED(2):Endian type of dictionary set to little
PDB$SEED(2):Undo initialization finished serial:0 start:744276 end:744276 diff:0 ms (0.0 seconds)
PDB$SEED(2):Pluggable database PDB$SEED dictionary check beginning
2021-09-20T09:53:38.232036+00:00
Thread 1 advanced to log sequence 3 (LGWR switch),  current SCN: 4236634
  Current log# 3 seq# 3 mem# 0: +DATA/DGTEST2/ONLINELOG/group_3.291.1067949473
  Current log# 3 seq# 3 mem# 1: +FRA/DGTEST2/ONLINELOG/group_3.421.1067949473
2021-09-20T09:53:38.316860+00:00
PDB$SEED(2):Pluggable Database PDB$SEED Dictionary check complete
2021-09-20T09:53:38.327245+00:00
ARC5 (PID:11164): Archived Log entry 522 added for T-1.S-2 ID 0xf0d1268f LAD:1
2021-09-20T09:53:38.328930+00:00
PDB$SEED(2):Database Characterset for PDB$SEED is AL32UTF8
2021-09-20T09:53:38.503847+00:00
QPI: opatch file present, opatch
QPI: qopiprep.bat file present
2021-09-20T09:53:38.642191+00:00
ALTER SYSTEM SET log_archive_dest_state_2='ENABLE' SCOPE=MEMORY SID='*';
2021-09-20T09:53:38.835626+00:00
PDB$SEED(2):SUPLOG: Set PDB SUPLOG SGA at PDB OPEN, old 0x0, new 0x0 (no suplog)
PDB$SEED(2):Opening pdb with no Resource Manager plan active
2021-09-20T09:53:39.622239+00:00
Starting background process CJQ0
Completed: ALTER DATABASE OPEN

After the failover completes the data guard broker status shows old primary is disabled and need reinstating.

DGMGRL> show configuration

Configuration - test_dg

  Protection Mode: MaxAvailability
  Members:
  dgtest2 - Primary database
    dgtest  - Physical standby database
    dgtest3 - Physical standby database (disabled)
      ORA-16661: the standby database needs to be reinstated

Fast-Start Failover:  Disabled

Configuration Status:
SUCCESS   (status updated 40 seconds ago)

If the old primary issue is rectified it would be reinstated as a standby (instead of adding as a new standby). When the old primary is started the DG broker will detect failover has happened and will prevent the opening the opening of the DB in read/write mode.

Completed: ALTER DATABASE MOUNT /* db agent *//* {0:0:312} */
ALTER DATABASE OPEN /* db agent *//* {0:0:312} */
Data Guard Broker initializing...
2021-09-20T09:58:19.408297+00:00
Starting Data Guard Broker (DMON)
Starting background process INSV
2021-09-20T09:58:19.425650+00:00
INSV started with pid=49, OS id=5982
2021-09-20T09:58:23.448529+00:00
Data Guard Broker initialization complete
Data Guard: verifying database primary role...
2021-09-20T09:58:23.467893+00:00
Starting background process NSV1
2021-09-20T09:58:23.484569+00:00
NSV1 started with pid=50, OS id=5987
2021-09-20T09:58:27.484306+00:00
Starting background process NSV2
2021-09-20T09:58:27.501947+00:00
NSV2 started with pid=51, OS id=5992
2021-09-20T09:58:30.605237+00:00
Starting background process RSM0
2021-09-20T09:58:30.620725+00:00
RSM0 started with pid=52, OS id=5997
2021-09-20T09:58:31.448386+00:00
Data Guard: broker startup completed
Data Guard determines a failover has occurred - instance will not be opened
ORA-16649 signalled during: ALTER DATABASE OPEN /* db agent *//* {0:0:312} */...

The database will be left at mount mode.

SQL> select db_unique_name,open_mode from v$database;

DB_UNIQUE_NAME                 OPEN_MODE
------------------------------ --------------------
dgtest3                        MOUNTED

From the current primary DG broker run the reinstate command.

DGMGRL> reinstate database dgtest3
Reinstating database "dgtest3", please wait...
Reinstatement of database "dgtest3" succeeded

During the reinstate process the old primary will be flashback to the time where failover occured and converted to a physical standby.

2021-09-20T09:59:51.612704+00:00
FLASHBACK DATABASE TO SCN 4236454
2021-09-20T09:59:51.774069+00:00
Flashback Restore Start
Flashback Restore Complete
Flashback Media Recovery Start
 Started logmerger process
2021-09-20T09:59:52.058152+00:00
max_pdb is 3
2021-09-20T09:59:52.297907+00:00
Parallel Media Recovery started with 8 slaves
Flashback Media Recovery Log +FRA/DGTEST3/ARCHIVELOG/2021_09_20/thread_1_seq_373.332.1083750351
Flashback Media Recovery Log +FRA/DGTEST3/ARCHIVELOG/2021_09_20/thread_1_seq_374.264.1083750365
2021-09-20T09:59:52.812785+00:00
Flashback Media Recovery Log +FRA/DGTEST3/ARCHIVELOG/2021_09_20/thread_1_seq_375.265.1083750375
Flashback Media Recovery Log +FRA/DGTEST3/ARCHIVELOG/2021_09_20/thread_1_seq_376.267.1083750649
2021-09-20T09:59:53.157016+00:00
Media Recovery End-Of-Redo indicator encountered
2021-09-20T09:59:53.157085+00:00
Media Recovery Continuing
2021-09-20T09:59:53.202162+00:00
Flashback Media Recovery Log +FRA/DGTEST3/ARCHIVELOG/2021_09_20/thread_1_seq_377.271.1083750659
Flashback Media Recovery Log +FRA/DGTEST3/ARCHIVELOG/2021_09_20/thread_1_seq_378.269.1083750671
Flashback Media Recovery Log +FRA/DGTEST3/ARCHIVELOG/2021_09_20/thread_1_seq_379.268.1083750683
2021-09-20T09:59:53.834774+00:00
Recovery of Online Redo Log: Thread 1 Group 1 Seq 380 Reading mem 0
  Mem# 0: +DATA/DGTEST3/ONLINELOG/group_1.299.1069533175
  Mem# 1: +FRA/DGTEST3/ONLINELOG/group_1.460.1069533177
2021-09-20T09:59:53.955443+00:00
Recovery of Online Redo Log: Thread 1 Group 2 Seq 381 Reading mem 0
  Mem# 0: +DATA/DGTEST3/ONLINELOG/group_2.300.1069533177
  Mem# 1: +FRA/DGTEST3/ONLINELOG/group_2.459.1069533177
2021-09-20T09:59:54.017466+00:00
Recovery of Online Redo Log: Thread 1 Group 3 Seq 382 Reading mem 0
  Mem# 0: +DATA/DGTEST3/ONLINELOG/group_3.301.1069533179
  Mem# 1: +FRA/DGTEST3/ONLINELOG/group_3.458.1069533179
2021-09-20T09:59:54.086085+00:00
Incomplete Recovery applied until change 4236455 time 09/20/2021 09:52:41
2021-09-20T09:59:54.094097+00:00
Flashback Media Recovery Complete
Completed: FLASHBACK DATABASE TO SCN 4236454
alter database convert to physical standby
2021-09-20T09:59:54.263414+00:00
ALTER DATABASE CONVERT TO PHYSICAL STANDBY (dgtest3)
Clearing standby activation ID 4040262352 (0xf0d182d0)
The primary database controlfile was created using the
'MAXLOGFILES 16' clause.
There is space for up to 13 standby redo logfiles
Use the following SQL commands on the standby database to create
standby redo logfiles that match the primary database:
ALTER DATABASE ADD STANDBY LOGFILE 'srl1.f' SIZE 209715200;
ALTER DATABASE ADD STANDBY LOGFILE 'srl2.f' SIZE 209715200;
ALTER DATABASE ADD STANDBY LOGFILE 'srl3.f' SIZE 209715200;
ALTER DATABASE ADD STANDBY LOGFILE 'srl4.f' SIZE 209715200;
Offline data file 2 marked as online during convert to standby or switchover to standby.
Restore of backup may be required if the file is not physically accessible.
Offline data file 4 marked as online during convert to standby or switchover to standby.
Restore of backup may be required if the file is not physically accessible.
Offline data file 6 marked as online during convert to standby or switchover to standby.
Restore of backup may be required if the file is not physically accessible.
Offline data file 8 marked as online during convert to standby or switchover to standby.
Restore of backup may be required if the file is not physically accessible.
Offline data file 9 marked as online during convert to standby or switchover to standby.
Restore of backup may be required if the file is not physically accessible.
Offline data file 10 marked as online during convert to standby or switchover to standby.
Restore of backup may be required if the file is not physically accessible.
RSM0 (PID:5997): Database role changed from PRIMARY to PHYSICAL STANDBY [kcvs.c:8936]
2021-09-20T09:59:54.325231+00:00
RSM0 (PID:5997): Waiting for all non-current ORLs to be archived
2021-09-20T09:59:54.325324+00:00
RSM0 (PID:5997): All non-current ORLs have been archived
RSM0 (PID:5997): Clearing online redo logfile 1 +DATA/DGTEST3/ONLINELOG/group_1.299.1069533175
RSM0 (PID:5997): Clearing online redo logfile 2 +DATA/DGTEST3/ONLINELOG/group_2.300.1069533177
RSM0 (PID:5997): Clearing online redo logfile 3 +DATA/DGTEST3/ONLINELOG/group_3.301.1069533179
Clearing online log 1 of thread 1 sequence number 380
Clearing online log 2 of thread 1 sequence number 381
Clearing online log 3 of thread 1 sequence number 382
2021-09-20T10:00:03.959609+00:00
RSM0 (PID:5997): Clearing online redo logfile 1 complete
RSM0 (PID:5997): Clearing online redo logfile 2 complete
RSM0 (PID:5997): Clearing online redo logfile 3 complete
RSM0 (PID:5997): RT: Role transition work is not done
RSM0 (PID:5997): Redo network throttle feature is disabled at mount time
Physical Standby Database mounted.
2021-09-20T10:00:03.970236+00:00
In-memory operation on ADG is currently only supported on Engineered systems and PaaS.
inmemory_adg_enabled is turned off automatically.
Please contact our support team for EXADATA solutions
CONVERT TO PHYSICAL STANDBY: Complete - Database mounted as physical standby

At the same time the DG broker will add log archive destination on the primary for the reinstated physical standby.

ALTER SYSTEM SET log_archive_dest_3='service="dgtest3tns"','SYNC AFFIRM delay=0 optional compression=disable max_failure=0 reopen=300 db_unique_name="dgtest3" net_timeout=30','valid_for=(online_logfile,all_roles)' SCOPE=BOTH;

The other standby will get an update on its fal_server with reference to the reinstated standby.

ALTER SYSTEM SET fal_server='dgtest2tns','dgtest3tns' SCOPE=BOTH;

At the end the data guard configuration will have same number of standbys as before but with a different primary.

DGMGRL> show configuration

Configuration - test_dg

  Protection Mode: MaxAvailability
  Members:
  dgtest2 - Primary database
    dgtest  - Physical standby database
    dgtest3 - Physical standby database

Fast-Start Failover:  Disabled

Configuration Status:
SUCCESS   (status updated 58 seconds ago)

Configuring Statspack for Standby Database

Statspack provide separate set of scripts for setting it up for a standby database in a data guard configuration. If enterprise edition is used with diagnostic pack then Remote Management Framework could be configured to get AWR reports of the standby. Depending on the data guard configuration (multiple standbys, snapshot data needed only from read only instances etc) the statspack setup would result in less effort and complexity than setting up RMF for AWR.
RMF has database link from both primary to standby and vice versa. Before a role transition additional work must be done by way of creating database link between future primary and future standby. In contrast the statspack only create one database link between primary and standby. As long as the TNS entry used for creating the DB link is available in the future primary then the statspack would continue to function. This is especially useful in a configuration where only subset of instances are candidate to become primary and all others are part of a reader farm.
In a multi-tenant architecture the data guard works on CDB level. As such the standby statspack is installed on the root container. This will create two users perfstat and stdbyperf. To accomplish this the statspack and standby statspack must be installed using catcon.pl script (Refer 2020285.1). The setup of standby consists of running spcreate.sql (normal statspack) and sbcreate.sql (standby statspack) scripts.
A script defining the statspack setup parameters could be used to automate the statspack creation. Output below shows content of a such script, which defines tablespace to create the statspack related tables and the password for perfstat user among other things.

cat sp.sql
define default_tablespace='STATSPACKTBS'
define temporary_tablespace='TEMP'
define perfstat_password='asanga123'
@?/rdbms/admin/spcreate.sql;
@/home/oracle/statpack/chng.sql;

The chng.sql contains the additional work must be done after setting up statspcak. This include issues mentioned in 382993.1 and 2437142.1. Content of this files is shown below.

cat chng.sql
alter table perfstat.stats$mutex_sleep disable constraint STATS$MUTEX_SLEEP_PK;
create index perfstat.STATS$MUTEX_SLEEP_PK on STATS$MUTEX_SLEEP(SNAP_ID,DBID,INSTANCE_NUMBER,MUTEX_TYPE,LOCATION);
insert into stats$idle_event select name from v$event_name where wait_class='Idle' 
minus select event from stats$idle_event;

If perl binary out of ORACLE_HOME is not in the path then following error will occur when running catcon.pl

perl $ORACLE_HOME/rdbms/admin/catcon.pl -u sys -s -e -c 'CDB$ROOT' -n 1 -b st_spcreate sp.sql
Can't locate Term/ReadKey.pm in @INC (@INC contains: /usr/local/lib64/perl5 /usr/local/share/perl5 /usr/lib64/perl5/vendor_perl /usr/share/perl5/vendor_perl /usr/lib64/perl5 /usr/share/perl5 . /opt/cx/app/oracle/product/19.x.0/dbhome_1/rdbms/admin/) at /opt/cx/app/oracle/product/19.x.0/dbhome_1/rdbms/admin//catcon.pm line 497.
BEGIN failed--compilation aborted at /opt/cx/app/oracle/product/19.x.0/dbhome_1/rdbms/admin//catcon.pm line 497.
Compilation failed in require at /opt/cx/app/oracle/product/19.x.0/dbhome_1/rdbms/admin/catcon.pl line 165.
BEGIN failed--compilation aborted at /opt/cx/app/oracle/product/19.x.0/dbhome_1/rdbms/admin/catcon.pl line 165.

To rectify this issue put perl binary out of OH into the path and run the catcon.pl specifying sp.sql script.

export PATH=$ORACLE_HOME/perl/bin:$PATH
$ which perl
/opt/cx/app/oracle/product/19.x.0/dbhome_1/perl/bin/perl

perl $ORACLE_HOME/rdbms/admin/catcon.pl -u sys -s -e -c 'CDB$ROOT' -n 1 -b st_spcreate sp.sql

Next step is to create the standby statspack. A similar script could be created for that as well. The most important parameter in this case is the TNS alias which will be used to create the database link that will connect to the standby.

cat sb.sql
define default_tablespace='STATSPACKTBS'
define temporary_tablespace='TEMP'
define stdbyuser_password='asanga321'
define perfstat_password='asanga123'
define key='y'
define tns_alias='ppdbxdb5tns'
@?/rdbms/admin/sbcreate.sql;
@/home/oracle/statpack/sbchng.sql;

The sbchng.sql has additional work must be done on standby statspack. As stdbyperf user doesn't have access to v$event_name the idle event related workaround must be done after the setup. Also the STATS$MUTEX_SLEEP on standby statspack doesn't have DBID or instance number columns.

SQL> desc stdbyperf.stats$mutex_sleep
 Name                                      Null?    Type
 ----------------------------------------- -------- ----------------------------
 SNAP_ID                                   NOT NULL NUMBER
 DB_UNIQUE_NAME                            NOT NULL VARCHAR2(30)
 INSTANCE_NAME                             NOT NULL VARCHAR2(16)
 MUTEX_TYPE                                NOT NULL VARCHAR2(32)
 LOCATION                                  NOT NULL VARCHAR2(40)
 SLEEPS                                             NUMBER
 WAIT_TIME                                          NUMBER

Instead db_unique_name and instance_name must be used

cat sbchng.sql
alter table stdbyperf.stats$mutex_sleep disable constraint STATS$MUTEX_SLEEP_PK;
CREATE INDEX stdbyperf.STATS$MUTEX_SLEEP_PK ON stdbyperf.STATS$MUTEX_SLEEP(SNAP_ID,DB_UNIQUE_NAME,INSTANCE_NAME,MUTEX_TYPE, LOCATION);

Run the standby statspack creation with

perl $ORACLE_HOME/rdbms/admin/catcon.pl -u sys -s -e -c 'CDB$ROOT' -n 1 -b st_sbcreate sb.sql

Once standby statspack is created run as sys the following to update the idle events in stdbyperf user schema.

insert into stdbyperf.stats$idle_event select event from perfstat.stats$idle_event minus select event from stdbyperf.stats$idle_event;

Additional standby instaces could be added to the standby statspack by using sbaddins.sql. This must be run using stdbyperf and must specify the TNS entry to the standby db. A script similar to below could be used for accomplishing this. As all the standby database instnaces use same set of tables no need to do the additional work on the subsequent standby instances.

cat sbaddin.sql
connect stdbyperf/asanga321
define key='y'
define tns_alias='ppdbxdb6tns'
define perfstat_password='asanga123'
@?/rdbms/admin/sbaddins.sql;

perl $ORACLE_HOME/rdbms/admin/catcon.pl -u sys -s -e -c 'CDB$ROOT' -n 1 -b st_sbaddins sbaddin.sql

Login as stdbyperf and run the below to view the current statspack standby configuration. This only shows the standby instances added to statspack. Not the actual data guard configuation.

select * from stats$standby_config ;

DB_UNIQUE_NAME                 INST_NAME        DB_LINK                          PACKAGE_NAME
------------------------------ ---------------- -------------------------------- ----------------------------
ppdbxdb5                       ppdbxdb5         STDBY_LINK_ppdbxdb5tns           STATSPACK_ppdbxdb5_ppdbxdb5
ppdbxdb6                       ppdbxdb6         STDBY_LINK_ppdbxdb6tns           STATSPACK_ppdbxdb6_ppdbxdb6

As mentioned earlie the standby statspack uses common set of tables to store data from all the standby instances. Only unique objects created for each standby instance are the database link and PL/SQL package which has the name STATSPACK_<db_unique_name>_<instance_name>. The corresponding package for the particular standby instance must be used in order to work with the instance. This inlcuding taking snapshots, changing statspack parameters and etc. For example in orde to change the snap level on ppdbxdb5 should run the STATSPACK_PPdbxDB5_PPdbxDB5 package. Similarly to change snap level on ppdbxdb6 should use STATSPACK_PPdbxDB6_PPdbxDB6. Below output shows chaning snap level on ppdbxdb6.

exec STATSPACK_PPdbxDB6_PPdbxDB6.modify_statspack_parameter(i_snap_level => 7, i_modify_parameter=>'true');

select DB_UNIQUE_NAME,INSTANCE_NAME,snap_level from stats$statspack_parameter;

DB_UNIQUE_NAME                 INSTANCE_NAME    SNAP_LEVEL
------------------------------ ---------------- ----------
ppdbxdb5                       ppdbxdb5                  7
ppdbxdb6                       ppdbxdb6                  7

Standby statspack use a common sequence to create snap id. As such when multiple instances takes snapshots there will be gaps between two consecutive snap ids.
Manual snapshots could be taken by running STATSPACK_<db_unique_name>_<instance_name>.snap. Snapshot taking is not automated and has to be done manually. Scheduling it in a dbms_scheduler job would allow the snapshot taking to work even after swichover in a reader farm configuration mentioned at the begining of the post.
The user stdbyperf doesn't have permission to create scheduler job. Grant create job and execute on dbms_scheduler (if revoked from public) to stdbyperf. Could create one scheduler job for each standby or have a single wrapper procedure with snap call for each standby. Below shows two separate scheduler job for automating snapshot taking.

--ppdbxdb5                       

BEGIN
DBMS_SCHEDULER.CREATE_JOB (
job_name => 'STANDBY_STATSPACK_PPdbxDB5',
job_type => 'STORED_PROCEDURE',
job_action => 'STATSPACK_PPdbxDB5_PPdbxDB5.snap',
start_date => TO_TIMESTAMP('00' ,'MI'),
repeat_interval => 'FREQ=HOURLY',
enabled => TRUE);
END;
/

--ppdbxdb6
                       
BEGIN
DBMS_SCHEDULER.CREATE_JOB (
job_name => 'STANDBY_STATSPACK_PPdbxDB6',
job_type => 'STORED_PROCEDURE',
job_action => 'STATSPACK_PPdbxDB6_PPdbxDB6.snap',
start_date => TO_TIMESTAMP('01' ,'MI'),
repeat_interval => 'FREQ=HOURLY',
enabled => TRUE);
END;
/

When having multiple scheduler jobs are invoked at the same time it is possible to run into the following error.

2020-06-29T18:02:49.768983+00:00
Errors in file /opt/cx/app/oracle/diag/rdbms/ppdbxdb1/ppdbxdb1/trace/ppdbxdb1_j001_119494.trc:
ORA-12012: error on auto execute of job "STDBYPERF"."STANDBY_STATSPACK_PPdbxDB5"
ORA-02049: timeout: distributed transaction waiting for lock
ORA-06512: at "STDBYPERF.STATSPACK_PPdbxDB5_PPdbxDB5", line 3909
ORA-06512: at "STDBYPERF.STATSPACK_PPdbxDB5_PPdbxDB5", line 5486
ORA-06512: at "STDBYPERF.STATSPACK_PPdbxDB5_PPdbxDB5", line 101
ORA-06512: at line 1
2020-06-29T18:06:57.404823+00:00

To eliminate this have a time gap between two schedulers. Above schedulers are scheduled 1 minute apart.
Create standby statspack report by calling the sbreport. This will prompt to select the db_unique_name and instance_name and prompt the range of snap ids to create the statspack report.

SQL> @?/rdbms/admin/sbreport

Instances in this Statspack schema
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

DB Unique Name                 Instance Name
------------------------------ ----------------
ppdbxdb5                       ppdbxdb5
ppdbxdb6                       ppdbxdb6

Enter the DATABASE UNIQUE NAME of the standby database to report
Enter value for db_unique_name: ppdbxdb5
You entered: ppdbxdb5

Enter the INSTANCE NAME of the standby database instance to report
Enter value for inst_name: ppdbxdb5
You entered: ppdbxdb5


Specify the number of days of snapshots to choose from
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
Entering the number of days (n) will result in the most recent
(n) days of snapshots being listed.  Pressing  without
specifying a number lists all completed snapshots.

Listing all Completed Snapshots
                                          Snap
Instance       Snap Id   Snap Started    Level Comment
------------ --------- ----------------- ----- --------------------
ppdbxdb5            14 30 Jun 2020 00:00     7
                    16 30 Jun 2020 01:00     7
                    18 30 Jun 2020 02:00     7
                    20 30 Jun 2020 03:00     7
                    22 30 Jun 2020 04:00     7

Standby statspack report will have a separate section on standby recovery

Recovery Progress Stats  DB/Inst: ppdbxdb5/ppdbxdb5  End Snap: 32
-> End Snapshot Time: 30-Jun-20 09:00:04
-> ordered by Recovery Start Time desc, Units, Item asc

Recovery Start Time Item                       Sofar Units   Redo Timestamp
------------------- ----------------- -------------- ------- ------------------
15-Jun-20 15:18:07  Log Files                 11,360 Files
15-Jun-20 15:18:07  Active Apply Rate            659 KB/sec
15-Jun-20 15:18:07  Average Apply Rat             12 KB/sec
15-Jun-20 15:18:07  Maximum Apply Rat         13,280 KB/sec
15-Jun-20 15:18:07  Redo Applied              14,936 Megabyt
15-Jun-20 15:18:07  Recovery ID                    0 RCVID
15-Jun-20 15:18:07  Last Applied Redo              0 SCN+Tim 30-Jun-20 09:00:15
15-Jun-20 15:18:07  Active Time               29,277 Seconds
15-Jun-20 15:18:07  Apply Time per Lo              1 Seconds
15-Jun-20 15:18:07  Checkpoint Time p              0 Seconds
15-Jun-20 15:18:07  Elapsed Time           1,273,511 Seconds
15-Jun-20 15:18:07  Standby Apply Lag              2 Seconds
          -------------------------------------------------------------

Purging of snapshots must also be done calling the purge procedure of the relavent standby instance package. Following procedure delete snapshots after 32 days of creation.

create table deleted_snaps(delete_time timestamp,
min_snapid number, 
max_snapid number, 
DB_UNIQUE_NAME VARCHAR2(30), 
INSTANCE_NAME VARCHAR2(16), primary key (DB_UNIQUE_NAME,INSTANCE_NAME,delete_time));

create or replace
Procedure Delete_Snaps As
min_snap_id stats$snapshot.snap_id%type;
max_snap_id stats$snapshot.snap_id%type;
db_unq_name stats$snapshot.DB_UNIQUE_NAME%type;
ins_name stats$snapshot.INSTANCE_NAME%type;
snapshots_purged pls_integer;
stm varchar2(500);
cursor delsnaps is select DB_UNIQUE_NAME,INSTANCE_NAME,min(snap_id) as min,
max(snap_id) as max from stats$snapshot 
where to_char(snap_time,'YYYY-MM-DD') = to_char(sysdate-32,'YYYY-MM-DD') 
group by DB_UNIQUE_NAME,INSTANCE_NAME;
begin
open delsnaps;
loop
fetch delsnaps into db_unq_name,ins_name,min_snap_id,max_snap_id;
Exit When Delsnaps%Notfound;
stm := 'begin STATSPACK_'||db_unq_name||'_'||ins_name||'.purge(i_begin_snap =>'||min_snap_id||', i_end_snap =>'|| max_snap_id||', i_snap_range  => true, i_extended_purge  => true, I_DB_UNIQUE_NAME  => '''||db_unq_name||''', I_INSTANCE_NAME => '''||ins_name||'''); end;';
dbms_output.put_line(stm);
execute immediate stm;
commit;
Insert Into Deleted_Snaps values(Systimestamp,Min_Snap_Id,Max_Snap_Id,db_unq_name,ins_name);  
end loop;
Close Delsnaps;
commit;
End;
/

Schedule the deletion of snapshots.

BEGIN
DBMS_SCHEDULER.CREATE_JOB (
job_name => 'Delete_Snaps_job',
job_type => 'STORED_PROCEDURE',
job_action => 'Delete_Snaps',
start_date => trunc(SYSDATE + 1,'DD') + 15/1444,
repeat_interval => 'FREQ=DAILY',
enabled => TRUE);
END;
/

To remove a standby instance from statspack configuration use $ORACLE_HOME/rdbms/admin/sbdelins.sql.

To drop standby statspack and statspack from the database use catcon.pl.

#remove standby statspack
perl $ORACLE_HOME/rdbms/admin/catcon.pl -u sys -s -e -c 'CDB$ROOT' -n 1 -b st_sbdrop  $ORACLE_HOME/rdbms/admin/sbdrop.sql

# remove statspack
perl $ORACLE_HOME/rdbms/admin/catcon.pl -u sys -s -e -c 'CDB$ROOT' -n 1 -b st_spdrop  $ORACLE_HOME/rdbms/admin/spdrop.sql

Useful metalink notes
STATSPACK:Idle Wait Events Missing in STATS$IDLE_EVENT [ID 2673657.1]
STATSPACK REPORTS SHOW "IDLE" DATA GUARD WAIT EVENTS IN TOP 5 IN 12.2 [ID 2305287.1]
12.2 or later STATSPACK: Idle Wait Event Such as 'Data Guard: Timer' is Erroneously Included in Top 5 Timed Events [ID 2437142.1]
New Idle Events are Erroneously Listed in STATSPACK Report [ID 1998538.1]
STATSPACK:Idle Wait Events Missing in STATS$IDLE_EVENT [ID 2673657.1]

Related Posts
Statspack setup in Brief
AWR Reports on Standby when Active Data Guard is Used

ORA-03186: Cannot start Oracle ADG recovery on a non-Oracle Cloud database on a server that is not a primary server

ORA-03186 occurs on a data guard configuration if the standby (or one of the standbys in a multiple standby configuration) was open for read only when the primary is restarted (or switchover happens between primary and another standby in a multiple standby configuration).
The setup is same setup used in postupgrading from 12.2 to 18c. The exact version of 18c is 18.6. All the servers are hosted on AWS. The current setup is as follows.

DGMGRL> show configuration

Configuration - dg12c2

  Protection Mode: MaxAvailability
  Members:
  colombo - Primary database
    london  - Physical standby database

Fast-Start Failover: DISABLED

Configuration Status:
SUCCESS   (status updated 51 seconds ago)

When the primary is restarted

[oracle@ip-172-31-20-117 trace]$ srvctl stop database -d colombo
[oracle@ip-172-31-20-117 trace]$ srvctl start database -d colombo

redo shipping to standby stops.
Following message could be seen on the alert log of the standby instance (instance open in read only mode, london).

2019-05-29T05:38:20.414031-04:00
 rfs (PID:9340): Client is running on host ip-172-31-20-117.eu-west-1.compute.internal, not the current host ip-172-31-15-199.eu-west-1.compute.internal
2019-05-29T05:38:21.974469-04:00
 rfs (PID:9348): Client is running on host ip-172-31-20-117.eu-west-1.compute.internal, not the current host ip-172-31-15-199.eu-west-1.compute.internal

On the alert log of the primary (colombo) following could be seen.

 2019-05-29T05:38:22.283282-04:00
Errors in file /opt/app/oracle/diag/rdbms/colombo/colombo/trace/colombo_tt00_10673.trc:
ORA-03186: Cannot start Oracle ADG recovery on a non-Oracle Cloud database on a server that is not a primary server.

If the error message shown is

ORA-03816: Message 3816 not found;  product=RDBMS; facility=ORA

then apply patch 27539475 (on 18c) to get the above error message.
Inside the trace file following lines shows that attempts were made to ship redo to the instance open in read only mode.

*** 2019-05-29T05:38:22.282064-04:00
krsu_upi_status: Error 3186 attaching RFS server to standby instance at host 'londontns'
krsi_verify_network: Error 3186 attaching to LOG_ARCHIVE_DEST_2 standby host londontns
 at 0x7fffa4dc23c8 placed krsg.c@4998
ORA-03186: Cannot start Oracle ADG recovery on a non-Oracle Cloud database on a server that is not a primary server.

Data guard configuration shows error status

DGMGRL> show configuration

Configuration - dg12c2

  Protection Mode: MaxAvailability
  Members:
  colombo - Primary database
    Error: ORA-16810: multiple errors or warnings detected for the member

    london  - Physical standby database

Fast-Start Failover: DISABLED

Configuration Status:
ERROR   (status updated 10 seconds ago)

Primary database configuration and status is shown below.

DGMGRL> show database verbose colombo

Database - colombo

  Role:               PRIMARY
  Intended State:     TRANSPORT-ON
  Instance(s):
    colombo
      Error: ORA-16737: the redo transport service for member "london" has an error

  Database Warning(s):
    ORA-16629: database reports a different protection level from the protection mode

  Properties:
    DGConnectIdentifier             = 'colombotns'
    ObserverConnectIdentifier       = ''
    LogXptMode                      = 'SYNC'
    RedoRoutes                      = ''
    DelayMins                       = '0'
    Binding                         = 'optional'
    MaxFailure                      = '0'
    MaxConnections                  = '1'
    ReopenSecs                      = '300'
    NetTimeout                      = '30'
    RedoCompression                 = 'DISABLE'
    LogShipping                     = 'ON'
    PreferredApplyInstance          = ''
    ApplyInstanceTimeout            = '0'
    ApplyLagThreshold               = '30'
    TransportLagThreshold           = '30'
    TransportDisconnectedThreshold  = '30'
    ApplyParallel                   = 'AUTO'
    ApplyInstances                  = '0'
    StandbyFileManagement           = 'AUTO'
    ArchiveLagTarget                = '0'
    LogArchiveMaxProcesses          = '10'
    LogArchiveMinSucceedDest        = '1'
    DataGuardSyncLatency            = '0'
    DbFileNameConvert               = '/london/, /colombo/'
    LogFileNameConvert              = '/london/, /colombo/'
    FastStartFailoverTarget         = ''
    InconsistentProperties          = '(monitor)'
    InconsistentLogXptProps         = '(monitor)'
    SendQEntries                    = '(monitor)'
    LogXptStatus                    = '(monitor)'
    RecvQEntries                    = '(monitor)'
    PreferredObserverHosts          = ''
    HostName                        = 'ip-172-31-20-117.eu-west-1.compute.internal'
    StaticConnectIdentifier         = '(DESCRIPTION=(ADDRESS=(PROTOCOL=TCP)(HOST=ip-172-31-20-117.eu-west-1.compute.internal)(PORT=1521))(CONNECT_DATA=(SERVICE_NAME=colombo_DGMGRL.domain.net)(INSTANCE_NAME=colombo)(SERVER=DEDICATED)))'
    OnlineArchiveLocation           = ''
    OnlineAlternateLocation         = ''
    StandbyArchiveLocation          = ''
    StandbyAlternateLocation        = ''
    LogArchiveTrace                 = '2049'
    LogArchiveFormat                = '%t_%s_%r.dbf'
    TopWaitEvents                   = '(monitor)'
    SidName                         = '(monitor)'

  Log file locations:
    Alert log               : /opt/app/oracle/diag/rdbms/colombo/colombo/trace/alert_colombo.log
    Data Guard Broker log   : /opt/app/oracle/diag/rdbms/colombo/colombo/trace/drccolombo.log

Database Status:
ERROR

Standby database configuration

DGMGRL> show database verbose london

Database - london

  Role:               PHYSICAL STANDBY
  Intended State:     APPLY-ON
  Transport Lag:      0 seconds (computed 203 seconds ago)
  Apply Lag:          0 seconds (computed 203 seconds ago)
  Average Apply Rate: 62.00 KByte/s
  Active Apply Rate:  0 Byte/s
  Maximum Apply Rate: 0 Byte/s
  Real Time Query:    ON
  Instance(s):
    london

  Database Warning(s):
    ORA-16857: member disconnected from redo source for longer than specified threshold

  Properties:
    DGConnectIdentifier             = 'londontns'
    ObserverConnectIdentifier       = ''
    LogXptMode                      = 'SYNC'
    RedoRoutes                      = ''
    DelayMins                       = '0'
    Binding                         = 'OPTIONAL'
    MaxFailure                      = '0'
    MaxConnections                  = '1'
    ReopenSecs                      = '300'
    NetTimeout                      = '30'
    RedoCompression                 = 'DISABLE'
    LogShipping                     = 'ON'
    PreferredApplyInstance          = ''
    ApplyInstanceTimeout            = '0'
    ApplyLagThreshold               = '30'
    TransportLagThreshold           = '30'
    TransportDisconnectedThreshold  = '30'
    ApplyParallel                   = 'AUTO'
    ApplyInstances                  = '0'
    StandbyFileManagement           = 'AUTO'
    ArchiveLagTarget                = '0'
    LogArchiveMaxProcesses          = '10'
    LogArchiveMinSucceedDest        = '1'
    DataGuardSyncLatency            = '0'
    DbFileNameConvert               = '/colombo/, /london/'
    LogFileNameConvert              = '/colombo/, /london/'
    FastStartFailoverTarget         = ''
    InconsistentProperties          = '(monitor)'
    InconsistentLogXptProps         = '(monitor)'
    SendQEntries                    = '(monitor)'
    LogXptStatus                    = '(monitor)'
    RecvQEntries                    = '(monitor)'
    PreferredObserverHosts          = ''
    HostName                        = 'ip-172-31-15-199.eu-west-1.compute.internal'
    StaticConnectIdentifier         = '(DESCRIPTION=(ADDRESS=(PROTOCOL=TCP)(HOST=ip-172-31-15-199.eu-west-1.compute.internal)(PORT=1521))(CONNECT_DATA=(SERVICE_NAME=london_DGMGRL.domain.net)(INSTANCE_NAME=london)(SERVER=DEDICATED)))'
    OnlineArchiveLocation           = ''
    OnlineAlternateLocation         = ''
    StandbyArchiveLocation          = ''
    StandbyAlternateLocation        = ''
    LogArchiveTrace                 = '2049'
    LogArchiveFormat                = '%t_%s_%r.dbf'
    TopWaitEvents                   = '(monitor)'
    SidName                         = '(monitor)'

  Log file locations:
    Alert log               : /opt/app/oracle/diag/rdbms/london/london/trace/alert_london.log
    Data Guard Broker log   : /opt/app/oracle/diag/rdbms/london/london/trace/drclondon.log

Database Status:
WARNING

There are no inconsistent properties in any of the databases.

DGMGRL> show database colombo 'InconsistentProperties';
INCONSISTENT PROPERTIES
   INSTANCE_NAME        PROPERTY_NAME         MEMORY_VALUE         SPFILE_VALUE         BROKER_VALUE

DGMGRL>  show database london  'InconsistentProperties';
INCONSISTENT PROPERTIES
   INSTANCE_NAME        PROPERTY_NAME         MEMORY_VALUE         SPFILE_VALUE         BROKER_VALUE

The archive dest status on the primary shows the same error as alert log.

SQL> select dest_id id,dest_name name, status, database_mode db_mode,recovery_mode,         protection_mode,standby_logfile_count "SRLs",        standby_logfile_active ACTIVE,         archived_seq#,error  from v$archive_dest_status where dest_id=2;

        ID NAME                 STATUS    DB_MODE         RECOVERY_MODE                      PROTECTION_MODE            SRLs     ACTIVE ARCHIVED_SEQ# ERROR
---------- -------------------- --------- --------------- ---------------------------------- -------------------- ---------- ---------- ------------- ----------------------------------------
         2 LOG_ARCHIVE_DEST_2   ERROR     UNKNOWN         IDLE                               RESYNCHRONIZATION             0          0             0 ORA-03186: Cannot start Oracle ADG
                                                                                                                                                      recovery on a non-Oracle Cloud database
                                                                                                                                                      on a server that is not a primary
                                                                                                                                                      server.

To resolve the redo shipping issue, restart the instance open in read only mode.

[oracle@ip-172-31-15-199 trace]$ srvctl stop database -d london
[oracle@ip-172-31-15-199 trace]$ srvctl start database -d london

Once it is started the redo shipping begins and log archive dest error clears.

DGMGRL> show configuration

Configuration - dg12c2

  Protection Mode: MaxAvailability
  Members:
  colombo - Primary database
    london  - Physical standby database

Fast-Start Failover: DISABLED

Configuration Status:
SUCCESS   (status updated 11 seconds ago)

The other workaround is to put the read only instance into mount mode before the primary is stopped or switchover happens and open it in read only mode afterwards. Either method would result in clients being disconnected to from the read only instance.
The same issue was observed on a 19c multiple data guard configuration hosted on AWS.

DGMGRL> show configuration
Configuration - fcdg
Protection Mode: MaxPerformance
Members:
gold - Primary database
silver - Physical standby database
bronze - Physical standby database

Fast-Start Failover: DISABLED

Configuration Status:
SUCCESS. 

The silver instance is in mount only mode and bronze is open read only mode. When there was a switchover to between gold and silver, redo transport to bronze stopped from the new primary (silver) with the same error as in 18c. Only way to resolve the issue was to restart the instance open in read only mode.
However, 11.2.0.4 data guard configuration hosted on AWS did not have this issue. The redo shipping was working fine across primary restarts when the standby was open in read only mode. Seems whatever issue or removal of capabilities on non-oracle clouds happened in later versions.

Update on 2019-09-24
The issue is fixed with 18.7 RU and for 19c with the 19.4 RU. For other versions apply patch 30289758 ( 30289758: MERGE ON DATABASE RU 18.5.0.0.0 OF 27539475 29430524 ) if available.

AWR Reports on Standby when Active Data Guard is Used

Oracle introduced Remote Management Framework (RMF) in 12.2 which allows creating AWR reports on standby database when active data guard is in use. This post list the steps for setting up the RMF so AWR reports could be generated on standby. The post use the data guard configuration set up on 18.3, which is mentioned in a previous post. Current data guard setup and standby open mode is as follows.

DGMGRL> show configuration

Configuration - paas_iaas_dg

  Protection Mode: MaxAvailability
  Members:
  fradb_fra1kk - Primary database
    londb        - Physical standby database

Fast-Start Failover: DISABLED

Configuration Status:
SUCCESS   (status updated 11 seconds ago)

DGMGRL> show database londb

Database - londb

  Role:               PHYSICAL STANDBY
  Intended State:     APPLY-ON
  Transport Lag:      0 seconds (computed 0 seconds ago)
  Apply Lag:          0 seconds (computed 0 seconds ago)
  Average Apply Rate: 1.00 KByte/s
  Real Time Query:    ON
  Instance(s):
    lonDB

Database Status:
SUCCESS

Oracle provide a pre-created user sys$umf (locked by default) which has all the necessary privileges to carry out the RMF related work. Unlock this user from primary DB and set a password.

alter user sys$umf identified by rmfuser account unlock;

Next create two database link that sys$umf user will use to connect to and from the standby DB. The TNS entries used here were created in the previous post. Before creating database link take a note of the global_names parameter. If this is set to true, then DB links are expected to be the same name as the DB they connect to. If not ORA-02085 error could be encountered when trying to use the DB links.

create database link fra_to_lon CONNECT TO sys$umf IDENTIFIED BY rmfuser  using 'LONDBTNS';
create database link lon_to_fra CONNECT TO sys$umf IDENTIFIED BY rmfuser  using 'FRADBTNS';

Check the DB links are working by querying remote instance using them. Run both on primary and standby.

SQL> select instance_name from v$instance@fra_to_lon;

INSTANCE_NAME
----------------
lonDB

SQL>  select instance_name from v$instance@lon_to_fra;

INSTANCE_NAME
----------------
fraDB

On primary run the following to configure the primary node with RMF. Configure node require a unique name for each node configured. If none is provided, the db_unique_name will be used instead.

SQL> exec dbms_umf.configure_node ('fraDB');

PL/SQL procedure successfully completed.

On standby run the following to configure the standby with RMF. In this case a unique name for standby and the db link name from standby to primary is given as inputs.

SQL> exec dbms_umf.configure_node ('lonDB','lon_to_fra');

PL/SQL procedure successfully completed.

Create the RMF topology by running following on primary.

SQL> exec DBMS_UMF.create_topology ('FRA_LON_TOPOLOGY');

PL/SQL procedure successfully completed.

SQL> select * from dba_umf_topology;

TOPOLOGY_NAME         TARGET_ID TOPOLOGY_VERSION TOPOLOGY
-------------------- ---------- ---------------- --------
FRA_LON_TOPOLOGY     1423735874                1 ACTIVE

View the registered nodes. Only primary is registered so far.

SQL> select * from dba_umf_registration;

TOPOLOGY_NAME        NODE_NAME     NODE_ID  NODE_TYPE AS_SO AS_CA STATE
-------------------- ---------- ---------- ---------- ----- ----- --------------------
FRA_LON_TOPOLOGY     fraDB      1423735874          0 FALSE FALSE OK

Register the standby with the topology. The meaning of the input parameters could be found here. Execute the following on primary.

exec DBMS_UMF.register_node ('FRA_LON_TOPOLOGY', 'lonDB', 'fra_to_lon', 'lon_to_fra', 'FALSE', 'FALSE');

Check both nodes are registered.

SQL> select * from dba_umf_registration;

TOPOLOGY_NAME        NODE_NAME     NODE_ID  NODE_TYPE AS_SO AS_CA STATE
-------------------- ---------- ---------- ---------- ----- ----- -----
FRA_LON_TOPOLOGY     fraDB      1423735874          0 FALSE FALSE OK
FRA_LON_TOPOLOGY     lonDB      4041047630          0 FALSE FALSE OK

Register the AWR service on the remote node.

SQL> exec DBMS_WORKLOAD_REPOSITORY.register_remote_database(node_name=>'lonDB');

PL/SQL procedure successfully completed.

Verify AWR service is active on the remote node

SQL> select * from dba_umf_service;

TOPOLOGY_NAME           NODE_ID SERVICE
-------------------- ---------- -------
FRA_LON_TOPOLOGY     4041047630 AWR

To generate AWR report create two snapshots on the remote database.

SQL> exec dbms_workload_repository.create_remote_snapshot('lonDB');

PL/SQL procedure successfully completed.

Once a snapshot is created the standby DB is listed in the AWR with the DB ID that is equal to the node ID (highlighted above) in the UMF registration.

Instances in this Workload Repository schema
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
  DB Id      Inst Num   DB Name      Instance     Host
------------ ---------- ---------    ----------   ------
  1042410484     1      FRADB        lonDB        lonvm
  4041047630     1      FRADB        lonDB        lonvm
* 1042410484     1      FRADB        fraDB        fravm

Enter value for dbid: 4041047630

The AWR instance report generated would list the role as physical standby.

The AWR control view list the standby database as well.

SQL> select * from dba_hist_wr_control;

      DBID SNAP_INTERVAL        RETENTION            TOPNSQL        CON_ID   SRC_DBID SRC_DBNAME
---------- -------------------- -------------------- ---------- ---------- ---------- ----------
1042410484 +00000 01:00:00.0    +00008 00:00:00.0    DEFAULT             0 1042410484
4041047630 +00000 01:00:00.0    +00008 00:00:00.0    DEFAULT               1042410484 lonDB

The remote snapshots will be automatically taken according to snapshot internal.

Instances in this Workload Repository schema
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
  DB Id      Inst Num   DB Name      Instance     Host
------------ ---------- ---------    ----------   ------
  1042410484     1      FRADB        lonDB        lonvm
  4041047630     1      FRADB        lonDB        lonvm
* 1042410484     1      FRADB        fraDB        fravm

Enter value for dbid: 4041047630
Using 4041047630 for database Id
Enter value for inst_num: 1
Using 1 for instance number


Specify the number of days of snapshots to choose from
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
Entering the number of days (n) will result in the most recent
(n) days of snapshots being listed.  Pressing  without
specifying a number lists all completed snapshots.


Enter value for num_days: 1

Listing the last day's Completed Snapshots
Instance     DB Name      Snap Id       Snap Started    Snap Level
------------ ------------ ---------- ------------------ ----------

lonDB        FRADB                1  29 Nov 2018 13:11    1
                                  2  29 Nov 2018 13:59    1
                                  3  29 Nov 2018 14:58    1
                                  4  29 Nov 2018 15:58    1

However, after a switchover, when roles changes the automatic snapshot taking will stop, both on new primary (old standby) and new standby (old primary). In order to automatic AWR snapshot to continue do the following after a switchover. As the first step unregistered the new primary (old standby) as a remote database.

SQL> exec DBMS_WORKLOAD_REPOSITORY.UNREGISTER_REMOTE_DATABASE('lonDB','FRA_LON_TOPOLOGY',false);

PL/SQL procedure successfully completed.

Drop the topology

exec DBMS_UMF.drop_topology('FRA_LON_TOPOLOGY');

PL/SQL procedure successfully completed.

Run un-configure procedure on each node

SQL>  exec DBMS_UMF.UNCONFIGURE_NODE;

PL/SQL procedure successfully completed.

Re-create the topology again with new primary and standby. On new primary

SQL> exec dbms_umf.configure_node ('lonDB');

PL/SQL procedure successfully completed.

On new standby

SQL> exec dbms_umf.configure_node ('fraDB','fra_to_lon');

PL/SQL procedure successfully completed.

Same topology name is used

exec DBMS_UMF.create_topology ('FRA_LON_TOPOLOGY');

Register new standby

exec DBMS_UMF.register_node ('FRA_LON_TOPOLOGY', 'fraDB', 'lon_to_fra', 'fra_to_lon', 'FALSE', 'FALSE');

Register remote DB for AWR by specifying the node ID of the DB shown in dba_umf_registration earlier. If the remote DB registration was to be done using DB name as before then "ORA-13516: AWR Operation failed: ORA-13516: AWR Operation failed: Remote source not registered for AWR" would be encountered when snapshots are taken. Doing log switches as suggested by 2409808.1 did not resolve this issue.

exec DBMS_WORKLOAD_REPOSITORY.register_remote_database(1423735874);

PL/SQL procedure successfully completed.

Once remote DB is registered remote snapshots could be taken

SQL> exec dbms_workload_repository.create_remote_snapshot('fraDB');

PL/SQL procedure successfully completed.

After the role reversal, automatic snapshots will continue on the snapshot interval. Run AWR instance reports same as before.

Instances in this Workload Repository schema
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
  DB Id      Inst Num   DB Name      Instance     Host
------------ ---------- ---------    ----------   ------
  1042410484     1      FRADB        fraDB        fravm
  1423735874     1      FRADB        fraDB        fravm
* 1042410484     1      FRADB        lonDB        lonvm

Enter value for dbid: 1423735874

The new primary (old standby) will be reflected on the AWR report taken against it.

Useful Metalink Note
How to Generate AWRs in Active Data Guard Standby Databases [ID 2409808.1]

Related Posts
Enabling Automatic AWR Snapshots on PDB

Update on 2020-07-13
Oracle documentation now has a separate section on "Managing ADG Role Transition". The doc shows the use of DBMS_UMF.SWITCH_DESTINATION for role reversal scenarios.

Start of Service Fails with ORA-16000 on Physical Standby Open for Read Only

Start of database service failed with ora-16000 on physical standby where both CDB and PDB are open read only mode. The DG setup is same one mentioned in earlier post Data Guard on 12.2 CDB.

SQL> show con_name

CON_NAME
----------
CDB$ROOT

SQL> select open_mode from v$database;

OPEN_MODE
--------------------
READ ONLY WITH APPLY

SQL> show pdbs

    CON_ID CON_NAME                       OPEN MODE  RESTRICTED
---------- ------------------------------ ---------- ----------
         2 PDB$SEED                       READ ONLY  NO
         3 PDBAPP1                        READ ONLY  NO

srvctl add service -db stbycdb -pdb pdbapp1 -service abc -role PHYSICAL_STANDBY -notification TRUE -failovertype NONE -failovermethod NONE -failoverdelay 0 -failoverretry 0
srvctl start service -db stbycdb -s abc
PRCD-1084 : Failed to start service abc
PRCR-1079 : Failed to start resource ora.stbycdb.abc.svc
CRS-5017: The resource action "ora.stbycdb.abc.svc start" encountered the following error:
ORA-16000: database or pluggable database open for read-only access
ORA-06512: at "SYS.DBMS_SERVICE", line 5
ORA-06512: at "SYS.DBMS_SERVICE", line 288
ORA-06512: at line 1
. For details refer to "(:CLSN00107:)" in "/opt/app/oracle/diag/crs/city7s/crs/trace/ohasd_oraagent_grid.trc".

CRS-2674: Start of 'ora.stbycdb.abc.svc' on 'city7s' failed

Reason seems to be creating the service with physical standby role is trying to add some rows to the read only database. To fix problem first add the service to the primary with role as physical standby and start the service. It may seems odd to start a database service that it is defined for a physical standby role but that's what needed to resolve this. Once the service is started stop it on the primary before the next steps.

srvctl add service -db prodcdb -pdb pdbapp1 -service abc -role PHYSICAL_STANDBY -notification TRUE -failovertype NONE -failovermethod NONE -failoverdelay 0 -failoverretry 0
srvctl start service -db prodcdb -s abc

lsnrctl status

...
  Instance "prodcdb", status READY, has 1 handler(s) for this service...
Service "prodcdb_DGB" has 1 instance(s).
  Instance "prodcdb", status READY, has 1 handler(s) for this service...
Service "prodcdb_DGMGRL" has 1 instance(s).
  Instance "prodcdb", status UNKNOWN, has 1 handler(s) for this service...
Service "abc" has 1 instance(s).
  Instance "prodcdb", status READY, has 1 handler(s) for this service...
The command completed successfully

srvctl stop service -db prodcdb -s abc

Do few log switches and wait until these logs are applied on the standby. Once logs are applied on standby create and start the service on standby.

srvctl add service -db stbycdb -pdb pdbapp1 -service abc -role PHYSICAL_STANDBY -notification TRUE -failovertype NONE -failovermethod NONE -failoverdelay 0 -failoverretry 0
srvctl start service -db stbycdb -s abc

lsnrctl status

...
  Instance "stbycdb", status READY, has 1 handler(s) for this service...
Service "stbycdb_DGB" has 1 instance(s).
  Instance "stbycdb", status READY, has 1 handler(s) for this service...
Service "stbycdb_DGMGRL" has 1 instance(s).
  Instance "stbycdb", status UNKNOWN, has 1 handler(s) for this service...
Service "abc" has 1 instance(s).
  Instance "stbycdb", status READY, has 1 handler(s) for this service...
The command completed successfully

Useful metalink notes
ORA-16000 Cannot Enable Auto Open of PDB On Physical Standby [ID 2377174.1]
How to create a RAC Database Service With Physical Standby Role Option? [ID 1129143.1]

Creating a PDB in a Data Guard Configuration

Oracle Data guard configuration used in this post is the 12.2 data guard configuration created earlier. The primary DB has enabled_PDBs_on_standby='*'. What needs to be done when creating a PDB in the primary DB of a data guard configuration depends whether standby is open with read only (active data guard) or whether the PDB was created with standbys=none or not. This post lists all three of these options.

    Creating a PDB when active data guard is in use
    Creating a PDB with standbys=none
    Creating a PDB with standbys=all (default) option and without active data guard

Creating a PDB when active data guard is in use
Oracle documentation states "to create a PDB as a local clone from a different PDB or from the seed PDB within the same primary CDB, copy the data files that belong to the source PDB over to the standby database. (This step is not necessary in an Oracle Active Data Guard environment because the data files are copied automatically at the standby when the PDB is created on the standby database.)". So when active data guard is in use (which require separation license options) PDB could be created same as in a non-data guard configuration. The PDB will be automatically created on standby without any manual intervention.
1. Standby is open for read only access and redo apply is on.

SQL> select open_mode from v$database;

OPEN_MODE
----------
READ ONLY

DGMGRL> show database stbycdb;

Database - stbycdb

  Role:               PHYSICAL STANDBY
  Intended State:     APPLY-ON

2. Create PDB without any standby options and open it.

SQL> create pluggable database pdbuat admin user pdbuat identified by pdbuat default tablespace users;
Pluggable database created.

SQL> alter pluggable database pdbuat open;
Pluggable database altered.

SQL> show pdbs;

    CON_ID CON_NAME                       OPEN MODE  RESTRICTED
---------- ------------------------------ ---------- ----------
         2 PDB$SEED                       READ ONLY  NO
         4 PDBUAT                         READ WRITE NO

3. The PDB will be created on the standby. The standby alert log will show data file being created for the PDB.

2018-03-14T15:24:55.972459+00:00
Recovery created pluggable database PDBUAT
*****************************************
WARNING: The converted filename '+DATA/stbycdb/6760aa08f64a2dd6e0535500a8c02c77/datafile/system.302.970753855'
         is an ASM fully qualified filename.
         Changing the filename to '+DATA/MUST_RENAME_THIS_DATAFILE_0.4294967295.4294967295'.
         Please rename it accordingly.
*****************************************
2018-03-14T15:25:00.621144+00:00
Recovery copied files for tablespace SYSTEM
Recovery successfully copied file +DATA/STBYCDB/6760AA08F64A2DD6E0535500A8C02C77/DATAFILE/system.289.970759497 from +DATA/STBYCDB/6325282AC695380EE0535500A8C0D89D/DATAFILE/system.281.968524885
PDBUAT(4):*****************************************
PDBUAT(4):WARNING: The converted filename '+DATA/stbycdb/6760aa08f64a2dd6e0535500a8c02c77/datafile/system.302.970753855'
PDBUAT(4):         is an ASM fully qualified filename.
PDBUAT(4):         Changing the filename to '+DATA/MUST_RENAME_THIS_DATAFILE_56006.4294967295.4294967295'.
PDBUAT(4):         Please rename it accordingly.
PDBUAT(4):*****************************************
PDBUAT(4):*****************************************
PDBUAT(4):WARNING: The converted filename '+DATA/stbycdb/6760aa08f64a2dd6e0535500a8c02c77/datafile/system.302.970753855'
PDBUAT(4):         is an ASM fully qualified filename.
PDBUAT(4):         Changing the filename to '+DATA/MUST_RENAME_THIS_DATAFILE_0.4294967295.4294967295'.
PDBUAT(4):         Please rename it accordingly.
PDBUAT(4):*****************************************
PDBUAT(4):Datafile 134 added to flashback set
PDBUAT(4):Successfully added datafile 134 to media recovery
PDBUAT(4):Datafile #134: '+DATA/STBYCDB/6760AA08F64A2DD6E0535500A8C02C77/DATAFILE/system.289.970759497'
*****************************************
WARNING: The converted filename '+DATA/stbycdb/6760aa08f64a2dd6e0535500a8c02c77/datafile/sysaux.279.970753857'
         is an ASM fully qualified filename.
         Changing the filename to '+DATA/MUST_RENAME_THIS_DATAFILE_0.4294967295.4294967295'.
         Please rename it accordingly.
*****************************************
2018-03-14T15:25:09.173271+00:00
Recovery copied files for tablespace SYSAUX
Recovery successfully copied file +DATA/STBYCDB/6760AA08F64A2DD6E0535500A8C02C77/DATAFILE/sysaux.306.970759501 from +DATA/STBYCDB/6325282AC695380EE0535500A8C0D89D/DATAFILE/sysaux.305.968524855
PDBUAT(4):*****************************************
PDBUAT(4):WARNING: The converted filename '+DATA/stbycdb/6760aa08f64a2dd6e0535500a8c02c77/datafile/sysaux.279.970753857'
PDBUAT(4):         is an ASM fully qualified filename.
PDBUAT(4):         Changing the filename to '+DATA/MUST_RENAME_THIS_DATAFILE_56006.4294967295.4294967295'.
PDBUAT(4):         Please rename it accordingly.
PDBUAT(4):*****************************************
PDBUAT(4):*****************************************
PDBUAT(4):WARNING: The converted filename '+DATA/stbycdb/6760aa08f64a2dd6e0535500a8c02c77/datafile/sysaux.279.970753857'
PDBUAT(4):         is an ASM fully qualified filename.
PDBUAT(4):         Changing the filename to '+DATA/MUST_RENAME_THIS_DATAFILE_0.4294967295.4294967295'.
PDBUAT(4):         Please rename it accordingly.
PDBUAT(4):*****************************************
PDBUAT(4):Datafile 135 added to flashback set
PDBUAT(4):Successfully added datafile 135 to media recovery
PDBUAT(4):Datafile #135: '+DATA/STBYCDB/6760AA08F64A2DD6E0535500A8C02C77/DATAFILE/sysaux.306.970759501'
*****************************************
WARNING: The converted filename '+DATA/stbycdb/6760aa08f64a2dd6e0535500a8c02c77/datafile/undotbs1.301.970753855'
         is an ASM fully qualified filename.
         Changing the filename to '+DATA/MUST_RENAME_THIS_DATAFILE_0.4294967295.4294967295'.
         Please rename it accordingly.
*****************************************
2018-03-14T15:25:13.544410+00:00
Recovery copied files for tablespace UNDOTBS1
Recovery successfully copied file +DATA/STBYCDB/6760AA08F64A2DD6E0535500A8C02C77/DATAFILE/undotbs1.274.970759509 from +DATA/STBYCDB/6325282AC695380EE0535500A8C0D89D/DATAFILE/undotbs1.300.968524893
PDBUAT(4):*****************************************
PDBUAT(4):WARNING: The converted filename '+DATA/stbycdb/6760aa08f64a2dd6e0535500a8c02c77/datafile/undotbs1.301.970753855'
PDBUAT(4):         is an ASM fully qualified filename.
PDBUAT(4):         Changing the filename to '+DATA/MUST_RENAME_THIS_DATAFILE_56006.4294967295.4294967295'.
PDBUAT(4):         Please rename it accordingly.
PDBUAT(4):*****************************************
PDBUAT(4):*****************************************
PDBUAT(4):WARNING: The converted filename '+DATA/stbycdb/6760aa08f64a2dd6e0535500a8c02c77/datafile/undotbs1.301.970753855'
PDBUAT(4):         is an ASM fully qualified filename.
PDBUAT(4):         Changing the filename to '+DATA/MUST_RENAME_THIS_DATAFILE_0.4294967295.4294967295'.
PDBUAT(4):         Please rename it accordingly.
PDBUAT(4):*****************************************
PDBUAT(4):Datafile 136 added to flashback set
PDBUAT(4):Successfully added datafile 136 to media recovery
PDBUAT(4):Datafile #136: '+DATA/STBYCDB/6760AA08F64A2DD6E0535500A8C02C77/DATAFILE/undotbs1.274.970759509'

4. PDB on the standby could be open read only.

SQL> show pdbs

    CON_ID CON_NAME                       OPEN MODE  RESTRICTED
---------- ------------------------------ ---------- ----------
         2 PDB$SEED                       READ ONLY  NO
         4 PDBUAT                         MOUNTED

SQL> alter pluggable database pdbuat open read only;
Pluggable database altered.

SQL> show pdbs

    CON_ID CON_NAME                       OPEN MODE  RESTRICTED
---------- ------------------------------ ---------- ----------
         2 PDB$SEED                       READ ONLY  NO
         4 PDBUAT                         READ ONLY  NO

5. When the PDB is open in read only mode, the alert log on standby will output a warning about the missing tempfile. If needed add the temp file for the PDB on the standby

2018-03-14T15:26:53.544876+00:00
alter pluggable database pdbuat open read only
PDBUAT(4):Autotune of undo retention is turned on.
2018-03-14T15:26:53.963014+00:00
PDBUAT(4):Endian type of dictionary set to little
PDBUAT(4):Undo initialization finished serial:0 start:2259280636 end:2259280636 diff:0 ms (0.0 seconds)
2018-03-14T15:26:54.540760+00:00
PDBUAT(4):Database Characterset for PDBUAT is AL32UTF8
PDBUAT(4):*********************************************************************
PDBUAT(4):WARNING: The following temporary tablespaces in container(PDBUAT)
PDBUAT(4):         contain no files.
PDBUAT(4):         This condition can occur when a backup controlfile has
PDBUAT(4):         been restored.  It may be necessary to add files to these
PDBUAT(4):         tablespaces.  That can be done using the SQL statement:
PDBUAT(4):
PDBUAT(4):         ALTER TABLESPACE  ADD TEMPFILE
PDBUAT(4):
PDBUAT(4):         Alternatively, if these temporary tablespaces are no longer
PDBUAT(4):         needed, then they can be dropped.
PDBUAT(4):           Empty temporary tablespace: TEMP
PDBUAT(4):*********************************************************************
2018-03-14T15:26:56.374791+00:00
PDBUAT(4):Opening pdb with no Resource Manager plan active
Pluggable database PDBUAT opened read only
Completed: alter pluggable database pdbuat open read only

SQL> alter session set container=pdbuat;
Session altered.

SQL> select name from v$tempfile;
no rows selected

SQL> alter tablespace temp add tempfile;
Tablespace altered.

SQL> select name from v$tempfile;

NAME
--------------------------------------------------------------------------------
+DATA/STBYCDB/6760AA08F64A2DD6E0535500A8C02C77/TEMPFILE/temp.278.970759733

Creating a PDB with standbys=none
1. 12c allows the option of creating a PDB on primary without creating the datafiles in the standby. In order to use this feature create the PDB with standbys=none option. This option is useful when standby cannot use the active data guard option.

SQL>  create pluggable database pdb2 admin user pdb2 identified by pdb2 standbys=none;
Pluggable database created.

SQL> alter pluggable database pdb2 open;
Pluggable database altered.

show pdbs;
SQL>
    CON_ID CON_NAME                       OPEN MODE  RESTRICTED
---------- ------------------------------ ---------- ----------
         2 PDB$SEED                       READ ONLY  NO
         3 PDBDEV                         READ WRITE NO
         5 PDB2                           READ WRITE NO

2. The stadby CDB will list the PDB name but it will have recovery status disable and unnamed datafiles with recover status.

SQL> show pdbs

    CON_ID CON_NAME                       OPEN MODE  RESTRICTED
---------- ------------------------------ ---------- ----------
         2 PDB$SEED                       MOUNTED
         3 PDBDEV                         MOUNTED
         5 PDB2                           MOUNTED

SQL> alter session set container=pdb2;
Session altered.

SQL> select name, recovery_status from v$pdbs;

NAME     RECOVERY
-------- --------
PDB2     DISABLED

SQL> select file#,name,status from v$datafile;

     FILE# NAME                                                         STATUS
---------- ------------------------------------------------------------ -------
       140 /opt/app/oracle/product/12.2.0/dbhome_1/dbs/UNNAMED00140     SYSOFF
       141 /opt/app/oracle/product/12.2.0/dbhome_1/dbs/UNNAMED00141     RECOVER
       142 /opt/app/oracle/product/12.2.0/dbhome_1/dbs/UNNAMED00142     RECOVER

The alert log on standby will have the following with regard to the datafiles

2018-03-14T16:50:21.769872+00:00
Recovery created pluggable database PDB2
PDB2(5):File #140 added to control file as 'UNNAMED00140'. Originally created as:
PDB2(5):'+DATA/PRODCDB/6761DB93F6D36101E0535500A8C00734/DATAFILE/system.293.970758983'
PDB2(5):because the pluggable database was created with nostandby
PDB2(5):or the tablespace belonging to the pluggable database is
PDB2(5):offline.
PDB2(5):File #141 added to control file as 'UNNAMED00141'. Originally created as:
PDB2(5):'+DATA/PRODCDB/6761DB93F6D36101E0535500A8C00734/DATAFILE/sysaux.294.970758983'
PDB2(5):because the pluggable database was created with nostandby
PDB2(5):or the tablespace belonging to the pluggable database is
PDB2(5):offline.
PDB2(5):File #142 added to control file as 'UNNAMED00142'. Originally created as:
PDB2(5):'+DATA/PRODCDB/6761DB93F6D36101E0535500A8C00734/DATAFILE/undotbs1.295.970758983'
PDB2(5):because the pluggable database was created with nostandby
PDB2(5):or the tablespace belonging to the pluggable database is
PDB2(5):offline.

3. To make the PDB part of the data guard by enabling recovery on it require copying the datafiles to the standby. The post list two methods of copying the datafiles. One using RMAN from primary and other using RMAN from Standby. Only need to do one of these methods to get the files copied.
  3.1 The first method of copying the datafile from primary to standby is by using an auxiliary channel to the standby. Connect to both primary and standby and issue a backup as copy on the pluggable database.

rman target / auxiliary sys/prodcdbdb@stbytns

connected to target database: PRODCDB (DBID=2963914998)
connected to auxiliary database: PRODCDB (DBID=2963914998, not open)

RMAN> backup as copy pluggable database pdb2 auxiliary format '+DATA';

Starting backup at 14-MAR-18
using target database control file instead of recovery catalog
allocated channel: ORA_DISK_1
channel ORA_DISK_1: SID=161 device type=DISK
channel ORA_DISK_1: starting datafile copy
input datafile file number=00141 name=+DATA/PRODCDB/6761DB93F6D36101E0535500A8C00734/DATAFILE/sysaux.294.970758983
output file name=+DATA/STBYCDB/6761DB93F6D36101E0535500A8C00734/DATAFILE/sysaux.275.970764731 tag=TAG20180314T151811
channel ORA_DISK_1: datafile copy complete, elapsed time: 00:00:15
channel ORA_DISK_1: starting datafile copy
input datafile file number=00140 name=+DATA/PRODCDB/6761DB93F6D36101E0535500A8C00734/DATAFILE/system.293.970758983
output file name=+DATA/STBYCDB/6761DB93F6D36101E0535500A8C00734/DATAFILE/system.292.970764747 tag=TAG20180314T151811
channel ORA_DISK_1: datafile copy complete, elapsed time: 00:00:07
channel ORA_DISK_1: starting datafile copy
input datafile file number=00142 name=+DATA/PRODCDB/6761DB93F6D36101E0535500A8C00734/DATAFILE/undotbs1.295.970758983
output file name=+DATA/STBYCDB/6761DB93F6D36101E0535500A8C00734/DATAFILE/undotbs1.286.970764753 tag=TAG20180314T151811
channel ORA_DISK_1: datafile copy complete, elapsed time: 00:00:07
Finished backup at 14-MAR-18

Once the files are copied to the standby they must be added to the PDB on the standby. This too could be done different ways. Below are few methods (some of the outputs are from different PDB creations. So some of the file names, file numbers may not match the above copy command file name and numbers).
    3.1.1. First method is using switch to datafile copies. This could be done at PDB level or file level. As the first step catalog the copied datafiles on standby.

 rman target /

connected to target database: PRODCDB (DBID=2963914998, not open)

RMAN> catalog start with '+data/stbycdb';

Starting implicit crosscheck backup at 19-MAR-18
using target database control file instead of recovery catalog
allocated channel: ORA_DISK_1
channel ORA_DISK_1: SID=149 device type=DISK
Crosschecked 12 objects
Finished implicit crosscheck backup at 19-MAR-18

Starting implicit crosscheck copy at 19-MAR-18
using channel ORA_DISK_1
Finished implicit crosscheck copy at 19-MAR-18

searching for all files in the recovery area
cataloging files...
no files cataloged

searching for all files that match the pattern +data/stbycdb

List of Files Unknown to the Database
=====================================
File Name: +DATA/STBYCDB/678982658F142261E0535500A8C05B68/DATAFILE/sysaux.297.971189621
File Name: +DATA/STBYCDB/678982658F142261E0535500A8C05B68/DATAFILE/system.281.971189637
File Name: +DATA/STBYCDB/678982658F142261E0535500A8C05B68/DATAFILE/undotbs1.287.971189643
File Name: +DATA/STBYCDB/678982658F142261E0535500A8C05B68/DATAFILE/users.266.971189659
File Name: +DATA/STBYCDB/CONTROLFILE/current.285.970934541

Do you really want to catalog the above files (enter YES or NO)? YES
cataloging files...
cataloging done

List of Cataloged Files
=======================
File Name: +DATA/STBYCDB/678982658F142261E0535500A8C05B68/DATAFILE/sysaux.297.971189621
File Name: +DATA/STBYCDB/678982658F142261E0535500A8C05B68/DATAFILE/system.281.971189637
File Name: +DATA/STBYCDB/678982658F142261E0535500A8C05B68/DATAFILE/undotbs1.287.971189643
File Name: +DATA/STBYCDB/678982658F142261E0535500A8C05B68/DATAFILE/users.266.971189659

List of Files Which Were Not Cataloged
=======================================
File Name: +DATA/STBYCDB/CONTROLFILE/current.285.970934541
  RMAN-07517: Reason: The file header is corrupted

RMAN> LIST DATAFILECOPY ALL;

List of Datafile Copies
=======================

Key     File S Completion Time Ckp SCN    Ckp Time        Sparse
------- ---- - --------------- ---------- --------------- ------
26      12   A 19-MAR-18       11731251   19-MAR-18       NO
        Name: +DATA/STBYCDB/678982658F142261E0535500A8C05B68/DATAFILE/system.281.971189637
        Tag: TAG20180319T131942

25      13   A 19-MAR-18       11731222   19-MAR-18       NO
        Name: +DATA/STBYCDB/678982658F142261E0535500A8C05B68/DATAFILE/sysaux.297.971189621
        Tag: TAG20180319T131942

27      14   A 19-MAR-18       11731268   19-MAR-18       NO
        Name: +DATA/STBYCDB/678982658F142261E0535500A8C05B68/DATAFILE/undotbs1.287.971189643
        Tag: TAG20180319T131942

28      15   A 19-MAR-18       11731297   19-MAR-18       NO
        Name: +DATA/STBYCDB/678982658F142261E0535500A8C05B68/DATAFILE/users.266.971189659
        Tag: TAG20180319T131942

      3.1.1.1. If PDB level copying is preferred switch the PDB to copy.

RMAN> switch pluggable database pdb2 to copy;

datafile 12 switched to datafile copy "+DATA/STBYCDB/678982658F142261E0535500A8C05B68/DATAFILE/system.281.971189637"
datafile 13 switched to datafile copy "+DATA/STBYCDB/678982658F142261E0535500A8C05B68/DATAFILE/sysaux.297.971189621"
datafile 14 switched to datafile copy "+DATA/STBYCDB/678982658F142261E0535500A8C05B68/DATAFILE/undotbs1.287.971189643"
datafile 15 switched to datafile copy "+DATA/STBYCDB/678982658F142261E0535500A8C05B68/DATAFILE/users.266.971189659"

      3.1.1.2. Or if invidual data file level switching is preferred then switch to individual datafile.

RMAN> list datafilecopy all;

using target database control file instead of recovery catalog
List of Datafile Copies
=======================

Key     File S Completion Time Ckp SCN    Ckp Time        Sparse
------- ---- - --------------- ---------- --------------- ------
23      178  A 16-MAR-18       10526422   16-MAR-18       NO
        Name: +DATA/STBYCDB/678784B473516D01E0535500A8C0AAEE/DATAFILE/system.278.970926583
        Tag: TAG20180316T121537

22      179  A 16-MAR-18       10526406   16-MAR-18       NO
        Name: +DATA/STBYCDB/678784B473516D01E0535500A8C0AAEE/DATAFILE/sysaux.274.970926577
        Tag: TAG20180316T121537

24      180  A 16-MAR-18       10526436   16-MAR-18       NO
        Name: +DATA/STBYCDB/678784B473516D01E0535500A8C0AAEE/DATAFILE/undotbs1.309.970926591
        Tag: TAG20180316T121537

25      181  A 16-MAR-18       10526450   16-MAR-18       NO
        Name: +DATA/STBYCDB/678784B473516D01E0535500A8C0AAEE/DATAFILE/users.263.970926597
        Tag: TAG20180316T121537

RMAN> switch datafile 178 to copy;
datafile 178 switched to datafile copy "+DATA/STBYCDB/678784B473516D01E0535500A8C0AAEE/DATAFILE/system.278.970926583"

RMAN> switch datafile 179 to copy;
datafile 179 switched to datafile copy "+DATA/STBYCDB/678784B473516D01E0535500A8C0AAEE/DATAFILE/sysaux.274.970926577"

RMAN> switch datafile 180 to copy;
datafile 180 switched to datafile copy "+DATA/STBYCDB/678784B473516D01E0535500A8C0AAEE/DATAFILE/undotbs1.309.970926591"

RMAN> switch datafile 181 to copy;
datafile 181 switched to datafile copy "+DATA/STBYCDB/678784B473516D01E0535500A8C0AAEE/DATAFILE/users.263.970926597"

When either 3.1.1.1 or 3.1.1.2 is done go to step 4.
  3.1.2. Second option is to rename the unamed file to copied file. This could require stopping redo apply and putting standby file management to manual.

DGMGRL> edit database stbycdb set state='APPLY-OFF';
Succeeded.

SQL> alter system set standby_file_management='manual';
System altered.

SQL> alter session set container=pdb2;
Session altered.

SQL> alter database rename file '/opt/app/oracle/product/12.2.0/dbhome_1/dbs/UNNAMED00140' to '+DATA/STBYCDB/6761DB93F6D36101E0535500A8C00734/DATAFILE/system.292.970764747';
Database altered.

SQL> alter database rename file '/opt/app/oracle/product/12.2.0/dbhome_1/dbs/UNNAMED00141' to '+DATA/STBYCDB/6761DB93F6D36101E0535500A8C00734/DATAFILE/sysaux.275.970764731';
Database altered.

SQL> alter database rename file '/opt/app/oracle/product/12.2.0/dbhome_1/dbs/UNNAMED00142' to '+DATA/STBYCDB/6761DB93F6D36101E0535500A8C00734/DATAFILE/undotbs1.286.970764753';
Database altered.


     FILE# NAME                                                                             STATUS
---------- -------------------------------------------------------------------------------- -------
       140 +DATA/STBYCDB/6761DB93F6D36101E0535500A8C00734/DATAFILE/system.292.970764747     SYSOFF
       141 +DATA/STBYCDB/6761DB93F6D36101E0535500A8C00734/DATAFILE/sysaux.275.970764731     RECOVER
       142 +DATA/STBYCDB/6761DB93F6D36101E0535500A8C00734/DATAFILE/undotbs1.286.970764753   RECOVER

alter system set standby_file_management='auto';

Once all the files are renamed, go to step 4.

3.2 The second method is restoring to the files onto the PDB on standby while connected to the standby CDB. This method uses FROM SERVICE caluse to get the datafile from primary.
  3.2.1. Disable redo apply on standby

DGMGRL> edit database stbycdb set state='apply-off';
Succeeded.

  3.2.2. From the standby CDB run the following RMAN commadns(PRODCDBTNSis a TNS entry to the primary CDB).

RMAN> run {
 set newname for pluggable database pdb2 to new;
 restore pluggable database pdb2 from service PRODCDBTNS;
 switch datafile all;
}

executing command: SET NEWNAME

Starting restore at 16-MAR-18
using target database control file instead of recovery catalog
allocated channel: ORA_DISK_1
channel ORA_DISK_1: SID=396 device type=DISK

channel ORA_DISK_1: starting datafile backup set restore
channel ORA_DISK_1: using network backup set from service PRODCDBTNS
channel ORA_DISK_1: specifying datafile(s) to restore from backup set
channel ORA_DISK_1: restoring datafile 00173 to +DATA
channel ORA_DISK_1: restore complete, elapsed time: 00:00:07
channel ORA_DISK_1: starting datafile backup set restore
channel ORA_DISK_1: using network backup set from service PRODCDBTNS
channel ORA_DISK_1: specifying datafile(s) to restore from backup set
channel ORA_DISK_1: restoring datafile 00174 to +DATA
channel ORA_DISK_1: restore complete, elapsed time: 00:00:07
channel ORA_DISK_1: starting datafile backup set restore
channel ORA_DISK_1: using network backup set from service PRODCDBTNS
channel ORA_DISK_1: specifying datafile(s) to restore from backup set
channel ORA_DISK_1: restoring datafile 00175 to +DATA
channel ORA_DISK_1: restore complete, elapsed time: 00:00:02
channel ORA_DISK_1: starting datafile backup set restore
channel ORA_DISK_1: using network backup set from service PRODCDBTNS
channel ORA_DISK_1: specifying datafile(s) to restore from backup set
channel ORA_DISK_1: restoring datafile 00176 to +DATA
channel ORA_DISK_1: restore complete, elapsed time: 00:00:01
Finished restore at 16-MAR-18

datafile 173 switched to datafile copy
input datafile copy RECID=18 STAMP=970925746 file name=+DATA/STBYCDB/67875857381C604EE0535500A8C01310/DATAFILE/system.274.970925743
datafile 174 switched to datafile copy
input datafile copy RECID=19 STAMP=970925755 file name=+DATA/STBYCDB/67875857381C604EE0535500A8C01310/DATAFILE/sysaux.278.970925751
datafile 175 switched to datafile copy
input datafile copy RECID=20 STAMP=970925758 file name=+DATA/STBYCDB/67875857381C604EE0535500A8C01310/DATAFILE/undotbs1.309.970925759
datafile 176 switched to datafile copy
input datafile copy RECID=21 STAMP=970925759 file name=+DATA/STBYCDB/67875857381C604EE0535500A8C01310/DATAFILE/users.263.970925759

Once RMAN commands complete follow step 4.

4. Once the datafiles are copied the next step is to enable recovery on the PDB. Stop redo apply if enabled and execute the recovery enable command on the PDB.

DGMGRL> edit database stbycdb set state='apply-off';
Succeeded.

SQL> alter session set container=pdb2;
Session altered.

SQL> select name, recovery_status from v$pdbs;

NAME     RECOVERY
-------- --------
PDB2     DISABLED

SQL> alter pluggable database enable recovery;
Pluggable database altered.

SQL> select name, recovery_status from v$pdbs;

NAME       RECOVERY
---------- --------
PDB2       ENABLED

DGMGRL> edit database stbycdb set state='APPLY-ON';
DGMGRL> show configuration

5. To add a temp file to the PDB, stop the redo apply and open the PDB in read only mode (stopping redo and opening PDB in read only mode doesn't require active data guard). Then add the temp file as before.



Creating a PDB with standbys=all (default) option and without active data guard
In this scenario the PDB is created with default standbys option, which will create the PDB in all the standbys, but the standby CDB is not using active data guard (running in mount mode). In this case once the PDB is created on the primary standby will encounter error and redo apply will stop for the standby CDB.
1. PDB created on primary

SQL> create pluggable database pdb1 admin user pdb1 identified by pdb1;
Pluggable database created.

SQL> alter pluggable database pdb1 open;
Pluggable database altered.

2. On standby alert log following could be seen. Note the stopping of MRP

2018-03-14T17:28:31.871065+00:00
Recovery created pluggable database PDB1
*****************************************
WARNING: The converted filename '+DATA/stbycdb/67626414ddd22ec4e0535500a8c0cddd/datafile/system.293.970761273'
         is an ASM fully qualified filename.
         Changing the filename to '+DATA/MUST_RENAME_THIS_DATAFILE_0.4294967295.4294967295'.
         Please rename it accordingly.
*****************************************
2018-03-14T17:28:33.593037+00:00
Recovery copied files for tablespace SYSTEM
Recovery successfully copied file +DATA/STBYCDB/67626414DDD22EC4E0535500A8C0CDDD/DATAFILE/system.292.970766911 from +DATA/STBYCDB/6325282AC695380EE0535500A8C0D89D/DATAFILE/system.281.968524885
PDB1(4):*****************************************
PDB1(4):WARNING: The converted filename '+DATA/stbycdb/67626414ddd22ec4e0535500a8c0cddd/datafile/system.293.970761273'
PDB1(4):         is an ASM fully qualified filename.
PDB1(4):         Changing the filename to '+DATA/MUST_RENAME_THIS_DATAFILE_27685.4294967295.4294967295'.
PDB1(4):         Please rename it accordingly.
PDB1(4):*****************************************
PDB1(4):*****************************************
PDB1(4):WARNING: The converted filename '+DATA/stbycdb/67626414ddd22ec4e0535500a8c0cddd/datafile/system.293.970761273'
PDB1(4):         is an ASM fully qualified filename.
PDB1(4):         Changing the filename to '+DATA/MUST_RENAME_THIS_DATAFILE_144.4294967295.4294967295'.
PDB1(4):         Please rename it accordingly.
PDB1(4):*****************************************
2018-03-14T17:28:33.720328+00:00
PDB1(4):Errors in file /opt/app/oracle/diag/rdbms/stbycdb/stbycdb/trace/stbycdb_pr00_5655.trc:
ORA-01516: nonexistent log file, data file, or temporary file "+DATA/MUST_RENAME_THIS_DATAFILE_144.4294967295.4294967295" in the current container
PDB1(4):Recovery was unable to create the file as:
PDB1(4):'+DATA/STBYCDB/67626414DDD22EC4E0535500A8C0CDDD/DATAFILE/system.292.970766911'
PDB1(4):*****************************************
PDB1(4):WARNING: The converted filename '+DATA/stbycdb/67626414ddd22ec4e0535500a8c0cddd/datafile/system.293.970761273'
PDB1(4):         is an ASM fully qualified filename.
PDB1(4):         Changing the filename to '+DATA/MUST_RENAME_THIS_DATAFILE_144.4294967295.4294967295'.
PDB1(4):         Please rename it accordingly.
PDB1(4):*****************************************
MRP0: Background Media Recovery terminated with error 1274
2018-03-14T17:28:33.848228+00:00
Errors in file /opt/app/oracle/diag/rdbms/stbycdb/stbycdb/trace/stbycdb_pr00_5655.trc:
ORA-01274: cannot add data file that was originally created as '+DATA/PRODCDB/67626414DDD22EC4E0535500A8C0CDDD/DATAFILE/system.293.970761273'
Managed Standby Recovery not using Real Time Apply
Recovery interrupted!
Recovery stopped due to failure in applying recovery marker (opcode 17.34).
Datafiles are recovered to a consistent state at change 9544096 but controlfile could be ahead of datafiles.
2018-03-14T17:28:34.163336+00:00
Errors in file /opt/app/oracle/diag/rdbms/stbycdb/stbycdb/trace/stbycdb_pr00_5655.trc:
ORA-01274: cannot add data file that was originally created as '+DATA/PRODCDB/67626414DDD22EC4E0535500A8C0CDDD/DATAFILE/system.293.970761273'
2018-03-14T17:28:34.265287+00:00
MRP0: Background Media Recovery process shutdown (stbycdb)
2018-03-14T17:28:34.896975+00:00
Errors in file /opt/app/oracle/diag/rdbms/stbycdb/stbycdb/trace/stbycdb_m000_5889.trc:
ORA-01110: data file 130: '/opt/app/oracle/product/12.2.0/dbhome_1/dbs/UNNAMED00130'
ORA-01565: error in identifying file '/opt/app/oracle/product/12.2.0/dbhome_1/dbs/UNNAMED00130'
ORA-27037: unable to obtain file status
Linux-x86_64 Error: 2: No such file or directory
Additional information: 7
2018-03-14T17:28:35.631197+00:00
Errors in file /opt/app/oracle/diag/rdbms/stbycdb/stbycdb/trace/stbycdb_m000_5889.trc:
ORA-01110: data file 131: '/opt/app/oracle/product/12.2.0/dbhome_1/dbs/UNNAMED00131'
ORA-01565: error in identifying file '/opt/app/oracle/product/12.2.0/dbhome_1/dbs/UNNAMED00131'
ORA-27037: unable to obtain file status
Linux-x86_64 Error: 2: No such file or directory
Additional information: 7
2018-03-14T17:28:36.057283+00:00
Errors in file /opt/app/oracle/diag/rdbms/stbycdb/stbycdb/trace/stbycdb_m000_5889.trc:
ORA-01110: data file 132: '/opt/app/oracle/product/12.2.0/dbhome_1/dbs/UNNAMED00132'
ORA-01565: error in identifying file '/opt/app/oracle/product/12.2.0/dbhome_1/dbs/UNNAMED00132'
ORA-27037: unable to obtain file status
Linux-x86_64 Error: 2: No such file or directory
Additional information: 7
2018-03-14T17:28:36.654044+00:00
Errors in file /opt/app/oracle/diag/rdbms/stbycdb/stbycdb/trace/stbycdb_m000_5889.trc:
ORA-01110: data file 133: '/opt/app/oracle/product/12.2.0/dbhome_1/dbs/UNNAMED00133'
ORA-01565: error in identifying file '/opt/app/oracle/product/12.2.0/dbhome_1/dbs/UNNAMED00133'
ORA-27037: unable to obtain file status
Linux-x86_64 Error: 2: No such file or directory
Additional information: 7
2018-03-14T17:28:37.161034+00:00
Errors in file /opt/app/oracle/diag/rdbms/stbycdb/stbycdb/trace/stbycdb_m000_5889.trc:
ORA-01110: data file 144: '/opt/app/oracle/product/12.2.0/dbhome_1/dbs/UNNAMED00144'
ORA-01565: error in identifying file '/opt/app/oracle/product/12.2.0/dbhome_1/dbs/UNNAMED00144'

Check DG Broker status

DGMGRL> show configuration

Configuration - dg12c2

  Protection Mode: MaxAvailability
  Members:
  prodcdb - Primary database
    stbycdb - Physical standby database
      Error: ORA-16810: multiple errors or warnings detected for the member

Fast-Start Failover: DISABLED

Configuration Status:
ERROR   (status updated 5669 seconds ago)

DGMGRL> show database stbycdb

Database - stbycdb

  Role:               PHYSICAL STANDBY
  Intended State:     APPLY-ON
  Transport Lag:      0 seconds (computed 0 seconds ago)
  Apply Lag:          1 minute 49 seconds (computed 0 seconds ago)
  Average Apply Rate: 4.00 KByte/s
  Real Time Query:    OFF
  Instance(s):
    stbycdb

  Database Error(s):
    ORA-16766: Redo Apply is stopped

  Database Warning(s):
    ORA-16853: apply lag has exceeded specified threshold

Database Status:
ERROR

3. It was noticed several times that in this situation the PDB that's been created doens't appear on the standby CDB. However the PDB becomes visible after the restart of the standby CDB.

SQL> show pdbs

    CON_ID CON_NAME                       OPEN MODE  RESTRICTED
---------- ------------------------------ ---------- ----------
         2 PDB$SEED                       MOUNTED
         3 PDBDEV                         MOUNTED

SQL> shutdown immediate;
SQL> startup mount;

show pdbs
SQL>
    CON_ID CON_NAME                       OPEN MODE  RESTRICTED
---------- ------------------------------ ---------- ----------
         2 PDB$SEED                       MOUNTED
         3 PDBDEV                         MOUNTED
         4 PDB1                           MOUNTED

Unlike the previous case the recovery status of the PDB will be enabled.

SQL> select name,recovery_status from v$pdbs;

NAME       RECOVERY
---------- --------
PDB1       ENABLED

4. Querying the datafiles for the PDB shows only one unnamed datafile.

SQL> alter session set container=pdb1;
Session altered.

SQL> select file#,name from v$datafile;

     FILE# NAME
---------- ------------------------------------------------------------
       144 /opt/app/oracle/product/12.2.0/dbhome_1/dbs/UNNAMED00144

5. Copy the datafile 144 from primary to standby

rman target / auxiliary sys/prodcdbdb@stbytns

connected to target database: PRODCDB (DBID=2963914998)
connected to auxiliary database: PRODCDB (DBID=2963914998, not open)

RMAN> backup as copy datafile 144 auxiliary format '+DATA';

Starting backup at 14-MAR-18
using target database control file instead of recovery catalog
allocated channel: ORA_DISK_1
channel ORA_DISK_1: SID=30 device type=DISK
channel ORA_DISK_1: starting datafile copy
input datafile file number=00144 name=+DATA/PRODCDB/67626414DDD22EC4E0535500A8C0CDDD/DATAFILE/system.293.970761273
output file name=+DATA/STBYCDB/67626414DDD22EC4E0535500A8C0CDDD/DATAFILE/system.275.970767465 tag=TAG20180314T160346
channel ORA_DISK_1: datafile copy complete, elapsed time: 00:00:07
Finished backup at 14-MAR-18

Alternatively could use use the FROM SERVICE clause to get the datafile from primary while connected to standby as target (output below is from a different PDB). This will eliminate the need for manual renaming and changing standby file management setting.

rman target /

connected to target database: PRODCDB (DBID=2963914998, not open)

run {
 set newname for datafile 30 to new;
 restore datafile 30 from service PRODCDBTNS;
 switch datafile 30;
 }

executing command: SET NEWNAME

Starting restore at 19-MAR-18
using channel ORA_DISK_1

channel ORA_DISK_1: starting datafile backup set restore
channel ORA_DISK_1: using network backup set from service PRODCDBTNS
channel ORA_DISK_1: specifying datafile(s) to restore from backup set
channel ORA_DISK_1: restoring datafile 00030 to +DATA
channel ORA_DISK_1: restore complete, elapsed time: 00:00:07
Finished restore at 19-MAR-18

datafile 30 switched to datafile copy
input datafile copy RECID=47 STAMP=971197093 file name=+DATA/STBYCDB/67C6738768FC06BAE0535500A8C0138A/DATAFILE/sysaux.304.971197089

6. Once the file is copied to standby, rename the un-named datafiles (if files were copied using auxiliary channel)

SQL> alter system set standby_file_management='manual';
System altered.

SQL> alter session set container=pdb1;
Session altered.

SQL> alter database rename file  '/opt/app/oracle/product/12.2.0/dbhome_1/dbs/UNNAMED00144' to '+DATA/STBYCDB/67626414DDD22EC4E0535500A8C0CDDD/DATAFILE/system.275.970767465';
Database altered.

SQL>  alter system set standby_file_management='auto';
System altered.
    
DGMGRL>  edit database stbycdb set state='APPLY-ON';
Succeeded.

7. Starting of the redo apply on last step cause the next data file to be added to the PDB as an unnamed datafile and redo apply will stop with an error.

SQL> alter session set container=pdb1;
SQL>  select file#,name from v$datafile;

     FILE# NAME
---------- --------------------------------------------------------------------------------
       144 +DATA/STBYCDB/67626414DDD22EC4E0535500A8C0CDDD/DATAFILE/system.275.970767465
       145 /opt/app/oracle/product/12.2.0/dbhome_1/dbs/UNNAMED00145

PDB1(4):Recovery was unable to create the file as:
PDB1(4):'+DATA/STBYCDB/67626414DDD22EC4E0535500A8C0CDDD/DATAFILE/sysaux.286.970767625'
PDB1(4):*****************************************
PDB1(4):WARNING: The converted filename '+DATA/stbycdb/67626414ddd22ec4e0535500a8c0cddd/datafile/sysaux.295.970761273'
PDB1(4):         is an ASM fully qualified filename.
PDB1(4):         Changing the filename to '+DATA/MUST_RENAME_THIS_DATAFILE_145.4294967295.4294967295'.
PDB1(4):         Please rename it accordingly.
PDB1(4):*****************************************
MRP0: Background Media Recovery terminated with error 1274

8. To resolve the issue copy the next datafile and rename the unnamed datafile.

RMAN> backup as copy datafile 145 auxiliary format '+DATA';

Starting backup at 14-MAR-18
using channel ORA_DISK_1
channel ORA_DISK_1: starting datafile copy
input datafile file number=00145 name=+DATA/PRODCDB/67626414DDD22EC4E0535500A8C0CDDD/DATAFILE/sysaux.295.970761273
output file name=+DATA/STBYCDB/67626414DDD22EC4E0535500A8C0CDDD/DATAFILE/sysaux.282.970767731 tag=TAG20180314T160811
channel ORA_DISK_1: datafile copy complete, elapsed time: 00:00:07
Finished backup at 14-MAR-18

SQL> alter system set standby_file_management='manual';
System altered.

SQL> alter database rename file '/opt/app/oracle/product/12.2.0/dbhome_1/dbs/UNNAMED00145' to '+DATA/STBYCDB/67626414DDD22EC4E0535500A8C0CDDD/DATAFILE/sysaux.282.970767731';
Database altered.

SQL> select file#,name from v$datafile;

     FILE# NAME
---------- --------------------------------------------------------------------------------
       144 +DATA/STBYCDB/67626414DDD22EC4E0535500A8C0CDDD/DATAFILE/system.275.970767465
       145 +DATA/STBYCDB/67626414DDD22EC4E0535500A8C0CDDD/DATAFILE/sysaux.282.970767731

SQL>  alter system set standby_file_management='auto';

DGMGRL>  edit database stbycdb set state='APPLY-ON';
Succeeded

9. Same as before starting redo apply will fail on the next datafile.

DGMGRL>  edit database stbycdb set state='APPLY-ON';    
PDB1(4):*****************************************
PDB1(4):WARNING: The converted filename '+DATA/stbycdb/67626414ddd22ec4e0535500a8c0cddd/datafile/undotbs1.294.970761273'
PDB1(4):         is an ASM fully qualified filename.
PDB1(4):         Changing the filename to '+DATA/MUST_RENAME_THIS_DATAFILE_146.4294967295.4294967295'.
PDB1(4):         Please rename it accordingly.
PDB1(4):*****************************************

SQL> select file#,name from v$datafile;

     FILE# NAME
---------- --------------------------------------------------------------------------------
       144 +DATA/STBYCDB/67626414DDD22EC4E0535500A8C0CDDD/DATAFILE/system.275.970767465
       145 +DATA/STBYCDB/67626414DDD22EC4E0535500A8C0CDDD/DATAFILE/sysaux.282.970767731
       146 /opt/app/oracle/product/12.2.0/dbhome_1/dbs/UNNAMED00146

10. Copy all the files part of the PDB to standby as they error. This PDB only needed to copy 3 files.

RMAN> backup as copy datafile 146  auxiliary format '+DATA';

Starting backup at 14-MAR-18
using channel ORA_DISK_1
channel ORA_DISK_1: starting datafile copy
input datafile file number=00146 name=+DATA/PRODCDB/67626414DDD22EC4E0535500A8C0CDDD/DATAFILE/undotbs1.294.970761273
output file name=+DATA/STBYCDB/67626414DDD22EC4E0535500A8C0CDDD/DATAFILE/undotbs1.277.970767877 tag=TAG20180314T161038
channel ORA_DISK_1: datafile copy complete, elapsed time: 00:00:07
Finished backup at 14-MAR-18

SQL> alter system set standby_file_management='manual';
System altered.

SQL> alter database rename file '/opt/app/oracle/product/12.2.0/dbhome_1/dbs/UNNAMED00146' to '+DATA/STBYCDB/67626414DDD22EC4E0535500A8C0CDDD/DATAFILE/undotbs1.277.970767877';
Database altered.

SQL> select file#,name from v$datafile;

     FILE# NAME
---------- --------------------------------------------------------------------------------
       144 +DATA/STBYCDB/67626414DDD22EC4E0535500A8C0CDDD/DATAFILE/system.275.970767465
       145 +DATA/STBYCDB/67626414DDD22EC4E0535500A8C0CDDD/DATAFILE/sysaux.282.970767731
       146 +DATA/STBYCDB/67626414DDD22EC4E0535500A8C0CDDD/DATAFILE/undotbs1.277.970767877

11. Once all the files are copied the redo apply will continue without any errors. Since PDB already has the recovery status enable no other steps are necessary.

DGMGRL> edit database stbycdb set state='APPLY-ON';
Succeeded.
DGMGRL> show configuration

Configuration - dg12c2

  Protection Mode: MaxAvailability
  Members:
  prodcdb - Primary database
    stbycdb - Physical standby database

Fast-Start Failover: DISABLED

Configuration Status:
SUCCESS   (status updated 5648 seconds ago)