Monday, April 5, 2021

LOG_FILE_NAME_CONVERT is not defined, stop clearing ORLs

LOG_FILE_NAME_CONVERT name parameter is used to correct the file location for log files if the database is being created from another database and source database had a different directory structure. This is used in data guard configuration when standby and primary has different file structure. However, there are few limitations to using this parameter and one of them is that it cannot be used with Oracle managed files or OMF (refer 1367014.1).
However, an odd behaviour was observed in a data guard configuration where even when OMF is used there database was looking for LOG_FILE_NAME_CONVERT and if not set stopping the automatic logfile clearing.
The databases (both primary and standby) had following OMF related parameters
NAME                                 TYPE        VALUE
------------------------------------ ----------- -------
db_create_file_dest                  string      +DATA
db_create_online_log_dest_1          string
db_create_online_log_dest_2          string
db_create_online_log_dest_3          string
db_create_online_log_dest_4          string
db_create_online_log_dest_5          string
db_recovery_file_dest                string      +FRA
The *convert* parameters were not set.
NAME                                 TYPE        VALUE
------------------------------------ ----------- ------------
db_file_name_convert                 string
log_file_name_convert                string
pdb_file_name_convert                string
During the active database duplication the duplication processed detected use of OMF as seen from below output.
...
sql statement: alter system set  local_listener =  ''LISTENER_DGTEST2'' comment= '''' scope=spfile

Oracle instance shut down

connected to auxiliary database (not started)
Oracle instance started

Total System Global Area    9965665616 bytes

Fixed Size                    12684624 bytes
Variable Size               1543503872 bytes
Database Buffers            8388608000 bytes
Redo Buffers                  20869120 bytes
duplicating Online logs to Oracle Managed File (OMF) location
duplicating Datafiles to Oracle Managed File (OMF) location

contents of Memory Script:
{
   sql clone "alter system set  control_files =
  ''+DATA/DGTEST2/CONTROLFILE/current.264.1067949389'', ''+FRA/DGTEST2/CONTROLFILE/current.418.1067949389'' comment=
  ...
Data guard consists of two databases.
DGMGRL>Configuration - test_dg

  Protection Mode: MaxAvailability
  Members:
  dgtest  - Primary database
    dgtest2 - Physical standby database

Fast-Start Failover:  Disabled

Configuration Status:
SUCCESS
With these setting on, after a switchover following was seen on the alert log of the old primary (new standby)
2021-03-23T12:55:20.314959+00:00
TT02 (PID:8514): LOG_FILE_NAME_CONVERT is not defined, stop clearing ORLs
Validate of the old primary (new standby) showed that online redo logs are not cleared
DGMGRL> validate database dgtest

  Database Role:     Physical standby database
  Primary Database:  dgtest2

  Ready for Switchover:  Yes
  Ready for Failover:    Yes (Primary Running)

  Managed by Clusterware:
    dgtest2:  YES
    dgtest :  YES

  Log Files Cleared:
    dgtest2 Standby Redo Log Files:  Cleared
    dgtest Online Redo Log Files:    Not Cleared
    dgtest Standby Redo Log Files:   Available
It was the same case when the new primary (old standby) did a switchover as well.
2021-03-23T13:02:25.800263+00:00
TT02 (PID:9657): LOG_FILE_NAME_CONVERT is not defined, stop clearing ORLs

DGMGRL> validate database dgtest2

  Database Role:     Physical standby database
  Primary Database:  dgtest

  Ready for Switchover:  Yes
  Ready for Failover:    Yes (Primary Running)

  Managed by Clusterware:
    dgtest :  YES
    dgtest2:  YES

  Log Files Cleared:
    dgtest Standby Redo Log Files:   Cleared
    dgtest2 Online Redo Log Files:   Not Cleared
    dgtest2 Standby Redo Log Files:  Available


This behaviour was observed even after applying release updates (RU) 19.3, 19.4, 19.5, 19.6 and 19.7. If the databases are in one of those RUs then only way to achieve the automatic ORL clearing is to explicitly set the log file name convert parameter. For example in this case for the primary DB dgtest
alter system set log_file_name_convert='/dgtest2/','/dgtest/' scope=spfile;
Once log_file_name_convert is set the automatic ORL clearence was observed even though MOS doc states log_file_name_convert cannot be used with OMF.

The issue is fixed on RU 19.8 and up (tested on 19.9 , 19.10). Once these RU are applied automatic ORL clearing happens as expected. If previously (in one of the affected RUs) log_file_name_convert was set then it could be removed once RU 19.8 or up is applied.
When automatic ORL clearing happens following could be seen on the alert log of the old primary
2021-03-23T13:20:23.135283+00:00
TT02 (PID:10641): Waiting for all non-current ORLs to be archived
2021-03-23T13:20:23.135414+00:00
TT02 (PID:10641): All non-current ORLs have been archived
2021-03-23T13:20:23.273380+00:00
TT02 (PID:10641): Clearing ORL LNO:1 +DATA/DGTEST/ONLINELOG/group_1.264.1067878075
2021-03-23T13:20:24.512745+00:00
TT02 (PID:10641): Clearing ORL LNO:1 complete
TT02 (PID:10641): Clearing ORL LNO:2 +DATA/DGTEST/ONLINELOG/group_2.262.1067878077
Clearing online log 2 of thread 1 sequence number 34
2021-03-23T13:20:26.525504+00:00
TT02 (PID:10641): Clearing ORL LNO:2 complete
TT02 (PID:10641): Clearing ORL LNO:3 +DATA/DGTEST/ONLINELOG/group_3.261.1067878079
Clearing online log 3 of thread 1 sequence number 35
2021-03-23T13:20:28.163683+00:00
TT02 (PID:10641): Clearing ORL LNO:3 complete
2021-03-23T13:20:33.174192+00:00
TT02 (PID:10641): Waiting for all non-current ORLs to be archived
2021-03-23T13:20:33.174386+00:00
TT02 (PID:10641): All non-current ORLs have been archived
Validating the standby shows no log files to be cleared.
DGMGRL> validate database dgtest2

  Database Role:     Physical standby database
  Primary Database:  dgtest

  Ready for Switchover:  Yes
  Ready for Failover:    Yes (Primary Running)

  Managed by Clusterware:
    dgtest :  YES
    dgtest2:  YES

Useful Metalink notes
Usage and Limitation of db_file_name_convert and log_file_name_convert [ID 1367014.1]

Monday, March 29, 2021

Creating a Standby From Backup of Another Standby

A previous post showed the steps for adding a physical standby to an existing data guard configuration. In it the creation of the standby was done using active database duplication. This post shows the steps for creating a standby using backups of another standby. This situation is useful if copying backups from primary to new standby location is not feasbile due to the backup size and geographical distance between the two sites.
Current DG configuration is shown below.
DGMGRL> show configuration

Configuration - dbx_dg

  Protection Mode: MaxAvailability
  Members:
  ppdbxdb1- Primary database
    ppdbxdb2 - Physical standby database
    ppdbxfs1 - Far sync instance
      ppdbxdb6 - Physical standby database

Fast-Start Failover:  Disabled

Configuration Status:
SUCCESS
A new standby instance named ppdbxdb5 is added to the same region as ppdbxdb6 and redo is shipped via far sync instance ppdbxfs1. The post doesn't show the pre-reqs that need to be completed before adding the new standby. Refer the previous post for those steps.
The database duplication in this case is done connecting the auxiliary database. This method is also used before on a data guard creation.
On the existing standby database (ppdbxdb6 in this case) run following to create backups of spfile, controlfile, database and archive logs.
backup spfile format '/backup/spbackup.bkp';
backup current controlfile for standby format '/backup/stdbycontro.ctl';
backup database format '/backup/dbbackup%U' plus archivelog format '/backup/archbackup%U' delete all input;
switch few log files in the primary
sql 'alter system switch logfile';
sql 'alter system switch logfile';
sql 'alter system switch logfile';
and backup the archive logs on the standby
backup archivelog all format '/backup/archbkp%U' delete all input;
Copy the backup files created earlier to a location on the new standby host. Exclude any controfile backups (explicit backups or auto backups) being copied to new location except for the controlfile backup taken above. If not during the duplication, due to selecting a controlfile that has a higher checkpoint sequence following error will be thrown.
RMAN-03002: failure of Duplicate Db command at 03/18/2021 12:04:04
RMAN-05501: aborting duplication of target database
RMAN-05507: standby control file checkpoint (5381793) is more recent than duplication point-in-time (5379510)
Once backups are copied start the new standby instance with nomount option.


Connect to the standby using rman auxiliary connection and run the duplication comamnd. In this case the parameter conversion occurs between existing standby DB ppdbxdb6 and new standby ppdbxdb5. Both these databses are terminal standby as such there's no reference to other databases. Only
rman auxiliary sys@ppdbxdb5tns

run {
allocate auxiliary channel ch1 device type disk;
allocate auxiliary channel ch2 device type disk;
allocate auxiliary channel ch3 device type disk;
allocate auxiliary channel ch4 device type disk;
allocate auxiliary channel ch5 device type disk;
allocate auxiliary channel ch6 device type disk;
allocate auxiliary channel ch7 device type disk;
allocate auxiliary channel ch8 device type disk;
duplicate database for standby
spfile
parameter_value_convert 'ppdbxdb6','ppdbxdb5','PPdbxDB6','PPdbxDB5'
set db_name='ppdbxdb1'
set db_unique_name='ppdbxdb5'
set log_file_name_convert='/ppdbxdb1/','/ppdbxdb5/','/ppdbxdb2/','/ppdbxdb5/'
set log_archive_max_processes='10'
set fal_server='ppdbxDB1TNS','ppdbxDB2TNS','ppdbxDB6TNS','ppdbxFS1TNS'
set log_archive_dest_1='location=use_db_recovery_file_dest valid_for=(all_logfiles,all_roles) db_unique_name=ppdbxdb5 NOREOPEN ALTERNATE=log_archive_dest_2' 
set local_listener='LISTENER_PPdbxDB5,DGLISTENER_PPdbxDB5'
set dg_broker_start='false'
reset log_archive_dest_3
reset log_archive_dest_4
BACKUP LOCATION '/backup' dorecover nofilenamecheck;
}


At the end of the duplication processes the new standby is ready to receive and apply redo from the primary and could be added to the data guard broker configuraiton.
show configuration

Configuration - dbx_dg

  Protection Mode: MaxAvailability
  Members:
  ppdbxdb1- Primary database
    ppdbxdb2 - Physical standby database
    ppdbxfs1 - Far sync instance
      ppdbxdb5 - Physical standby database
      ppdbxdb6 - Physical standby database

Fast-Start Failover:  Disabled

Configuration Status:
SUCCESS
Useful Metalink Notes
Step by Step method to create Primary/Standby Database from Standby Backup [ID 1604251.1]

Related Posts
Adding a New Physical Standby to Existing Data Guard Setup
Oracle Data Guard on 12.2 CDB with Oracle Restart

Monday, March 22, 2021

Removing a Standby Database From a Data Guard Configuration With RedoRoutes

There is an earlier post which shows how to remove a standby database from a data guard configuration when redo routes are not used. This post shows steps for removing standby databases when redo routes are in use.
The existing data guard setup is as follows.
DGMGRL> show configuration

Configuration - db_dg

  Protection Mode: MaxAvailability
  Members:
  db2  - Primary database
    db1  - Physical standby database
    db3  - Physical standby database
      db4  - Physical standby database (receiving current redo)
    fs1  - Far sync instance
      db5  - Physical standby database
      db6  - Physical standby database
      db7  - Physical standby database
      db8  - Physical standby database
      db9  - Physical standby database
      db10 - Physical standby database

  Members Not Receiving Redo:
  fs2  - Far sync instance (alternate of fs1)

Fast-Start Failover:  Disabled

Configuration Status:
SUCCESS   (status updated 57 seconds ago)
It was decided to remove standby databases db7-db10 and far sync instance fs2 from the data guard configuration. Simply running the remove database command will result in an error due to references in the redoroutes for the standby databases that are being removed.
DGMGRL> remove database db10;
Error: ORA-16691: cannot remove a configuration member that is specified in a RedoRoutes property

Failed.
If the redoroute property of the databases that will remain the data guard configuration is to be checked then references to databases being removed could be found.
DGMGRL> show database db2 redoroutes
  RedoRoutes = '(LOCAL : ( db1 SYNC), ( db3 SYNC PRIORITY=1, db4 SYNC PRIORITY=2),( fs1 SYNC PRIORITY=1, fs2 SYNC PRIORITY=2, db5 ASYNC, db6 ASYNC, db7 ASYNC, db8 ASYNC, db9 ASYNC, db10 ASYNC))'

Therefore before the standby databases are removed update the redoroutes of the databases that will remain in the data guard, by removing any references to those standby databsaes that are being removed. One exmaple redoroute update for above data guard configuration is given below.
edit database  db1 set property redoroutes='(LOCAL : ( db2 SYNC), ( db3 SYNC PRIORITY=1,  db4 SYNC PRIORITY=2),( fs1 SYNC PRIORITY=1,  fs2 SYNC PRIORITY=2,  db5 ASYNC, db6 ASYNC))( db3 :  db2 ASYNC)( db4 :  db2 ASYNC)';




Once all the redo routes are updated the data guard configuratoin may show error state while it revalidate, however, the error state could be ignored and could proceed with the removal of the standby databases.
DGMGRL> show configuration

Configuration - db_dg

  Protection Mode: MaxAvailability
  Members:
  db2  - Primary database
    db1  - Physical standby database
    db3  - Physical standby database
      db4  - Physical standby database (receiving current redo)
    fs1  - Far sync instance
      db5  - Physical standby database
      db6  - Physical standby database

  Members Not Receiving Redo:
  db7  - Physical standby database
    Error: ORA-16810: multiple errors or warnings detected for the member

  db8  - Physical standby database
    Error: ORA-16810: multiple errors or warnings detected for the member

  db9  - Physical standby database
    Error: ORA-16810: multiple errors or warnings detected for the member

  db10 - Physical standby database
    Error: ORA-16810: multiple errors or warnings detected for the member

  fs2  - Far sync instance (alternate of fs1)

Fast-Start Failover:  Disabled

Configuration Status:
ERROR   (status updated 71 seconds ago)

DGMGRL> remove database db10;
Removed database " db10" from the configuration
DGMGRL> remove database db9;
Removed database " db9" from the configuration
DGMGRL> remove database db8;
Removed database " db8" from the configuration
DGMGRL> remove database db7
Removed database " db7" from the configuration

DGMGRL> show configuration

Configuration - db_dg

  Protection Mode: MaxAvailability
  Members:
  db2 - Primary database
    db1 - Physical standby database
    db3 - Physical standby database
      db4 - Physical standby database (receiving current redo)
    fs1 - Far sync instance
      db5 - Physical standby database
      db6 - Physical standby database

  Members Not Receiving Redo:
  fs2 - Far sync instance (alternate of fs1)

Fast-Start Failover:  Disabled

Configuration Status:
SUCCESS   (status updated 54 seconds ago)
Similarly once all references to far sync instance (fs2) is removed from existing redoroutes that too could be removed from the data guard configuraiton. Redo route update without any reference to fs2 on one of the databases is given below.
edit database  db1 set property redoroutes='(LOCAL : ( db2 SYNC), ( db3 SYNC PRIORITY=1,  db4 SYNC PRIORITY=2),( fs1 SYNC PRIORITY=1,  db5 ASYNC, db6 ASYNC))( db3 :  db2 ASYNC)( db4 :  db2 ASYNC)';

Execute remove far_sync to remove the far sync instance.
DGMGRL> remove far_sync fs2;
Removed far sync instance " fs2" from the configuration
DGMGRL>  show configuration

Configuration - db_dg

  Protection Mode: MaxAvailability
  Members:
  db2 - Primary database
    db1 - Physical standby database
    db3 - Physical standby database
      db4 - Physical standby database (receiving current redo)
    fs1 - Far sync instance
      db5 - Physical standby database
      db6 - Physical standby database

Fast-Start Failover:  Disabled

Configuration Status:
SUCCESS   (status updated 64 seconds ago)

Related Post
Removing a Standby Database From a Data Guard Configuration