Saturday, August 14, 2021

ORA-16713: The Oracle Data Guard broker command timed out When Changing LogXptMode

After setting up a new dataguard configuration and broker the change of log transport mode to sync fail as below.
DGMGRL> edit database mystby set property LogXptMode='SYNC';
Error: ORA-16713: The Oracle Data Guard broker command timed out.

Failed.
Sync mode was needed to change the protection mode. There was no other DG issue and redo transport and log apply was working fine. Transport mode change was only failing on the standby at that time. However, the sync mode parameters gets setup on the primary's log archive destination. Log transport mode of the two instances looks like below due to this failure.
  Transport-Related Property Settings:
    Property                        myprod Value             mystby Value
    LogXptMode                      SYNC                     ASYNC
MOS doc 2007507.1 says this is due to rsm* process hanging and to kill it so it will respawn. This didn't help.
 ps ax | grep rsm
26686 pts/3    S+     0:00 grep --color=auto rsm
30386 ?        Ss     0:01 ora_rsm0_myprod
$ kill -9 30386

 ps ax | grep rsm
27157 ?        Ss     0:00 ora_rsm0_myprod  <--------- new process spawn
27161 pts/3    S+     0:00 grep --color=auto rsm

DGMGRL> edit database mystby set property LogXptMode='SYNC';
Error: ORA-16713: The Oracle Data Guard broker command timed out.

Failed.
MOS doc 1322877.1 suggested increasing OperationTimeout. That too didn't help.
DGMGRL>  EDIT CONFIGURATION SET PROPERTY OperationTimeout=600;
Property "operationtimeout" updated

DGMGRL> edit database mystby set property LogXptMode='SYNC';
Error: ORA-16713: The Oracle Data Guard broker command timed out.

Failed.
Few other MOS suggested clearing up ARDCI location. But those docs were related to DG broker hanging during validation. In this case validation ran fine without any issue.



Finally what resolved the issue was disabling the configuration and updating the log transport mode and enabling it again (maybe this respwaned the rsm cleanly than killing it).
DGMGRL> disable configuration;
Disabled.

DGMGRL> edit database mystby set property LogXptMode='SYNC';
Property "logxptmode" updated

DGMGRL> enable configuration;
Enabled.
This worked as expected. The transprot node changes are now visible on the log_archive_dest on the primary.
log_archive_dest_3   string service="mystbytns", SYNC AFFIRM delay=0 optional c

  Transport-Related Property Settings:
    Property                        myprod Value             mystby Value
    LogXptMode                      SYNC                     SYNC
After this it was possible to change the protection mode.