Far Sync Targets are not Discovered Correctly by OEM Auto Discovery Process

If there are far sync instances in a data guard configuration then those will not be correctly identified as far sync instances by the OEM target discovery process. There were two posts related to the same issue where far sync types were not listed for template apply. The other issue was related to data guard performance page where far sync instace graphs had error status.
Root cause for all of this is that auto discovery will assign target type of "database instance" for the far sync instance.

After an SR that spaned year and half Oracle support said that currently OEM auto discovery process is unable to correctly detect the far sync instance types. Their solution was to manaully add the far sync instance.



To manually add the far sync go to add target -> add target manually and select far sync type and the host where far sync resides.

Afterwards the far sync instaces has be views via all targets and filtering by far syncs. They will not appear under database targets which is the incorrect behaviour.

The data guard performance graphs for far sync will appear without any error status.

As far sync has been around over 8 years now and for OEM to not correctly identify them was very puzzling to say the least. The issue was first discovered on 13.4 and exists on the 13.5 (latest at the time of this post). After further deliberations Oracle has created an enhancement request (ER-34673700) to address this issue. Whether this ER will be accepted and worked upon is still unknown.

ORA-16856: transport lag could not be determined on Far Sync After Archive Logs Delete With DBRU 19.12 Applied

The behaviour mentioned in this post only started to appear after applying the DBRU 19.12. Prior to applying the DBRU there was no issue of ORA-16856 with the combination of conditions mentioned in the post. System has been in operation for sometime.
After applying DBRU 19.12 noticed increase number of alerts related to ORA-16856 coming from a data guard configuration that has far sync instance in it. Behaviour was observed not one but several such environments. On the data guard broker log of the primary DB entries similar to below could be seen.

2021-08-19T14:08:17.271+00:00
Data Guard Broker Status Summary:
Type Name Severity Status
Configuration tst_dg Warning ORA-16608: one or more members have warnings
Primary Database tstdb1 Success ORA-0: normal, successful completion
Physical Standby Database tstdb2 Success ORA-0: normal, successful completion
Far Sync Instance tstfs1 Warning ORA-16856: transport lag could not be determined
Physical Standby Database tstdb6 Success ORA-0: normal, successful completion
Physical Standby Database tstdb5 Success ORA-0: normal, successful completion

No ORA-16856 related entreis were shown on the far sync instance logs. The data guard broker goes to a warning state for several minutes and rectify itself aftwards.
Analysis of the data guard broker logs showed that ORA-16856 was starting at the same minute of the hour and every hour. Further investigation showed that that first occurence of ORA-16856 corresponds to archive log deletion on the far sync instance. Far sync had archive deletion policy set to applied on all standby (later it was found that archive log deletion policy has no relavence to this issue) and deleting all the archive logs.
There were no overlapping rman activities between far sync and primary or other standby instances. Neverthless the archive log deletion on far sync was shifted to a different time. After that the ORA-16856 error started to appear at the newly scheduled time for the archive log deletion on far sync. This confirmed that issue is caused by the archive log deletion on far sync.
At this stage, looked at when did the new pattern of ORA-16856 started. It was confirmed current pattern of (every hour at the same minute) ORA-16856 startd after the DBRU 19.12 was applied. As the system has been operational for number of years there was enough historic log entries to compare against to see if pattern was there before as well. It was not.
To further isolate the possible root cause, archive log deletion was stopped on far sync in one of the non critical environments that exhibit the behaviour. Once the archive log deletion was stopped there was no recurring ORA-16856 as before. This indicated the log archive deletion could be the possible root cause.
As the next step the log archive deletion command was changed from

delete noprompt archivelog all;

to

delete noprompt archivelog all completed before 'sysdate - 1/24';

With this not all the archive logs were getting deleted. Some were omitted from the deletion as they fell out side the completed before time period. After the rman comamnd change the occurence of periodic ORA-16856 also stopped.



After this it was possible to manually recreate the issue, at will, by running a full archive log deletion on the far sync. Issue doesn't happen when all archive logs are deleted on standby databases. It only happens when all archive logs are deleted on the far sync instance.
As a workaround to the problem, in production systems, on far sync instances the archive log deletion command was changed so that some archive logs would always remain.
Hypothesis was that to calculate the transport lag far sync instnace uses archive log entries. Removing all of them makes it unable to calculate the transport lag hence the ORA-16856.
Raised a SR and Oracle support asked to recreate the issue with 'TraceLevel' = 'SUPPORT' in the DG broker. With this trace level when the issue was recreated far sync DG broker log also output the line related to ORA-16856 as shown below.

2021-10-01T11:10:21.900+00:00 DMON: Checking critical status of this database.
2021-10-01T11:10:21.900+00:00 00000000 1128079456 DMON: Broker configuration file successfully loaded for EDIT CONFIGURATION RE/SET PROPERTY
2021-10-01T11:10:21.900+00:00 INSV: Reply received for message with
2021-10-01T11:10:21.901+00:00 req ID 1.1.1128079456, opcode EDIT_DRC_PROP, phase RESYNCH
2021-10-01T11:10:21.901+00:00 00000000 1128079456 DMON: rfm_release_chief_lock() called for EDIT_DRC_PROP, called from rfmspd
2021-10-01T11:11:50.183+00:00 RSM0: HEALTH CHECK WARNING: ORA-16856: transport lag could not be determined
2021-10-01T11:11:50.234+00:00 03001000 1447213588 Operation HEALTH_CHECK continuing with warning, status = ORA-16856
2021-10-01T11:11:50.235+00:00 03001000 1447213588 Operation HEALTH_CHECK continuing with warning, status = ORA-16856

SR has resulted in an internal bug 33427973. No permanent solution yet. SR continues.

Far Sync Instance Not Detected For Template Apply

A data guard configuration (same configuration mentioned in a previous post) is promoted to EM 13.4 and has serveral far sync instances.

The instance type is identified as far sync in the instance page and in data guard administer page.

A monitoring template was created with far sync as the instance type.

However, when the template is to be applied to far sync instances non is detected and following error is shown on the EM console.

Seems even though far sync is identified in all other places when it comes to template it as there are no far sync instances to monitor.
SR raised and status is "development working".

Related Post
Far Sync Instance Stats Not Shown on Data Guard Performance Page on Cloud Control 13.4

Far Sync Instance Stats Not Shown on Data Guard Performance Page on Cloud Control 13.4

Enterprise Manager Cloud control has a separate performance page for viewing data gaurd performance. This gives one page view of redo genration, apply rate and transport lags times. However, in a recent deployment it was observed that no stats related to far sync instances was shown on this page. Below is the screenshot of the data guard performance page (not all standby instances are shown).

The data guard performance page is not available out of the box. One has to run the verify configuration under data guard administration tag twice (yes, twice. refere 1484028.1) to get the views (redo generation, apply rate and trasport lag times) populated with graphs. However, no matter how many times verify configuration was run the far sync instance views were not populated.
It's understandable far sync instances will not have any stats for apply lag or apply rate as they don't do any applying. But at least transport lag which is relevant to far sync instances could be populated. This is shown with dgmgrl.

DGMGRL> show far_sync fs1

Far Sync Instance - fs1

Transport Lag: 0 seconds (computed 0 seconds ago)
Instance(s):
fs1

Far Sync Instance Status:
SUCCESS

During the verify configuration run the EM (assumed) add a new database property called "dbDisplayName" to the DG configuration. This could be observed on the data guard broker log file.

EDIT DATABASE db3 SET PROPERTY dbDisplayName = db1_host_name
Property "dbDisplayName" set to "db1_host_name" for member "db1"
EDIT DATABASE db3 SET PROPERTY dbDisplayName = db1_host_name completed successfully

This property value is used by the EM interally. The "show database" comamnd on dgmgrl will also show this as the "Enterprise Manager Name".

DGMGRL> show database db1

Database - db1

Enterprise Manager Name: db1_host_name
Role: PRIMARY
Intended State: TRANSPORT-ON
Instance(s):
db1

Database Status:
SUCCESS 

However, no such command was executed for far sync (i.e edit far_sync fs1 SET PROPERTY dbDisplayName = fs1_host_name).
On the emoms.log could see the following, possibly due to the missing dbDisplayName value on the far sync instance.

2020-08-26 09:20:49,225 [Thread-624744] WARN em.dataguard getDBTargetInfo.1183 - StandbyTargetService.getDBTargetInfo : no match for db_unique_name = fs1, sid = fs1
2020-08-26 09:21:12,020 [EMUI_09_19_58_/console/database/dataguard] WARN em.dataguard getDBTargetInfo.1183 - StandbyTargetService.getDBTargetInfo : no match for db_unique_name = fs1, sid = fs1

The error message shown on the page which states "1. Error: Unable to obtain Agent location for target pptbxfs1. You may not currently be logged in." is ruled out as sys login is used for far sync instead of dbsnmp on the EM. Far sync are in mount mode and need a user that can connect to a mounted instance (refer 1526818.1).
After a SR was rasied, following a 4 month investigation which involved demoing the observed behavior oracle support concluded "this is expected behavior for 'Farsync instance' details not shown in OEM because there is no views are available in Farsync instance like other instance to capture details from farsync to OMS to display in OEM".
This means transport lag stats related to far sync instances will not be visible on the data guard performance page until additional development is done on the OEM.

Setting Up Data Guard Broker for an Existing Data Guard Configuration with Far Sync

This post list the steps for setting up data guard broker configuration for an existing data guard configuration with far sync. The data guard configuration is the same setup that had far sync instances added earlier. The current setup of the data guard configuration is given below.

Before setting up the dg broker, patch 19399918 (included in 12.1.0.2.4) and 19571599 (at the time of writing only available for system patched with 12.1.0.2.4) must be applied on all oracle homes involved, this include oracle homes used for far sync instances as well. Without the patch there'll be issues with the dg configuration and the switchover will fail when the redo route has an alternate option. However the switchover works without the alternate option on the redo route.

DGMGRL> show database ent12c1 LogXptStatus
Error: ORA-16777: unable to find the destination entry of a standby database in V$ARCHIVE_DEST

DGMGRL> switchover to ent12c1s;
Performing switchover NOW, please wait...
Error: ORA-16778: redo transport

Another pre-req is clearing of log_archvie_dest_* parameters in all the instances (including far sync instances). Trying to create a dg configuration or add a (far_sync) instance will fail.

DGMGRL> create configuration ent12c1_dgb as primary database is ent12c1 connect identifier is ent12c1tns;
Error: ORA-16698: LOG_ARCHIVE_DEST_n parameter set for object to be added

DGMGRL> add database ent12c1s as connect identifier is ent12c1stns;
Error: ORA-16698: LOG_ARCHIVE_DEST_n parameter set for object to be added

DGMGRL> ADD FAR_SYNC fs12c1 AS CONNECT IDENTIFIER IS fs12c1tns;
Error: ORA-16698: LOG_ARCHIVE_DEST_n parameter set for object to be added

This is a new requirement introduced in 12c which states "as of Oracle Database 12c Release 1 (12.1), for all databases to be added to a broker configuration, any LOG_ARCHIVE_DEST_n parameters that have the SERVICE attribute set, but not the NOREGISTER attribute, must be cleared."
1. To add the dg broker configuration clear the existing log_archive_dest_* parameter. In the existing setup destination 2 and 3 were used for log shipping.

alter system set log_Archive_dest_2='' scope=both;
alter system set log_Archive_dest_3='' scope=both;
alter system set log_archive_dest_state_3='enable' scope=both;

2. Create the dg broker configuration by adding primary, standby and far sync instances. In this case ENT12C1 is the primary database while ENT12C1S is the standby and FS12C1 is the far sync instance the primary database ships redo synchronously. After a role switch when the ENT12C1S becomes the new primary then it uses FS12C1S as the far sync instance to transport redo synchronously.

create configuration ent12c1_dgb as primary database is ent12c1 connect identifier is ent12c1tns;
add database ent12c1s as connect identifier is ent12c1stns;
add FAR_SYNC fs12c1 as connect identifier is fs12c1tns;
add FAR_SYNC fs12c1s as connect identifier is fs12c1stns;

show configuration;

Configuration - ent12c1_dgb

  Protection Mode: MaxPerformance
  Members:
  ent12c1  - Primary database
    ent12c1s - Physical standby database
    fs12c1   - Far sync instance
    fs12c1s  - Far sync instance

Fast-Start Failover: DISABLED

Configuration Status:
DISABLED

3. Enable the configuration and far_sync.

enable configuration;
Enabled.
enable far_sync fs12c1;
Enabled.
enable far_sync fs12c1s;
Enabled.

4. Set the redo transport modes sync for far_sync (default is async) and appropriate redo routes for each instance. In this first instance the redo routes are set such that primary ship to far sync and far sync ships to standby. There's no redo route between primary and standby. This is different to earlier setup and will be changed in subsequent steps. It's also worth noting that "if a database receives redo from a database or far sync instance where the RedoRoutes property has been configured with a redo transport mode, then the mode specified by that RedoRoutes property value overrides the value of the LogXptMode property".

EDIT far_sync fs12c1 SET PROPERTY LogXptMode='SYNC';
EDIT far_sync fs12c1s SET PROPERTY LogXptMode='SYNC';

EDIT DATABASE ent12c1 SET PROPERTY 'RedoRoutes' = '(LOCAL : FS12C1 SYNC)';
EDIT FAR_SYNC fs12c1 SET PROPERTY 'RedoRoutes' = '(ent12c1 : ent12c1s ASYNC)';
EDIT DATABASE ent12c1s SET PROPERTY 'RedoRoutes' = '(LOCAL : FS12C1S SYNC)';
EDIT FAR_SYNC fs12c1s SET PROPERTY 'RedoRoutes' = '(ent12c1s : ent12c1 ASYNC)';

5. Verify the configuration and validate the database and far sync instances

DGMGRL> show configuration

Configuration - ent12c1_dgb

  Protection Mode: MaxPerformance
  Members:
  ent12c1  - Primary database
    fs12c1   - Far sync instance
      ent12c1s - Physical standby database

  Members Not Receiving Redo:
  fs12c1s  - Far sync instance

Fast-Start Failover: DISABLED

Configuration Status:
SUCCESS   (status updated 49 seconds ago)

DGMGRL> show configuration when primary is ent12c1s

Configuration when ent12c1s is primary - ent12c1_dgb

  Members:
  ent12c1s - Primary database
    fs12c1s  - Far sync instance
      ent12c1  - Physical standby database

  Members Not Receiving Redo:
  fs12c1   - Far sync instance

DGMGRL> validate far_sync fs12c1
    Member Role:        Far Sync Instance
    When Primary Is:    ent12c1

    Active Redo Source: ent12c1
    Redo Destinations:
                        ent12c1s

    Thread #  Online Redo Log Groups  Standby Redo Log Groups Status
              ent12c1                 fs12c1
    1         3                       4                       Sufficient SRLs

DGMGRL> validate far_sync fs12c1s when primary is ent12c1s
    Member Role:        Far Sync Instance
    When Primary Is:    ent12c1s
    Redo Destinations:
                        ent12c1

    Thread #  Online Redo Log Groups  Standby Redo Log Groups Status
              ent12c1s                fs12c1s
    1         3                       4                       Sufficient SRLs


DGMGRL> validate database ent12c1s

  Database Role:     Physical standby database
  Primary Database:  ent12c1

  Ready for Switchover:  Yes
  Ready for Failover:    Yes (Primary Running)

  Flashback Database Status:
    ent12c1:   Off
    ent12c1s:  Off

  Transport-Related Property Settings:
    Property                        ent12c1 Value            ent12c1s Value
    RedoRoutes                      (LOCAL : FS12C1 SYNC)    (LOCAL : FS12C1S SYNC)

DGMGRL> validate database ent12c1

  Database Role:    Primary database

  Ready for Switchover:  Yes

  Flashback Database Status:
    ent12c1:  Off

6. During the validation of the databases the alert log will show the following warning.

SWITCHOVER VERIFY BEGIN
SWITCHOVER VERIFY WARNING: no standby database is defined in LOG_ARCHIVE_DEST_n to protect this database if it is converted to a primary database
SWITCHOVER VERIFY COMPLETE

However the switchover worked fine without any issues.

DGMGRL> switchover to ent12c1s
Performing switchover NOW, please wait...
Operation requires a connection to instance "ent12c1s" on database "ent12c1s"
Connecting to instance "ent12c1s"...
Connected as SYSDBA.
New primary database "ent12c1s" is opening...
Operation requires start up of instance "ent12c1" on database "ent12c1"
Starting instance "ent12c1"...
ORACLE instance started.
Database mounted.
Switchover succeeded, new primary is "ent12c1s"


DGMGRL> show configuration

Configuration - ent12c1_dgb

  Protection Mode: MaxPerformance
  Members:
  ent12c1s - Primary database
    fs12c1s  - Far sync instance
      ent12c1  - Physical standby database

  Members Not Receiving Redo:
  fs12c1   - Far sync instance

Fast-Start Failover: DISABLED

Configuration Status:
SUCCESS   (status updated 3 seconds ago)

DGMGRL> switchover to ent12c1
Performing switchover NOW, please wait...
Operation requires a connection to instance "ent12c1" on database "ent12c1"
Connecting to instance "ent12c1"...
Connected as SYSDBA.
New primary database "ent12c1" is opening...
Operation requires start up of instance "ent12c1s" on database "ent12c1s"
Starting instance "ent12c1s"...
ORACLE instance started.
Database mounted.
Switchover succeeded, new primary is "ent12c1"

7. Once the switchovers are completed successfully it's time to modify the redo routes so there is an alternate path between primary and standby in case of far_sync failure. The redo transport mode between primary and standby would be async. In order to provide an alternative redo route the maxfailure property must be set to a non-zero value. Without this change alternate route cannot be specified

DGMGRL> EDIT DATABASE ent12c1 SET PROPERTY RedoRoutes = '(LOCAL : FS12C1 SYNC ALT=(ent12c1s ASYNC FALLBACK))';
Error: ORA-16863: A member whose MaxFailure property is set to zero cannot have an alternate destination.

EDIT DATABASE ent12c1 SET PROPERTY 'MaxFailure' = 1;
EDIT FAR_SYNC fs12c1 SET PROPERTY 'MaxFailure' = 1;
EDIT DATABASE ent12c1s SET PROPERTY 'MaxFailure' = 1;
EDIT FAR_SYNC fs12c1s SET PROPERTY 'MaxFailure' = 1;

The maxfailure property must be set for far sync as well. Even though they don't have a alternate destination set on them, failure to do so will result in the above error and it's not possible to change the maxfailure to 0 once the alternate destination is set.

DGMGRL>  show far_sync fs12c1 RedoRoutes
  RedoRoutes = '(ent12c1 : ent12c1s ASYNC)'

DGMGRL> EDIT FAR_SYNC fs12c1 SET PROPERTY 'MaxFailure' =0;
Error: ORA-16864: The MaxFailure property cannot be set to zero for a member that has an alternate destination.

8. Set the redo routes for the databases so there's an alternate route for redo transport if far sync is not available. As mentioned earlier the patch 19399918 must be applied on all instances (inclusive of far_sync) for this to work.

EDIT DATABASE ent12c1 SET PROPERTY RedoRoutes = '(LOCAL : FS12C1 SYNC ALT=(ent12c1s ASYNC FALLBACK))';
EDIT DATABASE ent12c1s SET PROPERTY RedoRoutes = '(LOCAL : FS12C1s SYNC ALT=(ent12c1 ASYNC FALLBACK))';

9. Verify the configuration shows the alternate destination.

DGMGRL> show configuration verbose

Configuration - ent12c1_dgb

  Protection Mode: MaxPerformance
  Members:
  ent12c1  - Primary database
    fs12c1   - Far sync instance
      ent12c1s - Physical standby database
    ent12c1s - Physical standby database (alternate of fs12c1)

  Members Not Receiving Redo:
  fs12c1s  - Far sync instance

DGMGRL> show configuration when primary is ent12c1s

Configuration when ent12c1s is primary - ent12c1_dgb

  Members:
  ent12c1s - Primary database
    fs12c1s  - Far sync instance
      ent12c1  - Physical standby database
    ent12c1  - Physical standby database (alternate of fs12c1s)

  Members Not Receiving Redo:
  fs12c1   - Far sync instance

DGMGRL> validate database ent12c1s

  Database Role:     Physical standby database
  Primary Database:  ent12c1

  Ready for Switchover:  Yes
  Ready for Failover:    Yes (Primary Running)

  Flashback Database Status:
    ent12c1:   Off
    ent12c1s:  Off

  Transport-Related Property Settings:
    Property                        ent12c1 Value            ent12c1s Value
    RedoRoutes                      (LOCAL : FS12C1 SYNC ALT=(ent12c1s ASYNC(LOCAL : FS12C1s SYNC ALT=(ent12c1 ASYNC

Check if the alternate location is visible on the v$archive_dest view of the primary database

SQL> SELECT DEST_NAME,STATUS,DESTINATION,TRANSMIT_MODE FROM V$ARCHIVE_DEST WHERE DESTINATION IS NOT NULL;

DEST_NAME            STATUS    DESTINATION                    TRANSMIT_MOD
-------------------- --------- ------------------------------ ------------
LOG_ARCHIVE_DEST_1   VALID     USE_DB_RECOVERY_FILE_DEST      SYNCHRONOUS
LOG_ARCHIVE_DEST_2   VALID     fs12c1tns                      PARALLELSYNC
LOG_ARCHIVE_DEST_3   ALTERNATE ent12c1stns                    ASYNCHRONOUS

In case of far_sync failure the alternate destination will become valid.

SQL> SELECT DEST_NAME,STATUS,DESTINATION,TRANSMIT_MODE FROM V$ARCHIVE_DEST WHERE DESTINATION IS NOT NULL;

DEST_NAME                      STATUS    DESTINATION                    TRANSMIT_MOD
------------------------------ --------- ------------------------------ ------------
LOG_ARCHIVE_DEST_1             VALID     USE_DB_RECOVERY_FILE_DEST      SYNCHRONOUS
LOG_ARCHIVE_DEST_2             ALTERNATE fs12c1tns                      PARALLELSYNC
LOG_ARCHIVE_DEST_3             VALID     ent12c1stns                    ASYNCHRONOUS

10. Check of inconsistent properties and rectify if any exists. Finally carry out switchovers.

DGMGRL> show database ent12c1 InconsistentLogXptProps;
INCONSISTENT LOG TRANSPORT PROPERTIES
   INSTANCE_NAME         STANDBY_NAME        PROPERTY_NAME         MEMORY_VALUE         BROKER_VALUE

DGMGRL> show database ent12c1s InconsistentLogXptProps;
INCONSISTENT LOG TRANSPORT PROPERTIES
   INSTANCE_NAME         STANDBY_NAME        PROPERTY_NAME         MEMORY_VALUE         BROKER_VALUE

DGMGRL> show far_sync fs12c1 InconsistentLogXptProps;
INCONSISTENT LOG TRANSPORT PROPERTIES
   INSTANCE_NAME         STANDBY_NAME        PROPERTY_NAME         MEMORY_VALUE         BROKER_VALUE

DGMGRL>  show far_sync fs12c1s InconsistentLogXptProps;
INCONSISTENT LOG TRANSPORT PROPERTIES
   INSTANCE_NAME         STANDBY_NAME        PROPERTY_NAME         MEMORY_VALUE         BROKER_VALUE

DGMGRL> switchover to ent12c1s
Performing switchover NOW, please wait...
Operation requires a connection to instance "ent12c1s" on database "ent12c1s"
Connecting to instance "ent12c1s"...
Connected as SYSDBA.
New primary database "ent12c1s" is opening...
Operation requires start up of instance "ent12c1" on database "ent12c1"
Starting instance "ent12c1"...
ORACLE instance started.
Database mounted.
Switchover succeeded, new primary is "ent12c1s"

DGMGRL> switchover to ent12c1
Performing switchover NOW, please wait...
Operation requires a connection to instance "ent12c1" on database "ent12c1"
Connecting to instance "ent12c1"...
Connected as SYSDBA.
New primary database "ent12c1" is opening...
Operation requires start up of instance "ent12c1s" on database "ent12c1s"
Starting instance "ent12c1s"...
ORACLE instance started.
Database mounted.
Switchover succeeded, new primary is "ent12c1"

This completes setting up of dg broker for existing data guard configuration with far sync instances. The below section list some of the issues observed during various testing of the setup.





Observed Issues
1. It seems how the far sync is terminated has an effect on the fail over of the archive destination. If the far sync was terminated abruptly (shutdown abort) then the primary database's archive log destination fails over to alternative path give (v$archive_dest output given earlier). But when the far_sync starts up again the log archive destination status for the path from far_sync to standby comes up as idle. Following output is from far_sync

SQL> SELECT DEST_NAME,STATUS,DESTINATION,TRANSMIT_MODE FROM V$ARCHIVE_DEST WHERE DESTINATION IS NOT NULL;

DEST_NAME            STATUS    DESTINATION                              TRANSMIT_MOD
-------------------- --------- ---------------------------------------- ------------
LOG_ARCHIVE_DEST_1   VALID     USE_DB_RECOVERY_FILE_DEST                SYNCHRONOUS
LOG_ARCHIVE_DEST_2   IDLE      ent12c1stns                              ASYNCHRONOUS
STANDBY_ARCHIVE_DEST VALID     USE_DB_RECOVERY_FILE_DEST                SYNCHRONOUS

Even after leaving it for 24 hour period the destination did not resolve itself. Manually intervention was needed

ALTER SYSTEM SET log_archive_dest_state_2='enable' SCOPE=BOTH;

2. If the far_sync has a clean shutdown (shutdown immediate instead of shutdown abort) then when it comes up again the log archive dest status come up as valid. However the failover on the primary doesn't happen to alternate destination. The destination to far_sync remains deferred and alternate stays as it is. Following output from primary when far_sync is shutdown cleanly

SQL> SELECT DEST_NAME,STATUS,DESTINATION,TRANSMIT_MODE FROM V$ARCHIVE_DEST WHERE DESTINATION IS NOT NULL;

DEST_NAME                      STATUS    DESTINATION                    TRANSMIT_MOD
------------------------------ --------- ------------------------------ ------------
LOG_ARCHIVE_DEST_1             VALID     USE_DB_RECOVERY_FILE_DEST      SYNCHRONOUS
LOG_ARCHIVE_DEST_2             DEFERRED  fs12c1tns                      PARALLELSYNC
LOG_ARCHIVE_DEST_3             ALTERNATE ent12c1stns                    ASYNCHRONOUS

When the far_sync comes up again the deferred destination becomes valid again. In this scenario can lead to data loss as redo transport doesn't happen between primary and standby even though an alternate path was specified.

3. Increasing the protection mode was not possible with the redo routes defined with ALT and ASYNC.

DGMGRL> show database ent12c1 RedoRoutes
  RedoRoutes = '(LOCAL : FS12C1 SYNC ALT=(ent12c1s ASYNC FALLBACK))'
DGMGRL> show database ent12c1s RedoRoutes
  RedoRoutes = '(LOCAL : FS12C1s SYNC ALT=(ent12c1 ASYNC FALLBACK))'
DGMGRL> show far_sync fs12c1 RedoRoutes
  RedoRoutes = '(ent12c1 : ent12c1s ASYNC)'
DGMGRL> show far_sync fs12c1s RedoRoutes
  RedoRoutes = '(ent12c1s : ent12c1 ASYNC)'

DGMGRL> EDIT CONFIGURATION SET PROTECTION MODE AS MaxAvailability;
Error: ORA-16627: operation disallowed since no standby databases would remain to support protection mode

There was no such problem before when the setup was not using the dg broker. Whatever the problem seem to be localized to dg broker setup. Removing the alternate option on the redo route from primary and standby databases allowed the upgrade of the protection mode, even though between far sync and standby redo was shipping async.

DGMGRL> EDIT DATABASE ent12c1 SET PROPERTY 'RedoRoutes' = '(LOCAL : FS12C1 SYNC)';
Property "RedoRoutes" updated
DGMGRL> EDIT DATABASE ent12c1s SET PROPERTY 'RedoRoutes' = '(LOCAL : FS12C1S SYNC)';
Property "RedoRoutes" updated

On primary

SQL> SELECT DEST_NAME,STATUS,DESTINATION,TRANSMIT_MODE FROM V$ARCHIVE_DEST WHERE DESTINATION IS NOT NULL;

DEST_NAME            STATUS    DESTINATION                    TRANSMIT_MOD
-------------------- --------- ------------------------------ ------------
LOG_ARCHIVE_DEST_1   VALID     USE_DB_RECOVERY_FILE_DEST      SYNCHRONOUS
LOG_ARCHIVE_DEST_2   VALID     fs12c1tns                      PARALLELSYNC

On far sync

SQL> SELECT DEST_NAME,STATUS,DESTINATION,TRANSMIT_MODE FROM V$ARCHIVE_DEST WHERE DESTINATION IS NOT NULL;

DEST_NAME            STATUS    DESTINATION                              TRANSMIT_MOD
-------------------- --------- ---------------------------------------- ------------
LOG_ARCHIVE_DEST_1   VALID     USE_DB_RECOVERY_FILE_DEST                SYNCHRONOUS
LOG_ARCHIVE_DEST_2   VALID     ent12c1stns                              ASYNCHRONOUS
STANDBY_ARCHIVE_DEST VALID     USE_DB_RECOVERY_FILE_DEST                SYNCHRONOUS

Increase of protection mode works and switchvoer is possible without any issue

DGMGRL> EDIT CONFIGURATION SET PROTECTION MODE AS MaxAvailability;
Succeeded.

DGMGRL> show configuration;

Configuration - ent12c1_dgb

  Protection Mode: MaxAvailability
  Members:
  ent12c1  - Primary database
    fs12c1   - Far sync instance
      ent12c1s - Physical standby database

  Members Not Receiving Redo:
  fs12c1s  - Far sync instance

Fast-Start Failover: DISABLED

Configuration Status:
SUCCESS   (status updated 59 seconds ago)

DGMGRL> switchover to ent12c1s
Performing switchover NOW, please wait...
Operation requires a connection to instance "ent12c1s" on database "ent12c1s"
Connecting to instance "ent12c1s"...
Connected as SYSDBA.
New primary database "ent12c1s" is opening...
Operation requires start up of instance "ent12c1" on database "ent12c1"
Starting instance "ent12c1"...
ORACLE instance started.
Database mounted.
Switchover succeeded, new primary is "ent12c1s"


DGMGRL> show configuration

Configuration - ent12c1_dgb

  Protection Mode: MaxAvailability
  Members:
  ent12c1s - Primary database
    fs12c1s  - Far sync instance
      ent12c1  - Physical standby database

  Members Not Receiving Redo:
  fs12c1   - Far sync instance

Fast-Start Failover: DISABLED

Configuration Status:
SUCCESS   (status updated 55 seconds ago)

DGMGRL> switchover to ent12c1
Performing switchover NOW, please wait...
Operation requires a connection to instance "ent12c1" on database "ent12c1"
Connecting to instance "ent12c1"...
Connected as SYSDBA.
New primary database "ent12c1" is opening...
Operation requires start up of instance "ent12c1s" on database "ent12c1s"
Starting instance "ent12c1s"...
ORACLE instance started.
Database mounted.
Switchover succeeded, new primary is "ent12c1"

Even though ALT+ASYCN combination didn't work, ALT+SYNC combination allowed increase of protection mode and switchover without any issue.

DGMGRL> EDIT CONFIGURATION SET PROTECTION MODE AS maxperformance;
Succeeded.

DGMGRL> show configuration;

Configuration - ent12c1_dgb

  Protection Mode: MaxPerformance
  Members:
  ent12c1  - Primary database
    fs12c1   - Far sync instance
      ent12c1s - Physical standby database

  Members Not Receiving Redo:
  fs12c1s  - Far sync instance

Fast-Start Failover: DISABLED

Configuration Status:
SUCCESS   (status updated 31 seconds ago)

DGMGRL> EDIT DATABASE ent12c1 SET PROPERTY RedoRoutes = '(LOCAL : FS12C1 SYNC ALT=(ent12c1s SYNC FALLBACK))';
Property "redoroutes" updated
DGMGRL> EDIT DATABASE ent12c1s SET PROPERTY RedoRoutes = '(LOCAL : FS12C1s SYNC ALT=(ent12c1 SYNC FALLBACK))' ; 
Property "redoroutes" updated

DGMGRL> show database ent12c1 RedoRoutes
  RedoRoutes = '(LOCAL : FS12C1 SYNC ALT=(ent12c1s SYNC FALLBACK))'
DGMGRL>  show database ent12c1s RedoRoutes
  RedoRoutes = '(LOCAL : FS12C1s SYNC ALT=(ent12c1 SYNC FALLBACK))'
DGMGRL> show far_sync fs12c1 RedoRoutes
  RedoRoutes = '(ent12c1 : ent12c1s ASYNC)'
DGMGRL>  show far_sync fs12c1s RedoRoutes
  RedoRoutes = '(ent12c1s : ent12c1 ASYNC)'


DGMGRL> EDIT CONFIGURATION SET PROTECTION MODE AS MaxAvailability;
Succeeded.

DGMGRL> show configuration

Configuration - ent12c1_dgb

  Protection Mode: MaxAvailability
  Members:
  ent12c1  - Primary database
    fs12c1   - Far sync instance
      ent12c1s - Physical standby database

  Members Not Receiving Redo:
  fs12c1s  - Far sync instance

Fast-Start Failover: DISABLED

Configuration Status:
SUCCESS   (status updated 50 seconds ago)

DGMGRL> switchover to ent12c1s
Performing switchover NOW, please wait...
Operation requires a connection to instance "ent12c1s" on database "ent12c1s"
Connecting to instance "ent12c1s"...
Connected as SYSDBA.
New primary database "ent12c1s" is opening...
Operation requires start up of instance "ent12c1" on database "ent12c1"
Starting instance "ent12c1"...
ORACLE instance started.
Database mounted.
Switchover succeeded, new primary is "ent12c1s"

DGMGRL> show configuration

Configuration - ent12c1_dgb

  Protection Mode: MaxAvailability
  Members:
  ent12c1s - Primary database
    fs12c1s  - Far sync instance
      ent12c1  - Physical standby database

  Members Not Receiving Redo:
  fs12c1   - Far sync instance

Fast-Start Failover: DISABLED

Configuration Status:
SUCCESS   (status updated 2 seconds ago)

DGMGRL> switchover to ent12c1
Performing switchover NOW, please wait...
Operation requires a connection to instance "ent12c1" on database "ent12c1"
Connecting to instance "ent12c1"...
Connected as SYSDBA.
New primary database "ent12c1" is opening...
Operation requires start up of instance "ent12c1s" on database "ent12c1s"
Starting instance "ent12c1s"...
ORACLE instance started.
Database mounted.
Switchover succeeded, new primary is "ent12c1"

This creates a alternate route between primary and standby when far sync fails. In this case redo shipment happens in SYNC mode. When far sync is up

SQL> SELECT DEST_NAME,STATUS,DESTINATION,TRANSMIT_MODE FROM V$ARCHIVE_DEST WHERE DESTINATION IS NOT NULL;

DEST_NAME                                STATUS    DESTINATION                    TRANSMIT_MOD
---------------------------------------- --------- ------------------------------ ------------
LOG_ARCHIVE_DEST_1                       VALID     USE_DB_RECOVERY_FILE_DEST      SYNCHRONOUS
LOG_ARCHIVE_DEST_2                       VALID     fs12c1tns                      PARALLELSYNC
LOG_ARCHIVE_DEST_3                       ALTERNATE ent12c1stns                    PARALLELSYNC

After far sync fails

SQL> SELECT DEST_NAME,STATUS,DESTINATION,TRANSMIT_MODE FROM V$ARCHIVE_DEST WHERE DESTINATION IS NOT NULL;

DEST_NAME                                STATUS    DESTINATION                    TRANSMIT_MOD
---------------------------------------- --------- ------------------------------ ------------
LOG_ARCHIVE_DEST_1                       VALID     USE_DB_RECOVERY_FILE_DEST      SYNCHRONOUS
LOG_ARCHIVE_DEST_2                       ALTERNATE fs12c1tns                      PARALLELSYNC
LOG_ARCHIVE_DEST_3                       VALID     ent12c1stns                    PARALLELSYNC

Even though this works it's not same as the setup exited before dg broker was used and one of the key points of far sync to avoid sync redo shipping between primary and standby over long distance. It is also not possible to change the redo mode to ASYNC.

DGMGRL> EDIT DATABASE ent12c1 SET PROPERTY RedoRoutes = '(LOCAL : FS12C1 SYNC ALT=(ent12c1s ASYNC FALLBACK))';
Error: ORA-16900: change of LogXptMode or RedoRoutes property violates overall protection mode

A SR was raised about this issue. After some investigation Oracle came with a patch "Patch 20695119 MERGE REQUEST ON TOP OF DATABASE PSU 12.1.0.2.2 FOR BUGS 19399918 19571599" (only works of 12.1.0.2.2). After applying this patch the dg broke allowed increasing the protection mode but when a switchover was issued failed with "ORA-00600: internal error code, arguments: [kghstack_underflow_internal_1], [0x2B29FF483BA0], [rfrxpts2dfp.1]". There was no resolution for this from Oracle and SR went nowhere.
Searching MOS it was discovered that recently Oracle has released patch 19571599 (requires 12.1.0.2.4 applied DB Home) which was not available outside of the merge patch earlier. After applying this patch it was possible to create the dg broker configuration same as before.

EDIT DATABASE ent12c1 SET PROPERTY RedoRoutes = '(LOCAL : FS12C1 SYNC ALT=(ent12c1s ASYNC FALLBACK))';
EDIT DATABASE ent12c1s SET PROPERTY RedoRoutes = '(LOCAL : FS12C1s SYNC ALT=(ent12c1 ASYNC FALLBACK))';

DGMGRL> EDIT CONFIGURATION SET PROTECTION MODE AS MaxAvailability;
Succeeded.

DGMGRL> show configuration

Configuration - ent12c1_dgb

  Protection Mode: MaxAvailability
  Members:
  ent12c1  - Primary database
    fs12c1   - Far sync instance
      ent12c1s - Physical standby database

  Members Not Receiving Redo:
  fs12c1s  - Far sync instance

Fast-Start Failover: DISABLED

Configuration Status:
SUCCESS   (status updated 35 seconds ago)

With this configuration and patches in place when the far sync goes down the dg broker lowers the protection mode to maximum performance to reflect the new redo shipping mode of ASYNC. Following entries could be seen on standby's alert log

Primary database is in MAXIMUM PERFORMANCE mode
Changing standby controlfile to MAXIMUM PERFORMANCE mode

When the far sync comes up again the protection mode is upgraded to maximum availability. On far sync alert log

Primary database is in MAXIMUM AVAILABILITY mode
Changing standby controlfile to MAXIMUM AVAILABILITY mode
Changing standby controlfile to RESYNCHRONIZATION level

On standby's alert log

Primary database is in MAXIMUM PERFORMANCE mode
Changing standby controlfile to MAXIMUM AVAILABILITY mode
Changing standby controlfile to RESYNCHRONIZATION level

Even though standby's protection level goes up and down primary's protection level remains the same at maximum availability and this can cause warning to appear on the dg broker configuation

DGMGRL> show configuration
Configuration - ent12c1_dgb

  Protection Mode: MaxAvailability
  Members:
  ent12c1  - Primary database
    Warning: ORA-16629: database reports a different protection level from the protection mode

    ent12c1s - Physical standby database (alternate of fs12c1)

  Members Not Receiving Redo:
  fs12c1   - Far sync instance
    Warning: ORA-01034: ORACLE not available

  fs12c1s  - Far sync instance

Fast-Start Failover: DISABLED

Configuration Status:
WARNING   (status updated 31 seconds ago)

DGMGRL> show database ent12c1 statusreport
STATUS REPORT
       INSTANCE_NAME   SEVERITY ERROR_TEXT
                   *    WARNING ORA-16629: database reports a different protection level from the protection mode

Another issue observed is that at times the alternate route disappear from the log archive dest entries even though the state of the entries remains alternate. On primary (there's no alternate option)

SQL> SELECT DEST_NAME,STATUS,DESTINATION,TRANSMIT_MODE FROM V$ARCHIVE_DEST WHERE DESTINATION IS NOT NULL;

DEST_NAME                      STATUS    DESTINATION                    TRANSMIT_MOD
------------------------------ --------- ------------------------------ ------------
LOG_ARCHIVE_DEST_1             VALID     USE_DB_RECOVERY_FILE_DEST      SYNCHRONOUS
LOG_ARCHIVE_DEST_2             VALID     fs12c1stns                     PARALLELSYNC

This will lead to log transport properties being inconsistent and switchovers failing

DGMGRL> show database ent12c1 InconsistentLogXptProps;
INCONSISTENT LOG TRANSPORT PROPERTIES
   INSTANCE_NAME         STANDBY_NAME        PROPERTY_NAME         MEMORY_VALUE         BROKER_VALUE
         ent12c1               fs12c1           RedoRoutes                 NONE             ent12c1s

When standby becomes primary

DGMGRL> show database ent12c1s InconsistentLogXptProps;
INCONSISTENT LOG TRANSPORT PROPERTIES
   INSTANCE_NAME         STANDBY_NAME        PROPERTY_NAME         MEMORY_VALUE         BROKER_VALUE
        ent12c1s              fs12c1s           RedoRoutes                 NONE              ent12c1

Show database would have entries similar to following

Instance(s):
    ent12c1s
      Error: ORA-16736: unable to find the destination entry of standby database "ent12c1" in V$ARCHIVE_DEST
      Warning: ORA-16715: redo transport-related property RedoRoutes of standby database "fs12c1s" is inconsistent
      Warning: ORA-16728: consistency check for property LogXptMode found ORA-16777 error
      Warning: ORA-16777: unable to find the destination entry of a standby database in V$ARCHIVE_DEST

One of the easiest way to resolve this is to disable and enable redo transport on the primary

DGMGRL>  EDIT DATABASE ent12c1s SET state='transport-off';
DGMGRL> EDIT DATABASE ent12c1s SET state='transport-on';

But this may not be possible when the protection mode is maximum availability. In such cases simply enabling transport, while it's already on works

DGMGRL> edit database ent12c1s set state='transport-off';
Error: ORA-16627: operation disallowed since no standby databases would remain to support protection mode
Failed.

DGMGRL> edit database ent12c1s set state='transport-on';
Succeeded.

Check if the alternate route appears on the archive destination

SQL>  SELECT DEST_NAME,STATUS,DESTINATION,TRANSMIT_MODE FROM V$ARCHIVE_DEST WHERE DESTINATION IS NOT NULL;

DEST_NAME                      STATUS    DESTINATION                    TRANSMIT_MOD
------------------------------ --------- ------------------------------ ------------
LOG_ARCHIVE_DEST_1             VALID     USE_DB_RECOVERY_FILE_DEST      SYNCHRONOUS
LOG_ARCHIVE_DEST_2             VALID     fs12c1stns                     PARALLELSYNC
LOG_ARCHIVE_DEST_3             ALTERNATE ent12c1tns                     ASYNCHRONOUS

Continue with the switchover

DGMGRL> switchover to ent12c1;
Performing switchover NOW, please wait...
Operation requires a connection to instance "ent12c1" on database "ent12c1"
Connecting to instance "ent12c1"...
Connected as SYSDBA.
New primary database "ent12c1" is opening...
Operation requires start up of instance "ent12c1s" on database "ent12c1s"
Starting instance "ent12c1s"...
ORACLE instance started.
Database mounted.
Switchover succeeded, new primary is "ent12c1"

Related Posts
Creating Data Guard Broker on 12c
Adding Far Sync Instances to Existing Data Guard Configuration
11gR2 RAC to RAC Data Guard
11gR2 Standalone Data Guard (with ASM and Role Separation)

Useful metalink notes
Cascaded Standby Databases in Oracle 12c [ID 1542969.1]
Create Configuration Failing with ORA-16698 [ID 1582179.1]
Data Guard 12c New Feature: Far Sync Standby [ID 1565071.1]
12c Create Dataguard Broker Configuration - DGMGRL [ID 1583588.1]
Best Practices for Corruption Detection, Prevention, and Automatic Repair - in a Data Guard Configuration [ID 1302539.1]
Configuring Active Data Guard Far Sync Using a Terminal Standby Database as Alternate Log Archive Destination in Data Guard Broker [ID 1918316.1]

Adding Far Sync Instances to Existing Data Guard Configuration

This post list the steps of adding far sync instances to an existing data guard configuration (for both primary and standby databases, to be used when standby become primary). Far sync is an new feature introduced with 12c which allows transporting of redo data synchronously from primary DB to a "near by" far sync instance, which then transport the redo data asynchronously "over a longer distance". The idea is that there's low overhead on primary when transporting redo synchronously to a "near by" far sync instance compared to transporting "over a long distance" to achieve zero data loss as well as off-loading of the redo transport.
Oracle documentation provides a complete description of the far sync concept. Given below are few important excerpts
Many configurations have a primary database shipping redo to a standby database using asynchronous transport at the risk of some data loss at failover time. Using synchronous redo transport to achieve zero data loss may not be a viable option because of the impact on the commit response times at the primary due to network latency between the two databases. Creating a far sync instance close to the primary has the benefit of minimizing impact on commit response times to an acceptable threshold (due to the smaller network latency between primary and far sync instance) while allowing for higher data protection guarantees -- if the primary were to fail, and assuming the far sync instance was synchronized at the time of the failure, the far sync instance and the terminal standby would coordinate a final redo shipment from the far sync instance to the standby to ship any redo not yet available to the Standby and then perform a zero-data-loss failover.
A far sync instance manages a control file, receives redo into standby redo logs (SRLs), and archives those SRLs to local archived redo logs. A far sync instance does not have user data files, cannot be opened for access, cannot run redo apply, and can never function in the primary role or be converted to any type of standby database.
Far sync instances are part of the Oracle Active Data Guard Far Sync feature, which requires an Oracle Active Data Guard license.
In a configuration that contains a far sync instance, there must still be a direct network connection between the primary database and the remote standby database. The direct connection between the primary and the remote standby is used to perform health checks and switchover processing tasks. It is not used for redo transport unless the standby has been configured as an alternate destination in case the far sync instance fails and there is no alternate far sync configured to maintain the protection level.
The existing data guard configuration's primary database parameter setting and active data guard creation script of the standby database is given below. (other prerequisites for setting up data guard is omitted)
Primary database parameter changes

alter system set log_archive_config='dg_config=(ent12c1,ent12c1s)' scope=both ;
alter system set log_archive_dest_1='location=use_db_recovery_file_dest valid_for=(all_logfiles,all_roles) db_unique_name=ent12c1' scope=both;
alter system set log_archive_dest_2='service=ENT12C1STNS ASYNC NOAFFIRM max_failure=10 max_connections=5 reopen=180 valid_for=(online_logfiles,primary_role) db_unique_name=ent12c1s' scope=both;
alter system set log_archive_dest_state_2='defer' scope=both;
alter system set log_archive_dest_state_1='enable' scope=both;
alter system set fal_server='ENT12C1STNS' scope=both;
alter system set log_archive_max_processes=10 scope=both;
alter system set db_file_name_convert='/opt/app/oracle/oradata/ENT12C1S','/data/oradata/ENT12C1' scope=spfile;
alter system set log_file_name_convert='/opt/app/oracle/oradata/ENT12C1S','/data/oradata/ENT12C1' ,'/opt/app/oracle/fast_recovery_area/ENT12C1S','/data/flash_recovery/ENT12C1' scope=spfile;
alter system set standby_file_management='AUTO' scope=both;

Standby creation script

mkdir -p  /opt/app/oracle/oradata/ENT12C1S/controlfile
mkdir -p /opt/app/oracle/fast_recovery_area/ENT12C1S/controlfile

duplicate target database for standby from active database spfile
parameter_value_convert 'ent12c1','ent12c1s','ENT12C1','ENT12C1S','data','opt/app/oracle','flash_recovery','fast_recovery_area'
set db_unique_name='ent12c1s'
set db_create_file_dest='/opt/app/oracle/oradata'
set db_recovery_file_dest='/opt/app/oracle/fast_recovery_area'
set db_file_name_convert='/data/oradata/ENT12C1','/opt/app/oracle/oradata/ENT12C1S'
set log_file_name_convert='/data/oradata/ENT12C1','/opt/app/oracle/oradata/ENT12C1S','/data/flash_recovery/ENT12C1','/opt/app/oracle/fast_recovery_area/ENT12C1S'
set log_archive_max_processes='10'
set fal_server='ENT12C1TNS'
set log_archive_dest_2='service=ENT12C1TNS ASYNC NOAFFIRM max_failure=10 max_connections=5 reopen=180 valid_for=(online_logfiles,primary_role) db_unique_name=ent12c1'
set log_archive_dest_1='location=use_db_recovery_file_dest valid_for=(all_logfiles,all_roles) db_unique_name=ent12c1s';

This single instance data guard configuration which transport redo asynchronously is transformed as show below.

In situation 1 ENT12C1 is the primary database while ENT12C1S is the standby and FS12C1 is the far sync instance the primary database ships redo synchronously.
After a role switch when the ENT12C1S becomes the new primary then it uses FS12C1S as the far sync instance to transport redo synchronously.
In both situation there exists a direct redo transport path between primary and standby and this will transport redo asynchronously in case of far sync instance failure. Once the far sync is backup again the data guard configuration will revert to using the far sync instance for redo transport. If the standby redo logs were created on the primary then when far sync instances are in use for redo transport standby redo logs will be created automatically for them.
1. On the servers used for creating the far sync instances, install the oracle database software and create a listener. There's no requirement for creating static listener configuration as far sync instance automatically registers with the listener.
2. Create TNS entries for far sync instances on the existing databases (primary and standby) and copy the existing TNS entries into the far sync instances tnsnames.ora file.

cat tnsnames.ora

ENT12C1TNS =
  (DESCRIPTION =
    (ADDRESS = (PROTOCOL = TCP)(HOST = ent12c1-host)(PORT = 1521))
    (CONNECT_DATA =
      (SERVER = DEDICATED)
      (SERVICE_NAME = ent12c1)
    )
  )

ENT12C1STNS =
  (DESCRIPTION =
    (ADDRESS = (PROTOCOL = TCP)(HOST = ent12c1s-host)(PORT = 1521))
    (CONNECT_DATA =
      (SERVER = DEDICATED)
      (SERVICE_NAME = ent12c1)
    )
  )

FS12C1TNS =
  (DESCRIPTION =
    (ADDRESS = (PROTOCOL = TCP)(HOST = fs12c1-host)(PORT = 1521))
    (CONNECT_DATA =
      (SERVER = DEDICATED)
      (SERVICE_NAME = fs12c1)
    )
  )

FS12C1STNS =
  (DESCRIPTION =
    (ADDRESS = (PROTOCOL = TCP)(HOST = fs12c1s-host)(PORT = 1521))
    (CONNECT_DATA =
      (SERVER = DEDICATED)
      (SERVICE_NAME = fs12c1s)
    )
  )

3. Create a control file for the far sync instances by connecting to the primary database. Same control file is used for both far sync instances in this case (fs12c1 and fs12c1s).

SQL> ALTER DATABASE CREATE FAR SYNC INSTANCE CONTROLFILE AS '/home/oracle/controlfs.ctl';

4. Copy the control file to the far sync instances. In this case the control file is multiplexed and renamed as control01.ctl and control02.ctl on the far sync instances (shown on the pfiles in subsequent steps).

scp controlfs.ctl fs12c1-host:/opt/app/oracle/oradata/FS12C1/controlfile/controlfs01.ctl
scp controlfs.ctl fs12c1s-host:/opt/app/oracle/oradata/FS12C1S/controlfile/controlfs01.ctl

Similarly copy the password file from the primary to $ORACLE_HOME/dbs on the servers where far sync instances will be created. For far sync instances fs12c1 and fs12c1s the password file need to be renamed as orapwfs12c1 and orapwfs12c1s respectively.

5. Create pfile from the primary spfile. This will be modified to reflect the far sync instance settings.

SQL> create pfile='/home/oracle/pfilefs.ora' from spfile;

6. Copy the pfile to far sync instances ($ORACLE_HOME/dbs) and rename them to reflect the instance names (eg. initfs12c1.ora and initfs12c1s.ora). Modify the init file used for the primary far sync instance (fs12c1) as shown below. Not all the parameters are needed for far sync and those could be removed from the pfile.

cat initfs12c1.ora

*.audit_file_dest='/opt/app/oracle/admin/fs12c1/adump'
*.audit_trail='OS'
*.compatible='12.1.0.2'
*.control_files='/opt/app/oracle/oradata/FS12C1/controlfile/controlfs01.ctl','/opt/app/oracle/oradata/FS12C1/controlfile/controlfs02.ctl'
*.db_block_size=8192
*.db_create_file_dest='/opt/app/oracle/oradata'
*.db_name='ent12c1'
*.db_recovery_file_dest_size=21474836480
*.db_recovery_file_dest='/opt/app/oracle/fast_recovery_area'
*.db_unique_name='fs12c1'
*.diagnostic_dest='/opt/app/oracle'
*.dispatchers='(PROTOCOL=TCP) (SERVICE=fs12c1XDB)'
*.fal_server='ENT12C1TNS'
*.log_archive_config='dg_config=(ent12c1,ent12c1s,fs12c1,fs12c1s)'
*.log_archive_dest_1='location=use_db_recovery_file_dest valid_for=(all_logfiles,all_roles) db_unique_name=fs12c1'
*.log_archive_dest_2='service=ENT12C1STNS ASYNC NOAFFIRM valid_for=(STANDBY_LOGFILES,standby_role) db_unique_name=ent12c1s max_failure=10 max_connections=5 reopen=180'
*.log_archive_dest_state_1='enable'
*.log_archive_dest_state_2='enable'
*.log_archive_format='%t_%s_%r.dbf'
*.log_archive_max_processes=10
*.log_file_name_convert='/data/oradata/ENT12C1','/opt/app/oracle/oradata/FS12C1','/data/flash_recovery/ENT12C1','/opt/app/oracle/fast_recovery_area/FS12C1'
*.remote_login_passwordfile='EXCLUSIVE'
*.standby_file_management='AUTO'

7. Mount the far sync instance using the pfile and then create a spfile from the pfile. Without the spfile a warning is shown when the far sync is added to a data guad broker configuration. Beside that having a spfile also helps with any subsequent parameter changes without the need to restart the far sync instance. Restart (mount) the far sync instance using the spfile.



8. Similarly create the pfile for the far sync instance used by current standby (when it becomes the primary) FS12C1S.

cat initfs12c1s.ora

*.audit_file_dest='/opt/app/oracle/admin/fs12c1s/adump'
*.audit_trail='OS'
*.compatible='12.1.0.2'
*.control_files='/opt/app/oracle/oradata/FS12C1S/controlfile/control01.ctl','/opt/app/oracle/fast_recovery_area/FS12C1S/controlfile/control02.ctl'
*.db_create_file_dest='/opt/app/oracle/oradata'
*.db_name='ent12c1'
*.db_recovery_file_dest_size=21474836480
*.db_recovery_file_dest='/opt/app/oracle/fast_recovery_area'
*.db_unique_name='fs12c1s'
*.diagnostic_dest='/opt/app/oracle'
*.dispatchers='(PROTOCOL=TCP) (SERVICE=fs12c1sXDB)'
*.fal_server='ENT12C1STNS'
*.log_archive_config='dg_config=(ent12c1,ent12c1s,fs12c1,fs12c1s)'
*.log_archive_dest_1='location=use_db_recovery_file_dest valid_for=(all_logfiles,all_roles) db_unique_name=fs12c1s'
*.log_archive_dest_2='service=ENT12C1TNS ASYNC NOAFFIRM valid_for=(standby_logfiles,standby_role) db_unique_name=ent12c1 max_failure=10 max_connections=5 reopen=180'
*.log_archive_dest_state_1='enable'
*.log_archive_dest_state_2='enable'
*.log_archive_format='%t_%s_%r.dbf'
*.log_archive_max_processes=10
*.log_file_name_convert='/data/oradata/ENT12C1','/opt/app/oracle/oradata/FS12C1S','/data/flash_recovery/ENT12C1','/opt/app/oracle/fast_recovery_area/FS12C1S'
*.remote_login_passwordfile='EXCLUSIVE'
*.standby_file_management='AUTO'

9. Similar to earlier (step 7) create a spfile and restart (mount) the far sync instance using the spfile.

10. Update the log archive config parameter on both primary and standby to include the far sync instance information as well.

alter system set log_archive_config='dg_config=(ent12c1,ent12c1s,fs12c1,fs12c1s)' scope=both ;

11. Update the fal server parameter on the primary (ent12c1) as below which allows the ent12c1 to fetch archive logs (when it becomes a standby) either from the primary (ent12c1s) or from the far sync instance (fs12c1s).

alter system set fal_server='ENT12C1STNS','FS12C1STNS' scope=both;

12. Update the fal server parameter on the standby (ent12c1s) so that it can fetch the archive logs either from the ent12c1 (primary) or far sync instance (fs12c1).

alter system set fal_server='ENT12C1TNS','FS12C1TNS' scope=both;

13. Update the log archive destination and log archive destination state on the primary such that redo transport is synchronized between primary and far sync and asynchronous between the primary and standby (direct). Further more the asynchronous log archive destination is set with state alternate so that when the synchronous log archive destination fails the data guard configuration start shipping redo via this alternate log archive destination.

alter system set log_archive_dest_state_2='enable' scope=both;
alter system set log_archive_dest_2='service=FS12C1TNS SYNC AFFIRM  db_unique_name=fs12c1  max_failure=1 valid_for=(online_logfiles,primary_role) alternate=log_archive_dest_3 max_connections=5' scope=both;

alter system set log_archive_dest_state_3='alternate' scope=both;
alter system set log_archive_dest_3='service=ENT12C1STNS ASYNC NOAFFIRM  db_unique_name=ent12c1s  valid_for=(online_logfiles,primary_role) alternate=log_archive_dest_2 max_failure=10 max_connections=5 reopen=180' scope=both;

14. Change and add log archive destination settings on the standby database so that when it becomes primary (ent12c1s) it too can use the far sync instance for synchronous redo transport and failing that use asynchronous redo transport directly with the standby at the time (ent12c1)

alter system set log_archive_dest_state_2='enable' scope=both;
alter system set log_archive_dest_2='service=FS12C1STNS SYNC AFFIRM db_unique_name=fs12c1s valid_for=(online_logfiles,primary_role) alternate=log_archive_dest_3 max_connections=5  max_failure=1' scope=both;

alter system set log_archive_dest_state_3='alternate' scope=both;
alter system set log_archive_dest_3='service=ENT12C1TNS ASYNC NOAFFIRM  db_unique_name=ent12c1 valid_for=(online_logfiles,primary_role) alternate=log_archive_dest_2 max_failure=10 max_connections=5 reopen=180' scope=both;

15. Increase the protection mode of the primary database to maximum availability.

ALTER DATABASE SET STANDBY TO MAXIMIZE AVAILABILITY;

SQL> SELECT PROTECTION_MODE FROM V$DATABASE;

PROTECTION_MODE
--------------------
MAXIMUM AVAILABILITY

16. Do few log switches and verify that redo transport is happening via the far sync instance. Easiest way to monitor is through the alert log, which log switch will be logged on both far sync instance alert log and standby instance alert log if it was transported via the far sync. If there's any issues with regard to log archive destination this could be observed on the primary

SQL> SELECT DEST_NAME,STATUS,DESTINATION FROM V$ARCHIVE_DEST WHERE DESTINATION IS NOT NULL;

DEST_NAME                      STATUS    DESTINATION
------------------------------ --------- ------------------------------
LOG_ARCHIVE_DEST_1             VALID     USE_DB_RECOVERY_FILE_DEST
LOG_ARCHIVE_DEST_2             VALID     FS12C1TNS
LOG_ARCHIVE_DEST_3             ALTERNATE ENT12C1STNS

and on the far sync instance

SQL> SELECT DEST_NAME,STATUS,DESTINATION FROM V$ARCHIVE_DEST WHERE DESTINATION IS NOT NULL;

DEST_NAME                      STATUS    DESTINATION
------------------------------ --------- ------------------------------
LOG_ARCHIVE_DEST_1             VALID     USE_DB_RECOVERY_FILE_DEST
LOG_ARCHIVE_DEST_2             VALID     ENT12C1STNS
STANDBY_ARCHIVE_DEST           VALID     USE_DB_RECOVERY_FILE_DEST

Above output shows that primary is transporting to log archive dest 2 and status valid and dest 3 is still in alternative state. On the far sync output it shows that far sync instance is shipping redo as per its log archive dest 2 value and current status is valid.

17. Shutdown abort the far instance and view the archive dest output. If the configuration works properly then log archive dest 3 should be valid and redo transport should be happening directory between primary and standby in asynchronous mode.

SQL> SELECT DEST_NAME,STATUS,DESTINATION FROM V$ARCHIVE_DEST WHERE DESTINATION IS NOT NULL;

DEST_NAME                      STATUS    DESTINATION
------------------------------ --------- ------------------------------
LOG_ARCHIVE_DEST_1             VALID     USE_DB_RECOVERY_FILE_DEST
LOG_ARCHIVE_DEST_2             ALTERNATE FS12C1TNS
LOG_ARCHIVE_DEST_3             VALID     ENT12C1STNS

From the above output it could be seen that after far sync instance is terminated the log archive dest 3 has become the valid destination and log archive dest 2 is kept as an alternate destination. On the primary instance alert log following could be observed when the far sync instance is terminated

Destination LOG_ARCHIVE_DEST_2 is UNSYNCHRONIZED
LGWR: Failed to archive log 1 thread 1 sequence 1503 (3113)
Destination LOG_ARCHIVE_DEST_2 no longer supports SYNCHRONIZATION

and on the standby instance alert log the following

Primary database is in MAXIMUM PERFORMANCE mode
Changing standby controlfile to MAXIMUM PERFORMANCE mode

It must be also mentioned on few occasions where the far sync was abruptly terminated (shutdown abort) the recovery process on standby got stopped due to lost writes

MRP0: Background Media Recovery terminated with error 742  <-- far sync instance terminated
Fri Sep 12 17:32:28 2014
Errors in file /opt/app/oracle/diag/rdbms/ent12c1s/ent12c1s/trace/ent12c1s_pr00_10098.trc:
ORA-00742: Log read detects lost write in thread 1 sequence 1503 block 868
ORA-00312: online log 4 thread 1: '/opt/app/oracle/fast_recovery_area/ENT12C1S/onlinelog/o1_mf_4_b0y1dn8v_.log'
ORA-00312: online log 4 thread 1: '/opt/app/oracle/oradata/ENT12C1S/onlinelog/o1_mf_4_b0y1dml4_.log'
Managed Standby Recovery not using Real Time Apply
RFS[16]: Assigned to RFS process (PID:10165)
RFS[16]: Selected log 5 for thread 1 sequence 1504 dbid 209099011 branch 833730501
Fri Sep 12 17:32:28 2014
Recovery interrupted!
Recovered data files to a consistent state at change 19573793
Fri Sep 12 17:32:28 2014
Errors in file /opt/app/oracle/diag/rdbms/ent12c1s/ent12c1s/trace/ent12c1s_pr00_10098.trc:
ORA-00742: Log read detects lost write in thread 1 sequence 1503 block 868
ORA-00312: online log 4 thread 1: '/opt/app/oracle/fast_recovery_area/ENT12C1S/onlinelog/o1_mf_4_b0y1dn8v_.log'
ORA-00312: online log 4 thread 1: '/opt/app/oracle/oradata/ENT12C1S/onlinelog/o1_mf_4_b0y1dml4_.log'
Fri Sep 12 17:32:28 2014
MRP0: Background Media Recovery process shutdown (ent12c1s)
Fri Sep 12 17:32:28 2014
Archived Log entry 107 added for thread 1 sequence 1503 rlc 833730501 ID 0xde697d0 dest 3:
Fri Sep 12 17:38:51 2014
alter database recover managed standby database disconnect  <-- manual start at 17:38 after it was stopped 17:32

So it maybe good idea to keep an eye on the recovery when the far sync instance terminates. According to 1302539.1 when active data guard is in place the there's automatic block repair transparent to the user. Once the far sync instance is backup again the redo transport will go back to original setting

SQL>  SELECT DEST_NAME,STATUS,DESTINATION FROM V$ARCHIVE_DEST WHERE DESTINATION IS NOT NULL;

DEST_NAME                      STATUS    DESTINATION
------------------------------ --------- ------------------------------
LOG_ARCHIVE_DEST_1             VALID     USE_DB_RECOVERY_FILE_DEST
LOG_ARCHIVE_DEST_2             VALID     FS12C1TNS
LOG_ARCHIVE_DEST_3             ALTERNATE ENT12C1STNS

And following could be observed on the alert log of primary

Destination LOG_ARCHIVE_DEST_2 is UNSYNCHRONIZED
LGWR: Standby redo logfile selected to archive thread 1 sequence 1506
LGWR: Standby redo logfile selected for thread 1 sequence 1506 for destination LOG_ARCHIVE_DEST_2
Fri Sep 12 16:21:08 2014
Thread 1 advanced to log sequence 1506 (LGWR switch)
  Current log# 1 seq# 1506 mem# 0: /data/oradata/ENT12C1/onlinelog/o1_mf_1_9bcsl3ds_.log
  Current log# 1 seq# 1506 mem# 1: /data/flash_recovery/ENT12C1/onlinelog/o1_mf_1_9bcsl3hm_.log
Fri Sep 12 16:21:08 2014
Archived Log entry 1408 added for thread 1 sequence 1505 ID 0xde697d0 dest 1:
Fri Sep 12 16:21:11 2014
Thread 1 cannot allocate new log, sequence 1507
Checkpoint not complete
  Current log# 1 seq# 1506 mem# 0: /data/oradata/ENT12C1/onlinelog/o1_mf_1_9bcsl3ds_.log
  Current log# 1 seq# 1506 mem# 1: /data/flash_recovery/ENT12C1/onlinelog/o1_mf_1_9bcsl3hm_.log
Fri Sep 12 16:21:14 2014
Destination LOG_ARCHIVE_DEST_2 is SYNCHRONIZED

On the alert log of the far sync instance

Primary database is in MAXIMUM AVAILABILITY mode
Changing standby controlfile to RESYNCHRONIZATION level
Standby controlfile consistent with primary
RFS[1]: Assigned to RFS process (PID:3557)
RFS[1]: Selected log 5 for thread 1 sequence 1506 dbid 209099011 branch 833730501
Fri Sep 12 17:04:06 2014
******************************************************************
TT00: Setting 'active' archival for destination LOG_ARCHIVE_DEST_2
******************************************************************
TT00: Standby redo logfile selected for thread 1 sequence 1506 for destination LOG_ARCHIVE_DEST_2
Fri Sep 12 17:04:07 2014
Changing standby controlfile to MAXIMUM PERFORMANCE mode
RFS[2]: Assigned to RFS process (PID:3561)
RFS[2]: Opened log for thread 1 sequence 1505 dbid 209099011 branch 833730501
Fri Sep 12 17:04:08 2014
Archived Log entry 203 added for thread 1 sequence 1505 rlc 833730501 ID 0xde697d0 dest 2:
Fri Sep 12 17:04:13 2014
Changing standby controlfile to MAXIMUM AVAILABILITY mode

On the standby alert log

Primary database is in MAXIMUM PERFORMANCE mode
Changing standby controlfile to MAXIMUM AVAILABILITY mode
Changing standby controlfile to RESYNCHRONIZATION level

Similarly it is possible to shutdown the standby instance and see if primary is able to ship redo to far sync instance and if the redo is fetch by standby once it is started again. With these test the situation 1 on the figure shown at the beginning of the post is complete.

18. To test the situation 2 on the figure above do a switchover and check the redo transport via the far sync (fs12c1s) instance.

This conclude adding of far sync instances to existing data guard configuration on 12c.

Useful metalink notes
Cascaded Standby Databases in Oracle 12c [ID 1542969.1]
Data Guard 12c New Feature: Far Sync Standby [ID 1565071.1]
Best Practices for Corruption Detection, Prevention, and Automatic Repair - in a Data Guard Configuration [ID 1302539.1]