11gR2 RAC to RAC Data Guard

The initial setup is not much different between RAC-RAC and RAC-Single instance data guard, in a way setting up the data guard is same with some additional setps at the end of setup in the way of bringing the standby database under the control of clusterware.

The primary RAC environment consists of two nodes, hostname rac4 and rac5. The primary database name is rac11g2 and two instances are rac11g21 (running on rac4 node) and rac11g22 (on rac5).

The standby RAC environment also consists of two nodes (symmetrical data guard) hostname rac4b and rac5b. The standby database will be named rac11g2s and the corresponding instances will be rac11g2s1 (on rac4b) and rac11g2s2 (on rac5b).

The software version is 11.2.0.2 on both primary and standby (both grid home and oracle home). Primary site has database created and standby site has grid infrastructure and oracle home software installed.

From Data Guard Concepts and Administration

New 11.2 Features Common to Redo Apply and SQL Apply
As of Oracle Database 11g Release 2 (11.2.0.2), Oracle Data Guard is fully integrated with Oracle Real Application Clusters One Node (Oracle RAC One Node).
A Data Guard configuration can now consist of a primary database and up to 30 standby databases.
The FAL_CLIENT database initialization parameter is no longer required.
The default archive destination used by the Oracle Automatic Storage Management (Oracle ASM) feature and the fast recovery area feature has changed from LOG_ARCHIVE_DEST_10 to LOG_ARCHIVE_DEST_1.
Redo transport compression is no longer limited to compressing redo data only when a redo gap is being resolved. When compression is enabled for a destination, all redo data sent to that destination is compressed.
The new ALTER SYSTEM FLUSH REDO SQL statement can be used at failover time to flush unsent redo from a mounted primary database to a standby database, thereby allowing a zero data loss failover to be performed even if the primary database is not running in a zero data loss data protection mode. See Section 8.2.2 for more information.

New 11.2 Features Specific to Redo Apply
You can configure apply lag tolerance in a real-time query environment by using the new STANDBY_MAX_DATA_DELAY parameter.
You can use the new ALTER SESSION SYNC WITH PRIMARY SQL statement to ensure that a suitably configured physical standby database is synchronized with the primary database as of the time the statement is issued.
The V$DATAGUARD_STATS view has been enhanced to a greater degree of accuracy in many of its columns, including apply lag and transport lag.
You can view a histogram of apply lag values on the physical standby. To do so, query the new V$STANDBY_EVENT_HISTOGRAM view.
A corrupted data block in a primary database can be automatically replaced with an uncorrupted copy of that block from a physical standby database that is operating in real-time query mode. A corrupted block in a physical standby database can also be automatically replaced with an uncorrupted copy of the block from the primary database.

1. On the standby ASM instances would have been created when grid infrastructure was installed but not the ASM diskgroups. Using asmca create diskgroup on the standby ASM instances. In this case both primary and standby ASM diskgroups are named +DATA and +FLASH

2. Create following directory structures in all standby nodes

cd $ORACLE_BASE
mkdir admin
cd admin/
mkdir rac11g2s
cd rac11g2s/
mkdir adump  dpdump  hdump  pfile

3. It is assumed primary db is in archivelog mode if not put the db in archive log mode. This is required since duplication is done using active database.

4. Enable force logging on the primary database

alter database force logging;

5.Create standby log files for each thread on primary db. These should be same size as the online redo log files

SQL> alter database add standby logfile thread 1 size 52428800;

or

SQL> alter database add standby logfile thread 1;
SQL> alter database add standby logfile thread 2;

There should be at least one more redo log group per thread than the online redo logs.

6. Both on primary and standby add static listener entries in $GRID_HOME($CRS_HOME)/network/admin/listener.ora. If scan listener is used for connections create static listener for those as well. One key difference is both in 10gR2 and 11gR1 the static listener and default listeners have hostname appended to the end but on 11gR2 this is not the case. Find out the available listeners with

srvctl config listener
Name: LISTENER
Network: 1, Owner: oracle
Home: 
End points: TCP:1521

srvctl config scan_listener
SCAN Listener LISTENER_SCAN1 exists. Port: TCP:1521

and create the static listeners for default listener as well as the scan listener. For standby on rac4b

more listener.ora
LISTENER=(DESCRIPTION=(ADDRESS_LIST=(ADDRESS=(PROTOCOL=IPC)(KEY=LISTENER))))            # line added by Agent
LISTENER_SCAN1=(DESCRIPTION=(ADDRESS_LIST=(ADDRESS=(PROTOCOL=IPC)(KEY=LISTENER_SCAN1))))                # line added by Agent
ENABLE_GLOBAL_DYNAMIC_ENDPOINT_LISTENER_SCAN1=ON                # line added by Agent
ENABLE_GLOBAL_DYNAMIC_ENDPOINT_LISTENER=ON              # line added by Agent


SID_LIST_LISTENER =
(SID_LIST =
(SID_DESC =
(GLOBAL_DBNAME = rac11g2s.domain.net)
(SID_NAME = rac11g2s1)
(ORACLE_HOME = /opt/app/oracle/product/11.2.0/db_1)
)
)


SID_LIST_LISTENER_SCAN1 =
(SID_LIST =
(SID_DESC =
(GLOBAL_DBNAME = rac11g2s.domain.net)
(SID_NAME = rac11g2s1)
(ORACLE_HOME = /opt/app/oracle/product/11.2.0/db_1)
)
)

on rac5b

more listener.ora
LISTENER_SCAN1=(DESCRIPTION=(ADDRESS_LIST=(ADDRESS=(PROTOCOL=IPC)(KEY=LISTENER_SCAN1))))                # line added by Agent
LISTENER=(DESCRIPTION=(ADDRESS_LIST=(ADDRESS=(PROTOCOL=IPC)(KEY=LISTENER))))            # line added by Agent
ENABLE_GLOBAL_DYNAMIC_ENDPOINT_LISTENER=ON              # line added by Agent
ENABLE_GLOBAL_DYNAMIC_ENDPOINT_LISTENER_SCAN1=ON                # line added by Agent


SID_LIST_LISTENER =
(SID_LIST =
(SID_DESC =
(GLOBAL_DBNAME = rac11g2s.domain.net)
(SID_NAME = rac11g2s2)
(ORACLE_HOME = /opt/app/oracle/product/11.2.0/db_1)
)
)

SID_LIST_LISTENER_SCAN1 =
(SID_LIST =
(SID_DESC =
(GLOBAL_DBNAME = rac11g2s.domain.net)
(SID_NAME = rac11g2s2)
(ORACLE_HOME = /opt/app/oracle/product/11.2.0/db_1)
)
)

In 11gR2 there's another file in $ORACLE_HOME/network/admin folder called endpoints_listener.ora. This is there for backward compatibility with pre-11.2 databases. More on metalink note 11.2 Scan and Node TNS Listener Setup Examples [ID 1070607.1]
Stop and start the listeners on standby

srvctl stop listener -n `hostname -s`
srvctl start listener -n `hostname -s`

srvctl stop scan_listener
srvctl start scan_listener

After this stop and start of listeners doing a listener status listener would show the statically registered instance and service as below

Service "rac11g2s.domain.net" has 1 instance(s).
Instance "rac11g2s1", status UNKNOWN, has 1 handler(s) for this service...
The command completed successfully

Do the same for primary environment on rac4

more listener.ora
LISTENER=(DESCRIPTION=(ADDRESS_LIST=(ADDRESS=(PROTOCOL=IPC)(KEY=LISTENER))))            # line added by Agent
LISTENER_SCAN1=(DESCRIPTION=(ADDRESS_LIST=(ADDRESS=(PROTOCOL=IPC)(KEY=LISTENER_SCAN1))))                # line added by Agent
ENABLE_GLOBAL_DYNAMIC_ENDPOINT_LISTENER_SCAN1=ON                # line added by Agent
ENABLE_GLOBAL_DYNAMIC_ENDPOINT_LISTENER=ON              # line added by Agent


SID_LIST_LISTENER =
(SID_LIST =
(SID_DESC =
(GLOBAL_DBNAME = rac11g2.domain.net)
(SID_NAME = rac11g21)
(ORACLE_HOME = /opt/app/oracle/product/11.2.0/db_1)
)
)


SID_LIST_LISTENER_SCAN1 =
(SID_LIST =
(SID_DESC =
(GLOBAL_DBNAME = rac11g2.domain.net)
(SID_NAME = rac11g21)
(ORACLE_HOME = /opt/app/oracle/product/11.2.0/db_1)
)
)

on rac5

more listener.ora
LISTENER_SCAN1=(DESCRIPTION=(ADDRESS_LIST=(ADDRESS=(PROTOCOL=IPC)(KEY=LISTENER_SCAN1))))                # line added by Agent
LISTENER=(DESCRIPTION=(ADDRESS_LIST=(ADDRESS=(PROTOCOL=IPC)(KEY=LISTENER))))            # line added by Agent
ENABLE_GLOBAL_DYNAMIC_ENDPOINT_LISTENER=ON              # line added by Agent
ENABLE_GLOBAL_DYNAMIC_ENDPOINT_LISTENER_SCAN1=ON                # line added by Agent


SID_LIST_LISTENER =
(SID_LIST =
(SID_DESC =
(GLOBAL_DBNAME = rac11g2.domain.net)
(SID_NAME = rac11g22)
(ORACLE_HOME = /opt/app/oracle/product/11.2.0/db_1)
)
)


SID_LIST_LISTENER_SCAN1 =
(SID_LIST =
(SID_DESC =
(GLOBAL_DBNAME = rac11g2.domain.net)
(SID_NAME = rac11g22)
(ORACLE_HOME = /opt/app/oracle/product/11.2.0/db_1)
)
)

7. Create TNS entries on both primary and standby. On all primary node's $ORACLE_HOME/network/admin

RAC11G2SCANB =
(DESCRIPTION =
(ADDRESS = (PROTOCOL = TCP)(HOST = racb-scan)(PORT = 1521))
(CONNECT_DATA =
(SERVER = DEDICATED)
(SERVICE_NAME = rac11g2s.domain.net)
)
)

RAC11G2S =
(DESCRIPTION =
(ADDRESS = (PROTOCOL = TCP)(HOST = rac4b-vip)(PORT = 1521))
(ADDRESS = (PROTOCOL = TCP)(HOST = rac5b-vip)(PORT = 1521))
(CONNECT_DATA =
(SERVER = DEDICATED)
(SERVICE_NAME = rac11g2s.domain.net)
)
)

RAC11G2S1 =
(DESCRIPTION =
(ADDRESS = (PROTOCOL = TCP)(HOST = rac4b-vip)(PORT = 1521))
(CONNECT_DATA =
(SERVER = DEDICATED)
(SERVICE_NAME = rac11g2s.domain.net)
(INSTANCE_NAME = rac11g2s1)
)
)


RAC11G2S2 =
(DESCRIPTION =
(ADDRESS = (PROTOCOL = TCP)(HOST = rac5b-vip)(PORT = 1521))
(CONNECT_DATA =
(SERVER = DEDICATED)
(SERVICE_NAME = rac11g2s.domain.net)
(INSTANCE_NAME = rac11g2s2)
)
)

RAC11G2 =
(DESCRIPTION =
(ADDRESS = (PROTOCOL = TCP)(HOST = rac4-vip)(PORT = 1521))
(ADDRESS = (PROTOCOL = TCP)(HOST = rac5-vip)(PORT = 1521))
(CONNECT_DATA =
(SERVER = DEDICATED)
(SERVICE_NAME = rac11g2.domain.net)
)
)

RAC11G21 =
(DESCRIPTION =
(ADDRESS = (PROTOCOL = TCP)(HOST = rac4-vip)(PORT = 1521))
(CONNECT_DATA =
(SERVER = DEDICATED)
(SERVICE_NAME = rac11g2.domain.net)
(INSTANCE_NAME = rac11g21)
)
)


RAC11G22 =
(DESCRIPTION =
(ADDRESS = (PROTOCOL = TCP)(HOST = rac5-vip)(PORT = 1521))
(CONNECT_DATA =
(SERVER = DEDICATED)
(SERVICE_NAME = rac11g2.domain.net)
(INSTANCE_NAME = rac11g22)
)
)

On all standby node's $ORACLE_HOME/network/admin

RAC11G2SCAN =
(DESCRIPTION =
(ADDRESS = (PROTOCOL = TCP)(HOST = rac-scan)(PORT = 1521))
(CONNECT_DATA =
(SERVER = DEDICATED)
(SERVICE_NAME = rac11g2.domain.net)
)
)

RAC11G2 =
(DESCRIPTION =
(ADDRESS = (PROTOCOL = TCP)(HOST = rac4-vip)(PORT = 1521))
(ADDRESS = (PROTOCOL = TCP)(HOST = rac5-vip)(PORT = 1521))
(CONNECT_DATA =
(SERVER = DEDICATED)
(SERVICE_NAME = rac11g2.domain.net)
)
)

RAC11G21 =
(DESCRIPTION =
(ADDRESS = (PROTOCOL = TCP)(HOST = rac4-vip)(PORT = 1521))
(CONNECT_DATA =
(SERVER = DEDICATED)
(SERVICE_NAME = rac11g2.domain.net)
(INSTANCE_NAME = rac11g21)
)
)


RAC11G22 =
(DESCRIPTION =
(ADDRESS = (PROTOCOL = TCP)(HOST = rac5-vip)(PORT = 1521))
(CONNECT_DATA =
(SERVER = DEDICATED)
(SERVICE_NAME = rac11g2.domain.net)
(INSTANCE_NAME = rac11g22)
)
)

RAC11G2S =
(DESCRIPTION =
(ADDRESS = (PROTOCOL = TCP)(HOST = rac4b-vip)(PORT = 1521))
(ADDRESS = (PROTOCOL = TCP)(HOST = rac5b-vip)(PORT = 1521))
(CONNECT_DATA =
(SERVER = DEDICATED)
(SERVICE_NAME = rac11g2s.domain.net)
)
)

RAC11G2S1 =
(DESCRIPTION =
(ADDRESS = (PROTOCOL = TCP)(HOST = rac4b-vip)(PORT = 1521))
(CONNECT_DATA =
(SERVER = DEDICATED)
(SERVICE_NAME = rac11g2s.domain.net)
(INSTANCE_NAME = rac11g2s1)
)
)


RAC11G2S2 =
(DESCRIPTION =
(ADDRESS = (PROTOCOL = TCP)(HOST = rac5b-vip)(PORT = 1521))
(CONNECT_DATA =
(SERVER = DEDICATED)
(SERVICE_NAME = rac11g2s.domain.net)
(INSTANCE_NAME = rac11g2s2)
)
)

8. Before modifying initialization parameters create a pfile from the spfile on the primary db

create pfile='/home/oracle/primaryinitpfile.ora' from spfile;

9. Modify the initialization parameters on the primary db. Remote archive dest state is set to defer, enable it before running the standby duplication. LGWR with ASYNC and NOAFFIRM is used as the redo transport this could be changed to suite the protection mode. Also FAL_CLIENT parameter has been set to each instance. This could be set to RAC11G2 TNS entry which has all entries.

alter system set log_archive_config='dg_config=(rac11g2,rac11g2s)' scope=both ;
alter system set log_archive_dest_1='location=use_db_recovery_file_dest valid_for=(all_logfiles,all_roles) db_unique_name=rac11g2' scope=both;
alter system set log_archive_dest_2='service=RAC11G2S LGWR ASYNC NOAFFIRM max_failure=10 max_connections=5 reopen=180 valid_for=(online_logfiles,primary_role) db_unique_name=rac11g2s' scope=both;
alter system set log_archive_dest_state_2='defer' scope=both;
alter system set log_archive_dest_state_1='enable' scope=both;
alter system set fal_server='RAC11G2S1','RAC11G2S2' scope=both;
alter system set fal_client='RAC11G21' scope=both sid='rac11g21'; --- since not using scan
alter system set fal_client='RAC11G22' scope=both sid='rac11g22'; --- sicne not using scan
alter system set log_archive_max_processes=10 scope=both;
alter system set db_file_name_convert='+DATA/rac11g2s','+DATA/rac11g2' scope=spfile;
alter system set log_file_name_convert='+FLASH/rac11g2s','+FLASH/rac11g2','+DATA/rac11g2s','+DATA/rac11g2' scope=spfile;
alter system set standby_file_management='AUTO' scope=both;

To make these changes take effect it requires a database restart not an instance restart this is due to the fact that filename convert parameter must be same on all nodes. If an instance restart is done it will fail with the following error

on alert log

NOTE: Loaded library: System
SUCCESS: diskgroup DATA was mounted
SUCCESS: diskgroup FLASH was mounted
NOTE: dependency between database rac11g2 and diskgroup resource ora.DATA.dg is established
NOTE: dependency between database rac11g2 and diskgroup resource ora.FLASH.dg is established
ORA-1105 signalled during: ALTER DATABASE MOUNT /* db agent *//* {1:7917:278} */...
Fri Feb 18 14:49:17 2011
Shutting down instance (abort)
License high water mark = 1
USER (ospid: 9322): terminating the instance
Instance terminated by USER, pid = 9322
Fri Feb 18 14:49:18 2011
Instance shutdown complete

on the command shell

$ srvctl stop instance -d rac11g2 -i rac11g21
$ srvctl start instance -d rac11g2 -i rac11g21
PRCR-1013 : Failed to start resource ora.rac11g2.db
PRCR-1064 : Failed to start resource ora.rac11g2.db on node rac4
CRS-5017: The resource action "ora.rac11g2.db start" encountered the following error:
ORA-01105: mount is incompatible with mounts by other instances
ORA-01677: standby file name convert parameters differ from other instance
CRS-2674: Start of 'ora.rac11g2.db' on 'rac4' failed

If the dataguard traffic is only one way (don't plan to use current primary as a standby in the future, then filename convert parameters could be ommited and a instnace restart could be carried out), otherwise do a database restart which means brining the system down.

10. Another difference between 10gR2/11gR1 and 11gR2 is the local_listener and remote_listener parameters. According to Real Application Clusters Installation Guide During Oracle Database creation, the LOCAL_LISTENER parameter is automatically configured to point to the local listener for the database. The Database Agent sets the LOCAL_LISTENER parameter to a connect descriptor that does not require a TNS alias.
You can set a value manually for LOCAL_LISTENER. However, Oracle recommends that you leave the parameter unset so that the Database Agent can maintain it automatically. If you set LOCAL_LISTENER, then the Agent does not automatically update this value. If you do not set LOCAL_LISTENER, then the Database Agent automatically keeps the database associated with the Grid home's node listener updated, even as the ports or IP of that listener are changed.

With Oracle Clusterware 11g release 2 and later, Database Configuration Agent (DBCA) no longer sets the LOCAL_LISTENER parameter. The Oracle Clusterware agent that starts the database sets the LOCAL_LISTENER parameter dynamically, and it sets it to the actual value, not an alias. So listener_alias entries are no longer needed in the tnsnames.ora file. For the REMOTE_LISTENER parameter, Oracle Clusterware uses the EZ connect syntax scanname:scanport, so no entries are needed for the REMOTE_LISTENER parameter in the tnsnames.ora file.More on 11gR2 Grid Infrastructure Single Client Access Name (SCAN) Explained [ID 887522.1]

11. On 11gr2 remote listener is set to scan identify the standby scan name with

srvctl config scan
SCAN name: racb-scan, Network: 1/192.168.0.0/255.255.255.0/eth0
SCAN VIP name: scan1, IP: /racb-scan/192.168.0.92

This will be used later on.

12. Copy the primary password file to standby nodes and rename accordingly. On rac4b where instance rac11g2s1 will be running password file will be named orapwrac11g2s1 and on rac5b orapwrac11g2s2.

13. Select one node (rac4b in this case) on the standby cluster and create a pfile with db_name entry which will have the standby database name (not the instance name). Rename the pfile with the instance name.

cat initrac11g2s1.ora
db_name=rac11g2s

14. Start the instance in nomount mode

export ORACLE_SID=rac11g2s1
SQL> startup nomount pfile='?/dbs/initrac11g2s1.ora';
ORACLE instance started.

Total System Global Area  217157632 bytes
Fixed Size                  2225064 bytes
Variable Size             159386712 bytes
Database Buffers           50331648 bytes
Redo Buffers                5214208 bytes

15. On primary db enable the deferred log archive destination

SQL> alter system set log_archive_dest_state_2='enable' SCOPE=both sid='*';

16. On one node in the primary rac (rac4 in this case) connect to primary instance (target) as well standby instance (using rman auxiliary connection)

rman target / auxiliary sys/******@rac11g2s1

Recovery Manager: Release 11.2.0.2.0 - Production on Fri Feb 18 15:12:39 2011

Copyright (c) 1982, 2009, Oracle and/or its affiliates.  All rights reserved.

connected to target database: RAC11G2 (DBID=379668842)
connected to auxiliary database: RAC11G2S (not mounted)

17. Run the duplication command with

duplicate target database for standby from active database
spfile
parameter_value_convert 'rac11g2','rac11g2s','RAC11G2','RAC11G2S'
set db_unique_name='rac11g2s'
set db_file_name_convert='+DATA/rac11g2','+DATA/rac11g2s'
set log_file_name_convert='+FLASH/rac11g2','+FLASH/rac11g2s','+DATA/rac11g2','+DATA/rac11g2s'
set control_files='+DATA','+FLASH'
set instance_number='1'
set log_archive_max_processes='5'
set fal_client='RAC11G2S'
set fal_server='RAC11G21','RAC11G22'
set remote_listener='racb-scan:1521'
reset local_listener
set log_archive_dest_2='service=RAC11G2 LGWR ASYNC NOAFFIRM max_failure=10 max_connections=5 reopen=180 valid_for=(online_logfiles,primary_role) db_unique_name=rac11g2'
set log_archive_dest_1='location=use_db_recovery_file_dest valid_for=(all_logfiles,all_roles) db_unique_name=rac11g2s';

Key parameters are data file and log file name converters as well as the parameter value converter which maps primary environment values to appropriate standby environment values. local_listener is reset as it is auto set by grid infrastructure. remote_listener value is set the scan ip and port identified earlier.

rman target / auxiliary sys/******@rac11g2s1

Recovery Manager: Release 11.2.0.2.0 - Production on Fri Feb 18 16:10:16 2011

Copyright (c) 1982, 2009, Oracle and/or its affiliates.  All rights reserved.

connected to target database: RAC11G2 (DBID=379668842)
connected to auxiliary database: RAC11G2S (not mounted)

RMAN> duplicate target database for standby from active database
2> spfile
3> parameter_value_convert 'rac11g2','rac11g2s','RAC11G2','RAC11G2S'
4> set db_unique_name='rac11g2s'
5> set db_file_name_convert='+DATA/rac11g2','+DATA/rac11g2s'
6> set log_file_name_convert='+FLASH/rac11g2','+FLASH/rac11g2s','+DATA/rac11g2','+DATA/rac11g2s'
7> set control_files='+DATA','+FLASH'
8> set instance_number='1'
9> set log_archive_max_processes='5'
10> set fal_client='RAC11G2S'
11> set fal_server='RAC11G21','RAC11G22'
12> set remote_listener='racb-scan:1521'
13> reset local_listener
14> set log_archive_dest_2='service=RAC11G2 LGWR ASYNC NOAFFIRM max_failure=10 max_connections=5 reopen=180 valid_for=(online_logfiles,primary_role) db_unique_name=rac11g2'
15> set log_archive_dest_1='location=use_db_recovery_file_dest valid_for=(all_logfiles,all_roles) db_unique_name=rac11g2s';

Starting Duplicate Db at 18-FEB-11
using target database control file instead of recovery catalog
allocated channel: ORA_AUX_DISK_1
channel ORA_AUX_DISK_1: SID=13 device type=DISK

contents of Memory Script:
{
backup as copy reuse
targetfile  '/opt/app/oracle/product/11.2.0/db_1/dbs/orapwrac11g21' auxiliary format
'/opt/app/oracle/product/11.2.0/db_1/dbs/orapwrac11g2s1'   targetfile
'+DATA/rac11g2/spfilerac11g2.ora' auxiliary format
'/opt/app/oracle/product/11.2.0/db_1/dbs/spfilerac11g2s1.ora'   ;
sql clone "alter system set spfile= ''/opt/app/oracle/product/11.2.0/db_1/dbs/spfilerac11g2s1.ora''";
}
executing Memory Script

Starting backup at 18-FEB-11
allocated channel: ORA_DISK_1
channel ORA_DISK_1: SID=155 instance=rac11g21 device type=DISK
Finished backup at 18-FEB-11

sql statement: alter system set spfile= ''/opt/app/oracle/product/11.2.0/db_1/dbs/spfilerac11g2s1.ora''

contents of Memory Script:
{
sql clone "alter system set  audit_file_dest =
''/opt/app/oracle/admin/rac11g2s/adump'' comment=
'''' scope=spfile";
sql clone "alter system set  dispatchers =
''(PROTOCOL=TCP) (SERVICE=rac11g2sXDB)'' comment=
'''' scope=spfile";
sql clone "alter system set  db_unique_name =
''rac11g2s'' comment=
'''' scope=spfile";
sql clone "alter system set  db_file_name_convert =
''+DATA/rac11g2'', ''+DATA/rac11g2s'' comment=
'''' scope=spfile";
sql clone "alter system set  log_file_name_convert =
''+FLASH/rac11g2'', ''+FLASH/rac11g2s'', ''+DATA/rac11g2'', ''+DATA/rac11g2s'' comment=
'''' scope=spfile";
sql clone "alter system set  control_files =
''+DATA'', ''+FLASH'' comment=
'''' scope=spfile";
sql clone "alter system set  instance_number =
1 comment=
'''' scope=spfile";
sql clone "alter system set  log_archive_max_processes =
5 comment=
'''' scope=spfile";
sql clone "alter system set  fal_client =
''RAC11G2S'' comment=
'''' scope=spfile";
sql clone "alter system set  fal_server =
''RAC11G21'', ''RAC11G22'' comment=
'''' scope=spfile";
sql clone "alter system set  remote_listener =
''racb-scan:1521'' comment=
'''' scope=spfile";
sql clone "alter system set  log_archive_dest_2 =
''service=RAC11G2 LGWR ASYNC NOAFFIRM max_failure=10 max_connections=5 reopen=180 valid_for=(online_logfiles,primary_role) db_unique_name=rac11g2'' comment=
'''' scope=spfile";
sql clone "alter system set  log_archive_dest_1 =
''location=use_db_recovery_file_dest valid_for=(all_logfiles,all_roles) db_unique_name=rac11g2s'' comment=
'''' scope=spfile";
shutdown clone immediate;
startup clone nomount;
}
executing Memory Script

sql statement: alter system set  audit_file_dest =  ''/opt/app/oracle/admin/rac11g2s/adump'' comment= '''' scope=spfile

sql statement: alter system set  dispatchers =  ''(PROTOCOL=TCP) (SERVICE=rac11g2sXDB)'' comment= '''' scope=spfile

sql statement: alter system set  db_unique_name =  ''rac11g2s'' comment= '''' scope=spfile

sql statement: alter system set  db_file_name_convert =  ''+DATA/rac11g2'', ''+DATA/rac11g2s'' comment= '''' scope=spfile

sql statement: alter system set  log_file_name_convert =  ''+FLASH/rac11g2'', ''+FLASH/rac11g2s'', ''+DATA/rac11g2'', ''+DATA/rac11g2s'' comment= '''' scope=spfile

sql statement: alter system set  control_files =  ''+DATA'', ''+FLASH'' comment= '''' scope=spfile

sql statement: alter system set  instance_number =  1 comment= '''' scope=spfile

sql statement: alter system set  log_archive_max_processes =  5 comment= '''' scope=spfile

sql statement: alter system set  fal_client =  ''RAC11G2S'' comment= '''' scope=spfile

sql statement: alter system set  fal_server =  ''RAC11G21'', ''RAC11G22'' comment= '''' scope=spfile

sql statement: alter system set  remote_listener =  ''racb-scan:1521'' comment= '''' scope=spfile

sql statement: alter system set  log_archive_dest_2 =  ''service=RAC11G2 LGWR ASYNC NOAFFIRM max_failure=10 max_connections=5 reopen=180 valid_for=(online_logfiles,primary_role) db_unique_name=rac11g2'' comment= '''' scope=spfile

sql statement: alter system set  log_archive_dest_1 =  ''location=use_db_recovery_file_dest valid_for=(all_logfiles,all_roles) db_unique_name=rac11g2s'' comment= '''' scope=spfile

Oracle instance shut down

connected to auxiliary database (not started)
Oracle instance started

Total System Global Area     630501376 bytes

Fixed Size                     2229120 bytes
Variable Size                247467136 bytes
Database Buffers             373293056 bytes
Redo Buffers                   7512064 bytes

contents of Memory Script:
{
sql clone "alter system set  control_files =
''+DATA/rac11g2s/controlfile/current.256.743447505'', ''+FLASH/rac11g2s/controlfile/current.256.743447507'' comment=
''Set by RMAN'' scope=spfile";
backup as copy current controlfile for standby auxiliary format  '+DATA/rac11g2s/controlfile/current.257.743447507';
restore clone controlfile to  '+FLASH/rac11g2s/controlfile/current.257.743447507' from
'+DATA/rac11g2s/controlfile/current.257.743447507';
sql clone "alter system set  control_files =
''+DATA/rac11g2s/controlfile/current.257.743447507'', ''+FLASH/rac11g2s/controlfile/current.257.743447507'' comment=
''Set by RMAN'' scope=spfile";
shutdown clone immediate;
startup clone nomount;
}
executing Memory Script

sql statement: alter system set  control_files =   ''+DATA/rac11g2s/controlfile/current.256.743447505'', ''+FLASH/rac11g2s/controlfile/current.256.743447507'' comment= ''Set by RMAN'' scope=spfile

Starting backup at 18-FEB-11
using channel ORA_DISK_1
channel ORA_DISK_1: starting datafile copy
copying standby control file
output file name=/opt/app/oracle/product/11.2.0/db_1/dbs/snapcf_rac11g21.f tag=TAG20110218T161053 RECID=1 STAMP=743443854
channel ORA_DISK_1: datafile copy complete, elapsed time: 00:00:07
Finished backup at 18-FEB-11

Starting restore at 18-FEB-11
allocated channel: ORA_AUX_DISK_1
channel ORA_AUX_DISK_1: SID=19 instance=rac11g2s1 device type=DISK

channel ORA_AUX_DISK_1: copied control file copy
Finished restore at 18-FEB-11

sql statement: alter system set  control_files =   ''+DATA/rac11g2s/controlfile/current.257.743447507'', ''+FLASH/rac11g2s/controlfile/current.257.743447507'' comment= ''Set by RMAN'' scope=spfile

Oracle instance shut down

connected to auxiliary database (not started)
Oracle instance started

Total System Global Area     630501376 bytes

Fixed Size                     2229120 bytes
Variable Size                247467136 bytes
Database Buffers             373293056 bytes
Redo Buffers                   7512064 bytes

contents of Memory Script:
{
sql clone 'alter database mount standby database';
}
executing Memory Script

sql statement: alter database mount standby database
RMAN-05529: WARNING: DB_FILE_NAME_CONVERT resulted in invalid ASM names; names changed to disk group only.

contents of Memory Script:
{
set newname for tempfile  1 to
"+data";
switch clone tempfile all;
set newname for datafile  1 to
"+data";
set newname for datafile  2 to
"+data";
set newname for datafile  3 to
"+data";
set newname for datafile  4 to
"+data";
set newname for datafile  5 to
"+data";
set newname for datafile  6 to
"+data";
set newname for datafile  7 to
"+data";
backup as copy reuse
datafile  1 auxiliary format
"+data"   datafile
2 auxiliary format
"+data"   datafile
3 auxiliary format
"+data"   datafile
4 auxiliary format
"+data"   datafile
5 auxiliary format
"+data"   datafile
6 auxiliary format
"+data"   datafile
7 auxiliary format
"+data"   ;
sql 'alter system archive log current';
}
executing Memory Script

executing command: SET NEWNAME

renamed tempfile 1 to +data in control file

executing command: SET NEWNAME

executing command: SET NEWNAME

executing command: SET NEWNAME

executing command: SET NEWNAME

executing command: SET NEWNAME

executing command: SET NEWNAME

executing command: SET NEWNAME

Starting backup at 18-FEB-11
using channel ORA_DISK_1
channel ORA_DISK_1: starting datafile copy
input datafile file number=00002 name=+DATA/rac11g2/datafile/sysaux.257.740770047
output file name=+DATA/rac11g2s/datafile/sysaux.258.743447539 tag=TAG20110218T161124
channel ORA_DISK_1: datafile copy complete, elapsed time: 00:01:25
channel ORA_DISK_1: starting datafile copy
input datafile file number=00001 name=+DATA/rac11g2/datafile/system.256.740770045
output file name=+DATA/rac11g2s/datafile/system.259.743447625 tag=TAG20110218T161124
channel ORA_DISK_1: datafile copy complete, elapsed time: 00:01:15
channel ORA_DISK_1: starting datafile copy
input datafile file number=00003 name=+DATA/rac11g2/datafile/undotbs1.258.740770049
output file name=+DATA/rac11g2s/datafile/undotbs1.260.743447699 tag=TAG20110218T161124
channel ORA_DISK_1: datafile copy complete, elapsed time: 00:00:25
channel ORA_DISK_1: starting datafile copy
input datafile file number=00005 name=+DATA/rac11g2/datafile/undotbs2.264.740770355
output file name=+DATA/rac11g2s/datafile/undotbs2.261.743447725 tag=TAG20110218T161124
channel ORA_DISK_1: datafile copy complete, elapsed time: 00:00:07
channel ORA_DISK_1: starting datafile copy
input datafile file number=00004 name=+DATA/rac11g2/datafile/users.259.740770049
output file name=+DATA/rac11g2s/datafile/users.262.743447733 tag=TAG20110218T161124
channel ORA_DISK_1: datafile copy complete, elapsed time: 00:00:01
channel ORA_DISK_1: starting datafile copy
input datafile file number=00006 name=+DATA/rac11g2/datafile/test.268.741633047
output file name=+DATA/rac11g2s/datafile/test.263.743447733 tag=TAG20110218T161124
channel ORA_DISK_1: datafile copy complete, elapsed time: 00:00:01
channel ORA_DISK_1: starting datafile copy
input datafile file number=00007 name=+DATA/rac11g2/datafile/test.269.741633183
output file name=+DATA/rac11g2s/datafile/test.264.743447735 tag=TAG20110218T161124
channel ORA_DISK_1: datafile copy complete, elapsed time: 00:00:01
Finished backup at 18-FEB-11

sql statement: alter system archive log current

contents of Memory Script:
{
switch clone datafile all;
}
executing Memory Script

datafile 1 switched to datafile copy
input datafile copy RECID=1 STAMP=743447740 file name=+DATA/rac11g2s/datafile/system.259.743447625
datafile 2 switched to datafile copy
input datafile copy RECID=2 STAMP=743447740 file name=+DATA/rac11g2s/datafile/sysaux.258.743447539
datafile 3 switched to datafile copy
input datafile copy RECID=3 STAMP=743447740 file name=+DATA/rac11g2s/datafile/undotbs1.260.743447699
datafile 4 switched to datafile copy
input datafile copy RECID=4 STAMP=743447740 file name=+DATA/rac11g2s/datafile/users.262.743447733
datafile 5 switched to datafile copy
input datafile copy RECID=5 STAMP=743447740 file name=+DATA/rac11g2s/datafile/undotbs2.261.743447725
datafile 6 switched to datafile copy
input datafile copy RECID=6 STAMP=743447740 file name=+DATA/rac11g2s/datafile/test.263.743447733
datafile 7 switched to datafile copy
input datafile copy RECID=7 STAMP=743447740 file name=+DATA/rac11g2s/datafile/test.264.743447735
Finished Duplicate Db at 18-FEB-11

18. Once the above command suceeds on primary on the standby(rac4b node) run the log apply command

SQL> alter database recover managed standby database using current logfile disconnect;
Database altered.

On the database alert log it could be observed archived logs being applied

Fri Feb 18 17:19:56 2011
alter database recover managed standby database using current logfile disconnect
Attempt to start background Managed Standby Recovery process (rac11g2s1)
Fri Feb 18 17:19:56 2011
MRP0 started with pid=60, OS id=8639
MRP0: Background Managed Standby Recovery process started (rac11g2s1)
started logmerger process
Fri Feb 18 17:20:01 2011
Managed Standby Recovery starting Real Time Apply
Parallel Media Recovery started with 2 slaves
Waiting for all non-current ORLs to be archived...
All non-current ORLs have been archived.
Fri Feb 18 17:20:02 2011
Media Recovery Log +FLASH/rac11g2s/archivelog/2011_02_18/thread_1_seq_53.269.743447835
Media Recovery Log +FLASH/rac11g2s/archivelog/2011_02_18/thread_2_seq_43.268.743447835
Completed: alter database recover managed standby database using current logfile disconnect
Media Recovery Log +FLASH/rac11g2s/archivelog/2011_02_18/thread_1_seq_54.271.743447847
Media Recovery Log +FLASH/rac11g2s/archivelog/2011_02_18/thread_2_seq_44.270.743447847
Media Recovery Log +FLASH/rac11g2s/archivelog/2011_02_18/thread_2_seq_45.273.743447887
Media Recovery Log +FLASH/rac11g2s/archivelog/2011_02_18/thread_1_seq_55.272.743447883
Media Recovery Waiting for thread 1 sequence 56 (in transit)
Recovery of Online Redo Log: Thread 1 Group 6 Seq 56 Reading mem 0
Mem# 0: +DATA/rac11g2s/onlinelog/group_6.270.743447749
Mem# 1: +FLASH/rac11g2s/onlinelog/group_6.263.743447751
Media Recovery Waiting for thread 2 sequence 46 (in transit)
Recovery of Online Redo Log: Thread 2 Group 9 Seq 46 Reading mem 0
Mem# 0: +DATA/rac11g2s/onlinelog/group_9.273.743447755
Mem# 1: +FLASH/rac11g2s/onlinelog/group_9.266.743447755

Another way to find out is to connect into the standby instance (rac11g2s1 running on rac4b node) and issue

SQL> select sequence#,thread#,applied from v$archived_log;

SEQUENCE#    THREAD# APPLIED
---------- ---------- ---------
53          1 YES
43          2 YES
54          1 YES
44          2 YES
55          1 YES
45          2 IN-MEMORY
6 rows selected.

which shows YES on archive log applied column.



19. With above configuration inplace the data guard is setup and working. The next step is to bring the standby database under the control of the cluster. For this on the standby instance create a pfile from the spfile that was created during the duplication process.

SQL> create pfile='/home/oracle/stdbypfile.ora' from spfile;

20. Edit the pfile and remove any references to primary db (rac11g2) except db_name this must be same as primary db. Set instance specific instance_number, undo_tablespace, fal_client and thread. pfile content after the edit is given below

rac11g2s1.__db_cache_size=373293056
rac11g2s1.__java_pool_size=4194304
rac11g2s1.__large_pool_size=4194304
rac11g2s1.__oracle_base='/opt/app/oracle'#ORACLE_BASE set from environment
rac11g2s1.__pga_aggregate_target=209715200
rac11g2s1.__sga_target=633339904
rac11g2s1.__shared_io_pool_size=0
rac11g2s1.__shared_pool_size=239075328
rac11g2s1.__streams_pool_size=0
rac11g2s2.__db_cache_size=373293056
rac11g2s2.__java_pool_size=4194304
rac11g2s2.__large_pool_size=4194304
rac11g2s2.__oracle_base='/opt/app/oracle'#ORACLE_BASE set from environment
rac11g2s2.__pga_aggregate_target=209715200
rac11g2s2.__sga_target=633339904
rac11g2s2.__shared_io_pool_size=0
rac11g2s2.__shared_pool_size=239075328
rac11g2s2.__streams_pool_size=0
*.audit_file_dest='/opt/app/oracle/admin/rac11g2s/adump'
*.audit_trail='db'
*.cluster_database=true
*.compatible='11.2.0.0.0'
*.control_files='+DATA/rac11g2s/controlfile/current.257.743447507','+FLASH/rac11g2s/controlfile/current.257.743447507'#Set by RMAN
*.db_block_size=8192
*.db_create_file_dest='+DATA'
*.db_domain='domain.net'
*.db_file_name_convert='+DATA/rac11g2','+DATA/rac11g2s'
*.db_name='rac11g2'
*.db_recovery_file_dest='+FLASH'
*.db_recovery_file_dest_size=9470738432
*.db_unique_name='rac11g2s'
*.diagnostic_dest='/opt/app/oracle'
*.dispatchers='(PROTOCOL=TCP) (SERVICE=rac11g2sXDB)'
rac11g2s1.fal_client='RAC11G2S1'
rac1g2s22.fal_client='RAC11G2S2'
*.fal_server='RAC11G21','RAC11G22'
rac11g2s1.instance_number=1
rac11g2s2.instance_number=2
*.log_archive_config='dg_config=(rac11g2,rac11g2s)'
*.log_archive_dest_2='service=RAC11G2 LGWR ASYNC NOAFFIRM max_failure=10 max_connections=5 reopen=180 valid_for=(online_logfiles,primary_role) db_uniq
ue_name=rac11g2'
*.log_archive_dest_1='location=use_db_recovery_file_dest valid_for=(all_logfiles,all_roles) db_unique_name=rac11g2s'
*.log_archive_dest_state_2='enable'
*.log_archive_dest_state_1='enable'
*.log_archive_format='%t_%s_%r.dbf'
*.log_archive_max_processes=5
*.log_file_name_convert='+FLASH/rac11g2','+FLASH/rac11g2s','+DATA/rac11g2','+DATA/rac11g2s'
*.open_cursors=300
*.pga_aggregate_target=209715200
*.processes=150
*.remote_listener='racb-scan:1521'
*.remote_login_passwordfile='exclusive'
*.sga_target=631242752
*.standby_file_management='AUTO'
rac11g2s1.undo_tablespace='UNDOTBS1'
rac11g2s2.undo_tablespace='UNDOTBS2'
rac11g2s1.thread=1
rac11g2s2.thread=2

21. Shutdown the database and startup mount and create a spfile with +ASM diskgroup as the locaiton

SQL>alter database recover managed standby database cancel;

SQL>startup mount pfile='/home/oracle/stdbypfile.ora';

ORACLE instance started.

Total System Global Area  630501376 bytes
Fixed Size                  2229120 bytes
Variable Size             247467136 bytes
Database Buffers          373293056 bytes
Redo Buffers                7512064 bytes
Database mounted.

SQL> create spfile='+DATA/rac11g2s/spfilerac11g2s.ora' from pfile='/home/oracle/stdbypfile.ora';

File created.

SQL> shutdown immediate;

22. Create pfile file on each node's oracle home (ORACLE_HOME/dbs) that points to the spfile by adding one line that referes to the spfile

vi initrac11g2s1.ora
spfile='+DATA/rac11g2s/spfilerac11g2s.ora'

Above is for rac4b, create initrac11g2s2.ora on rac5b.

23. Once above pfiles are created remove the spfile in $ORACLE_HOME/dbs that was created as part of the duplication.

24. Before bringing the standby instance under cluster control check it could be started on all other nodes with their respective instances. In this case rac5b since this is a two node rac, export ORACLE_SID to rac11g2s2 and start the instance in mount mode. Resolve any issues (init parameter related) before proceeding to the next step)

25. To add the database and instances to the cluster configuration run

$GRID_HOME/bin/srvctl add database -d rac11g2s -o $ORACLE_HOME -m local -p "+DATA/rac11g2s/spfilerac11g2s.ora" -n rac11g2 -r physical_standby -s mount
$GRID_HOME/bin/srvctl add instance -d rac11g2s -i rac11g2s1 -n rac4b
$GRID_HOME/bin/srvctl add instance -d rac11g2s -i rac11g2s2 -n rac5b

Without the "-s mount" the standby database would be started in open (read only) mode by default (both R1 and R2). With 11g (both R1 and R2) it is possible to apply redo while the database is open in read only mode. But this requires active dataguard license and setting the default startup mode to mount prevents such activity from happenning when active dataguard is not licensed. On 11gR1 db instances could be made dependent on that node's asm instance but this option is deprecated in 11gr2

$CRS_HOME/bin/srvctl modify instance -d rac11g2s -i rac11g2s1 -s +ASM1
-s option has been deprecated and will be ignored.

srvctl config database -d rac11g2s
Database unique name: rac11g2s
Database name: rac11g2
Oracle home: /opt/app/oracle/product/11.2.0/db_1
Oracle user: oracle
Spfile: +DATA/rac11g2s/spfilerac11g2s.ora
Domain: local
Start options: mount
Stop options: immediate
Database role: PHYSICAL_STANDBY
Management policy: AUTOMATIC
Server pools: rac11g2s
Database instances: rac11g2s1,rac11g2s2
Disk Groups:
Mount point paths:
Services:
Type: RAC
Database is administrator managed

26. Test the configuration with starting the entire standby database

srvctl status database -d rac11g2s
Instance rac11g2s1 is not running on node rac4b
Instance rac11g2s2 is not running on node rac5b

srvctl start database -d rac11g2s

srvctl status database -d rac11g2s
Instance rac11g2s1 is running on node rac4b
Instance rac11g2s2 is running on node rac5b

Once the database is started as a cluster resource the asm dependencies are automatically added as well as the local_listener values which were reset during the duplication.

crsctl stat res ora.rac11g2s.db -p

NAME=ora.rac11g2s.db
TYPE=ora.database.type
..
SERVER_POOLS=ora.rac11g2s
SPFILE=+DATA/rac11g2s/spfilerac11g2s.ora
START_DEPENDENCIES=hard(ora.DATA.dg,ora.FLASH.dg)weak(type:ora.listener.type,global:type:ora.scan_listener.type,uniform:ora.ons,global:ora.gns)pullup(ora.DATA.dg,ora.FLASH.dg)
START_TIMEOUT=600
STATE_CHANGE_TEMPLATE=
STOP_DEPENDENCIES=hard(intermediate:ora.asm,shutdown:ora.DATA.dg,shutdown:ora.FLASH.dg)
..

SQL> show parameter local

NAME            TYPE        VALUE
--------------- ---------- ------------------------------
local_listener  string      (DESCRIPTION=(ADDRESS_LIST=(ADDRESS=(PROTOCOL=TCP(HOST=192.168.0.98)(PORT=1521))))

26. Start the recovery process on one of the standby instances.

SQL> alter database recover managed standby database using current logfile disconnect;

This concludes the setup of RAC-RAC data guard.

27. To make sure archive logs don't get deleted on primary without being applied on standby set the log deletion policy to applied on standby. For this at least one log archive locations must be mandatory.

SQL> alter system set log_archive_dest_2='service=RAC11G2S LGWR ASYNC NOAFFIRM max_failure=10 max_connections=5 reopen=180 valid_for=(online_logfiles,primary_role) db_unique_name=rac11g2s mandatory' scope=both sid='*';

Usign rman chagne the archive log deletion policy.

RMAN> CONFIGURE ARCHIVELOG DELETION POLICY TO APPLIED ON STANDBY;

old RMAN configuration parameters:
CONFIGURE ARCHIVELOG DELETION POLICY TO NONE;
new RMAN configuration parameters:
CONFIGURE ARCHIVELOG DELETION POLICY TO APPLIED ON STANDBY;
new RMAN configuration parameters are successfully stored

If it is configured APPLIED ON ALL STANDBY then archived redo log files are eligible for deletion after being applied or consumed on all remote destinations, whether mandatory or not.

With mandatory destination inplace if the standby is not up and accepting redo when the primary starts following error

PING[ARC1]: Heartbeat failed to connect to standby 'RAC11G2S'. Error is 12543.
Archived Log entry 202 added for thread 1 sequence 80 ID 0x16a13366 dest 10:
ARCH: Archival stopped, error occurred. Will continue retrying
ORACLE Instance rac11g21 - Archival Error
ORA-16038: log 2 sequence# 80 cannot be archived
ORA-12543: TNS:destination host unreachable
ORA-00312: online log 2 thread 1: '+DATA/rac11g2/onlinelog/group_2.262.740770163'
ORA-00312: online log 2 thread 1: '+FLASH/rac11g2/onlinelog/group_2.258.740770163'

and any alter system switch logfile commands will hang.

Distributing redo load

It is possible to distribute the redo load among all standby instnaces. In this case instead of sending the redo from all primary instance to one standby instance each primary instance will send its redo to corresponding standby instance. This configuration is ideal in a symmetric data guard environment and one signle standby instnace cannot keep up with all the redo generated. To do this set primary instance specific log_archive_dest values

alter system set log_archive_dest_2='service=RAC11G2S1 LGWR ASYNC NOAFFIRM max_failure=10 max_connections=5 reopen=180 valid_for=(online_logfiles,primary_role) db_unique_name=rac11g2s mandatory' scope=spfile sid='rac11g21';

alter system set log_archive_dest_2='service=RAC11G2S2 LGWR ASYNC NOAFFIRM max_failure=10 max_connections=5 reopen=180 valid_for=(online_logfiles,primary_role) db_unique_name=rac11g2s mandatory' scope=spfile sid='rac11g22';

Here rac11g21 will send its redo to rac11g2s1 and rac11g22 will send to rac11g2s22.

But there can only be one apply instance.

Otherway around, sending all primary redo once standby instance gives more flexibility than above one-to-one redo transport configuration. In this case if the current standby apply instance need to be shutdown all that is required is stop the apply service on that instance and start in another instance. Since there's no instance specific redo transport configuration no additional steps are required. (provided the TNS entry has addressed for all standby instances similar to rac11g2s TNS entry above).

Related Post
11gR2 Standalone Data Guard (with ASM and Role Separation)
Data Guard Broker for 11gR2 RAC
Enable Active Dataguard on 11gR2 RAC Standby
Creating Data Guard Broker on 12c
Adding Far Sync Instances to Existing Data Guard Configuration
Setting Up Data Guard Broker for an Existing Data Guard Configuration with Far Sync