Thursday, February 24, 2011

Data Guard Broker for 11gR2

Data guard broker makes managing the data guard configuration easy by masking lot of the sql commands and multiple steps into few (sometimes one as in the case of switchover) data guard broker commands. Once the dataguard broker is setup it's advisable to use that to manipulate the dataguard configuration rather than the sql commands.
The data guard configuration created earlier will be configured to be managed through dataguard broker. There are some dataguard broker specific setup and configuration need to be done to the earlier setup.

1. Data guard broker requires a special naming convention to be followed in the value entered for global_dbname in listener.ora. Database sid should be suffixed with _DGMGRL if not data guard broker will not function properly. See metalink note Automatic Restart of Databases during Switchover fail with ORA-12514 in DGMGRL [ID 308943.1]
Change the earlier listener.ora files as below, on standby rac4b
more listener.ora

LISTENER=(DESCRIPTION=(ADDRESS_LIST=(ADDRESS=(PROTOCOL=IPC)(KEY=LISTENER))))            # line added by Agent
LISTENER_SCAN1=(DESCRIPTION=(ADDRESS_LIST=(ADDRESS=(PROTOCOL=IPC)(KEY=LISTENER_SCAN1))))                # line added by Agent
ENABLE_GLOBAL_DYNAMIC_ENDPOINT_LISTENER_SCAN1=ON                # line added by Agent
ENABLE_GLOBAL_DYNAMIC_ENDPOINT_LISTENER=ON              # line added by Agent


SID_LIST_LISTENER =
(SID_LIST =
(SID_DESC =
(GLOBAL_DBNAME = rac11g2s.domain.net)
(SID_NAME = rac11g2s1)
(ORACLE_HOME = /opt/app/oracle/product/11.2.0/db_1)
)
(SID_DESC =
(GLOBAL_DBNAME = rac11g2s_DGMGRL.domain.net)
(SID_NAME = rac11g2s1)
(ORACLE_HOME = /opt/app/oracle/product/11.2.0/db_1)
)
)


SID_LIST_LISTENER_SCAN1 =
(SID_LIST =
(SID_DESC =
(GLOBAL_DBNAME = rac11g2s.domain.net)
(SID_NAME = rac11g2s1)
(ORACLE_HOME = /opt/app/oracle/product/11.2.0/db_1)
)
(SID_DESC =
(GLOBAL_DBNAME = rac11g2s_DGMGRL.domain.net)
(SID_NAME = rac11g2s1)
(ORACLE_HOME = /opt/app/oracle/product/11.2.0/db_1)
)
)
on rac5b
more listener.ora

LISTENER_SCAN1=(DESCRIPTION=(ADDRESS_LIST=(ADDRESS=(PROTOCOL=IPC)(KEY=LISTENER_SCAN1))))                # line added by Agent
LISTENER=(DESCRIPTION=(ADDRESS_LIST=(ADDRESS=(PROTOCOL=IPC)(KEY=LISTENER))))            # line added by Agent
ENABLE_GLOBAL_DYNAMIC_ENDPOINT_LISTENER=ON              # line added by Agent
ENABLE_GLOBAL_DYNAMIC_ENDPOINT_LISTENER_SCAN1=ON                # line added by Agent


SID_LIST_LISTENER =
(SID_LIST =
(SID_DESC =
(GLOBAL_DBNAME = rac11g2s.domain.net)
(SID_NAME = rac11g2s2)
(ORACLE_HOME = /opt/app/oracle/product/11.2.0/db_1)
)
(SID_DESC =
(GLOBAL_DBNAME = rac11g2s_DGMGRL.domain.net)
(SID_NAME = rac11g2s2)
(ORACLE_HOME = /opt/app/oracle/product/11.2.0/db_1)
)

)

SID_LIST_LISTENER_SCAN1 =
(SID_LIST =
(SID_DESC =
(GLOBAL_DBNAME = rac11g2s.domain.net)
(SID_NAME = rac11g2s2)
(ORACLE_HOME = /opt/app/oracle/product/11.2.0/db_1)
)
(SID_DESC =
(GLOBAL_DBNAME = rac11g2s_DGMGRL.domain.net)
(SID_NAME = rac11g2s2)
(ORACLE_HOME = /opt/app/oracle/product/11.2.0/db_1)
)
)
Stop and start the listeners and check the static listener is available
srvctl stop listener -n `hostname -s`
srvctl start listener -n `hostname -s`

srvctl stop scan_listener
srvctl start scan_listener

lsnrctl status listener

LSNRCTL for Linux: Version 11.2.0.2.0 - Production on 23-FEB-2011 17:51:05

Copyright (c) 1991, 2010, Oracle.  All rights reserved.

Connecting to (DESCRIPTION=(ADDRESS=(PROTOCOL=IPC)(KEY=LISTENER)))
STATUS of the LISTENER
------------------------
Alias                     LISTENER
Version                   TNSLSNR for Linux: Version 11.2.0.2.0 - Production
Start Date                23-FEB-2011 17:42:04
Uptime                    0 days 0 hr. 9 min. 1 sec
Trace Level               off
Security                  ON: Local OS Authentication
SNMP                      OFF
Listener Parameter File   /opt/app/11.2.0/grid/network/admin/listener.ora
Listener Log File         /opt/app/oracle/diag/tnslsnr/rac5b/listener/alert/log.xml
Listening Endpoints Summary...
(DESCRIPTION=(ADDRESS=(PROTOCOL=ipc)(KEY=LISTENER)))
(DESCRIPTION=(ADDRESS=(PROTOCOL=tcp)(HOST=192.168.0.93)(PORT=1521)))
(DESCRIPTION=(ADDRESS=(PROTOCOL=tcp)(HOST=192.168.0.97)(PORT=1521)))
Services Summary...
Service "+ASM" has 1 instance(s).
Instance "+ASM2", status READY, has 1 handler(s) for this service...
Service "rac11g2s.domain.net" has 1 instance(s).
Instance "rac11g2s2", status UNKNOWN, has 1 handler(s) for this service...
Service "rac11g2s.domain.net" has 1 instance(s).
Instance "rac11g2s2", status READY, has 1 handler(s) for this service...
Service "rac11g2s_DGMGRL.domain.net" has 1 instance(s).
Instance "rac11g2s2", status UNKNOWN, has 1 handler(s) for this service...
The command completed successfully
To view the scan listener status it is important to set the ORACLE_HOME variable to GRID_HOME (or CRS_HOME whatever used) if not
lsnrctl status listener_scan1

TNS-01101: Could not find service name listener_scan1
will be shown.
export ORACLE_HOME=$CRS_HOME
$CRS_HOME/bin/lsnrctl status listener_scan1

LSNRCTL for Linux: Version 11.2.0.2.0 - Production on 23-FEB-2011 17:50:18

Copyright (c) 1991, 2010, Oracle.  All rights reserved.

Connecting to (DESCRIPTION=(ADDRESS=(PROTOCOL=IPC)(KEY=LISTENER_SCAN1)))
STATUS of the LISTENER
------------------------
Alias                     LISTENER_SCAN1
Version                   TNSLSNR for Linux: Version 11.2.0.2.0 - Production
Start Date                23-FEB-2011 17:49:44
Uptime                    0 days 0 hr. 0 min. 34 sec
Trace Level               off
Security                  ON: Local OS Authentication
SNMP                      OFF
Listener Parameter File   /opt/app/11.2.0/grid/network/admin/listener.ora
Listener Log File         /opt/app/11.2.0/grid/log/diag/tnslsnr/rac5b/listener_scan1/alert/log.xml
Listening Endpoints Summary...
(DESCRIPTION=(ADDRESS=(PROTOCOL=ipc)(KEY=LISTENER_SCAN1)))
(DESCRIPTION=(ADDRESS=(PROTOCOL=tcp)(HOST=192.168.0.92)(PORT=1521)))
Services Summary...
Service "rac11g2s.domain.net" has 1 instance(s).
Instance "rac11g2s2", status UNKNOWN, has 1 handler(s) for this service...
Service "rac11g2s_DGMGRL.domain.net" has 1 instance(s).
Instance "rac11g2s2", status UNKNOWN, has 1 handler(s) for this service...
The command completed successfully
Similarly make the changes to primary listener.ora files as well. On rac4
LISTENER=(DESCRIPTION=(ADDRESS_LIST=(ADDRESS=(PROTOCOL=IPC)(KEY=LISTENER))))            # line added by Agent
LISTENER_SCAN1=(DESCRIPTION=(ADDRESS_LIST=(ADDRESS=(PROTOCOL=IPC)(KEY=LISTENER_SCAN1))))                # line added by Agent
ENABLE_GLOBAL_DYNAMIC_ENDPOINT_LISTENER_SCAN1=ON                # line added by Agent
ENABLE_GLOBAL_DYNAMIC_ENDPOINT_LISTENER=ON              # line added by Agent


SID_LIST_LISTENER =
(SID_LIST =
(SID_DESC =
(GLOBAL_DBNAME = rac11g2.domain.net)
(SID_NAME = rac11g21)
(ORACLE_HOME = /opt/app/oracle/product/11.2.0/db_1)
)
(SID_DESC =
(GLOBAL_DBNAME = rac11g2_DGMGRL.domain.net)
(SID_NAME = rac11g21)
(ORACLE_HOME = /opt/app/oracle/product/11.2.0/db_1)
)
)


SID_LIST_LISTENER_SCAN1 =
(SID_LIST =
(SID_DESC =
(GLOBAL_DBNAME = rac11g2.domain.net)
(SID_NAME = rac11g21)
(ORACLE_HOME = /opt/app/oracle/product/11.2.0/db_1)
)
(SID_DESC =
(GLOBAL_DBNAME = rac11g2_DGMGRL.domain.net)
(SID_NAME = rac11g21)
(ORACLE_HOME = /opt/app/oracle/product/11.2.0/db_1)
)
)
On rac5
LISTENER_SCAN1=(DESCRIPTION=(ADDRESS_LIST=(ADDRESS=(PROTOCOL=IPC)(KEY=LISTENER_SCAN1))))                # line added by Agent
LISTENER=(DESCRIPTION=(ADDRESS_LIST=(ADDRESS=(PROTOCOL=IPC)(KEY=LISTENER))))            # line added by Agent
ENABLE_GLOBAL_DYNAMIC_ENDPOINT_LISTENER=ON              # line added by Agent
ENABLE_GLOBAL_DYNAMIC_ENDPOINT_LISTENER_SCAN1=ON                # line added by Agent


SID_LIST_LISTENER =
(SID_LIST =
(SID_DESC =
(GLOBAL_DBNAME = rac11g2.domain.net)
(SID_NAME = rac11g22)
(ORACLE_HOME = /opt/app/oracle/product/11.2.0/db_1)
)
(SID_DESC =
(GLOBAL_DBNAME = rac11g2_DGMGRL.domain.net)
(SID_NAME = rac11g22)
(ORACLE_HOME = /opt/app/oracle/product/11.2.0/db_1)
)
)


SID_LIST_LISTENER_SCAN1 =
(SID_LIST =
(SID_DESC =
(GLOBAL_DBNAME = rac11g2.domain.net)
(SID_NAME = rac11g22)
(ORACLE_HOME = /opt/app/oracle/product/11.2.0/db_1)
)
(SID_DESC =
(GLOBAL_DBNAME = rac11g2_DGMGRL.domain.net)
(SID_NAME = rac11g22)
(ORACLE_HOME = /opt/app/oracle/product/11.2.0/db_1)
)
)
As with the standby restart the listeners and verify the static listener is available.

2. There are three initialization parameters that is related to data guard broker
SQL> show parameter dg

NAME                                 TYPE        VALUE
------------------------------------ ----------- ------------------------------
dg_broker_config_file1               string      /opt/app/oracle/product/11.2.0
/db_1/dbs/dr1rac11g2.dat
dg_broker_config_file2               string      /opt/app/oracle/product/11.2.0
/db_1/dbs/dr2rac11g2.dat
dg_broker_start                      boolean     FALSE
In a RAC environment dg_broker_config_file1 and dg_broker_config_file2 must be stored in a shared location that could be accessed by all instances. The location could be Cluster file system,Raw devices or ASM. These location must be same on all instances. The file need not exist but if a new location is created to store them create it beforehand. In this case these files were stored in ASM and a directory was created inside asm diskgroup. On primary
ASMCMD>cd data/rac11g2
pwd
+data/rac11g2
ASMCMD> mkdir dgbroker
On standby
ASMCMD>cd data/rac11g2s
pwd
+data/rac11g2s
ASMCMD> mkdir dgbroker
Then set the intialization parameters on primary with
SQL> alter system set dg_broker_config_file1='+DATA/rac11g2/dgbroker/dr1rac11g2.dat' scope=both sid='*';
System altered.

SQL> alter system set dg_broker_config_file2='+DATA/rac11g2/dgbroker/dr2rac11g2.dat' scope=both sid='*';
System altered.
On standby
SQL> alter system set dg_broker_config_file1='+DATA/rac11g2s/dgbroker/dr1rac11g2s.dat' scope=both sid='*';
System altered.

SQL> alter system set dg_broker_config_file2='+DATA/rac11g2s/dgbroker/dr2rac11g2s.dat' scope=both sid='*';
System altered.
3. Enterprise Manager will automatically set the next required initialization parameter DG_BROKER_START to TRUE for new standby databases that it creates. But as in this case when using DGMGRL, this must be explicitly set to true otherwise dataguard broker daemon will not start. Both on primary and standby set
SQL> alter system set dg_broker_start=true scope=both sid='*';
System altered.
When this gets executed following message will appear on the alert log indicating the start of data guard broker daemon
Thu Feb 24 11:04:54 2011
DMON started with pid=56, OS id=16310
Starting Data Guard Broker (DMON)
Thu Feb 24 11:05:08 2011
INSV started with pid=60, OS id=16327
4. Next step is to create the data guard broker configuration. This must be created on the primary running on standby would give the following error
DGMGRL> create configuration rac11g2_dgb as primary database is rac11g2 connect identifier is rac11g2;
Error: ORA-16584: operation cannot be performed on a standby database

Failed.
5. On primary run dgmgrl command line tool to create the configuration
$ dgmgrl
DGMGRL for Linux: Version 11.2.0.2.0 - 64bit Production

Copyright (c) 2000, 2009, Oracle. All rights reserved.

Welcome to DGMGRL, type "help" for information.
DGMGRL> connect sys
Password:
Connected.

DGMGRL> create configuration rac11g2_dgb as primary database is rac11g2 connect identifier is rac11g2;
Configuration "rac11g2_dgb" created with primary database "rac11g2"

DGMGRL> show configuration

Configuration - rac11g2_dgb

Protection Mode: MaxPerformance
Databases:
rac11g2 - Primary database

Fast-Start Failover: DISABLED

Configuration Status:
DISABLED
Syntax to add the physical standby could be of many forms. In 11gR1 documentation "maintained as physical" was used to explicity indicate the database added is a standby. But in 11gR2 this could be ommited and configuration recognize the database role and configure it accordingly. Both commands are shown below
DGMGRL> add database rac11g2s as connect identifier is rac11g2s;
Database "rac11g2s" added
DGMGRL> show configuration

Configuration - rac11g2_dgb

Protection Mode: MaxPerformance
Databases:
rac11g2  - Primary database
rac11g2s - Physical standby database

Fast-Start Failover: DISABLED

Configuration Status:
DISABLED
To remove a database from the configuration use
DGMGRL> remove database rac11g2s;
Removed database "rac11g2s" from the configuration
The 11gR1 syntax
DGMGRL> add database rac11g2s as connect identifier is rac11g2s maintained as physical;
Database "rac11g2s" added
DGMGRL> show configuration

Configuration - rac11g2_dgb

Protection Mode: MaxPerformance
Databases:
rac11g2  - Primary database
rac11g2s - Physical standby database

Fast-Start Failover: DISABLED

Configuration Status:
DISABLED
6. Even though configuration is created it is disabled. As the final step enable the configuration
DGMGRL> enable configuration;
Enabled.
Following could be observed on the primary alert log
Thu Feb 24 11:12:53 2011
RSM0 started with pid=61, OS id=16735
ALTER SYSTEM SET log_archive_trace=0 SCOPE=BOTH SID='rac11g21';
ALTER SYSTEM SET log_archive_format='%t_%s_%r.dbf' SCOPE=SPFILE SID='rac11g21';
ALTER SYSTEM SET standby_file_management='AUTO' SCOPE=BOTH SID='*';
ALTER SYSTEM SET archive_lag_target=0 SCOPE=BOTH SID='*';
ALTER SYSTEM SET log_archive_max_processes=10 SCOPE=BOTH SID='*';
ALTER SYSTEM SET log_archive_min_succeed_dest=1 SCOPE=BOTH SID='*';
ALTER SYSTEM SET db_file_name_convert='+DATA/rac11g2s','+DATA/rac11g2' SCOPE=SPFILE;
ALTER SYSTEM SET log_file_name_convert='+FLASH/rac11g2s','+FLASH/rac11g2','+DATA/rac11g2s','+DATA/rac11g2' SCOPE=SPFILE;
ALTER SYSTEM ARCHIVE LOG
Thu Feb 24 11:13:01 2011
Thread 1 advanced to log sequence 125 (LGWR switch)
Current log# 1 seq# 125 mem# 0: +DATA/rac11g2/onlinelog/group_1.261.740770161
Current log# 1 seq# 125 mem# 1: +FLASH/rac11g2/onlinelog/group_1.257.740770161
Thu Feb 24 11:13:03 2011
LNS: Standby redo logfile selected for thread 1 sequence 125 for destination LOG_ARCHIVE_DEST_1
Thu Feb 24 11:13:03 2011
Archived Log entry 353 added for thread 1 sequence 124 ID 0x16d045c1 dest 10:
and on standby alert log
Thu Feb 24 12:13:50 2011
RSM0 started with pid=54, OS id=19480
ALTER SYSTEM SET log_archive_trace=0 SCOPE=BOTH SID='rac11g2s1';
ALTER SYSTEM SET log_archive_format='%t_%s_%r.dbf' SCOPE=SPFILE SID='rac11g2s1';
ALTER SYSTEM SET standby_file_management='AUTO' SCOPE=BOTH SID='*';
ALTER SYSTEM SET archive_lag_target=0 SCOPE=BOTH SID='*';
ALTER SYSTEM SET log_archive_max_processes=5 SCOPE=BOTH SID='*';
ALTER SYSTEM SET log_archive_min_succeed_dest=1 SCOPE=BOTH SID='*';
ALTER SYSTEM SET db_file_name_convert='+DATA/rac11g2','+DATA/rac11g2s' SCOPE=SPFILE;
ALTER SYSTEM SET log_file_name_convert='+FLASH/rac11g2','+FLASH/rac11g2s','+DATA/rac11g2','+DATA/rac11g2s' SCOPE=SPFILE;
ALTER SYSTEM SET fal_server='rac11g2' SCOPE=BOTH;
Thu Feb 24 12:14:00 2011
RFS[2]: Selected log 5 for thread 1 sequence 125 dbid 379668842 branch 740770160
Thu Feb 24 12:14:00 2011
Archived Log entry 189 added for thread 1 sequence 124 ID 0x16d045c1 dest 10:
Thu Feb 24 12:14:01 2011
RFS[5]: Selected log 8 for thread 2 sequence 117 dbid 379668842 branch 740770160
Thu Feb 24 12:14:01 2011
Archived Log entry 190 added for thread 2 sequence 116 ID 0x16d045c1 dest 10:
7. Check the configuration status with
DGMGRL> show configuration

Configuration - rac11g2_dgb

Protection Mode: MaxPerformance
Databases:
rac11g2  - Primary database
rac11g2s - Physical standby database

Fast-Start Failover: DISABLED

Configuration Status:
SUCCESS
If the status is not success then resolve any issues. Sometimes it is possible to have a warning status
DGMGRL> show configuration

Configuration - rac11g2_dgb

Protection Mode: MaxPerformance
Databases:
rac11g2  - Primary database
Warning: ORA-16792: configurable property value is inconsistent with database setting

rac11g2s - Physical standby database

Fast-Start Failover: DISABLED

Configuration Status:
WARNING
Looking at the error number
oerr ora 16792
16792, 0000, "configurable property value is inconsistent with database setting"
// *Cause:  The values of one or more configurable properties were
//          inconsistent with database in-memory settings or server parameter
//          file settings. This may happen by directly altering initialization
//          parameters instead of editing configurable property values using
//          Data Guard broker.
// *Action: Query the InconsistentProperties property on the database or check
//          the Data Guard broker log to find which properties are set
//          inconsistently. Reset these properties to make them consistent
//          with the database settings. Alternatively, enable the database
//          or the entire configuration to allow the configurable property
//          settings to be propagated to the initialization parameters.
it was possible to bring all the instance into a consistent state with a database restart
srvctl stop database -d rac11g2
srvctl start database -d rac11g2

gmgrl
DGMGRL for Linux: Version 11.2.0.2.0 - 64bit Production

Copyright (c) 2000, 2009, Oracle. All rights reserved.

Welcome to DGMGRL, type "help" for information.
DGMGRL> connect sys
Password:
Connected.
DGMGRL> show configuration

Configuration - rac11g2_dgb

Protection Mode: MaxPerformance
Databases:
rac11g2  - Primary database
rac11g2s - Physical standby database

Fast-Start Failover: DISABLED

Configuration Status:
SUCCESS
8. Check the status of each database in the data guard broker configuration
DGMGRL> show database verbose rac11g2

Database - rac11g2

Role:            PRIMARY
Intended State:  TRANSPORT-ON
Instance(s):
rac11g21
rac11g22

Properties:
DGConnectIdentifier             = 'rac11g2'
ObserverConnectIdentifier       = ''
LogXptMode                      = 'ASYNC'
DelayMins                       = '0'
Binding                         = 'optional'
MaxFailure                      = '0'
MaxConnections                  = '1'
ReopenSecs                      = '300'
NetTimeout                      = '30'
RedoCompression                 = 'DISABLE'
LogShipping                     = 'ON'
PreferredApplyInstance          = ''
ApplyInstanceTimeout            = '0'
ApplyParallel                   = 'AUTO'
StandbyFileManagement           = 'AUTO'
ArchiveLagTarget                = '0'
LogArchiveMaxProcesses          = '10'
LogArchiveMinSucceedDest        = '1'
DbFileNameConvert               = '+DATA/rac11g2s, +DATA/rac11g2'
LogFileNameConvert              = '+FLASH/rac11g2s, +FLASH/rac11g2, +DATA/rac11g2s, +DATA/rac11g2'
FastStartFailoverTarget         = ''
InconsistentProperties          = '(monitor)'
InconsistentLogXptProps         = '(monitor)'
SendQEntries                    = '(monitor)'
LogXptStatus                    = '(monitor)'
RecvQEntries                    = '(monitor)'
SidName(*)
StaticConnectIdentifier(*)
StandbyArchiveLocation(*)
AlternateLocation(*)
LogArchiveTrace(*)
LogArchiveFormat(*)
TopWaitEvents(*)
(*) - Please check specific instance for the property value

Database Status:
SUCCESS


DGMGRL> show database verbose rac11g2s

Database - rac11g2s

Role:            PHYSICAL STANDBY
Intended State:  APPLY-ON
Transport Lag:   0 seconds
Apply Lag:       0 seconds
Real Time Query: OFF
Instance(s):
rac11g2s1 (apply instance)
rac11g2s2

Properties:
DGConnectIdentifier             = 'rac11g2s'
ObserverConnectIdentifier       = ''
LogXptMode                      = 'ASYNC'
DelayMins                       = '0'
Binding                         = 'MANDATORY'
MaxFailure                      = '10'
MaxConnections                  = '5'
ReopenSecs                      = '180'
NetTimeout                      = '30'
RedoCompression                 = 'DISABLE'
LogShipping                     = 'ON'
PreferredApplyInstance          = ''
ApplyInstanceTimeout            = '0'
ApplyParallel                   = 'AUTO'
StandbyFileManagement           = 'AUTO'
ArchiveLagTarget                = '0'
LogArchiveMaxProcesses          = '5'
LogArchiveMinSucceedDest        = '1'
DbFileNameConvert               = '+DATA/rac11g2, +DATA/rac11g2s'
LogFileNameConvert              = '+FLASH/rac11g2, +FLASH/rac11g2s, +DATA/rac11g2, +DATA/rac11g2s'
FastStartFailoverTarget         = ''
InconsistentProperties          = '(monitor)'
InconsistentLogXptProps         = '(monitor)'
SendQEntries                    = '(monitor)'
LogXptStatus                    = '(monitor)'
RecvQEntries                    = '(monitor)'
SidName(*)
StaticConnectIdentifier(*)
StandbyArchiveLocation(*)
AlternateLocation(*)
LogArchiveTrace(*)
LogArchiveFormat(*)
TopWaitEvents(*)
(*) - Please check specific instance for the property value

Database Status:
SUCCESS
From 11gR2 data guard broker documentation "If the standby database is an Oracle RAC database, the broker starts Redo Apply on exactly one standby instance, called the apply instance. If this instance fails, the broker automatically chooses another instance that is either mounted or open read-only. This new instance then becomes the apply instance."




Switchover Using Data Guard Broker


To test all the configuration parameters are setup properly a switchover will be carried out using the data guard broker. Data guard broker will take care of all the steps that were carried out manually during earlier switchover scenario using sql commands.

To switchover using data guard broker all that is required is issue the command "switchover to" followed by the database name to switchover to.
DGMGRL> switchover to rac11g2s
Performing switchover NOW, please wait...
New primary database "rac11g2s" is opening...
Operation requires shutdown of instance "rac11g21" on database "rac11g2"
Shutting down instance "rac11g21"...
ORACLE instance shut down.
Operation requires startup of instance "rac11g21" on database "rac11g2"
Starting instance "rac11g21"...
ORACLE instance started.
Database mounted.
Switchover succeeded, new primary is "rac11g2s"
. At the time issuing this there were two primary instances (rac11g21 and rac11g22) in open read,write mode and two standby instance (rac11g2s1 and rac11g2s2) in mount mode.

The command was issued from rac11g21 instance and as soon as it was issued rac11g22 was shutdown by the data guard broker(with shutdown abort) as seen from rac11g22 alert log
Thu Feb 24 15:35:50 2011
Shutting down instance (abort)
License high water mark = 5
USER (ospid: 22346): terminating the instance
Instance terminated by USER, pid = 22346
Thu Feb 24 15:35:50 2011
Instance shutdown complete
rac11g21 disconnect all the sessions and commit to switchover with session shutdown as seen from rac11g21 alert log (only partial information is shown here, not all the entries generated at the time of switchover)
Thu Feb 24 15:35:53 2011
ALTER DATABASE COMMIT TO SWITCHOVER TO PHYSICAL STANDBY WITH SESSION SHUTDOWN
ALTER DATABASE COMMIT TO SWITCHOVER TO PHYSICAL STANDBY [Process Id: 26886] (rac11g21)
Thu Feb 24 15:35:53 2011
Thread 1 advanced to log sequence 184 (LGWR switch)
Current log# 1 seq# 184 mem# 0: +DATA/rac11g2/onlinelog/group_1.261.740770161
Current log# 1 seq# 184 mem# 1: +FLASH/rac11g2/onlinelog/group_1.257.740770161
Thu Feb 24 15:35:53 2011
Stopping background process CJQ0
Stopping background process QMNC
Thu Feb 24 15:35:54 2011
Archived Log entry 541 added for thread 2 sequence 173 ID 0x16d1ac74 dest 10:
All dispatchers and shared servers shutdown
CLOSE: killing server sessions.
Active process 32445 user 'oracle' program 'oracle@rac4.domain.net (W000)'
Active process 32445 user 'oracle' program 'oracle@rac4.domain.net (W000)'
Active process 437 user 'oracle' program 'oracle@rac4.domain.net (TNS V1-V3)'
Active process 32445 user 'oracle' program 'oracle@rac4.domain.net (W000)'
..
..
Switchover: Complete - Database shutdown required
Completed: ALTER DATABASE COMMIT TO SWITCHOVER TO PHYSICAL STANDBY WITH SESSION SHUTDOWN
Once the shutdown is complete rac11g21 is started as the new standby and log apply process is started
ALTER SYSTEM SET fal_server='rac11g2s' SCOPE=BOTH;
ALTER DATABASE RECOVER MANAGED STANDBY DATABASE  THROUGH ALL SWITCHOVER DISCONNECT  USING CURRENT LOGFILE
Attempt to start background Managed Standby Recovery process (rac11g21)
Thu Feb 24 15:36:29 2011
MRP0 started with pid=64, OS id=719
MRP0: Background Managed Standby Recovery process started (rac11g21)
started logmerger process
Thu Feb 24 15:36:34 2011
Managed Standby Recovery starting Real Time Apply
Parallel Media Recovery started with 2 slaves
Waiting for all non-current ORLs to be archived...
All non-current ORLs have been archived.
Clearing online redo logfile 1 +DATA/rac11g2/onlinelog/group_1.261.740770161
Clearing online log 1 of thread 1 sequence number 189
Completed: ALTER DATABASE RECOVER MANAGED STANDBY DATABASE  THROUGH ALL SWITCHOVER DISCONNECT  USING CURRENT LOGFILE
Clearing online redo logfile 1 complete
The other instance in this new standby (old primary) is also stated in the mount mode.


While the above is going on there's activity on the old standby (rac11g2s, new primary) as well. As soon as the switchover command is issued the apply instance (rac11g2s1) at standby disconnect from the primary and stop the log apply process
Thu Feb 24 16:36:47 2011
RFS[11]: Possible network disconnect with primary database
Thu Feb 24 16:36:47 2011
RFS[9]: Possible network disconnect with primary database
Thu Feb 24 16:36:47 2011
RFS[7]: Possible network disconnect with primary database
Thu Feb 24 16:36:47 2011
RFS[8]: Possible network disconnect with primary database
Thu Feb 24 16:36:47 2011
RFS[14]: Assigned to RFS process 29873
RFS[14]: Possible network disconnect with primary database
..
..
Resetting standby activation ID 382839924 (0x16d1ac74)
Thu Feb 24 16:36:56 2011
ALTER DATABASE RECOVER MANAGED STANDBY DATABASE CANCEL
Media Recovery Waiting for thread 1 sequence 185
MRP0: Background Media Recovery cancelled with status 16037
Once this completes it switchover to the primary database role and open the database
MRP0: Background Media Recovery process shutdown (rac11g2s1)
Managed Standby Recovery Canceled (rac11g2s1)
Completed: ALTER DATABASE RECOVER MANAGED STANDBY DATABASE CANCEL
ALTER DATABASE COMMIT TO SWITCHOVER TO PRIMARY WAIT WITH SESSION SHUTDOWN
ALTER DATABASE SWITCHOVER TO PRIMARY (rac11g2s1)
Maximum wait for role transition is 15 minutes.
As soon as the switchover is complete other instances in the (old)standby are also brought up in open mode and the new primary is available for use. Following from the rac11g2s2 alert log
Thu Feb 24 16:36:56 2011
Managed Standby Recovery not using Real Time Apply
Switchover: Complete - Database mounted as primary
Thu Feb 24 16:37:02 2011
ALTER DATABASE OPEN
Data Guard Broker initializing...
Picked broadcast on commit scheme to generate SCNs
Checking the new configuration
DGMGRL> show configuration

Configuration - rac11g2_dgb

Protection Mode: MaxPerformance
Databases:
rac11g2s - Primary database
rac11g2  - Physical standby database

Fast-Start Failover: DISABLED

Configuration Status:
SUCCESS

DGMGRL> show database verbose rac11g2

Database - rac11g2

Role:            PHYSICAL STANDBY
Intended State:  APPLY-ON
Transport Lag:   0 seconds
Apply Lag:       0 seconds
Real Time Query: OFF
Instance(s):
 rac11g21 (apply instance)
rac11g22
..
..
Unlike 11gR1 configuration 11gR2 does use the VIPs for StaticConnectIdentifier.

Wednesday, February 23, 2011

11gR2 RAC to RAC Switchover

Versions prior to 11gR2 (10gR1, 10gR2 and 11gR1) all required only one primary and one standby instance to be online during the switchover. But in 11gR2 it seem this behavior has been changed.

From 10gR1 Dataguard and concepts documentation "For a Real Application Clusters database, only one primary instance and one standby instance can be online during the switchover. Shut down all other instances before starting the switchover. Then, after the switchover completes, bring these instances back online."

From 10gR2 documentation "Verify that all but one RAC instance on the standby databases in a Real Application Clusters configuration are shut down. For a Real Application Clusters database, only one RAC instance on the standby database can be online during the role transition. Shut down all other instances before starting the role transition. Then, after the role transition completes, bring these instances back online. Even though only one RAC instance on the standby database is open during the switchover, all other standby database instances will automatically undergo a transition to their new role correctly when they are opened." This doesn't say anything about the primary RAC but it's same as 10gR1.

From 11gR1 documentation "Before performing a switchover from an Oracle RAC primary database to a physical standby database, shut down all but one primary database instance. Any primary database instances shut down at this time can be started after the switchover completes.
Before performing a switchover or a failover to an Oracle RAC physical standby database, shut down all but one standby database instance. Any standby database instances shut down at this time can be restarted after the role transition completes.
"

Now the 11gR2 documentation "Before performing a switchover from an Oracle RAC primary database to a physical standby database, shut down all but one primary database instance. Any primary database instances shut down at this time can be started after the switchover completes.

Before performing a switchover to a physical standby database that is in real-time query mode, consider bringing all instances of that standby database to the mounted but not open state to achieve the fastest possible role transition and to cleanly terminate any user sessions connected to the physical standby database prior to the role transition.
" It seems in 11gR2 if active dataguard is on (another word for real time query mode) then it must be stopped before the switchover but it is possible to keep ALL instances in the mount mode during the switchover.

Dataguard used here is the one created earlier.

First the switchover was carried out as per 11gR1 documentation with only one primary and one standby active during switchover. This worked fine as expected.

Next the switchover was carried out with all the standby instance in the mount state. They weren't in real time query mode so this setup is similar to a traditional physical standby setup where standby databases are in the mount mode. Steps below are from this second scenario.

Here rac11g2 is the primary db with two instances and rac11g2s is the standby database also with two instances.

1. On the primary stop all but one instance (as stated by 11gR2 documentation)
srvctl stop instance -d rac11g2 -i rac11g21
2. SQL Plus in to online instance and query the switchover status
SQL> SELECT SWITCHOVER_STATUS FROM V$DATABASE;

SWITCHOVER_STATUS
-----------------
TO STANDBY
3. If the switchover status is to_standby then
SQL> alter database commit to switchover to physical standby;
If the switchover status is session_active then
SQL> alter database commit to switchover to physical standby with session shutdown;
Once the command finishes following could be seen on the instance's alert log
standby redo logfiles that match the primary database:
ALTER DATABASE ADD STANDBY LOGFILE 'srl1.f' SIZE 52428800;
ALTER DATABASE ADD STANDBY LOGFILE 'srl2.f' SIZE 52428800;
ALTER DATABASE ADD STANDBY LOGFILE 'srl3.f' SIZE 52428800;
ALTER DATABASE ADD STANDBY LOGFILE 'srl4.f' SIZE 52428800;
ALTER DATABASE ADD STANDBY LOGFILE 'srl5.f' SIZE 52428800;
Archivelog for thread 2 sequence 95 required for standby recovery
Switchover: Primary controlfile converted to standby controlfile succesfully.
Switchover: Complete - Database shutdown required
Completed: alter database commit to switchover to physical standby
4. Shutdown the database and change cluster properties of the database to physical standby and default startup mode to mount and start the "new" standby databse (old primary)
SQL> shutdown abort

srvctl modify database -d rac11g2 -r physical_standby -s mount
srvctl start database -d rac11g2
Redo apply process could be started now or wait until the standby switchover to primary role

5. On the "old" standby (one that's going to be the new primary) check the switchover status
SQL> SELECT SWITCHOVER_STATUS FROM V$DATABASE;

SWITCHOVER_STATUS
-----------------
TO PRIMARY
Since both instances were left in mount mode this could be run in any instance and switchover status would be the same.

6. If the switchover status is to_primary then then run the following command on only one instnace
SQL> alter database commit to switchover to primary;
If the switchover status is session_active then (again only on one instance)
SQL> alter database commit to switchover to primary with session shutdown;
On the alert log of the instnace the command was run
Wed Feb 23 15:00:08 2011
alter database commit to switchover to primary
ALTER DATABASE SWITCHOVER TO PRIMARY (rac11g2s1)
Maximum wait for role transition is 15 minutes.
Switchover: Media recovery is still active
Role Change: Canceling MRP - no more redo to apply
Wed Feb 23 15:00:09 2011
MRP0: Background Media Recovery cancelled with status 16037
Errors in file /opt/app/oracle/diag/rdbms/rac11g2s/rac11g2s1/trace/rac11g2s1_pr00_9833.trc:
ORA-16037: user requested cancel of managed recovery operation
Wed Feb 23 15:00:09 2011
Managed Standby Recovery not using Real Time Apply
Recovery interrupted!
Wed Feb 23 15:00:09 2011
MRP0: Background Media Recovery process shutdown (rac11g2s1)
Role Change: Canceled MRP
Backup controlfile written to trace file /opt/app/oracle/diag/rdbms/rac11g2s/rac11g2s1/trace/rac11g2s1_ora_14026.trc
SwitchOver after complete recovery through change 3671973
Online log +DATA/rac11g2s/onlinelog/group_1.265.743447741: Thread 1 Group 1 was previously cleared
Online log +FLASH/rac11g2s/onlinelog/group_1.258.743447741: Thread 1 Group 1 was previously cleared
Online log +DATA/rac11g2s/onlinelog/group_2.266.743447743: Thread 1 Group 2 was previously cleared
Online log +FLASH/rac11g2s/onlinelog/group_2.259.743447743: Thread 1 Group 2 was previously cleared
Online log +DATA/rac11g2s/onlinelog/group_3.267.743447745: Thread 2 Group 3 was previously cleared
Online log +FLASH/rac11g2s/onlinelog/group_3.260.743447745: Thread 2 Group 3 was previously cleared
Online log +DATA/rac11g2s/onlinelog/group_4.268.743447747: Thread 2 Group 4 was previously cleared
Online log +FLASH/rac11g2s/onlinelog/group_4.261.743447747: Thread 2 Group 4 was previously cleared
Standby became primary SCN: 3671971
Switchover: Complete - Database mounted as primary
Completed: alter database commit to switchover to primary
Alert log of the other instance
Wed Feb 23 15:00:09 2011
Managed Standby Recovery not using Real Time Apply
Switchover: Complete - Database mounted as primary
7. Once the above command completes open all the instances with
SQL> ALTER DATABASE OPEN;
Database altered.
8. Change the cluster information of the database from physical standby to primary with
srvctl modify database -d rac11g2s -r  primary -s open
9. Start the log apply process on the standby with
SQL> alter database recover managed standby database using current logfile disconnect;
Database altered.
10. Verify the redo transport either querying the v$archive_log view.

Enable Active Dataguard on 11gR2 RAC Standby

The RAC-to-RAC dataguard created earlier is used to enable active dataguard.

The standby site has two instances all in mounted mode and log apply enable on one instnace. To enable active dataguard

1. In this case log apply process is enabled on rac4b therefore cancel log apply process on that node. No action is required on other node rac5b.
SQL> alter database recover managed standby database cancel;
Database altered.
2. Open the database in read only mode. Since database added to cluster to startup default on mount mode with
$GIRD_HOME/bin/srvctl add database -d rac11g2s -o $ORACLE_HOME -m local -p "+DATA/rac11g2s/spfilerac11g2s.ora" -n rac11g2 -r physical_standby -s mount
each instance must be started on read only mode if all of them are used for read only application. On rac4b
SQL> alter database open read only;
Database altered.

SQL> select open_mode from v$database;

OPEN_MODE
---------
READ ONLY
After this on rac5b query the open mode to verify its still in mount mode and then open in read only mode
SQL>  select open_mode from v$database;

OPEN_MODE
---------
MOUNTED

SQL> alter database open read only;
Database altered.

SQL> select open_mode from v$database;

OPEN_MODE
---------
READ ONLY
3. Start the log apply service on one instance with
SQL> alter database recover managed standby database using current logfile disconnect;
Database altered.
4. On both standby instances querying the open mode will give
SQL> select open_mode from v$database;

OPEN_MODE
--------------------
READ ONLY WITH APPLY
If data gurad broker is configured and redo apply is started then to enable active data guard

1.Stop the log apply on the standby using dgmgrl
DGMGRL> edit database rac11g2s set state='APPLY-OFF';
Succeeded.

DGMGRL> show database rac11g2s

Database - rac11g2s

Role: PHYSICAL STANDBY
Intended State: APPLY-OFF
Transport Lag: 0 seconds
Apply Lag: 0 seconds
Real Time Query: OFF
Instance(s):
rac11g2s1 (apply instance)
rac11g2s2

Database Status:
SUCCESS
2. Open the standby database instances in read only mode using sql plus
SQL> alter database open read only;
Database altered.
3. Start the redo apply service again using dgmgrl
DGMGRL> edit database rac11g2s set state='APPLY-ON';
Succeeded.


Disabling Active Dataguard

1. Cancel the log apply service
SQL> alter database recover managed standby database cancel;
Database altered.
This will change open mode from READ ONLY WITH APPLY to READ ONLY
SQL> select open_mode from v$database;

OPEN_MODE
---------
READ ONLY
2. Shutdown and start (in mount mode) the standby database
srvctl stop database -d rac11g2s
srvctl start database -d rac11g2s

SQL> select open_mode from v$database;

OPEN_MODE
---------
MOUNTED
3. Enable log apply service
SQL> alter database recover managed standby database using current logfile disconnect;
Database altered.


Saturday, February 19, 2011

11gR2 RAC to RAC Data Guard

The initial setup is not much different between RAC-RAC and RAC-Single instance data guard, in a way setting up the data guard is same with some additional setps at the end of setup in the way of bringing the standby database under the control of clusterware.

The primary RAC environment consists of two nodes, hostname rac4 and rac5. The primary database name is rac11g2 and two instances are rac11g21 (running on rac4 node) and rac11g22 (on rac5).

The standby RAC environment also consists of two nodes (symmetrical data guard) hostname rac4b and rac5b. The standby database will be named rac11g2s and the corresponding instances will be rac11g2s1 (on rac4b) and rac11g2s2 (on rac5b).

The software version is 11.2.0.2 on both primary and standby (both grid home and oracle home). Primary site has database created and standby site has grid infrastructure and oracle home software installed.

From Data Guard Concepts and Administration

New 11.2 Features Common to Redo Apply and SQL Apply
As of Oracle Database 11g Release 2 (11.2.0.2), Oracle Data Guard is fully integrated with Oracle Real Application Clusters One Node (Oracle RAC One Node).
A Data Guard configuration can now consist of a primary database and up to 30 standby databases.
The FAL_CLIENT database initialization parameter is no longer required.
The default archive destination used by the Oracle Automatic Storage Management (Oracle ASM) feature and the fast recovery area feature has changed from LOG_ARCHIVE_DEST_10 to LOG_ARCHIVE_DEST_1.
Redo transport compression is no longer limited to compressing redo data only when a redo gap is being resolved. When compression is enabled for a destination, all redo data sent to that destination is compressed.
The new ALTER SYSTEM FLUSH REDO SQL statement can be used at failover time to flush unsent redo from a mounted primary database to a standby database, thereby allowing a zero data loss failover to be performed even if the primary database is not running in a zero data loss data protection mode. See Section 8.2.2 for more information.


New 11.2 Features Specific to Redo Apply
You can configure apply lag tolerance in a real-time query environment by using the new STANDBY_MAX_DATA_DELAY parameter.
You can use the new ALTER SESSION SYNC WITH PRIMARY SQL statement to ensure that a suitably configured physical standby database is synchronized with the primary database as of the time the statement is issued.
The V$DATAGUARD_STATS view has been enhanced to a greater degree of accuracy in many of its columns, including apply lag and transport lag.
You can view a histogram of apply lag values on the physical standby. To do so, query the new V$STANDBY_EVENT_HISTOGRAM view.
A corrupted data block in a primary database can be automatically replaced with an uncorrupted copy of that block from a physical standby database that is operating in real-time query mode. A corrupted block in a physical standby database can also be automatically replaced with an uncorrupted copy of the block from the primary database.


1. On the standby ASM instances would have been created when grid infrastructure was installed but not the ASM diskgroups. Using asmca create diskgroup on the standby ASM instances. In this case both primary and standby ASM diskgroups are named +DATA and +FLASH

2. Create following directory structures in all standby nodes
cd $ORACLE_BASE
mkdir admin
cd admin/
mkdir rac11g2s
cd rac11g2s/
mkdir adump  dpdump  hdump  pfile
3. It is assumed primary db is in archivelog mode if not put the db in archive log mode. This is required since duplication is done using active database.

4. Enable force logging on the primary database
alter database force logging;
5.Create standby log files for each thread on primary db. These should be same size as the online redo log files
SQL> alter database add standby logfile thread 1 size 52428800;
or
SQL> alter database add standby logfile thread 1;
SQL> alter database add standby logfile thread 2;
There should be at least one more redo log group per thread than the online redo logs.

6. Both on primary and standby add static listener entries in $GRID_HOME($CRS_HOME)/network/admin/listener.ora. If scan listener is used for connections create static listener for those as well. One key difference is both in 10gR2 and 11gR1 the static listener and default listeners have hostname appended to the end but on 11gR2 this is not the case. Find out the available listeners with
srvctl config listener
Name: LISTENER
Network: 1, Owner: oracle
Home: 
End points: TCP:1521

srvctl config scan_listener
SCAN Listener LISTENER_SCAN1 exists. Port: TCP:1521
and create the static listeners for default listener as well as the scan listener. For standby on rac4b
more listener.ora
LISTENER=(DESCRIPTION=(ADDRESS_LIST=(ADDRESS=(PROTOCOL=IPC)(KEY=LISTENER))))            # line added by Agent
LISTENER_SCAN1=(DESCRIPTION=(ADDRESS_LIST=(ADDRESS=(PROTOCOL=IPC)(KEY=LISTENER_SCAN1))))                # line added by Agent
ENABLE_GLOBAL_DYNAMIC_ENDPOINT_LISTENER_SCAN1=ON                # line added by Agent
ENABLE_GLOBAL_DYNAMIC_ENDPOINT_LISTENER=ON              # line added by Agent


SID_LIST_LISTENER =
(SID_LIST =
(SID_DESC =
(GLOBAL_DBNAME = rac11g2s.domain.net)
(SID_NAME = rac11g2s1)
(ORACLE_HOME = /opt/app/oracle/product/11.2.0/db_1)
)
)


SID_LIST_LISTENER_SCAN1 =
(SID_LIST =
(SID_DESC =
(GLOBAL_DBNAME = rac11g2s.domain.net)
(SID_NAME = rac11g2s1)
(ORACLE_HOME = /opt/app/oracle/product/11.2.0/db_1)
)
)
on rac5b
more listener.ora
LISTENER_SCAN1=(DESCRIPTION=(ADDRESS_LIST=(ADDRESS=(PROTOCOL=IPC)(KEY=LISTENER_SCAN1))))                # line added by Agent
LISTENER=(DESCRIPTION=(ADDRESS_LIST=(ADDRESS=(PROTOCOL=IPC)(KEY=LISTENER))))            # line added by Agent
ENABLE_GLOBAL_DYNAMIC_ENDPOINT_LISTENER=ON              # line added by Agent
ENABLE_GLOBAL_DYNAMIC_ENDPOINT_LISTENER_SCAN1=ON                # line added by Agent


SID_LIST_LISTENER =
(SID_LIST =
(SID_DESC =
(GLOBAL_DBNAME = rac11g2s.domain.net)
(SID_NAME = rac11g2s2)
(ORACLE_HOME = /opt/app/oracle/product/11.2.0/db_1)
)
)

SID_LIST_LISTENER_SCAN1 =
(SID_LIST =
(SID_DESC =
(GLOBAL_DBNAME = rac11g2s.domain.net)
(SID_NAME = rac11g2s2)
(ORACLE_HOME = /opt/app/oracle/product/11.2.0/db_1)
)
)
In 11gR2 there's another file in $ORACLE_HOME/network/admin folder called endpoints_listener.ora. This is there for backward compatibility with pre-11.2 databases. More on metalink note 11.2 Scan and Node TNS Listener Setup Examples [ID 1070607.1]
Stop and start the listeners on standby
srvctl stop listener -n `hostname -s`
srvctl start listener -n `hostname -s`

srvctl stop scan_listener
srvctl start scan_listener
After this stop and start of listeners doing a listener status listener would show the statically registered instance and service as below
Service "rac11g2s.domain.net" has 1 instance(s).
Instance "rac11g2s1", status UNKNOWN, has 1 handler(s) for this service...
The command completed successfully
Do the same for primary environment on rac4
more listener.ora
LISTENER=(DESCRIPTION=(ADDRESS_LIST=(ADDRESS=(PROTOCOL=IPC)(KEY=LISTENER))))            # line added by Agent
LISTENER_SCAN1=(DESCRIPTION=(ADDRESS_LIST=(ADDRESS=(PROTOCOL=IPC)(KEY=LISTENER_SCAN1))))                # line added by Agent
ENABLE_GLOBAL_DYNAMIC_ENDPOINT_LISTENER_SCAN1=ON                # line added by Agent
ENABLE_GLOBAL_DYNAMIC_ENDPOINT_LISTENER=ON              # line added by Agent


SID_LIST_LISTENER =
(SID_LIST =
(SID_DESC =
(GLOBAL_DBNAME = rac11g2.domain.net)
(SID_NAME = rac11g21)
(ORACLE_HOME = /opt/app/oracle/product/11.2.0/db_1)
)
)


SID_LIST_LISTENER_SCAN1 =
(SID_LIST =
(SID_DESC =
(GLOBAL_DBNAME = rac11g2.domain.net)
(SID_NAME = rac11g21)
(ORACLE_HOME = /opt/app/oracle/product/11.2.0/db_1)
)
)
on rac5
more listener.ora
LISTENER_SCAN1=(DESCRIPTION=(ADDRESS_LIST=(ADDRESS=(PROTOCOL=IPC)(KEY=LISTENER_SCAN1))))                # line added by Agent
LISTENER=(DESCRIPTION=(ADDRESS_LIST=(ADDRESS=(PROTOCOL=IPC)(KEY=LISTENER))))            # line added by Agent
ENABLE_GLOBAL_DYNAMIC_ENDPOINT_LISTENER=ON              # line added by Agent
ENABLE_GLOBAL_DYNAMIC_ENDPOINT_LISTENER_SCAN1=ON                # line added by Agent


SID_LIST_LISTENER =
(SID_LIST =
(SID_DESC =
(GLOBAL_DBNAME = rac11g2.domain.net)
(SID_NAME = rac11g22)
(ORACLE_HOME = /opt/app/oracle/product/11.2.0/db_1)
)
)


SID_LIST_LISTENER_SCAN1 =
(SID_LIST =
(SID_DESC =
(GLOBAL_DBNAME = rac11g2.domain.net)
(SID_NAME = rac11g22)
(ORACLE_HOME = /opt/app/oracle/product/11.2.0/db_1)
)
)
7. Create TNS entries on both primary and standby. On all primary node's $ORACLE_HOME/network/admin
RAC11G2SCANB =
(DESCRIPTION =
(ADDRESS = (PROTOCOL = TCP)(HOST = racb-scan)(PORT = 1521))
(CONNECT_DATA =
(SERVER = DEDICATED)
(SERVICE_NAME = rac11g2s.domain.net)
)
)

RAC11G2S =
(DESCRIPTION =
(ADDRESS = (PROTOCOL = TCP)(HOST = rac4b-vip)(PORT = 1521))
(ADDRESS = (PROTOCOL = TCP)(HOST = rac5b-vip)(PORT = 1521))
(CONNECT_DATA =
(SERVER = DEDICATED)
(SERVICE_NAME = rac11g2s.domain.net)
)
)

RAC11G2S1 =
(DESCRIPTION =
(ADDRESS = (PROTOCOL = TCP)(HOST = rac4b-vip)(PORT = 1521))
(CONNECT_DATA =
(SERVER = DEDICATED)
(SERVICE_NAME = rac11g2s.domain.net)
(INSTANCE_NAME = rac11g2s1)
)
)


RAC11G2S2 =
(DESCRIPTION =
(ADDRESS = (PROTOCOL = TCP)(HOST = rac5b-vip)(PORT = 1521))
(CONNECT_DATA =
(SERVER = DEDICATED)
(SERVICE_NAME = rac11g2s.domain.net)
(INSTANCE_NAME = rac11g2s2)
)
)

RAC11G2 =
(DESCRIPTION =
(ADDRESS = (PROTOCOL = TCP)(HOST = rac4-vip)(PORT = 1521))
(ADDRESS = (PROTOCOL = TCP)(HOST = rac5-vip)(PORT = 1521))
(CONNECT_DATA =
(SERVER = DEDICATED)
(SERVICE_NAME = rac11g2.domain.net)
)
)

RAC11G21 =
(DESCRIPTION =
(ADDRESS = (PROTOCOL = TCP)(HOST = rac4-vip)(PORT = 1521))
(CONNECT_DATA =
(SERVER = DEDICATED)
(SERVICE_NAME = rac11g2.domain.net)
(INSTANCE_NAME = rac11g21)
)
)


RAC11G22 =
(DESCRIPTION =
(ADDRESS = (PROTOCOL = TCP)(HOST = rac5-vip)(PORT = 1521))
(CONNECT_DATA =
(SERVER = DEDICATED)
(SERVICE_NAME = rac11g2.domain.net)
(INSTANCE_NAME = rac11g22)
)
)
On all standby node's $ORACLE_HOME/network/admin
RAC11G2SCAN =
(DESCRIPTION =
(ADDRESS = (PROTOCOL = TCP)(HOST = rac-scan)(PORT = 1521))
(CONNECT_DATA =
(SERVER = DEDICATED)
(SERVICE_NAME = rac11g2.domain.net)
)
)

RAC11G2 =
(DESCRIPTION =
(ADDRESS = (PROTOCOL = TCP)(HOST = rac4-vip)(PORT = 1521))
(ADDRESS = (PROTOCOL = TCP)(HOST = rac5-vip)(PORT = 1521))
(CONNECT_DATA =
(SERVER = DEDICATED)
(SERVICE_NAME = rac11g2.domain.net)
)
)

RAC11G21 =
(DESCRIPTION =
(ADDRESS = (PROTOCOL = TCP)(HOST = rac4-vip)(PORT = 1521))
(CONNECT_DATA =
(SERVER = DEDICATED)
(SERVICE_NAME = rac11g2.domain.net)
(INSTANCE_NAME = rac11g21)
)
)


RAC11G22 =
(DESCRIPTION =
(ADDRESS = (PROTOCOL = TCP)(HOST = rac5-vip)(PORT = 1521))
(CONNECT_DATA =
(SERVER = DEDICATED)
(SERVICE_NAME = rac11g2.domain.net)
(INSTANCE_NAME = rac11g22)
)
)

RAC11G2S =
(DESCRIPTION =
(ADDRESS = (PROTOCOL = TCP)(HOST = rac4b-vip)(PORT = 1521))
(ADDRESS = (PROTOCOL = TCP)(HOST = rac5b-vip)(PORT = 1521))
(CONNECT_DATA =
(SERVER = DEDICATED)
(SERVICE_NAME = rac11g2s.domain.net)
)
)

RAC11G2S1 =
(DESCRIPTION =
(ADDRESS = (PROTOCOL = TCP)(HOST = rac4b-vip)(PORT = 1521))
(CONNECT_DATA =
(SERVER = DEDICATED)
(SERVICE_NAME = rac11g2s.domain.net)
(INSTANCE_NAME = rac11g2s1)
)
)


RAC11G2S2 =
(DESCRIPTION =
(ADDRESS = (PROTOCOL = TCP)(HOST = rac5b-vip)(PORT = 1521))
(CONNECT_DATA =
(SERVER = DEDICATED)
(SERVICE_NAME = rac11g2s.domain.net)
(INSTANCE_NAME = rac11g2s2)
)
)
8. Before modifying initialization parameters create a pfile from the spfile on the primary db
create pfile='/home/oracle/primaryinitpfile.ora' from spfile;
9. Modify the initialization parameters on the primary db. Remote archive dest state is set to defer, enable it before running the standby duplication. LGWR with ASYNC and NOAFFIRM is used as the redo transport this could be changed to suite the protection mode. Also FAL_CLIENT parameter has been set to each instance. This could be set to RAC11G2 TNS entry which has all entries.
alter system set log_archive_config='dg_config=(rac11g2,rac11g2s)' scope=both ;
alter system set log_archive_dest_1='location=use_db_recovery_file_dest valid_for=(all_logfiles,all_roles) db_unique_name=rac11g2' scope=both;
alter system set log_archive_dest_2='service=RAC11G2S LGWR ASYNC NOAFFIRM max_failure=10 max_connections=5 reopen=180 valid_for=(online_logfiles,primary_role) db_unique_name=rac11g2s' scope=both;
alter system set log_archive_dest_state_2='defer' scope=both;
alter system set log_archive_dest_state_1='enable' scope=both;
alter system set fal_server='RAC11G2S1','RAC11G2S2' scope=both;
alter system set fal_client='RAC11G21' scope=both sid='rac11g21'; --- since not using scan
alter system set fal_client='RAC11G22' scope=both sid='rac11g22'; --- sicne not using scan
alter system set log_archive_max_processes=10 scope=both;
alter system set db_file_name_convert='+DATA/rac11g2s','+DATA/rac11g2' scope=spfile;
alter system set log_file_name_convert='+FLASH/rac11g2s','+FLASH/rac11g2','+DATA/rac11g2s','+DATA/rac11g2' scope=spfile;
alter system set standby_file_management='AUTO' scope=both;
To make these changes take effect it requires a database restart not an instance restart this is due to the fact that filename convert parameter must be same on all nodes. If an instance restart is done it will fail with the following error
on alert log

NOTE: Loaded library: System
SUCCESS: diskgroup DATA was mounted
SUCCESS: diskgroup FLASH was mounted
NOTE: dependency between database rac11g2 and diskgroup resource ora.DATA.dg is established
NOTE: dependency between database rac11g2 and diskgroup resource ora.FLASH.dg is established
ORA-1105 signalled during: ALTER DATABASE MOUNT /* db agent *//* {1:7917:278} */...
Fri Feb 18 14:49:17 2011
Shutting down instance (abort)
License high water mark = 1
USER (ospid: 9322): terminating the instance
Instance terminated by USER, pid = 9322
Fri Feb 18 14:49:18 2011
Instance shutdown complete
on the command shell
$ srvctl stop instance -d rac11g2 -i rac11g21
$ srvctl start instance -d rac11g2 -i rac11g21
PRCR-1013 : Failed to start resource ora.rac11g2.db
PRCR-1064 : Failed to start resource ora.rac11g2.db on node rac4
CRS-5017: The resource action "ora.rac11g2.db start" encountered the following error:
ORA-01105: mount is incompatible with mounts by other instances
ORA-01677: standby file name convert parameters differ from other instance
CRS-2674: Start of 'ora.rac11g2.db' on 'rac4' failed
If the dataguard traffic is only one way (don't plan to use current primary as a standby in the future, then filename convert parameters could be ommited and a instnace restart could be carried out), otherwise do a database restart which means brining the system down.

10. Another difference between 10gR2/11gR1 and 11gR2 is the local_listener and remote_listener parameters. According to Real Application Clusters Installation Guide During Oracle Database creation, the LOCAL_LISTENER parameter is automatically configured to point to the local listener for the database. The Database Agent sets the LOCAL_LISTENER parameter to a connect descriptor that does not require a TNS alias.
You can set a value manually for LOCAL_LISTENER. However, Oracle recommends that you leave the parameter unset so that the Database Agent can maintain it automatically. If you set LOCAL_LISTENER, then the Agent does not automatically update this value. If you do not set LOCAL_LISTENER, then the Database Agent automatically keeps the database associated with the Grid home's node listener updated, even as the ports or IP of that listener are changed.

With Oracle Clusterware 11g release 2 and later, Database Configuration Agent (DBCA) no longer sets the LOCAL_LISTENER parameter. The Oracle Clusterware agent that starts the database sets the LOCAL_LISTENER parameter dynamically, and it sets it to the actual value, not an alias. So listener_alias entries are no longer needed in the tnsnames.ora file. For the REMOTE_LISTENER parameter, Oracle Clusterware uses the EZ connect syntax scanname:scanport, so no entries are needed for the REMOTE_LISTENER parameter in the tnsnames.ora file.
More on 11gR2 Grid Infrastructure Single Client Access Name (SCAN) Explained [ID 887522.1]

11. On 11gr2 remote listener is set to scan identify the standby scan name with
srvctl config scan
SCAN name: racb-scan, Network: 1/192.168.0.0/255.255.255.0/eth0
SCAN VIP name: scan1, IP: /racb-scan/192.168.0.92
This will be used later on.

12. Copy the primary password file to standby nodes and rename accordingly. On rac4b where instance rac11g2s1 will be running password file will be named orapwrac11g2s1 and on rac5b orapwrac11g2s2.

13. Select one node (rac4b in this case) on the standby cluster and create a pfile with db_name entry which will have the standby database name (not the instance name). Rename the pfile with the instance name.
cat initrac11g2s1.ora
db_name=rac11g2s
14. Start the instance in nomount mode
export ORACLE_SID=rac11g2s1
SQL> startup nomount pfile='?/dbs/initrac11g2s1.ora';
ORACLE instance started.

Total System Global Area  217157632 bytes
Fixed Size                  2225064 bytes
Variable Size             159386712 bytes
Database Buffers           50331648 bytes
Redo Buffers                5214208 bytes
15. On primary db enable the deferred log archive destination
SQL> alter system set log_archive_dest_state_2='enable' SCOPE=both sid='*';
16. On one node in the primary rac (rac4 in this case) connect to primary instance (target) as well standby instance (using rman auxiliary connection)
rman target / auxiliary sys/******@rac11g2s1

Recovery Manager: Release 11.2.0.2.0 - Production on Fri Feb 18 15:12:39 2011

Copyright (c) 1982, 2009, Oracle and/or its affiliates.  All rights reserved.

connected to target database: RAC11G2 (DBID=379668842)
connected to auxiliary database: RAC11G2S (not mounted)
17. Run the duplication command with
duplicate target database for standby from active database
spfile
parameter_value_convert 'rac11g2','rac11g2s','RAC11G2','RAC11G2S'
set db_unique_name='rac11g2s'
set db_file_name_convert='+DATA/rac11g2','+DATA/rac11g2s'
set log_file_name_convert='+FLASH/rac11g2','+FLASH/rac11g2s','+DATA/rac11g2','+DATA/rac11g2s'
set control_files='+DATA','+FLASH'
set instance_number='1'
set log_archive_max_processes='5'
set fal_client='RAC11G2S'
set fal_server='RAC11G21','RAC11G22'
set remote_listener='racb-scan:1521'
reset local_listener
set log_archive_dest_2='service=RAC11G2 LGWR ASYNC NOAFFIRM max_failure=10 max_connections=5 reopen=180 valid_for=(online_logfiles,primary_role) db_unique_name=rac11g2'
set log_archive_dest_1='location=use_db_recovery_file_dest valid_for=(all_logfiles,all_roles) db_unique_name=rac11g2s';
Key parameters are data file and log file name converters as well as the parameter value converter which maps primary environment values to appropriate standby environment values. local_listener is reset as it is auto set by grid infrastructure. remote_listener value is set the scan ip and port identified earlier.
rman target / auxiliary sys/******@rac11g2s1

Recovery Manager: Release 11.2.0.2.0 - Production on Fri Feb 18 16:10:16 2011

Copyright (c) 1982, 2009, Oracle and/or its affiliates.  All rights reserved.

connected to target database: RAC11G2 (DBID=379668842)
connected to auxiliary database: RAC11G2S (not mounted)

RMAN> duplicate target database for standby from active database
2> spfile
3> parameter_value_convert 'rac11g2','rac11g2s','RAC11G2','RAC11G2S'
4> set db_unique_name='rac11g2s'
5> set db_file_name_convert='+DATA/rac11g2','+DATA/rac11g2s'
6> set log_file_name_convert='+FLASH/rac11g2','+FLASH/rac11g2s','+DATA/rac11g2','+DATA/rac11g2s'
7> set control_files='+DATA','+FLASH'
8> set instance_number='1'
9> set log_archive_max_processes='5'
10> set fal_client='RAC11G2S'
11> set fal_server='RAC11G21','RAC11G22'
12> set remote_listener='racb-scan:1521'
13> reset local_listener
14> set log_archive_dest_2='service=RAC11G2 LGWR ASYNC NOAFFIRM max_failure=10 max_connections=5 reopen=180 valid_for=(online_logfiles,primary_role) db_unique_name=rac11g2'
15> set log_archive_dest_1='location=use_db_recovery_file_dest valid_for=(all_logfiles,all_roles) db_unique_name=rac11g2s';

Starting Duplicate Db at 18-FEB-11
using target database control file instead of recovery catalog
allocated channel: ORA_AUX_DISK_1
channel ORA_AUX_DISK_1: SID=13 device type=DISK

contents of Memory Script:
{
backup as copy reuse
targetfile  '/opt/app/oracle/product/11.2.0/db_1/dbs/orapwrac11g21' auxiliary format
'/opt/app/oracle/product/11.2.0/db_1/dbs/orapwrac11g2s1'   targetfile
'+DATA/rac11g2/spfilerac11g2.ora' auxiliary format
'/opt/app/oracle/product/11.2.0/db_1/dbs/spfilerac11g2s1.ora'   ;
sql clone "alter system set spfile= ''/opt/app/oracle/product/11.2.0/db_1/dbs/spfilerac11g2s1.ora''";
}
executing Memory Script

Starting backup at 18-FEB-11
allocated channel: ORA_DISK_1
channel ORA_DISK_1: SID=155 instance=rac11g21 device type=DISK
Finished backup at 18-FEB-11

sql statement: alter system set spfile= ''/opt/app/oracle/product/11.2.0/db_1/dbs/spfilerac11g2s1.ora''

contents of Memory Script:
{
sql clone "alter system set  audit_file_dest =
''/opt/app/oracle/admin/rac11g2s/adump'' comment=
'''' scope=spfile";
sql clone "alter system set  dispatchers =
''(PROTOCOL=TCP) (SERVICE=rac11g2sXDB)'' comment=
'''' scope=spfile";
sql clone "alter system set  db_unique_name =
''rac11g2s'' comment=
'''' scope=spfile";
sql clone "alter system set  db_file_name_convert =
''+DATA/rac11g2'', ''+DATA/rac11g2s'' comment=
'''' scope=spfile";
sql clone "alter system set  log_file_name_convert =
''+FLASH/rac11g2'', ''+FLASH/rac11g2s'', ''+DATA/rac11g2'', ''+DATA/rac11g2s'' comment=
'''' scope=spfile";
sql clone "alter system set  control_files =
''+DATA'', ''+FLASH'' comment=
'''' scope=spfile";
sql clone "alter system set  instance_number =
1 comment=
'''' scope=spfile";
sql clone "alter system set  log_archive_max_processes =
5 comment=
'''' scope=spfile";
sql clone "alter system set  fal_client =
''RAC11G2S'' comment=
'''' scope=spfile";
sql clone "alter system set  fal_server =
''RAC11G21'', ''RAC11G22'' comment=
'''' scope=spfile";
sql clone "alter system set  remote_listener =
''racb-scan:1521'' comment=
'''' scope=spfile";
sql clone "alter system set  log_archive_dest_2 =
''service=RAC11G2 LGWR ASYNC NOAFFIRM max_failure=10 max_connections=5 reopen=180 valid_for=(online_logfiles,primary_role) db_unique_name=rac11g2'' comment=
'''' scope=spfile";
sql clone "alter system set  log_archive_dest_1 =
''location=use_db_recovery_file_dest valid_for=(all_logfiles,all_roles) db_unique_name=rac11g2s'' comment=
'''' scope=spfile";
shutdown clone immediate;
startup clone nomount;
}
executing Memory Script

sql statement: alter system set  audit_file_dest =  ''/opt/app/oracle/admin/rac11g2s/adump'' comment= '''' scope=spfile

sql statement: alter system set  dispatchers =  ''(PROTOCOL=TCP) (SERVICE=rac11g2sXDB)'' comment= '''' scope=spfile

sql statement: alter system set  db_unique_name =  ''rac11g2s'' comment= '''' scope=spfile

sql statement: alter system set  db_file_name_convert =  ''+DATA/rac11g2'', ''+DATA/rac11g2s'' comment= '''' scope=spfile

sql statement: alter system set  log_file_name_convert =  ''+FLASH/rac11g2'', ''+FLASH/rac11g2s'', ''+DATA/rac11g2'', ''+DATA/rac11g2s'' comment= '''' scope=spfile

sql statement: alter system set  control_files =  ''+DATA'', ''+FLASH'' comment= '''' scope=spfile

sql statement: alter system set  instance_number =  1 comment= '''' scope=spfile

sql statement: alter system set  log_archive_max_processes =  5 comment= '''' scope=spfile

sql statement: alter system set  fal_client =  ''RAC11G2S'' comment= '''' scope=spfile

sql statement: alter system set  fal_server =  ''RAC11G21'', ''RAC11G22'' comment= '''' scope=spfile

sql statement: alter system set  remote_listener =  ''racb-scan:1521'' comment= '''' scope=spfile

sql statement: alter system set  log_archive_dest_2 =  ''service=RAC11G2 LGWR ASYNC NOAFFIRM max_failure=10 max_connections=5 reopen=180 valid_for=(online_logfiles,primary_role) db_unique_name=rac11g2'' comment= '''' scope=spfile

sql statement: alter system set  log_archive_dest_1 =  ''location=use_db_recovery_file_dest valid_for=(all_logfiles,all_roles) db_unique_name=rac11g2s'' comment= '''' scope=spfile

Oracle instance shut down

connected to auxiliary database (not started)
Oracle instance started

Total System Global Area     630501376 bytes

Fixed Size                     2229120 bytes
Variable Size                247467136 bytes
Database Buffers             373293056 bytes
Redo Buffers                   7512064 bytes

contents of Memory Script:
{
sql clone "alter system set  control_files =
''+DATA/rac11g2s/controlfile/current.256.743447505'', ''+FLASH/rac11g2s/controlfile/current.256.743447507'' comment=
''Set by RMAN'' scope=spfile";
backup as copy current controlfile for standby auxiliary format  '+DATA/rac11g2s/controlfile/current.257.743447507';
restore clone controlfile to  '+FLASH/rac11g2s/controlfile/current.257.743447507' from
'+DATA/rac11g2s/controlfile/current.257.743447507';
sql clone "alter system set  control_files =
''+DATA/rac11g2s/controlfile/current.257.743447507'', ''+FLASH/rac11g2s/controlfile/current.257.743447507'' comment=
''Set by RMAN'' scope=spfile";
shutdown clone immediate;
startup clone nomount;
}
executing Memory Script

sql statement: alter system set  control_files =   ''+DATA/rac11g2s/controlfile/current.256.743447505'', ''+FLASH/rac11g2s/controlfile/current.256.743447507'' comment= ''Set by RMAN'' scope=spfile

Starting backup at 18-FEB-11
using channel ORA_DISK_1
channel ORA_DISK_1: starting datafile copy
copying standby control file
output file name=/opt/app/oracle/product/11.2.0/db_1/dbs/snapcf_rac11g21.f tag=TAG20110218T161053 RECID=1 STAMP=743443854
channel ORA_DISK_1: datafile copy complete, elapsed time: 00:00:07
Finished backup at 18-FEB-11

Starting restore at 18-FEB-11
allocated channel: ORA_AUX_DISK_1
channel ORA_AUX_DISK_1: SID=19 instance=rac11g2s1 device type=DISK

channel ORA_AUX_DISK_1: copied control file copy
Finished restore at 18-FEB-11

sql statement: alter system set  control_files =   ''+DATA/rac11g2s/controlfile/current.257.743447507'', ''+FLASH/rac11g2s/controlfile/current.257.743447507'' comment= ''Set by RMAN'' scope=spfile

Oracle instance shut down

connected to auxiliary database (not started)
Oracle instance started

Total System Global Area     630501376 bytes

Fixed Size                     2229120 bytes
Variable Size                247467136 bytes
Database Buffers             373293056 bytes
Redo Buffers                   7512064 bytes

contents of Memory Script:
{
sql clone 'alter database mount standby database';
}
executing Memory Script

sql statement: alter database mount standby database
RMAN-05529: WARNING: DB_FILE_NAME_CONVERT resulted in invalid ASM names; names changed to disk group only.

contents of Memory Script:
{
set newname for tempfile  1 to
"+data";
switch clone tempfile all;
set newname for datafile  1 to
"+data";
set newname for datafile  2 to
"+data";
set newname for datafile  3 to
"+data";
set newname for datafile  4 to
"+data";
set newname for datafile  5 to
"+data";
set newname for datafile  6 to
"+data";
set newname for datafile  7 to
"+data";
backup as copy reuse
datafile  1 auxiliary format
"+data"   datafile
2 auxiliary format
"+data"   datafile
3 auxiliary format
"+data"   datafile
4 auxiliary format
"+data"   datafile
5 auxiliary format
"+data"   datafile
6 auxiliary format
"+data"   datafile
7 auxiliary format
"+data"   ;
sql 'alter system archive log current';
}
executing Memory Script

executing command: SET NEWNAME

renamed tempfile 1 to +data in control file

executing command: SET NEWNAME

executing command: SET NEWNAME

executing command: SET NEWNAME

executing command: SET NEWNAME

executing command: SET NEWNAME

executing command: SET NEWNAME

executing command: SET NEWNAME

Starting backup at 18-FEB-11
using channel ORA_DISK_1
channel ORA_DISK_1: starting datafile copy
input datafile file number=00002 name=+DATA/rac11g2/datafile/sysaux.257.740770047
output file name=+DATA/rac11g2s/datafile/sysaux.258.743447539 tag=TAG20110218T161124
channel ORA_DISK_1: datafile copy complete, elapsed time: 00:01:25
channel ORA_DISK_1: starting datafile copy
input datafile file number=00001 name=+DATA/rac11g2/datafile/system.256.740770045
output file name=+DATA/rac11g2s/datafile/system.259.743447625 tag=TAG20110218T161124
channel ORA_DISK_1: datafile copy complete, elapsed time: 00:01:15
channel ORA_DISK_1: starting datafile copy
input datafile file number=00003 name=+DATA/rac11g2/datafile/undotbs1.258.740770049
output file name=+DATA/rac11g2s/datafile/undotbs1.260.743447699 tag=TAG20110218T161124
channel ORA_DISK_1: datafile copy complete, elapsed time: 00:00:25
channel ORA_DISK_1: starting datafile copy
input datafile file number=00005 name=+DATA/rac11g2/datafile/undotbs2.264.740770355
output file name=+DATA/rac11g2s/datafile/undotbs2.261.743447725 tag=TAG20110218T161124
channel ORA_DISK_1: datafile copy complete, elapsed time: 00:00:07
channel ORA_DISK_1: starting datafile copy
input datafile file number=00004 name=+DATA/rac11g2/datafile/users.259.740770049
output file name=+DATA/rac11g2s/datafile/users.262.743447733 tag=TAG20110218T161124
channel ORA_DISK_1: datafile copy complete, elapsed time: 00:00:01
channel ORA_DISK_1: starting datafile copy
input datafile file number=00006 name=+DATA/rac11g2/datafile/test.268.741633047
output file name=+DATA/rac11g2s/datafile/test.263.743447733 tag=TAG20110218T161124
channel ORA_DISK_1: datafile copy complete, elapsed time: 00:00:01
channel ORA_DISK_1: starting datafile copy
input datafile file number=00007 name=+DATA/rac11g2/datafile/test.269.741633183
output file name=+DATA/rac11g2s/datafile/test.264.743447735 tag=TAG20110218T161124
channel ORA_DISK_1: datafile copy complete, elapsed time: 00:00:01
Finished backup at 18-FEB-11

sql statement: alter system archive log current

contents of Memory Script:
{
switch clone datafile all;
}
executing Memory Script

datafile 1 switched to datafile copy
input datafile copy RECID=1 STAMP=743447740 file name=+DATA/rac11g2s/datafile/system.259.743447625
datafile 2 switched to datafile copy
input datafile copy RECID=2 STAMP=743447740 file name=+DATA/rac11g2s/datafile/sysaux.258.743447539
datafile 3 switched to datafile copy
input datafile copy RECID=3 STAMP=743447740 file name=+DATA/rac11g2s/datafile/undotbs1.260.743447699
datafile 4 switched to datafile copy
input datafile copy RECID=4 STAMP=743447740 file name=+DATA/rac11g2s/datafile/users.262.743447733
datafile 5 switched to datafile copy
input datafile copy RECID=5 STAMP=743447740 file name=+DATA/rac11g2s/datafile/undotbs2.261.743447725
datafile 6 switched to datafile copy
input datafile copy RECID=6 STAMP=743447740 file name=+DATA/rac11g2s/datafile/test.263.743447733
datafile 7 switched to datafile copy
input datafile copy RECID=7 STAMP=743447740 file name=+DATA/rac11g2s/datafile/test.264.743447735
Finished Duplicate Db at 18-FEB-11
18. Once the above command suceeds on primary on the standby(rac4b node) run the log apply command
SQL> alter database recover managed standby database using current logfile disconnect;
Database altered.
On the database alert log it could be observed archived logs being applied
Fri Feb 18 17:19:56 2011
alter database recover managed standby database using current logfile disconnect
Attempt to start background Managed Standby Recovery process (rac11g2s1)
Fri Feb 18 17:19:56 2011
MRP0 started with pid=60, OS id=8639
MRP0: Background Managed Standby Recovery process started (rac11g2s1)
started logmerger process
Fri Feb 18 17:20:01 2011
Managed Standby Recovery starting Real Time Apply
Parallel Media Recovery started with 2 slaves
Waiting for all non-current ORLs to be archived...
All non-current ORLs have been archived.
Fri Feb 18 17:20:02 2011
Media Recovery Log +FLASH/rac11g2s/archivelog/2011_02_18/thread_1_seq_53.269.743447835
Media Recovery Log +FLASH/rac11g2s/archivelog/2011_02_18/thread_2_seq_43.268.743447835
Completed: alter database recover managed standby database using current logfile disconnect
Media Recovery Log +FLASH/rac11g2s/archivelog/2011_02_18/thread_1_seq_54.271.743447847
Media Recovery Log +FLASH/rac11g2s/archivelog/2011_02_18/thread_2_seq_44.270.743447847
Media Recovery Log +FLASH/rac11g2s/archivelog/2011_02_18/thread_2_seq_45.273.743447887
Media Recovery Log +FLASH/rac11g2s/archivelog/2011_02_18/thread_1_seq_55.272.743447883
Media Recovery Waiting for thread 1 sequence 56 (in transit)
Recovery of Online Redo Log: Thread 1 Group 6 Seq 56 Reading mem 0
Mem# 0: +DATA/rac11g2s/onlinelog/group_6.270.743447749
Mem# 1: +FLASH/rac11g2s/onlinelog/group_6.263.743447751
Media Recovery Waiting for thread 2 sequence 46 (in transit)
Recovery of Online Redo Log: Thread 2 Group 9 Seq 46 Reading mem 0
Mem# 0: +DATA/rac11g2s/onlinelog/group_9.273.743447755
Mem# 1: +FLASH/rac11g2s/onlinelog/group_9.266.743447755
Another way to find out is to connect into the standby instance (rac11g2s1 running on rac4b node) and issue
SQL> select sequence#,thread#,applied from v$archived_log;

SEQUENCE#    THREAD# APPLIED
---------- ---------- ---------
53          1 YES
43          2 YES
54          1 YES
44          2 YES
55          1 YES
45          2 IN-MEMORY
6 rows selected.
which shows YES on archive log applied column.



19. With above configuration inplace the data guard is setup and working. The next step is to bring the standby database under the control of the cluster. For this on the standby instance create a pfile from the spfile that was created during the duplication process.
SQL> create pfile='/home/oracle/stdbypfile.ora' from spfile;
20. Edit the pfile and remove any references to primary db (rac11g2) except db_name this must be same as primary db. Set instance specific instance_number, undo_tablespace, fal_client and thread. pfile content after the edit is given below
rac11g2s1.__db_cache_size=373293056
rac11g2s1.__java_pool_size=4194304
rac11g2s1.__large_pool_size=4194304
rac11g2s1.__oracle_base='/opt/app/oracle'#ORACLE_BASE set from environment
rac11g2s1.__pga_aggregate_target=209715200
rac11g2s1.__sga_target=633339904
rac11g2s1.__shared_io_pool_size=0
rac11g2s1.__shared_pool_size=239075328
rac11g2s1.__streams_pool_size=0
rac11g2s2.__db_cache_size=373293056
rac11g2s2.__java_pool_size=4194304
rac11g2s2.__large_pool_size=4194304
rac11g2s2.__oracle_base='/opt/app/oracle'#ORACLE_BASE set from environment
rac11g2s2.__pga_aggregate_target=209715200
rac11g2s2.__sga_target=633339904
rac11g2s2.__shared_io_pool_size=0
rac11g2s2.__shared_pool_size=239075328
rac11g2s2.__streams_pool_size=0
*.audit_file_dest='/opt/app/oracle/admin/rac11g2s/adump'
*.audit_trail='db'
*.cluster_database=true
*.compatible='11.2.0.0.0'
*.control_files='+DATA/rac11g2s/controlfile/current.257.743447507','+FLASH/rac11g2s/controlfile/current.257.743447507'#Set by RMAN
*.db_block_size=8192
*.db_create_file_dest='+DATA'
*.db_domain='domain.net'
*.db_file_name_convert='+DATA/rac11g2','+DATA/rac11g2s'
*.db_name='rac11g2'
*.db_recovery_file_dest='+FLASH'
*.db_recovery_file_dest_size=9470738432
*.db_unique_name='rac11g2s'
*.diagnostic_dest='/opt/app/oracle'
*.dispatchers='(PROTOCOL=TCP) (SERVICE=rac11g2sXDB)'
rac11g2s1.fal_client='RAC11G2S1'
rac1g2s22.fal_client='RAC11G2S2'
*.fal_server='RAC11G21','RAC11G22'
rac11g2s1.instance_number=1
rac11g2s2.instance_number=2
*.log_archive_config='dg_config=(rac11g2,rac11g2s)'
*.log_archive_dest_2='service=RAC11G2 LGWR ASYNC NOAFFIRM max_failure=10 max_connections=5 reopen=180 valid_for=(online_logfiles,primary_role) db_uniq
ue_name=rac11g2'
*.log_archive_dest_1='location=use_db_recovery_file_dest valid_for=(all_logfiles,all_roles) db_unique_name=rac11g2s'
*.log_archive_dest_state_2='enable'
*.log_archive_dest_state_1='enable'
*.log_archive_format='%t_%s_%r.dbf'
*.log_archive_max_processes=5
*.log_file_name_convert='+FLASH/rac11g2','+FLASH/rac11g2s','+DATA/rac11g2','+DATA/rac11g2s'
*.open_cursors=300
*.pga_aggregate_target=209715200
*.processes=150
*.remote_listener='racb-scan:1521'
*.remote_login_passwordfile='exclusive'
*.sga_target=631242752
*.standby_file_management='AUTO'
rac11g2s1.undo_tablespace='UNDOTBS1'
rac11g2s2.undo_tablespace='UNDOTBS2'
rac11g2s1.thread=1
rac11g2s2.thread=2
21. Shutdown the database and startup mount and create a spfile with +ASM diskgroup as the locaiton
SQL>alter database recover managed standby database cancel;

SQL>startup mount pfile='/home/oracle/stdbypfile.ora';

ORACLE instance started.

Total System Global Area  630501376 bytes
Fixed Size                  2229120 bytes
Variable Size             247467136 bytes
Database Buffers          373293056 bytes
Redo Buffers                7512064 bytes
Database mounted.

SQL> create spfile='+DATA/rac11g2s/spfilerac11g2s.ora' from pfile='/home/oracle/stdbypfile.ora';

File created.

SQL> shutdown immediate;
22. Create pfile file on each node's oracle home (ORACLE_HOME/dbs) that points to the spfile by adding one line that referes to the spfile
vi initrac11g2s1.ora
spfile='+DATA/rac11g2s/spfilerac11g2s.ora'
Above is for rac4b, create initrac11g2s2.ora on rac5b.

23. Once above pfiles are created remove the spfile in $ORACLE_HOME/dbs that was created as part of the duplication.

24. Before bringing the standby instance under cluster control check it could be started on all other nodes with their respective instances. In this case rac5b since this is a two node rac, export ORACLE_SID to rac11g2s2 and start the instance in mount mode. Resolve any issues (init parameter related) before proceeding to the next step)

25. To add the database and instances to the cluster configuration run
$GRID_HOME/bin/srvctl add database -d rac11g2s -o $ORACLE_HOME -m local -p "+DATA/rac11g2s/spfilerac11g2s.ora" -n rac11g2 -r physical_standby -s mount
$GRID_HOME/bin/srvctl add instance -d rac11g2s -i rac11g2s1 -n rac4b
$GRID_HOME/bin/srvctl add instance -d rac11g2s -i rac11g2s2 -n rac5b
Without the "-s mount" the standby database would be started in open (read only) mode by default (both R1 and R2). With 11g (both R1 and R2) it is possible to apply redo while the database is open in read only mode. But this requires active dataguard license and setting the default startup mode to mount prevents such activity from happenning when active dataguard is not licensed. On 11gR1 db instances could be made dependent on that node's asm instance but this option is deprecated in 11gr2
$CRS_HOME/bin/srvctl modify instance -d rac11g2s -i rac11g2s1 -s +ASM1
-s option has been deprecated and will be ignored.

srvctl config database -d rac11g2s
Database unique name: rac11g2s
Database name: rac11g2
Oracle home: /opt/app/oracle/product/11.2.0/db_1
Oracle user: oracle
Spfile: +DATA/rac11g2s/spfilerac11g2s.ora
Domain: local
Start options: mount
Stop options: immediate
Database role: PHYSICAL_STANDBY
Management policy: AUTOMATIC
Server pools: rac11g2s
Database instances: rac11g2s1,rac11g2s2
Disk Groups:
Mount point paths:
Services:
Type: RAC
Database is administrator managed
26. Test the configuration with starting the entire standby database
srvctl status database -d rac11g2s
Instance rac11g2s1 is not running on node rac4b
Instance rac11g2s2 is not running on node rac5b

srvctl start database -d rac11g2s

srvctl status database -d rac11g2s
Instance rac11g2s1 is running on node rac4b
Instance rac11g2s2 is running on node rac5b
Once the database is started as a cluster resource the asm dependencies are automatically added as well as the local_listener values which were reset during the duplication.
crsctl stat res ora.rac11g2s.db -p

NAME=ora.rac11g2s.db
TYPE=ora.database.type
..
SERVER_POOLS=ora.rac11g2s
SPFILE=+DATA/rac11g2s/spfilerac11g2s.ora
START_DEPENDENCIES=hard(ora.DATA.dg,ora.FLASH.dg)weak(type:ora.listener.type,global:type:ora.scan_listener.type,uniform:ora.ons,global:ora.gns)pullup(ora.DATA.dg,ora.FLASH.dg)
START_TIMEOUT=600
STATE_CHANGE_TEMPLATE=
STOP_DEPENDENCIES=hard(intermediate:ora.asm,shutdown:ora.DATA.dg,shutdown:ora.FLASH.dg)
..

SQL> show parameter local

NAME            TYPE        VALUE
--------------- ---------- ------------------------------
local_listener  string      (DESCRIPTION=(ADDRESS_LIST=(ADDRESS=(PROTOCOL=TCP(HOST=192.168.0.98)(PORT=1521))))
26. Start the recovery process on one of the standby instances.
SQL> alter database recover managed standby database using current logfile disconnect;
This concludes the setup of RAC-RAC data guard.

27. To make sure archive logs don't get deleted on primary without being applied on standby set the log deletion policy to applied on standby. For this at least one log archive locations must be mandatory.
SQL> alter system set log_archive_dest_2='service=RAC11G2S LGWR ASYNC NOAFFIRM max_failure=10 max_connections=5 reopen=180 valid_for=(online_logfiles,primary_role) db_unique_name=rac11g2s mandatory' scope=both sid='*';
Usign rman chagne the archive log deletion policy.
RMAN> CONFIGURE ARCHIVELOG DELETION POLICY TO APPLIED ON STANDBY;

old RMAN configuration parameters:
CONFIGURE ARCHIVELOG DELETION POLICY TO NONE;
new RMAN configuration parameters:
CONFIGURE ARCHIVELOG DELETION POLICY TO APPLIED ON STANDBY;
new RMAN configuration parameters are successfully stored
If it is configured APPLIED ON ALL STANDBY then archived redo log files are eligible for deletion after being applied or consumed on all remote destinations, whether mandatory or not.

With mandatory destination inplace if the standby is not up and accepting redo when the primary starts following error
PING[ARC1]: Heartbeat failed to connect to standby 'RAC11G2S'. Error is 12543.
Archived Log entry 202 added for thread 1 sequence 80 ID 0x16a13366 dest 10:
ARCH: Archival stopped, error occurred. Will continue retrying
ORACLE Instance rac11g21 - Archival Error
ORA-16038: log 2 sequence# 80 cannot be archived
ORA-12543: TNS:destination host unreachable
ORA-00312: online log 2 thread 1: '+DATA/rac11g2/onlinelog/group_2.262.740770163'
ORA-00312: online log 2 thread 1: '+FLASH/rac11g2/onlinelog/group_2.258.740770163'
and any alter system switch logfile commands will hang.

Distributing redo load

It is possible to distribute the redo load among all standby instnaces. In this case instead of sending the redo from all primary instance to one standby instance each primary instance will send its redo to corresponding standby instance. This configuration is ideal in a symmetric data guard environment and one signle standby instnace cannot keep up with all the redo generated. To do this set primary instance specific log_archive_dest values
alter system set log_archive_dest_2='service=RAC11G2S1 LGWR ASYNC NOAFFIRM max_failure=10 max_connections=5 reopen=180 valid_for=(online_logfiles,primary_role) db_unique_name=rac11g2s mandatory' scope=spfile sid='rac11g21';

alter system set log_archive_dest_2='service=RAC11G2S2 LGWR ASYNC NOAFFIRM max_failure=10 max_connections=5 reopen=180 valid_for=(online_logfiles,primary_role) db_unique_name=rac11g2s mandatory' scope=spfile sid='rac11g22';
Here rac11g21 will send its redo to rac11g2s1 and rac11g22 will send to rac11g2s22.

But there can only be one apply instance.

Otherway around, sending all primary redo once standby instance gives more flexibility than above one-to-one redo transport configuration. In this case if the current standby apply instance need to be shutdown all that is required is stop the apply service on that instance and start in another instance. Since there's no instance specific redo transport configuration no additional steps are required. (provided the TNS entry has addressed for all standby instances similar to rac11g2s TNS entry above).

Related Post
11gR2 Standalone Data Guard (with ASM and Role Separation)
Data Guard Broker for 11gR2 RAC
Enable Active Dataguard on 11gR2 RAC Standby
Creating Data Guard Broker on 12c
Adding Far Sync Instances to Existing Data Guard Configuration
Setting Up Data Guard Broker for an Existing Data Guard Configuration with Far Sync

Friday, February 11, 2011

Adding a Node to 11gR2 RAC

Steps to add a node to 11gR2 RAC is same as in 11gR1 RACFirst add the grid infrastructure then Oracle Home and finally extend the database by adding a new instance on the new node.
Cluster used here is the one used earlier on the node delete blog.

1. Check nodeapp setup is complete with cluvfy
cluvfy stage -pre nodeadd -n rac5
rac5 is the hostname of the new node that's being added. Output of the above command might have the following lines
ERROR:
PRVF-5449 : Check of Voting Disk location "ORCL:CLUS1(ORCL:CLUS1)" failed on the following nodes:
Check failed on nodes:
rac5
rac5:No such file or directory
ERROR:
PRVF-5449 : Check of Voting Disk location "ORCL:CLUS2(ORCL:CLUS2)" failed on the following nodes:
Check failed on nodes:
rac5
rac5:No such file or directory
ERROR:
PRVF-5449 : Check of Voting Disk location "ORCL:CLUS3(ORCL:CLUS3)" failed on the following nodes:
Check failed on nodes:
rac5
rac5:No such file or directory
PRVF-5431 : Oracle Cluster Voting Disk configuration check failed
Time zone consistency check passed
As for metalink note PRVF-5449 : Check of Voting Disk location "ORCL:(ORCL:)" failed [ID 1267569.1], this is a bug on cluvfy and is not fixed at the time of this blog.

Workaround is set
export IGNORE_PREADDNODE_CHECKS=Y
without it silent node add would fail without showing any errors.

2. Run addNode.sh in $CRS_HOME/oui/bin from an existing node
export IGNORE_PREADDNODE_CHECKS=Y
./addNode.sh "CLUSTER_NEW_NODES={rac5}" "CLUSTER_NEW_VIRTUAL_HOSTNAMES={rac5-vip}" -silent
Starting Oracle Universal Installer...

Checking swap space: must be greater than 500 MB.   Actual 3313 MB    Passed
Checking monitor: must be configured to display at least 256 colors.    Actual 65536    Passed
Oracle Universal Installer, Version 11.2.0.2.0 Production
Copyright (C) 1999, 2010, Oracle. All rights reserved.


Performing tests to see whether nodes rac5 are available
............................................................... 100% Done.

------------------------------------------------------------------------
Cluster Node Addition Summary
Global Settings
Source: /opt/app/11.2.0/grid
New Nodes
Space Requirements
New Nodes
rac5
/: Required 4.11GB : Available 10.33GB
Installed Products
Product Names
Oracle Grid Infrastructure 11.2.0.2.0
Sun JDK 1.5.0.24.08
Installer SDK Component 11.2.0.2.0
Oracle One-Off Patch Installer 11.2.0.0.2
Oracle Universal Installer 11.2.0.2.0
Oracle USM Deconfiguration 11.2.0.2.0
Oracle Configuration Manager Deconfiguration 10.3.1.0.0
Enterprise Manager Common Core Files 10.2.0.4.3
Oracle DBCA Deconfiguration 11.2.0.2.0
Oracle RAC Deconfiguration 11.2.0.2.0
Oracle Quality of Service Management (Server) 11.2.0.2.0
Installation Plugin Files 11.2.0.2.0
Universal Storage Manager Files 11.2.0.2.0
Oracle Text Required Support Files 11.2.0.2.0
Automatic Storage Management Assistant 11.2.0.2.0
Oracle Database 11g Multimedia Files 11.2.0.2.0
Oracle Multimedia Java Advanced Imaging 11.2.0.2.0
Oracle Globalization Support 11.2.0.2.0
Oracle Multimedia Locator RDBMS Files 11.2.0.2.0
Oracle Core Required Support Files 11.2.0.2.0
Bali Share 1.1.18.0.0
Oracle Database Deconfiguration 11.2.0.2.0
Oracle Quality of Service Management (Client) 11.2.0.2.0
Expat libraries 2.0.1.0.1
Oracle Containers for Java 11.2.0.2.0
Perl Modules 5.10.0.0.1
Secure Socket Layer 11.2.0.2.0
Oracle JDBC/OCI Instant Client 11.2.0.2.0
Oracle Multimedia Client Option 11.2.0.2.0
LDAP Required Support Files 11.2.0.2.0
Character Set Migration Utility 11.2.0.2.0
Perl Interpreter 5.10.0.0.1
PL/SQL Embedded Gateway 11.2.0.2.0
OLAP SQL Scripts 11.2.0.2.0
Database SQL Scripts 11.2.0.2.0
Oracle Extended Windowing Toolkit 3.4.47.0.0
SSL Required Support Files for InstantClient 11.2.0.2.0
SQL*Plus Files for Instant Client 11.2.0.2.0
Oracle Net Required Support Files 11.2.0.2.0
Oracle Database User Interface 2.2.13.0.0
RDBMS Required Support Files for Instant Client 11.2.0.2.0
RDBMS Required Support Files Runtime 11.2.0.2.0
XML Parser for Java 11.2.0.2.0
Oracle Security Developer Tools 11.2.0.2.0
Oracle Wallet Manager 11.2.0.2.0
Enterprise Manager plugin Common Files 11.2.0.2.0
Platform Required Support Files 11.2.0.2.0
Oracle JFC Extended Windowing Toolkit 4.2.36.0.0
RDBMS Required Support Files 11.2.0.2.0
Oracle Ice Browser 5.2.3.6.0
Oracle Help For Java 4.2.9.0.0
Enterprise Manager Common Files 10.2.0.4.3
Deinstallation Tool 11.2.0.2.0
Oracle Java Client 11.2.0.2.0
Cluster Verification Utility Files 11.2.0.2.0
Oracle Notification Service (eONS) 11.2.0.2.0
Oracle LDAP administration 11.2.0.2.0
Cluster Verification Utility Common Files 11.2.0.2.0
Oracle Clusterware RDBMS Files 11.2.0.2.0
Oracle Locale Builder 11.2.0.2.0
Oracle Globalization Support 11.2.0.2.0
Buildtools Common Files 11.2.0.2.0
Oracle RAC Required Support Files-HAS 11.2.0.2.0
SQL*Plus Required Support Files 11.2.0.2.0
XDK Required Support Files 11.2.0.2.0
Agent Required Support Files 10.2.0.4.3
Parser Generator Required Support Files 11.2.0.2.0
Precompiler Required Support Files 11.2.0.2.0
Installation Common Files 11.2.0.2.0
Required Support Files 11.2.0.2.0
Oracle JDBC/THIN Interfaces 11.2.0.2.0
Oracle Multimedia Locator 11.2.0.2.0
Oracle Multimedia 11.2.0.2.0
HAS Common Files 11.2.0.2.0
Assistant Common Files 11.2.0.2.0
PL/SQL 11.2.0.2.0
HAS Files for DB 11.2.0.2.0
Oracle Recovery Manager 11.2.0.2.0
Oracle Database Utilities 11.2.0.2.0
Oracle Notification Service 11.2.0.2.0
SQL*Plus 11.2.0.2.0
Oracle Netca Client 11.2.0.2.0
Oracle Net 11.2.0.2.0
Oracle JVM 11.2.0.2.0
Oracle Internet Directory Client 11.2.0.2.0
Oracle Net Listener 11.2.0.2.0
Cluster Ready Services Files 11.2.0.2.0
Oracle Database 11g 11.2.0.2.0
-----------------------------------------------------------------------------


Instantiating scripts for add node (Thursday, February 10, 2011 2:46:24 PM GMT)
.                                                                 1% Done.
Instantiation of add node scripts complete

Copying to remote nodes (Thursday, February 10, 2011 2:46:29 PM GMT)
.......................................................................          96% Done.
Home copied to new nodes

Saving inventory on nodes (Thursday, February 10, 2011 2:50:43 PM GMT)
.                                                               100% Done.
Save inventory complete
WARNING:
The following configuration scripts need to be executed as the "root" user in each cluster node.
/opt/app/11.2.0/grid/root.sh #On nodes rac5
To execute the configuration scripts:
1. Open a terminal window
2. Log in as "root"
3. Run the scripts in each cluster node

The Cluster Node Addition of /opt/app/11.2.0/grid was successful.
Please check '/tmp/silentInstall.log' for more details.
3. Run root.sh on the new node
./root.sh
Running Oracle 11g root script...

The following environment variables are set as:
ORACLE_OWNER= oracle
ORACLE_HOME=  /opt/app/11.2.0/grid

Enter the full pathname of the local bin directory: [/usr/local/bin]:
The contents of "dbhome" have not changed. No need to overwrite.
The contents of "oraenv" have not changed. No need to overwrite.
The contents of "coraenv" have not changed. No need to overwrite.


Creating /etc/oratab file...
Entries will be added to the /etc/oratab file as needed by
Database Configuration Assistant when a database is created
Finished running generic part of root script.
Now product-specific root actions will be performed.
Using configuration parameter file: /opt/app/11.2.0/grid/crs/install/crsconfig_params
Creating trace directory
LOCAL ADD MODE
Creating OCR keys for user 'root', privgrp 'root'..
Operation successful.
OLR initialization - successful
Adding daemon to inittab
ACFS-9200: Supported
ACFS-9300: ADVM/ACFS distribution files found.
ACFS-9307: Installing requested ADVM/ACFS software.
ACFS-9308: Loading installed ADVM/ACFS drivers.
ACFS-9321: Creating udev for ADVM/ACFS.
ACFS-9323: Creating module dependencies - this may take some time.
ACFS-9327: Verifying ADVM/ACFS devices.
ACFS-9309: ADVM/ACFS installation correctness verified.
CRS-4402: The CSS daemon was started in exclusive mode but found an active CSS daemon on node rac4, number 1, and is terminating
An active cluster was found during exclusive startup, restarting to join the cluster
clscfg: EXISTING configuration version 5 detected.
clscfg: version 5 is 11g Release 2.
Successfully accumulated necessary OCR keys.
Creating OCR keys for user 'root', privgrp 'root'..
Operation successful.
Preparing packages for installation...
cvuqdisk-1.0.9-1
Configure Oracle Grid Infrastructure for a Cluster ... succeeded
At the end of this inventory.xml is updated on all nodes to reflect the newly added node
<HOME NAME="Ora11g_gridinfrahome1" LOC="/opt/app/11.2.0/grid" TYPE="O" IDX="1" CRS="true">
<NODE_LIST>
<NODE NAME="rac4"/>
<NODE NAME="rac5"/>
</NODE_LIST>
</HOME>
4. Use
cluvfy stage -post nodeadd -n rac5
to verify node addition and that cluster is not divided into two seperate parts.This may give the following error
Checking name resolution setup for "rac-scan"...

ERROR:
PRVF-4664 : Found inconsistent name resolution entries for SCAN name "rac-scan"
ERROR:
PRVF-4657 : Name resolution setup check for "rac-scan" (IP address: 192.168.0.91) failed
ERROR:
PRVF-4664 : Found inconsistent name resolution entries for SCAN name "rac-scan"
Verification of SCAN VIP and Listener setup failed
Reasons are explained in metalink note PRVF-4664 PRVF-4657: Found inconsistent name resolution entries for SCAN name [ID 887471.1]In this case it was because the scan ip was listed on host file. This concludes the first phase of adding a node. Next is to add the oracle home to the new node.




5. To extend the ORACLE_HOME (RAC) run addNode.sh from $ORACLE_HOME/oui/bin
./addNode.sh -silent "CLUSTER_NEW_NODES={rac5}"
Starting Oracle Universal Installer...

Checking swap space: must be greater than 500 MB.   Actual 3758 MB    Passed
Oracle Universal Installer, Version 11.2.0.2.0 Production
Copyright (C) 1999, 2010, Oracle. All rights reserved.


Performing tests to see whether nodes rac5 are available
............................................................... 100% Done.
-----------------------------------------------------------------------------
Cluster Node Addition Summary
Global Settings
Source: /opt/app/oracle/product/11.2.0/db_1
New Nodes
Space Requirements
New Nodes
rac5
/opt/app/oracle/product/11.2.0/db_1: Required 4.56GB : Available 5.06GB
Installed Products
Product Names
Oracle Database 11g 11.2.0.2.0
Sun JDK 1.5.0.24.08
Installer SDK Component 11.2.0.2.0
Oracle One-Off Patch Installer 11.2.0.0.2
Oracle Universal Installer 11.2.0.2.0
Oracle USM Deconfiguration 11.2.0.2.0
Oracle Configuration Manager Deconfiguration 10.3.1.0.0
Oracle DBCA Deconfiguration 11.2.0.2.0
Oracle RAC Deconfiguration 11.2.0.2.0
Oracle Database Deconfiguration 11.2.0.2.0
Oracle Configuration Manager Client 10.3.2.1.0
Oracle Configuration Manager 10.3.3.1.1
Oracle ODBC Driverfor Instant Client 11.2.0.2.0
LDAP Required Support Files 11.2.0.2.0
SSL Required Support Files for InstantClient 11.2.0.2.0
Bali Share 1.1.18.0.0
Oracle Extended Windowing Toolkit 3.4.47.0.0
Oracle JFC Extended Windowing Toolkit 4.2.36.0.0
Oracle Real Application Testing 11.2.0.2.0
Oracle Database Vault J2EE Application 11.2.0.2.0
Oracle Label Security 11.2.0.2.0
Oracle Data Mining RDBMS Files 11.2.0.2.0
Oracle OLAP RDBMS Files 11.2.0.2.0
Oracle OLAP API 11.2.0.2.0
Platform Required Support Files 11.2.0.2.0
Oracle Database Vault option 11.2.0.2.0
Oracle RAC Required Support Files-HAS 11.2.0.2.0
SQL*Plus Required Support Files 11.2.0.2.0
Oracle Display Fonts 9.0.2.0.0
Oracle Ice Browser 5.2.3.6.0
Oracle JDBC Server Support Package 11.2.0.2.0
Oracle SQL Developer 11.2.0.2.0
Oracle Application Express 11.2.0.2.0
XDK Required Support Files 11.2.0.2.0
RDBMS Required Support Files for Instant Client 11.2.0.2.0
SQLJ Runtime 11.2.0.2.0
Database Workspace Manager 11.2.0.2.0
RDBMS Required Support Files Runtime 11.2.0.2.0
Oracle Globalization Support 11.2.0.2.0
Exadata Storage Server 11.2.0.1.0
Provisioning Advisor Framework 10.2.0.4.3
Enterprise Manager Database Plugin -- Repository Support 11.2.0.2.0
Enterprise Manager Repository Core Files 10.2.0.4.3
Enterprise Manager Database Plugin -- Agent Support 11.2.0.2.0
Enterprise Manager Grid Control Core Files 10.2.0.4.3
Enterprise Manager Common Core Files 10.2.0.4.3
Enterprise Manager Agent Core Files 10.2.0.4.3
RDBMS Required Support Files 11.2.0.2.0
regexp 2.1.9.0.0
Agent Required Support Files 10.2.0.4.3
Oracle 11g Warehouse Builder Required Files 11.2.0.2.0
Oracle Notification Service (eONS) 11.2.0.2.0
Oracle Text Required Support Files 11.2.0.2.0
Parser Generator Required Support Files 11.2.0.2.0
Oracle Database 11g Multimedia Files 11.2.0.2.0
Oracle Multimedia Java Advanced Imaging 11.2.0.2.0
Oracle Multimedia Annotator 11.2.0.2.0
Oracle JDBC/OCI Instant Client 11.2.0.2.0
Oracle Multimedia Locator RDBMS Files 11.2.0.2.0
Precompiler Required Support Files 11.2.0.2.0
Oracle Core Required Support Files 11.2.0.2.0
Sample Schema Data 11.2.0.2.0
Oracle Starter Database 11.2.0.2.0
Oracle Message Gateway Common Files 11.2.0.2.0
Oracle XML Query 11.2.0.2.0
XML Parser for Oracle JVM 11.2.0.2.0
Oracle Help For Java 4.2.9.0.0
Installation Plugin Files 11.2.0.2.0
Enterprise Manager Common Files 10.2.0.4.3
Expat libraries 2.0.1.0.1
Deinstallation Tool 11.2.0.2.0
Oracle Quality of Service Management (Client) 11.2.0.2.0
Perl Modules 5.10.0.0.1
JAccelerator (COMPANION) 11.2.0.2.0
Oracle Containers for Java 11.2.0.2.0
Perl Interpreter 5.10.0.0.1
Oracle Net Required Support Files 11.2.0.2.0
Secure Socket Layer 11.2.0.2.0
Oracle Universal Connection Pool 11.2.0.2.0
Oracle JDBC/THIN Interfaces 11.2.0.2.0
Oracle Multimedia Client Option 11.2.0.2.0
Oracle Java Client 11.2.0.2.0
Character Set Migration Utility 11.2.0.2.0
Oracle Code Editor 1.2.1.0.0I
PL/SQL Embedded Gateway 11.2.0.2.0
OLAP SQL Scripts 11.2.0.2.0
Database SQL Scripts 11.2.0.2.0
Oracle Locale Builder 11.2.0.2.0
Oracle Globalization Support 11.2.0.2.0
SQL*Plus Files for Instant Client 11.2.0.2.0
Required Support Files 11.2.0.2.0
Oracle Database User Interface 2.2.13.0.0
Oracle ODBC Driver 11.2.0.2.0
Oracle Notification Service 11.2.0.2.0
XML Parser for Java 11.2.0.2.0
Oracle Security Developer Tools 11.2.0.2.0
Oracle Wallet Manager 11.2.0.2.0
Cluster Verification Utility Common Files 11.2.0.2.0
Oracle Clusterware RDBMS Files 11.2.0.2.0
Oracle UIX 2.2.24.6.0
Enterprise Manager plugin Common Files 11.2.0.2.0
HAS Common Files 11.2.0.2.0
Precompiler Common Files 11.2.0.2.0
Installation Common Files 11.2.0.2.0
Oracle Help for the  Web 2.0.14.0.0
Oracle LDAP administration 11.2.0.2.0
Buildtools Common Files 11.2.0.2.0
Assistant Common Files 11.2.0.2.0
Oracle Recovery Manager 11.2.0.2.0
PL/SQL 11.2.0.2.0
Generic Connectivity Common Files 11.2.0.2.0
Oracle Database Gateway for ODBC 11.2.0.2.0
Oracle Programmer 11.2.0.2.0
Oracle Database Utilities 11.2.0.2.0
Enterprise Manager Agent 10.2.0.4.3
SQL*Plus 11.2.0.2.0
Oracle Netca Client 11.2.0.2.0
Oracle Multimedia Locator 11.2.0.2.0
Oracle Call Interface (OCI) 11.2.0.2.0
Oracle Multimedia 11.2.0.2.0
Oracle Net 11.2.0.2.0
Oracle XML Development Kit 11.2.0.2.0
Database Configuration and Upgrade Assistants 11.2.0.2.0
Oracle JVM 11.2.0.2.0
Oracle Advanced Security 11.2.0.2.0
Oracle Internet Directory Client 11.2.0.2.0
Oracle Enterprise Manager Console DB 11.2.0.2.0
HAS Files for DB 11.2.0.2.0
Oracle Net Listener 11.2.0.2.0
Oracle Text 11.2.0.2.0
Oracle Net Services 11.2.0.2.0
Oracle Database 11g 11.2.0.2.0
Oracle OLAP 11.2.0.2.0
Oracle Spatial 11.2.0.2.0
Oracle Partitioning 11.2.0.2.0
Enterprise Edition Options 11.2.0.2.0
-----------------------------------------------------------------------------


Instantiating scripts for add node (Thursday, February 10, 2011 4:38:33 PM GMT)
.                                                                 1% Done.
Instantiation of add node scripts complete
Copying to remote nodes (Thursday, February 10, 2011 4:38:41 PM GMT)
.......................................................................      96% Done.
Home copied to new nodes

Saving inventory on nodes (Thursday, February 10, 2011 4:46:43 PM GMT)
.                                                               100% Done.
Save inventory complete
WARNING:
The following configuration scripts need to be executed as the "root" user in each cluster node.
/opt/app/oracle/product/11.2.0/db_1/root.sh #On nodes rac5
To execute the configuration scripts:
1. Open a terminal window
2. Log in as "root"
3. Run the scripts in each cluster node

The Cluster Node Addition of /opt/app/oracle/product/11.2.0/db_1 was successful.
Please check '/tmp/silentInstall.log' for more details.
Run root.sh on the new node. This concludes the second phase. Final phase is to add an instance on the new node.

6. Use DBAC to add a instance to the new node via instance management

7. Check addition of the new node with cluvfy
cluvfy comp admprv -o db_config -d $ORACLE_HOME -n rac4,rac5

Verifying administrative privileges

Checking user equivalence...
User equivalence check passed for user "oracle"

Checking administrative privileges...
Check for multiple users with UID value 500 passed
User existence check passed for "oracle"
Group existence check passed for "oinstall"
Membership check for user "oracle" in group "oinstall" [as Primary] passed
Group existence check passed for "dba"
Membership check for user "oracle" in group "dba" passed

Administrative privileges check passed

Verification of administrative privileges was successful.
This concludes the addition of node to 11gR2 RAC.

Related Post
Adding a Node to 11gR1 RAC