Showing posts with label 11g. Show all posts
Showing posts with label 11g. Show all posts

Friday, July 6, 2012

11gR2 Standalone Data Guard (with ASM and Role Separation)

Post list the steps to configure a data guard environment with 11gR2 (11.2.0.3) standalone server using ASM and role separation. Following table list the names and conventions used during the setup.

Item On Primary On Standby
Host namerac1rac2
Database Nameent11g2ent11g2s
Diskgroup NamesDATA
FLASH
APPS
DG_DATA
DG_FLASH
APPS


1. Create the relevant *dump directories in the standby server
cd $ORACLE_BASE/admin
mkdir ent11g2s
cd ent11g2s
mkdir adump  dpdump  hdump  pfile
2. It is assumed primary db is in archive log mode if not put the db in archive log mode. This is required since duplication is done using active database.

3. Enable force logging on the primary database
alter database force logging;
4. Create standby log files for each thread on primary db. These should be same size as the online redo log. There should be at least one more redo log group per thread than the online redo log groups.
SQL> alter database add standby logfile thread 1 size 52428800;
5. Add static listener entries on standby and primary $GI_HOME's listener.ora file. This must be done as grid user. On primary add the following static listener entry to listener.ora
SID_LIST_LISTENER =
(SID_LIST =
        (SID_DESC =
                (GLOBAL_DBNAME = ent11g2.domain.net)
                (SID_NAME = ent11g2)
                (ORACLE_HOME = /opt/app/oracle/product/11.2.0/dbhome_1)
        )
)
On standby
SID_LIST_LISTENER =
(SID_LIST =
        (SID_DESC =
                (GLOBAL_DBNAME = ent11g2s.domain.net)
                (SID_NAME = ent11g2s)
                (ORACLE_HOME = /opt/app/oracle/product/11.2.0/dbhome_1)
        )
)
Restart both primary and standby listeners for static listener entries to take effect. After the listener restart the listener status would show on primary
...
Listening Endpoints Summary...
  (DESCRIPTION=(ADDRESS=(PROTOCOL=ipc)(KEY=EXTPROC1521)))
  (DESCRIPTION=(ADDRESS=(PROTOCOL=tcp)(HOST=rac1)(PORT=1521)))
Services Summary...
Service "+ASM" has 1 instance(s).
  Instance "+ASM", status READY, has 1 handler(s) for this service...
Service "ent11g2.domain.net" has 2 instance(s).
  Instance "ent11g2", status UNKNOWN, has 1 handler(s) for this service...
  Instance "ent11g2", status READY, has 1 handler(s) for this service...
Service "ent11g2XDB.domain.net" has 1 instance(s).
  Instance "ent11g2", status READY, has 1 handler(s) for this service...
The command completed successfully
On standby
...
Listening Endpoints Summary...
  (DESCRIPTION=(ADDRESS=(PROTOCOL=ipc)(KEY=EXTPROC1521)))
  (DESCRIPTION=(ADDRESS=(PROTOCOL=tcp)(HOST=rac2)(PORT=1521)))
Services Summary...
Service "+ASM" has 1 instance(s).
  Instance "+ASM", status READY, has 1 handler(s) for this service...
Service "ent11g2s.domain.net" has 1 instance(s).
  Instance "ent11g2s", status UNKNOWN, has 1 handler(s) for this service...
The command completed successfully
6. As oracle user in ORACLE_HOME'S create tns entries. On primary server's oracle home create a tns entry pointing to the standby instance
ENT11G2STNS =
  (DESCRIPTION =
    (ADDRESS = (PROTOCOL = TCP)(HOST = rac2)(PORT = 1521))
    (CONNECT_DATA =
      (SERVER = DEDICATED)
      (SERVICE_NAME = ent11g2s.domain.net)
    )
  )
On standby's tnsnames.ora file add an entry pointing to the primary instance
ENT11G2TNS =
  (DESCRIPTION =
    (ADDRESS = (PROTOCOL = TCP)(HOST = rac1)(PORT = 1521))
    (CONNECT_DATA =
      (SERVER = DEDICATED)
      (SERVICE_NAME = ent11g2.domain.net)
    )
  )
7. Create a backup pfile from the primary spfile
SQL> create pfile='/home/oracle/pfile.ora' from spfile;
8. Modify the initialization parameters on the primary db. Remote archive dest state is set to defer, enable it before running the standby duplication. LGWR with ASYNC and NOAFFIRM is used as the redo transport this could be changed to suite the protection mode.
alter system set log_archive_config='dg_config=(ent11g2,ent11g2s)' scope=both ;
alter system set log_archive_dest_1='location=use_db_recovery_file_dest valid_for=(all_logfiles,all_roles) db_unique_name=ent11g2' scope=both;
alter system set log_archive_dest_2='service=ENT11G2STNS LGWR ASYNC NOAFFIRM max_failure=10 max_connections=5 reopen=180 valid_for=(online_logfiles,primary_role) db_unique_name=ent11g2s' scope=both;
alter system set log_archive_dest_state_2='defer' scope=both;
alter system set log_archive_dest_state_1='enable' scope=both;
alter system set fal_server='ENT11G2STNS' scope=both;
alter system set fal_client='ENT11G2TNS' scope=both ;
alter system set log_archive_max_processes=10 scope=both;
alter system set db_file_name_convert='+DG_DATA/ent11g2s','+DATA/ent11g2','+APPS/ent11g2s','+APPS/ent11g2' scope=spfile;
alter system set log_file_name_convert='+DG_FLASH/ent11g2s','+FLASH/ent11g2','+DG_DATA/ent11g2s','+DATA/ent11g2' scope=spfile;
alter system set standby_file_management='AUTO' scope=both;
Restart the database server for the changes to take effect
srvctl stop database -d ent11g2
srvctl start database -d ent11g2
9. Copy password to standby and rename with the instance name
scp orapwent11g2 rac2:/opt/app/oracle/product/11.2.0/dbhome_1/dbs/orapwent11g2s
10. Create pfile on standby with db_name parameter only
export ORACLE_SID=ent11g2s
cat initent11g2s.ora
db_name=ent11g2s
10. Start the standby instance in nomount mode
SQL> startup nomount
ORACLE instance started.

Total System Global Area  238034944 bytes
Fixed Size                  2227136 bytes
Variable Size             180356160 bytes
Database Buffers           50331648 bytes
Redo Buffers                5120000 bytes
11. Enable the remote log archive destination on primary
alter system set log_archive_dest_state_2='enable' SCOPE=both sid='*';
12. Run the duplicate from active database command from primary. Local listner is set automatically and will be reset during duplication process.
duplicate target database for standby from active database
spfile
parameter_value_convert 'ent11g2','ent11g2s','ENT11G2','ENT11G2S'
set db_unique_name='ent11g2s'
set db_file_name_convert='+DATA/ent11g2','+DG_DATA/ent11g2s','+APPS/ent11g2','+APPS/ent11g2s'
set log_file_name_convert='+FLASH/ent11g2','+DG_FLASH/ent11g2s','+DATA/ent11g2','+DG_DATA/ent11g2s'
set control_files='+DG_DATA','+DG_FLASH'
set db_create_file_dest='+DG_DATA'
set db_recovery_file_dest='+DG_FLASH'
set log_archive_max_processes='10'
set fal_client='ENT11G2STNS'
set fal_server='ENT11G2TNS'
reset local_listener
set log_archive_dest_2='service=ENT11G2TNS LGWR ASYNC NOAFFIRM max_failure=10 max_connections=5 reopen=180 valid_for=(online_logfiles,primary_role) db_unique_name=ent11g2'
set log_archive_dest_1='location=use_db_recovery_file_dest valid_for=(all_logfiles,all_roles) db_unique_name=ent11g2s';
The complete output of the duplication process is given below
rman target / auxiliary sys/ent11g2db@ent11g2stns

Recovery Manager: Release 11.2.0.3.0 - Production on Wed Jul 4 16:56:36 2012

Copyright (c) 1982, 2011, Oracle and/or its affiliates.  All rights reserved.

connected to target database: ENT11G2 (DBID=2557133120)
connected to auxiliary database: ENT11G2S (not mounted)

RMAN> duplicate target database for standby from active database
2> spfile
3> parameter_value_convert 'ent11g2','ent11g2s','ENT11G2','ENT11G2S'
4> set db_unique_name='ent11g2s'
5> set db_file_name_convert='+DATA/ent11g2','+DG_DATA/ent11g2s','+APPS/ent11g2','+APPS/ent11g2s'
6> set log_file_name_convert='+FLASH/ent11g2','+DG_FLASH/ent11g2s','+DATA/ent11g2','+DG_DATA/ent11g2s'
7> set control_files='+DG_DATA','+DG_FLASH'
8> set db_create_file_dest='+DG_DATA'
9> set db_recovery_file_dest='+DG_FLASH'
10> set log_archive_max_processes='10'
11> set fal_client='ENT11G2STNS'
12> set fal_server='ENT11G2TNS'
13> reset local_listener
14> set log_archive_dest_2='service=ENT11G2TNS LGWR ASYNC NOAFFIRM max_failure=10 max_connections=5 reopen=180 valid_for=(online_logfiles,primary_role) db_unique_name=ent11g2'
15> set log_archive_dest_1='location=use_db_recovery_file_dest valid_for=(all_logfiles,all_roles) db_unique_name=ent11g2s';

Starting Duplicate Db at 04-JUL-12
using target database control file instead of recovery catalog
allocated channel: ORA_AUX_DISK_1
channel ORA_AUX_DISK_1: SID=10 device type=DISK

contents of Memory Script:
{
   backup as copy reuse
   targetfile  '/opt/app/oracle/product/11.2.0/dbhome_1/dbs/orapwent11g2' auxiliary format
 '/opt/app/oracle/product/11.2.0/dbhome_1/dbs/orapwent11g2s'   targetfile
 '+DATA/ent11g2/spfileent11g2.ora' auxiliary format
 '/opt/app/oracle/product/11.2.0/dbhome_1/dbs/spfileent11g2s.ora'   ;
   sql clone "alter system set spfile= ''/opt/app/oracle/product/11.2.0/dbhome_1/dbs/spfileent11g2s.ora''";
}
executing Memory Script

Starting backup at 04-JUL-12
allocated channel: ORA_DISK_1
channel ORA_DISK_1: SID=18 device type=DISK
Finished backup at 04-JUL-12

sql statement: alter system set spfile= ''/opt/app/oracle/product/11.2.0/dbhome_1/dbs/spfileent11g2s.ora''

contents of Memory Script:
{
   sql clone "alter system set  audit_file_dest =
 ''/opt/app/oracle/admin/ent11g2s/adump'' comment=
 '''' scope=spfile";
   sql clone "alter system set  dispatchers =
 ''(PROTOCOL=TCP) (SERVICE=ent11g2sXDB)'' comment=
 '''' scope=spfile";
   sql clone "alter system set  db_unique_name =
 ''ent11g2s'' comment=
 '''' scope=spfile";
   sql clone "alter system set  db_file_name_convert =
 ''+DATA/ent11g2'', ''+DG_DATA/ent11g2s'', ''+APPS/ent11g2'', ''+APPS/ent11g2s'' comment=
 '''' scope=spfile";
   sql clone "alter system set  log_file_name_convert =
 ''+FLASH/ent11g2'', ''+DG_FLASH/ent11g2s'', ''+DATA/ent11g2'', ''+DG_DATA/ent11g2s'' comment=
 '''' scope=spfile";
   sql clone "alter system set  control_files =
 ''+DG_DATA'', ''+DG_FLASH'' comment=
 '''' scope=spfile";
   sql clone "alter system set  db_create_file_dest =
 ''+DG_DATA'' comment=
 '''' scope=spfile";
   sql clone "alter system set  db_recovery_file_dest =
 ''+DG_FLASH'' comment=
 '''' scope=spfile";
   sql clone "alter system set  log_archive_max_processes =
 10 comment=
 '''' scope=spfile";
   sql clone "alter system set  fal_client =
 ''ENT11G2STNS'' comment=
 '''' scope=spfile";
   sql clone "alter system set  fal_server =
 ''ENT11G2TNS'' comment=
 '''' scope=spfile";
   sql clone "alter system set  log_archive_dest_2 =
 ''service=ENT11G2TNS LGWR ASYNC NOAFFIRM max_failure=10 max_connections=5 reopen=180 valid_for=(online_logfiles,primary_role) db_unique_name=ent11g2'' comment=
 '''' scope=spfile";
   sql clone "alter system set  log_archive_dest_1 =
 ''location=use_db_recovery_file_dest valid_for=(all_logfiles,all_roles) db_unique_name=ent11g2s'' comment=
 '''' scope=spfile";
   shutdown clone immediate;
   startup clone nomount;
}
executing Memory Script

sql statement: alter system set  audit_file_dest =  ''/opt/app/oracle/admin/ent11g2s/adump'' comment= '''' scope=spfile

sql statement: alter system set  dispatchers =  ''(PROTOCOL=TCP) (SERVICE=ent11g2sXDB)'' comment= '''' scope=spfile

sql statement: alter system set  db_unique_name =  ''ent11g2s'' comment= '''' scope=spfile

sql statement: alter system set  db_file_name_convert =  ''+DATA/ent11g2'', ''+DG_DATA/ent11g2s'', ''+APPS/ent11g2'', ''+APPS/ent11g2s'' comment= '''' scope=spfile

sql statement: alter system set  log_file_name_convert =  ''+FLASH/ent11g2'', ''+DG_FLASH/ent11g2s'', ''+DATA/ent11g2'', ''+DG_DATA/ent11g2s'' comment= '''' scope=spfile

sql statement: alter system set  control_files =  ''+DG_DATA'', ''+DG_FLASH'' comment= '''' scope=spfile

sql statement: alter system set  db_create_file_dest =  ''+DG_DATA'' comment= '''' scope=spfile

sql statement: alter system set  db_recovery_file_dest =  ''+DG_FLASH'' comment= '''' scope=spfile

sql statement: alter system set  log_archive_max_processes =  10 comment= '''' scope=spfile

sql statement: alter system set  fal_client =  ''ENT11G2STNS'' comment= '''' scope=spfile

sql statement: alter system set  fal_server =  ''ENT11G2TNS'' comment= '''' scope=spfile

sql statement: alter system set  log_archive_dest_2 =  ''service=ENT11G2TNS LGWR ASYNC NOAFFIRM max_failure=10 max_connections=5 reopen=180 valid_for=(online_logfiles,primary_role) db_unique_name=ent11g2'' comment= '''' scope=spfile

sql statement: alter system set  log_archive_dest_1 =  ''location=use_db_recovery_file_dest valid_for=(all_logfiles,all_roles) db_unique_name=ent11g2s'' comment= '''' scope=spfile

Oracle instance shut down

connected to auxiliary database (not started)
Oracle instance started

Total System Global Area     839282688 bytes

Fixed Size                     2233000 bytes
Variable Size                230690136 bytes
Database Buffers             599785472 bytes
Redo Buffers                   6574080 bytes

contents of Memory Script:
{
   sql clone "alter system set  control_files =
  ''+DG_DATA/ent11g2s/controlfile/current.268.787763727'', ''+DG_FLASH/ent11g2s/controlfile/current.259.787763731'' comment=
 ''Set by RMAN'' scope=spfile";
   backup as copy current controlfile for standby auxiliary format  '+DG_DATA/ent11g2s/controlfile/current.267.787763731';
   restore clone controlfile to  '+DG_FLASH/ent11g2s/controlfile/current.258.787763731' from
 '+DG_DATA/ent11g2s/controlfile/current.267.787763731';
   sql clone "alter system set  control_files =
  ''+DG_DATA/ent11g2s/controlfile/current.267.787763731'', ''+DG_FLASH/ent11g2s/controlfile/current.258.787763731'' comment=
 ''Set by RMAN'' scope=spfile";
   shutdown clone immediate;
   startup clone nomount;
}
executing Memory Script

sql statement: alter system set  control_files =   ''+DG_DATA/ent11g2s/controlfile/current.268.787763727'', ''+DG_FLASH/ent11g2s/controlfile/current.259.787763731'' comment= ''Set by RMAN'' scope=spfile

Starting backup at 04-JUL-12
using channel ORA_DISK_1
channel ORA_DISK_1: starting datafile copy
copying standby control file
output file name=/opt/app/oracle/product/11.2.0/dbhome_1/dbs/snapcf_ent11g2.f tag=TAG20120704T143626 RECID=5 STAMP=787761386
channel ORA_DISK_1: datafile copy complete, elapsed time: 00:00:03
Finished backup at 04-JUL-12

Starting restore at 04-JUL-12
allocated channel: ORA_AUX_DISK_1
channel ORA_AUX_DISK_1: SID=138 device type=DISK

channel ORA_AUX_DISK_1: copied control file copy
Finished restore at 04-JUL-12

sql statement: alter system set  control_files =   ''+DG_DATA/ent11g2s/controlfile/current.267.787763731'', ''+DG_FLASH/ent11g2s/controlfile/current.258.787763731'' comment= ''Set by RMAN'' scope=spfile

Oracle instance shut down

connected to auxiliary database (not started)
Oracle instance started

Total System Global Area     839282688 bytes

Fixed Size                     2233000 bytes
Variable Size                230690136 bytes
Database Buffers             599785472 bytes
Redo Buffers                   6574080 bytes

contents of Memory Script:
{
   sql clone 'alter database mount standby database';
}
executing Memory Script

sql statement: alter database mount standby database
RMAN-05529: WARNING: DB_FILE_NAME_CONVERT resulted in invalid ASM names; names changed to disk group only.

contents of Memory Script:
{
   set newname for tempfile  1 to
 "+dg_data";
   switch clone tempfile all;
   set newname for datafile  1 to
 "+dg_data";
   set newname for datafile  2 to
 "+dg_data";
   set newname for datafile  3 to
 "+dg_data";
   set newname for datafile  4 to
 "+dg_data";
   set newname for datafile  5 to
 "+apps";
   backup as copy reuse
   datafile  1 auxiliary format
 "+dg_data"   datafile
 2 auxiliary format
 "+dg_data"   datafile
 3 auxiliary format
 "+dg_data"   datafile
 4 auxiliary format
 "+dg_data"   datafile
 5 auxiliary format
 "+apps"   ;
   sql 'alter system archive log current';
}
executing Memory Script

executing command: SET NEWNAME

renamed tempfile 1 to +dg_data in control file

executing command: SET NEWNAME

executing command: SET NEWNAME

executing command: SET NEWNAME

executing command: SET NEWNAME

executing command: SET NEWNAME

Starting backup at 04-JUL-12
using channel ORA_DISK_1
channel ORA_DISK_1: starting datafile copy
input datafile file number=00001 name=+DATA/ent11g2/datafile/system.256.787750113
output file name=+DG_DATA/ent11g2s/datafile/system.260.787763751 tag=TAG20120704T143647
channel ORA_DISK_1: datafile copy complete, elapsed time: 00:01:15
channel ORA_DISK_1: starting datafile copy
input datafile file number=00002 name=+DATA/ent11g2/datafile/sysaux.257.787750113
output file name=+DG_DATA/ent11g2s/datafile/sysaux.261.787763827 tag=TAG20120704T143647
channel ORA_DISK_1: datafile copy complete, elapsed time: 00:00:55
channel ORA_DISK_1: starting datafile copy
input datafile file number=00003 name=+DATA/ent11g2/datafile/undotbs1.258.787750113
output file name=+DG_DATA/ent11g2s/datafile/undotbs1.262.787763883 tag=TAG20120704T143647
channel ORA_DISK_1: datafile copy complete, elapsed time: 00:00:15
channel ORA_DISK_1: starting datafile copy
input datafile file number=00005 name=+APPS/ent11g2/datafile/myapps.256.787760733
output file name=+APPS/ent11g2s/datafile/myapps.256.787763897 tag=TAG20120704T143647
channel ORA_DISK_1: datafile copy complete, elapsed time: 00:00:07
channel ORA_DISK_1: starting datafile copy
input datafile file number=00004 name=+DATA/ent11g2/datafile/users.259.787750113
output file name=+DG_DATA/ent11g2s/datafile/users.263.787763905 tag=TAG20120704T143647
channel ORA_DISK_1: datafile copy complete, elapsed time: 00:00:07
Finished backup at 04-JUL-12

sql statement: alter system archive log current

contents of Memory Script:
{
   switch clone datafile all;
}
executing Memory Script

datafile 1 switched to datafile copy
input datafile copy RECID=5 STAMP=787763913 file name=+DG_DATA/ent11g2s/datafile/system.260.787763751
datafile 2 switched to datafile copy
input datafile copy RECID=6 STAMP=787763913 file name=+DG_DATA/ent11g2s/datafile/sysaux.261.787763827
datafile 3 switched to datafile copy
input datafile copy RECID=7 STAMP=787763913 file name=+DG_DATA/ent11g2s/datafile/undotbs1.262.787763883
datafile 4 switched to datafile copy
input datafile copy RECID=8 STAMP=787763913 file name=+DG_DATA/ent11g2s/datafile/users.263.787763905
datafile 5 switched to datafile copy
input datafile copy RECID=9 STAMP=787763913 file name=+APPS/ent11g2s/datafile/myapps.256.787763897
Finished Duplicate Db at 04-JUL-12
13. Once the duplication for standby is completed start the log apply process on standby
SQL> alter database recover managed standby database using current logfile disconnect;
and monitor the standby instance's alert log for recover process in action
alter database recover managed standby database using current logfile disconnect
Attempt to start background Managed Standby Recovery process (ent11g2s)
Wed Jul 04 17:07:05 2012
MRP0 started with pid=48, OS id=5459
MRP0: Background Managed Standby Recovery process started (ent11g2s)
 started logmerger process
Wed Jul 04 17:07:11 2012
Managed Standby Recovery starting Real Time Apply
Parallel Media Recovery started with 2 slaves
Waiting for all non-current ORLs to be archived...
All non-current ORLs have been archived.
Media Recovery Log +DG_FLASH/ent11g2s/archivelog/2012_07_04/thread_1_seq_36.265.787770401
Media Recovery Log +DG_FLASH/ent11g2s/archivelog/2012_07_04/thread_1_seq_37.264.787770403
Media Recovery Log +DG_FLASH/ent11g2s/archivelog/2012_07_04/thread_1_seq_38.256.787770405
Media Recovery Waiting for thread 1 sequence 39 (in transit)
Recovery of Online Redo Log: Thread 1 Group 4 Seq 39 Reading mem 0
  Mem# 0: +DG_DATA/ent11g2s/onlinelog/group_4.258.787770345
  Mem# 1: +DG_FLASH/ent11g2s/onlinelog/group_4.278.787770345
Completed: alter database recover managed standby database using current logfile disconnect
Do a log switch on primary and see archive log application on standby
QL> select sequence#,thread#,applied from v$archived_log;

 SEQUENCE#    THREAD# APPLIED
---------- ---------- ---------
        36          1 YES
        37          1 YES
        38          1 IN-MEMORY
14. This conclude the data guard configuration. Next step is to bring the standby instance under oracle restart (high availability service) similar to primary db
srvctl config database -d ent11g2
Database unique name: ent11g2
Database name: ent11g2
Oracle home: /opt/app/oracle/product/11.2.0/dbhome_1
Oracle user: oracle
Spfile: +DATA/ent11g2/spfileent11g2.ora
Domain: domain.net
Start options: open
Stop options: immediate
Database role: PRIMARY
Management policy: AUTOMATIC
Database instance: ent11g2
Disk Groups: DATA,FLASH,APPS
Services:
Create a pfile from current spfile used by standby. At the end of the duplication standby's spfile is in the $ORACLE_HOME/dbs, this could be confirmed from the alert log which will have the following entry
Using parameter settings in server-side spfile /opt/app/oracle/product/11.2.0/dbhome_1/dbs/spfileent11g2s.ora
Create the pfile from spfile
SQL> create pfile='/home/oracle/pfileinitent11g2s.ora' from spfile;

File created.
Stop the log apply and shutdown the instance
SQL> alter database recover managed standby database cancel;

Database altered.

SQL> shutdown immediate;
Edit the pfile and remove any references to primary instance. The final pfile would only have standby entries
ent11g2s.__db_cache_size=599785472
ent11g2s.__java_pool_size=4194304
ent11g2s.__large_pool_size=4194304
ent11g2s.__pga_aggregate_target=419430400
ent11g2s.__sga_target=843055104
ent11g2s.__shared_io_pool_size=0
ent11g2s.__shared_pool_size=222298112
ent11g2s.__streams_pool_size=0
*.audit_file_dest='/opt/app/oracle/admin/ent11g2s/adump'
*.audit_trail='db'
*.compatible='11.2.0.0.0'
*.control_files='+DG_DATA/ent11g2s/controlfile/current.274.787770165','+DG_FLASH/ent11g2s/controlfile/current.282.787770167'#Set by RMAN
*.db_block_size=8192
*.db_create_file_dest='+DG_DATA'
*.db_domain='domain.net'
*.db_file_name_convert='+DATA/ent11g2','+DG_DATA/ent11g2s','+APPS/ent11g2','+APPS/ent11g2s'
*.db_name='ent11g2'
*.db_recovery_file_dest_size=9437184000
*.db_recovery_file_dest='+DG_FLASH'
*.db_unique_name='ent11g2s'
*.diagnostic_dest='/opt/app/oracle'
*.dispatchers='(PROTOCOL=TCP) (SERVICE=ent11g2sXDB)'
*.fal_client='ENT11G2STNS'
*.fal_server='ENT11G2TNS'
*.log_archive_config='dg_config=(ent11g2,ent11g2s)'
*.log_archive_dest_1='location=use_db_recovery_file_dest valid_for=(all_logfiles,all_roles) db_unique_name=ent11g2s'
*.log_archive_dest_2='service=ENT11G2TNS LGWR ASYNC NOAFFIRM max_failure=10 max_connections=5 reopen=180 valid_for=(online_logfiles,primary_role) db_u
nique_name=ent11g2'
*.log_archive_dest_state_1='enable'
*.log_archive_dest_state_2='enable'
*.log_archive_format='%t_%s_%r.dbf'
*.log_archive_max_processes=10
*.log_file_name_convert='+FLASH/ent11g2','+DG_FLASH/ent11g2s','+DATA/ent11g2','+DG_DATA/ent11g2s'
*.open_cursors=300
*.pga_aggregate_target=419430400
*.processes=150
*.remote_login_passwordfile='EXCLUSIVE'
*.sga_target=840957952
*.standby_file_management='AUTO'
*.undo_tablespace='UNDOTBS1'
Start the standby instance with the modified pfile
SQL> startup mount pfile='/home/oracle/pfileinitent11g2s.ora';
Create a spfile from the pfile give diskgroup as the spfile location
SQL> create spfile='+DG_DATA/ent11g2s/spfileent11g2s.ora' from pfile='/home/oracle/pfileinitent11g2s.ora';

SQL> shutdown immediate;
in $ORACLE_HOME/dbs create new pfile with following line
vi initent11g2s.ora
spfile='+DG_DATA/ent11g2s/spfileent11g2s.ora'
Remove the existing spfile created part of the duplication. Bring the standby instance under the control of oracle restart by running the following command as the oracle user. Startup mode is set to mount to prevent opening in read only mode which requires active data guard license.
$GI_HOME/bin/srvctl add database -d ent11g2s -o $ORACLE_HOME -m local -p "+DG_DATA/ent11g2s/spfileent11g2s.ora" -r physical_standby -s mount
If the database was running when the above command was executed then stop it using sqlplus and then start with
srvctl start database -d ent11g2s

srvctl config database -d ent11g2s
Database unique name: ent11g2s
Database name:
Oracle home: /opt/app/oracle/product/11.2.0/dbhome_1
Oracle user: oracle
Spfile: +DG_DATA/ent11g2s/spfileent11g2s.ora
Domain: domain.net
Start options: mount
Stop options: immediate
Database role: PHYSICAL_STANDBY
Management policy: AUTOMATIC
Database instance: ent11g2s
Disk Groups: DG_DATA,DG_FLASH,APPS
Services:
This will establish all the dependencies of the database and following lines will disappear from the alert log. Before instance is brought under the oracle restart
SUCCESS: diskgroup DG_FLASH was mounted
ERROR: failed to establish dependency between database ent11g2s and diskgroup resource ora.DG_DATA.dg
ERROR: failed to establish dependency between database ent11g2s and diskgroup resource ora.DG_FLASH.dg
ARCH: STARTING ARCH PROCESSES
After
ALTER DATABASE MOUNT /* db agent *//* {0:0:307} */
NOTE: Loaded library: /opt/oracle/extapi/64/asm/orcl/1/libasm.so
NOTE: Loaded library: System
SUCCESS: diskgroup DG_DATA was mounted
SUCCESS: diskgroup DG_FLASH was mounted
ARCH: STARTING ARCH PROCESSES
Wed Jul 04 17:27:58 2012
ARC0 started with pid=26, OS id=6206
ARC0: Archival started
This conclude the configuration and setup of the standalone role separated ASM data guard configuration.



Alternately a pfile for the standby could be created with all the necessary parameter entries and use that to start the duplicate in nomount state. Then when running the duplicate command and omit the spfile clause. In this scenario pfile created on step 10 would have all the necessary parameters, this could be created by modifying a pfile created from the primary.
more initent11g2s.ora
*.audit_file_dest='/opt/app/oracle/admin/ent11g2s/adump'
*.audit_trail='db'
*.compatible='11.2.0.0.0'
*.control_files='+DG_DATA','+DG_FLASH'
*.db_block_size=8192
*.db_create_file_dest='+DG_DATA'
*.db_domain='domain.net'
*.db_file_name_convert='+DATA/ent11g2','+DG_DATA/ent11g2s','+APPS/ent11g2','+APPS/ent11g2s'
*.db_name='ent11g2'
*.db_unique_name='ent11g2s'
*.db_recovery_file_dest='+DG_FLASH'
*.db_recovery_file_dest_size=9437184000
*.diagnostic_dest='/opt/app/oracle'
*.dispatchers='(PROTOCOL=TCP) (SERVICE=ent11g2sXDB)'
*.fal_client='ENT11G2STNS'
*.fal_server='ENT11G2TNS'
*.log_archive_config='dg_config=(ent11g2,ent11g2s)'
*.log_archive_dest_1='location=use_db_recovery_file_dest valid_for=(all_logfiles,all_roles) db_unique_name=ent11g2s'
*.log_archive_dest_2='service=ENT11G2TNS LGWR ASYNC NOAFFIRM max_failure=10 max_connections=5 reopen=180 valid_for=(online_logfiles,primary_role) db_u
nique_name=ent11g2'
*.log_archive_dest_state_1='enable'
*.log_archive_dest_state_2='enable'
*.log_archive_format='%t_%s_%r.dbf'
*.log_archive_max_processes=10
*.log_file_name_convert='+FLASH/ent11g2','+DG_FLASH/ent11g2s','+DATA/ent11g2','+DG_DATA/ent11g2s'
*.open_cursors=300
*.pga_aggregate_target=419430400
*.processes=150
*.remote_login_passwordfile='EXCLUSIVE'
*.sga_target=840957952
*.standby_file_management='AUTO'
*.undo_tablespace='UNDOTBS1'
Then execute the duplicate command as follows. In this case the auxiliary database name will be same as the primary name unlike in the method mentioned above (step 12)
rman target / auxiliary sys/ent11g2db@ent11g2stns

Recovery Manager: Release 11.2.0.3.0 - Production on Wed Jul 4 15:26:35 2012

Copyright (c) 1982, 2011, Oracle and/or its affiliates.  All rights reserved.

connected to target database: ENT11G2 (DBID=2557133120)
connected to auxiliary database: ENT11G2 (not mounted)

RMAN> duplicate target database for standby from active database;

Starting Duplicate Db at 04-JUL-12
using target database control file instead of recovery catalog
allocated channel: ORA_AUX_DISK_1
channel ORA_AUX_DISK_1: SID=12 device type=DISK

contents of Memory Script:
{
   backup as copy reuse
   targetfile  '/opt/app/oracle/product/11.2.0/dbhome_1/dbs/orapwent11g2' auxiliary format
 '/opt/app/oracle/product/11.2.0/dbhome_1/dbs/orapwent11g2s'   ;
}
executing Memory Script

Starting backup at 04-JUL-12
allocated channel: ORA_DISK_1
channel ORA_DISK_1: SID=151 device type=DISK
Finished backup at 04-JUL-12

contents of Memory Script:
{
   backup as copy current controlfile for standby auxiliary format  '+DG_DATA/ent11g2s/controlfile/current.267.787766873';
   restore clone controlfile to  '+DG_FLASH/ent11g2s/controlfile/current.258.787766875' from
 '+DG_DATA/ent11g2s/controlfile/current.267.787766873';
   sql clone "create spfile from memory";
   shutdown clone immediate;
   startup clone nomount;
   sql clone "alter system set  control_files =
  ''+DG_DATA/ent11g2s/controlfile/current.267.787766873'', ''+DG_FLASH/ent11g2s/controlfile/current.258.787766875'' comment=
 ''Set by RMAN'' scope=spfile";
   shutdown clone immediate;
   startup clone nomount;
}
executing Memory Script

Starting backup at 04-JUL-12
using channel ORA_DISK_1
channel ORA_DISK_1: starting datafile copy
copying standby control file
output file name=/opt/app/oracle/product/11.2.0/dbhome_1/dbs/snapcf_ent11g2.f tag=TAG20120704T152853 RECID=6 STAMP=787764533
channel ORA_DISK_1: datafile copy complete, elapsed time: 00:00:07
Finished backup at 04-JUL-12

Starting restore at 04-JUL-12
using channel ORA_AUX_DISK_1

channel ORA_AUX_DISK_1: copied control file copy
Finished restore at 04-JUL-12

sql statement: create spfile from memory

Oracle instance shut down

connected to auxiliary database (not started)
Oracle instance started

Total System Global Area     839282688 bytes

Fixed Size                     2233000 bytes
Variable Size                234884440 bytes
Database Buffers             595591168 bytes
Redo Buffers                   6574080 bytes

sql statement: alter system set  control_files =   ''+DG_DATA/ent11g2s/controlfile/current.267.787766873'', ''+DG_FLASH/ent11g2s/controlfile/current.258.787766875'' comment= ''Set by RMAN'' scope=spfile

Oracle instance shut down

connected to auxiliary database (not started)
Oracle instance started

Total System Global Area     839282688 bytes

Fixed Size                     2233000 bytes
Variable Size                234884440 bytes
Database Buffers             595591168 bytes
Redo Buffers                   6574080 bytes

contents of Memory Script:
{
   sql clone 'alter database mount standby database';
}
executing Memory Script

sql statement: alter database mount standby database
RMAN-05529: WARNING: DB_FILE_NAME_CONVERT resulted in invalid ASM names; names changed to disk group only.

contents of Memory Script:
{
   set newname for tempfile  1 to
 "+dg_data";
   switch clone tempfile all;
   set newname for datafile  1 to
 "+dg_data";
   set newname for datafile  2 to
 "+dg_data";
   set newname for datafile  3 to
 "+dg_data";
   set newname for datafile  4 to
 "+dg_data";
   set newname for datafile  5 to
 "+apps";
   backup as copy reuse
   datafile  1 auxiliary format
 "+dg_data"   datafile
 2 auxiliary format
 "+dg_data"   datafile
 3 auxiliary format
 "+dg_data"   datafile
 4 auxiliary format
 "+dg_data"   datafile
 5 auxiliary format
 "+apps"   ;
   sql 'alter system archive log current';
}
executing Memory Script

executing command: SET NEWNAME

renamed tempfile 1 to +dg_data in control file

executing command: SET NEWNAME

executing command: SET NEWNAME

executing command: SET NEWNAME

executing command: SET NEWNAME

executing command: SET NEWNAME

Starting backup at 04-JUL-12
using channel ORA_DISK_1
channel ORA_DISK_1: starting datafile copy
input datafile file number=00001 name=+DATA/ent11g2/datafile/system.256.787750113
output file name=+DG_DATA/ent11g2s/datafile/system.263.787766913 tag=TAG20120704T152928
channel ORA_DISK_1: datafile copy complete, elapsed time: 00:01:15
channel ORA_DISK_1: starting datafile copy
input datafile file number=00002 name=+DATA/ent11g2/datafile/sysaux.257.787750113
output file name=+DG_DATA/ent11g2s/datafile/sysaux.262.787766989 tag=TAG20120704T152928
channel ORA_DISK_1: datafile copy complete, elapsed time: 00:00:55
channel ORA_DISK_1: starting datafile copy
input datafile file number=00003 name=+DATA/ent11g2/datafile/undotbs1.258.787750113
output file name=+DG_DATA/ent11g2s/datafile/undotbs1.261.787767045 tag=TAG20120704T152928
channel ORA_DISK_1: datafile copy complete, elapsed time: 00:00:15
channel ORA_DISK_1: starting datafile copy
input datafile file number=00005 name=+APPS/ent11g2/datafile/myapps.256.787760733
output file name=+APPS/ent11g2s/datafile/myapps.256.787767061 tag=TAG20120704T152928
channel ORA_DISK_1: datafile copy complete, elapsed time: 00:00:03
channel ORA_DISK_1: starting datafile copy
input datafile file number=00004 name=+DATA/ent11g2/datafile/users.259.787750113
output file name=+DG_DATA/ent11g2s/datafile/users.260.787767063 tag=TAG20120704T152928
channel ORA_DISK_1: datafile copy complete, elapsed time: 00:00:03
Finished backup at 04-JUL-12

sql statement: alter system archive log current

contents of Memory Script:
{
   switch clone datafile all;
}
executing Memory Script

datafile 1 switched to datafile copy
input datafile copy RECID=6 STAMP=787767066 file name=+DG_DATA/ent11g2s/datafile/system.263.787766913
datafile 2 switched to datafile copy
input datafile copy RECID=7 STAMP=787767066 file name=+DG_DATA/ent11g2s/datafile/sysaux.262.787766989
datafile 3 switched to datafile copy
input datafile copy RECID=8 STAMP=787767066 file name=+DG_DATA/ent11g2s/datafile/undotbs1.261.787767045
datafile 4 switched to datafile copy
input datafile copy RECID=9 STAMP=787767066 file name=+DG_DATA/ent11g2s/datafile/users.260.787767063
datafile 5 switched to datafile copy
input datafile copy RECID=10 STAMP=787767066 file name=+APPS/ent11g2s/datafile/myapps.256.787767061
Finished Duplicate Db at 04-JUL-12
Reset of the steps are same as above to bring the standby under oracle restart's control.

Some other related notes
When the oracle software is installed on the standby node the permission of the Oracle binary is as follows.
ls -l /opt/app/oracle/product/11.2.0/grid/bin/oracle # GI HOME
-rwsr-s--x 1 grid oinstall 203972955 Jul  4 10:05 /opt/app/oracle/product/11.2.0/grid/bin/oracle
ls -l /opt/app/oracle/product/11.2.0/dbhome_1/bin/oracle # ORACLE HOME
-rwsr-s--x 1 oracle oinstall 232399473 Jul  4 12:29 /opt/app/oracle/product/11.2.0/dbhome_1/bin/oracle
See the post related to oracle binary permissions. On primary the permission of the oracle binary was "oracle:asmadmin". The permission was left as it is (oracle:oinstall) and this did not cause any problem during the standby creation and at the end of the duplication the oracle binary permission was changed to "oracle:asmadmin"
ls -l /opt/app/oracle/product/11.2.0/dbhome_1/bin/oracle
-rwsr-s--x 1 oracle asmadmin 232399473 Jul  4 12:29 /opt/app/oracle/product/11.2.0/dbhome_1/bin/oracle
s -l /opt/app/oracle/product/11.2.0/grid/bin/oracle
-rwsr-s--x 1 grid oinstall 203972955 Jul  4 10:05 /opt/app/oracle/product/11.2.0/grid/bin/oracle
It was confirmed oracle binary permissions are in a smilar state after the installation and that when the database is created with dbca the permission gets changed to "oracle:asmadmin".

It is normal to see message similar to below in the standby alert log when the duplication process and this does not mean a problem
Errors in file /opt/app/oracle/diag/rdbms/ent11g2s/ent11g2s/trace/ent11g2s_lgwr_27303.trc:
ORA-00313: open failed for members of log group 1 of thread 1
ORA-00312: online log 1 thread 1: '+DG_FLASH/ent11g2s/onlinelog/group_1.257.787750219'
ORA-17503: ksfdopn:2 Failed to open file +DG_FLASH/ent11g2s/onlinelog/group_1.257.787750219
ORA-15012: ASM file '+DG_FLASH/ent11g2s/onlinelog/group_1.257.787750219' does not exist
ORA-00312: online log 1 thread 1: '+DG_DATA/ent11g2s/onlinelog/group_1.261.787750215'
ORA-17503: ksfdopn:2 Failed to open file +DG_DATA/ent11g2s/onlinelog/group_1.261.787750215
ORA-15012: ASM file '+DG_DATA/ent11g2s/onlinelog/group_1.261.787750215' does not exist
Errors in file /opt/app/oracle/diag/rdbms/ent11g2s/ent11g2s/trace/ent11g2s_lgwr_27303.trc:
ORA-00313: open failed for members of log group 1 of thread 1
ORA-00312: online log 1 thread 1: '+DG_FLASH/ent11g2s/onlinelog/group_1.257.787750219'
ORA-17503: ksfdopn:2 Failed to open file +DG_FLASH/ent11g2s/onlinelog/group_1.257.787750219
ORA-15012: ASM file '+DG_FLASH/ent11g2s/onlinelog/group_1.257.787750219' does not exist
ORA-00312: online log 1 thread 1: '+DG_DATA/ent11g2s/onlinelog/group_1.261.787750215'
ORA-17503: ksfdopn:2 Failed to open file +DG_DATA/ent11g2s/onlinelog/group_1.261.787750215
ORA-15012: ASM file '+DG_DATA/ent11g2s/onlinelog/group_1.261.787750215' does not exist
The alert logs will be created without an issue.

Useful metalink notes
RMAN Duplicate Database From RAC ASM To RAC ASM [ID 461479.1]
Performing duplicate database with ASM/OMF/RMAN [ID 340848.1]
ORA-7445 [kcrf_val_log_bsz] Creating Physical Standby With Rman Duplicate From Active [ID 1320757.1]
Dataguard DB/LOG FILE NAME CONVERT has been set but files are created in a different directory [ID 1348512.1]
OERR:RMAN 5529 WARNING: DB_FILE_NAME_CONVERT resulted in invalid ASM names; names changed to disk group only [ID 1367794.1]

Related Posts
RAC to Single Instance Physical Standby
11gR2 RAC to RAC Data Guard

Friday, June 22, 2012

Upgrading ASMLib and OS in 11gR1 RAC Environment

Upgrading operating system (OS) requires several consideration and tasks.
1. If ASMLib is used then ASMLib libraries must be upgraded as part of the OS upgrade.
2. After the upgrade Oracle/clusterware binaries should be relinked.
In this case the two nodes 11gR1 (11.1.0.7) RAC environment running on RHEL 5 (2.6.18-194.el5) will be upgraded to RHEL 5 (2.6.18-308.el5).
As per (743649.1) Oralce support rolling upgrade of OS when both OS are certified on the database that's running. This is supported only for the duration of the upgrade. But according to (1391807.1) using multiple versions of ASMLib across the cluster may not be compatible. To avoid these incompatible issues and to minimize the total downtime one node could be upgraded first and then shutting down the other node before the upgraded one with new version of ASMLib is started. This way only one node will be active in the cluster when two nodes are having different ASMLib versions. In this case all nodes are upgraded at the same time incurring total system outage.

1. Find out if ASMLibs are available for the kernel version to which the system is being upgraded to. If not move asmlib devices to block devices before the upgrade (also true if upgrading to RHEL 6 where to get asmlibs require ULN account) or see if (462618.1) could be of any help.

2. Make a note of the ownership and permission of the following files in $CRS_HOME/bin directory using the script below. Some files might be missing this is not a problem.
#!/bin/sh

if [ "$CRS_HOME" != "" ]; then
                echo "crs home exist"

        for i in clntsh.map clntst_1.lis clntst_2.lis clntst.lis libclntsh.so libclntsh.so.10.1 libclntst10.so libclntsh.so.11.1 libclntst11.so clntst
        do
                if [ -f $CRS_HOME/lib/$i ]; then
                        echo "file exists"
                        ls -l $CRS_HOME/lib/$i

                        if [ -f $CRS_HOME/lib32/$i ]; then
                                echo "file in lib32"
                                ls -l $CRS_HOME/lib32/$i
                        else
                                echo "file not exists"
                        fi
                else
                        if [ -f $CRS_HOME/lib32/$i ]; then
                                echo "file in lib32"
                                ls -l $CRS_HOME/lib32/$i
                        else
                                echo "file not exists"
                        fi
                fi
        done
else
                echo "Please set CRS_HOME and run again"
fi

./cluster_file_check.sh
crs home exist
file exists
-rwxr--r-- 1 root root 4765562 Feb  8  2011 /opt/crs/oracle/product/11.1.0/crs/lib/clntsh.map
file in lib32
-rw-r--r-- 1 root root 3985432 Feb  8  2011 /opt/crs/oracle/product/11.1.0/crs/lib32/clntsh.map
file not exists
file not exists
file not exists
file exists
lrwxrwxrwx 1 root root 17 Feb  8  2011 /opt/crs/oracle/product/11.1.0/crs/lib/libclntsh.so -> libclntsh.so.11.1
file in lib32
lrwxrwxrwx 1 root root 17 Feb  8  2011 /opt/crs/oracle/product/11.1.0/crs/lib32/libclntsh.so -> libclntsh.so.11.1
file exists
lrwxrwxrwx 1 oracle oinstall 51 Feb  8  2011 /opt/crs/oracle/product/11.1.0/crs/lib/libclntsh.so.10.1 -> /opt/crs/oracle/product/11.1.0/crs/lib/libclntsh.so
file in lib32
lrwxrwxrwx 1 oracle oinstall 53 Feb  8  2011 /opt/crs/oracle/product/11.1.0/crs/lib32/libclntsh.so.10.1 -> /opt/crs/oracle/product/11.1.0/crs/lib32/libclntsh.so
file not exists
file exists
-rwxr-xr-x 1 root root 48316147 Feb  8  2011 /opt/crs/oracle/product/11.1.0/crs/lib/libclntsh.so.11.1
file in lib32
-rwxr-xr-x 1 root root 37081093 Feb  8  2011 /opt/crs/oracle/product/11.1.0/crs/lib32/libclntsh.so.11.1
file not exists
file not exists
3. Make a note of the asm disk's scsi_ids , this setup is a test setup using Virtualbox. The scsi_ids will be compared after the OS upgrade and must be the same as before the upgrade.
# blkid|grep sd.*oracleasm|while read a b;do echo -n $a$b" scsi_id=";(echo $a|tr -d [:digit:]|tr -d [:]|cut -d"/" -f3|xargs -i scsi_id -g -s /block/{})done;
/dev/sdc1:LABEL="DATA" TYPE="oracleasm" scsi_id=SATA     VBOX HARDDISK  VB35fd6dbb-14d87b49
/dev/sdd1:LABEL="FLASH" TYPE="oracleasm" scsi_id=SATA     VBOX HARDDISK  VB4b88ed44-eff71eb1
4. Shutdown the cluster stack across the cluster. If one node at a time is upgraded then do this only on one node. In this case since all nodes are upgraded at the same time entire cluster is brought down
crs_stop -all
5. Stop the crs and disbale auto start of the crs. Do this on all ndoes.
# crsctl stop crs
Stopping resources.
This could take several minutes.
Successfully stopped Oracle Clusterware resources
Stopping Cluster Synchronization Services.
Shutting down the Cluster Synchronization Services daemon.
Shutdown request successfully issued.

# crsctl disable crs
Oracle Clusterware is disabled for start-up after a reboot.
6. Comment out the crs spwaning processes entries in /etc/inittab. Do this on all nodes
#h1:35:respawn:/etc/init.d/init.evmd run >/dev/null 2>&1 </dev/null
#h2:35:respawn:/etc/init.d/init.cssd fatal >/dev/null 2>&1 </dev/null
#h3:35:respawn:/etc/init.d/init.crsd run >/dev/null 2>&1 </dev/null
Run init on the systme and verify that no crs related processes are running.
/sbin/init q

ps -ef|grep css
ps -ef|grep crs
ps -ef|grep evm
ps -ef|grep init
7. Stop oracleasm running and disable and prevent it loading on reboot. Do this on all nodes.
# /etc/init.d/oracleasm stop
Dropping Oracle ASMLib disks:                              [  OK  ]
Shutting down the Oracle ASMLib driver:                    [  OK  ]

# /etc/init.d/oracleasm disable
Writing Oracle ASM library driver configuration: done
Dropping Oracle ASMLib disks:                              [  OK  ]
Shutting down the Oracle ASMLib driver:                    [  OK  ]

# /sbin/chkconfig --list | grep oracleasm
oracleasm       0:off   1:off   2:on    3:on    4:on    5:on    6:off
# /sbin/chkconfig oracleasm off
# /sbin/chkconfig --list | grep oracleasm
oracleasm       0:off   1:off   2:off   3:off   4:off   5:off   6:off
8. Make a backup of /etc/sysconfig/oracleasm-_dev_oracleasm on all nodes.
# cp /etc/sysconfig/oracleasm-_dev_oracleasm /etc/sysconfig/oracleasm-_dev_oracleasm.bak
At this point the system is ready for the OS upgrade.

9. Carry out the OS ugprade. All Oracle Home locations (ORACLE_HOME,ASM_HOME and CRS_HOME) are to be same before and after the upgrade





10. Install the new version of oracleasm and remove the old version. Old version could only be removed after installing the new version due to dependencies.
rpm -e oracleasm-2.6.18-194.el5-2.0.5-1.el5
error: Failed dependencies:
        oracleasm >= 1.0.4 is needed by (installed) oracleasmlib-2.0.4-1.el5.x86_64

# rpm -ivh oracleasm-2.6.18-308.el5-2.0.5-1.el5.x86_64.rpm
warning: oracleasm-2.6.18-308.el5-2.0.5-1.el5.x86_64.rpm: Header V3 DSA signature: NOKEY, key ID 1e5e0159
Preparing...                ########################################### [100%]
   1:oracleasm-2.6.18-308.el########################################### [100%]

# rpm -e oracleasm-2.6.18-194.el5-2.0.5-1.el5

# rpm -qa | grep oracleasm
oracleasmlib-2.0.4-1.el5
oracleasm-2.6.18-308.el5-2.0.5-1.el5
oracleasm-support-2.1.3-1.el5
11. Configure oracleasm and make sure content on oracleasm-_dev_oracleasm.bak and oracleasm-_dev_oracleasm are the same.
/etc/init.d/oracleasm configure
Configuring the Oracle ASM library driver.

This will configure the on-boot properties of the Oracle ASM library
driver.  The following questions will determine whether the driver is
loaded on boot and what permissions it will have.  The current values
will be shown in brackets ('[]').  Hitting  without typing an
answer will keep that current value.  Ctrl-C will abort.

Default user to own the driver interface [oracle]:
Default group to own the driver interface [dba]:
Start Oracle ASM library driver on boot (y/n) [n]: y
Scan for Oracle ASM disks on boot (y/n) [y]:
Writing Oracle ASM library driver configuration: done
Initializing the Oracle ASMLib driver:                     [  OK  ]
Scanning the system for Oracle ASMLib disks:               [  OK  ]
12. Enable oracleasm and set it up to start on reboot.
/etc/init.d/oracleasm enable
Writing Oracle ASM library driver configuration: done
Initializing the Oracle ASMLib driver:                     [  OK  ]
Scanning the system for Oracle ASMLib disks:               [  OK  ]

chkconfig --list | grep oracleasm
oracleasm       0:off   1:off   2:off   3:off   4:off   5:off   6:off

# chkconfig oracleasm on

# chkconfig --list | grep oracleasm
oracleasm       0:off   1:off   2:on    3:on    4:on    5:on    6:off
13. Verify scsi_id are same as before the upgrade
# blkid|grep sd.*oracleasm|while read a b;do echo -n $a$b" scsi_id=";(echo $a|tr -d [:digit:]|tr -d [:]|cut -d"/" -f3|xargs -i scsi_id -g -s /block/{})done;
/dev/sdc1:LABEL="DATA" TYPE="oracleasm" scsi_id=SATA     VBOX HARDDISK  VB35fd6dbb-14d87b49
/dev/sdd1:LABEL="FLASH" TYPE="oracleasm" scsi_id=SATA     VBOX HARDDISK  VB4b88ed44-eff71eb1
With this the ASMLib upgrade is complete. Next is to relink the Oracle binaries.

14. As oracle user relink both ASM_HOME and ORACLE_HOME. Verify which relink is in the path variable using
which relink
Relink ORACLE_HOME
$ cd $ORACLE_HOME/bin
$ relink all
Reset the PATH and ORACLE_HOME before running relink for ASM_HOME
export PATH=$ASM_HOME/bin:$PATH
export ORACLE_HOME=$ASM_HOME
Verify the relink binary from ASM_HOME is in path
which relink
Run the relink of ASM_HOME
$ cd $ASM_HOME/bin
$ relink all
15. As per (743649.1 step 10) clusterware client shared libraries should be relinked as the clusterware software owner. But running this as software owner result in permission issue
export ORACLE_HOME=$CRS_HOME (important to set ORACLE_HOME variable to CRS_HOME location)
cd $ORACLE_HOME/bin

./genclntsh
/bin/rm: cannot remove `/opt/crs/oracle/product/11.1.0/crs/lib/libclntsh.so.11.1': Permission denied
/bin/rm: cannot remove `/opt/crs/oracle/product/11.1.0/crs/lib/clntsh.map': Permission denied
./genclntsh: line 309: /opt/crs/oracle/product/11.1.0/crs/lib/clntsh.map: Permission denied
genclntsh: Failed to link libclntsh.so.11.1
Permission issue persist even after setting the permission to oracle:oinstall. Running ./genclntsh as root results in files get created with new timestamp permission same as before (metalink note could be wrong!??). When genclntsh runs without any issue no message is shown and return to shell prompt. After run check the file permissions are same as before (step 2 above).
./cluster_file_check.sh
crs home exist
file exists
-rw-r--r-- 1 root root 4764652 Jun 20 16:49 /opt/crs/oracle/product/11.1.0/crs/lib/clntsh.map
file in lib32
-rw-r--r-- 1 root root 3984490 Jun 20 16:49 /opt/crs/oracle/product/11.1.0/crs/lib32/clntsh.map
file not exists
file not exists
file not exists
file exists
lrwxrwxrwx 1 root root 17 Jun 20 16:49 /opt/crs/oracle/product/11.1.0/crs/lib/libclntsh.so -> libclntsh.so.11.1
file in lib32
lrwxrwxrwx 1 root root 17 Jun 20 16:49 /opt/crs/oracle/product/11.1.0/crs/lib32/libclntsh.so -> libclntsh.so.11.1
file exists
lrwxrwxrwx 1 oracle oinstall 51 Feb  8  2011 /opt/crs/oracle/product/11.1.0/crs/lib/libclntsh.so.10.1 -> /opt/crs/oracle/product/11.1.0/crs/lib/libclntsh.so
file in lib32
lrwxrwxrwx 1 oracle oinstall 53 Feb  8  2011 /opt/crs/oracle/product/11.1.0/crs/lib32/libclntsh.so.10.1 -> /opt/crs/oracle/product/11.1.0/crs/lib32/libclntsh.so
file not exists
file exists
-rwxr-xr-x 1 root root 48316147 Jun 20 16:49 /opt/crs/oracle/product/11.1.0/crs/lib/libclntsh.so.11.1
file in lib32
-rwxr-xr-x 1 root root 37081093 Jun 20 16:49 /opt/crs/oracle/product/11.1.0/crs/lib32/libclntsh.so.11.1
file not exists
file not exists
16. Enable crs and remove the comments on /etc/inittab and start crs
# crsctl enable crs
Oracle Clusterware is enabled for start-up after a reboot.

h1:35:respawn:/etc/init.d/init.evmd run >/dev/null 2>&1 </dev/null
h2:35:respawn:/etc/init.d/init.cssd fatal >/dev/null 2>&1 </dev/null
h3:35:respawn:/etc/init.d/init.crsd run >/dev/null 2>&1 </dev/null
17. Start crs with crsctl or with OS init
crsctl start crs
Attempting to start Oracle Clusterware stack
The CRS stack will be started shortly
or
/sbin/init q
This conclude the ASMLib and OS upgrade.

Useful Metalink Notes
How to Relink Oracle Database Software on UNIX [ID 131321.1]
Is It Necessary To Relink Oracle Following OS Upgrade? [ID 444595.1]
Will an Operating System Upgrade Affect Oracle Clusterware? [ID 743649.1]
How To Upgrade ASMLib Kernel Driver as Part of Kernel Upgrade? [ID 1391807.1]
Do You Need to Relink Oracle Clusterware When Upgrading the Operating System? [ID 743649.1]
Is It Required To Relink The Oracle Binaries When Booting With The Old Kernel? [ID 461138.1]
Cannot Find Exact Kernel Version Match For ASMLib (Workaround using oracleasm_debug_link tool) [ID 462618.1]
RAC: Frequently Asked Questions [ID 220970.1] (Is a relink required for the clusterware home after an OS upgrade?)

Useful White Paper
Best Practices for Optimizing Availability During Planned Maintenance Using Oracle Clusterware and Oracle Real Application Clusters. Oracle Maximum Availability Architecture White Paper September 2007

Related Posts
Upgrading RHEL 6 OS in a 11gR2 RAC Environment
Upgrading OS in 11gR2 RAC Environment (RHEL 5)

Moving ASMLib disk to block devices (Non-ASMLib) in 11gR1 RAC

This post list steps specific to moving ASMLib disk to block devices in two nodes 11gR1 RAC environment. There's an earlier post which shows steps involved with moving ASMLib to block devices in a 11gR2 standalone setup. Most of the steps are similar as the earlier post but in the RAC environment the moving from ASMLib to block device could be done in a rolling fashion reducing down time.

1. Create a pfile from the asm spfile. This is more for protection (to start ASM if there's a mistake in asm_string and can't get the ASM up with spfile) and is not part of the moving process.
SQL> create pfile='/home/oracle/asmpfile.ora' from spfile;
File created.

more asmpfile.ora
+ASM1.__oracle_base='/opt/app/oracle'#ORACLE_BASE set from environment
+ASM2.__oracle_base='/opt/app/oracle'#ORACLE_BASE set from environment
+ASM1.asm_diskgroups='DATA','FLASH'
+ASM2.asm_diskgroups='DATA','FLASH'
*.cluster_database=true
*.diagnostic_dest='/opt/app/oracle'
+ASM2.instance_number=2
+ASM1.instance_number=1
*.instance_type='asm'
*.large_pool_size=12M
2. Current ASM disks
SQL> select name,label,path from v$asm_disk;

NAME       LABEL      PATH
---------- ---------- ----------
DATA       DATA       ORCL:DATA
FLASH      FLASH      ORCL:FLASH
3. It is possible to dismount diskgroup in one node and test the procedure before applying the changes permanently.
SQL> alter diskgroup flash dismount
From ASM alert log
SQL> alter diskgroup flash dismount
NOTE: cache dismounting group 2/0x73784638 (FLASH)
Fri Jun 22 13:29:49 2012
kjbdomdet send to node 1
detach from dom 2, sending detach message to node 1
Fri Jun 22 13:29:49 2012
NOTE: detached from domain 2
NOTE: cache dismounted group 2/0x73784638 (FLASH)
kfdp_dismount(): 9
Fri Jun 22 13:29:49 2012
kfdp_dismountBg(): 9
NOTE: De-assigning number (2,0) from disk (ORCL:FLASH)
SUCCESS: diskgroup FLASH was dismounted
SUCCESS: alter diskgroup flash dismount
Status of ASM disks accross cluster which shows FLASH disgroup in node 1 dismounted
SQL> select inst_id,name,state from gv$asm_diskgroup; 

   INST_ID NAME      STATE
    ------- --------- -----------
         2 DATA      MOUNTED
         2 FLASH     MOUNTED
         1 DATA      MOUNTED
         1 FLASH     DISMOUNTED
4. Query the ASMLib disk and find out the block device used for creating ASMLib disk
# /etc/init.d/oracleasm querydisk -d FLASH
Disk "FLASH" is a valid ASM disk on device [8, 49]

ls -l /dev/sd*
...
brw-r----- 1 root   disk     8, 32 Jun 20 17:16 /dev/sdc
brw-r----- 1 root   disk     8, 33 Jun 20 17:17 /dev/sdc1
brw-r----- 1 root   disk     8, 48 Jun 20 17:16 /dev/sdd
brw-r----- 1 root   disk     8, 49 Jun 20 17:17 /dev/sdd1
Change the ownership of the block device
chown oracle:dba /dev/sdd1

ls -l /dev/sd*
...
brw-r----- 1 root   disk     8, 32 Jun 20 17:16 /dev/sdc
brw-r----- 1 root   disk     8, 33 Jun 20 17:17 /dev/sdc1
brw-r----- 1 root   disk     8, 48 Jun 20 17:16 /dev/sdd
brw-r----- 1 oracle dba      8, 49 Jun 20 17:17 /dev/sdd1
5. Change the asm_diskstring to include the block device and mount the flash diskgroup
SQL> alter system set asm_diskstring ='ORCL:DATA','/dev/sdd1' scope=memory;
System altered.
Mount the FLASH diskgroup and verify block device is used for ASM diskgroup
SQL> alter diskgroup flash mount

NOTE: cache registered group FLASH number=2 incarn=0x73784644
NOTE: cache began mount (not first) of group FLASH number=2 incarn=0x73784644
NOTE: Assigning number (2,0) to disk (/dev/sdd1)
kfdp_query(FLASH): 11
Fri Jun 22 13:35:46 2012
kfdp_queryBg(): 11
NOTE: cache opening disk 0 of grp 2: FLASH path:/dev/sdd1
NOTE: F1X0 found on disk 0 fcn 0.0
NOTE: cache mounting (not first) group 2/0x73784644 (FLASH)
Fri Jun 22 13:35:46 2012
kjbdomatt send to node 1
From the data views
SQL> select inst_id,name,label,path from gv$asm_disk;

   INST_ID NAME       LABEL      PATH
---------- ---------- ---------- ----------
         1 DATA       DATA       ORCL:DATA
         1 FLASH                 /dev/sdd1
         2 DATA       DATA       ORCL:DATA
         2 FLASH      FLASH      ORCL:FLASH
6. Do the same with the other diskgroup. Query the disk to find out the block device and change the ownership of the block device so oracle process could write to it directly
/etc/init.d/oracleasm querydisk -d data
Disk "DATA" is a valid ASM disk on device [8, 33]

# chown oracle:dba /dev/sdc1

ls /dev/sd*
...
brw-r----- 1 root   disk     8, 32 Jun 20 17:16 /dev/sdc
brw-r----- 1 oracle dba      8, 33 Jun 20 17:17 /dev/sdc1
brw-r----- 1 root   disk     8, 48 Jun 20 17:16 /dev/sdd
brw-r----- 1 oracle dba      8, 49 Jun 22 13:39 /dev/sdd1
7. Unmount all the diskgroups in the node and set the asm_diskstring. Trying to set asm_diskstring while one is mounted would result in an error. See earlier post's step 9
SQL> alter diskgroup flash dismount;
Diskgroup altered.

SQL> alter diskgroup data dismount;
Diskgroup altered.

SQL> alter system set asm_diskstring ='/dev/sdc1','/dev/sdd1' scope=memory;
System altered.

SQL> alter diskgroup data mount;
Diskgroup altered.

SQL> alter diskgroup flash mount;
Diskgroup altered.
From ASM alert log
NOTE: cache registered group DATA number=1 incarn=0x7378464e
NOTE: cache began mount (not first) of group DATA number=1 incarn=0x7378464e
NOTE: Assigning number (1,0) to disk (/dev/sdc1)
kfdp_query(DATA): 16
Fri Jun 22 13:41:12 2012
kfdp_queryBg(): 16
NOTE: cache opening disk 0 of grp 1: DATA path:/dev/sdc1
NOTE: F1X0 found on disk 0 fcn 0.0
NOTE: cache mounting (not first) group 1/0x7378464E (DATA)
Fri Jun 22 13:41:12 2012
kjbdomatt send to node 1
...
...
SQL> alter diskgroup flash mount
NOTE: cache registered group FLASH number=2 incarn=0x73784652
NOTE: cache began mount (not first) of group FLASH number=2 incarn=0x73784652
NOTE: Assigning number (2,0) to disk (/dev/sdd1)
kfdp_query(FLASH): 19
kfdp_queryBg(): 19
NOTE: cache opening disk 0 of grp 2: FLASH path:/dev/sdd1
NOTE: F1X0 found on disk 0 fcn 0.0
NOTE: cache mounting (not first) group 2/0x73784652 (FLASH)
kjbdomatt send to node 1
Confirm first node's ASM diskgroups are now using block devices
SQL> select inst_id,name,label,path from gv$asm_disk;

   INST_ID NAME       LABEL      PATH
---------- ---------- ---------- ----------
         2 DATA       DATA       ORCL:DATA
         2 FLASH      FLASH      ORCL:FLASH
         1 DATA                  /dev/sdc1
         1 FLASH                 /dev/sdd1
8. Create udev rules so the block device ownerships are persistent across reboots.
# ASM DATA
KERNEL=="sdc[1]", OWNER="oracle", GROUP="dba", MODE="660"

# ASM FLASH
KERNEL=="sdd[1]", OWNER="oracle", GROUP="dba", MODE="660"
9. Set the new asm_diskstring values to ASM spfile
SQL> alter system set asm_diskstring ='/dev/sdc1','/dev/sdd1' scope=both;
10. Unload the oracleasm module. To unload the oracleasm module the ASM instance must be shutdown.
lsmod  | grep oracleasm
oracleasm              84136  1

# /etc/init.d/oracleasm stop
Dropping Oracle ASMLib disks:                              [  OK  ]
Shutting down the Oracle ASMLib driver:                    [FAILED]

srvctl stop asm -n rac1

# /etc/init.d/oracleasm stop
Dropping Oracle ASMLib disks:                              [  OK  ]
Shutting down the Oracle ASMLib driver:                    [  OK  ]

# /etc/init.d/oracleasm disable
Writing Oracle ASM library driver configuration: done
Dropping Oracle ASMLib disks:                              [  OK  ]
Shutting down the Oracle ASMLib driver:                    [  OK  ]

# chkconfig oracleasm off
11. Start the ASM instance on the node
srvctl start asm -n rac1
Verify the block devices are used by monitoring the ASM alert log
NOTE: cache opening disk 0 of grp 1: DATA path:/dev/sdc1
NOTE: F1X0 found on disk 0 fcn 0.0
NOTE: cache mounting (not first) group 1/0x737E1A2E (DATA)
kjbdomatt send to node 1
NOTE: attached to recovery domain 1
NOTE: LGWR attempting to mount thread 1 for diskgroup 1
NOTE: LGWR mounted thread 1 for disk group 1
NOTE: opening chunk 1 at fcn 0.69396 ABA
NOTE: seq=102 blk=4866
NOTE: cache mounting group 1/0x737E1A2E (DATA) succeeded
NOTE: cache ending mount (success) of group DATA number=1 incarn=0x737e1a2e
kfdp_query(FLASH): 4
kfdp_queryBg(): 4
NOTE: cache opening disk 0 of grp 2: FLASH path:/dev/sdd1
NOTE: F1X0 found on disk 0 fcn 0.0
NOTE: cache mounting (not first) group 2/0x737E1A2F (FLASH)
Start the database instance on the node

12. Next step is to mount the asm diskgroup on the remaining node with the block devices. Stop the database instance running on the node and change the ownership of the block devices. These were identified earlier
# chown oracle:dba /dev/sdc1
# chown oracle:dba /dev/sdd1
13. Since asm_diskstring was set as both no need to make any changes simply dismount and mount one diskgroup to test the configuration
SQL> alter diskgroup flash dismount;

Diskgroup altered.

SQL> alter diskgroup flash mount;

Diskgroup altered.
14. Create a udev rules file in this node as well
# ASM DATA
KERNEL=="sdc[1]", OWNER="oracle", GROUP="dba", MODE="660"

# ASM FLASH
KERNEL=="sdd[1]", OWNER="oracle", GROUP="dba", MODE="660"
15. Stop the asm instance and unload the oracleasm module
srvctl stop asm -n rac2

# /etc/init.d/oracleasm stop
Dropping Oracle ASMLib disks:                              [  OK  ]
Shutting down the Oracle ASMLib driver:                    [  OK  ]

# /etc/init.d/oracleasm disable
Writing Oracle ASM library driver configuration: done
Dropping Oracle ASMLib disks:                              [  OK  ]
Shutting down the Oracle ASMLib driver:                    [  OK  ]

#chkconfig oracleasm off
15. Start asm instance and verify block devices being used through ASM alert log
srvctl start asm -n rac2

NOTE: cache opening disk 0 of grp 1: DATA path:/dev/sdc1
NOTE: F1X0 found on disk 0 fcn 0.0
NOTE: cache mounting (not first) group 1/0x73701F4C (DATA)
kjbdomatt send to node 0
NOTE: attached to recovery domain 1
NOTE: LGWR attempting to mount thread 2 for diskgroup 1
NOTE: LGWR mounted thread 2 for disk group 1
NOTE: opening chunk 2 at fcn 0.69424 ABA
NOTE: seq=86 blk=4843
NOTE: cache mounting group 1/0x73701F4C (DATA) succeeded
NOTE: cache ending mount (success) of group DATA number=1 incarn=0x73701f4c
kfdp_query(FLASH): 4
kfdp_queryBg(): 4
NOTE: cache opening disk 0 of grp 2: FLASH path:/dev/sdd1
NOTE: F1X0 found on disk 0 fcn 0.0
NOTE: cache mounting (not first) group 2/0x73701F4D (FLASH)
Querying the data view
SQL> select inst_id,name,label,path from gv$asm_disk order by 1;

   INST_ID NAME       LABEL      PATH
---------- ---------- ---------- ----------
         1 DATA                  /dev/sdc1
         1 FLASH                 /dev/sdd1
         2 DATA                  /dev/sdc1
         2 FLASH                 /dev/sdd1
16. As the last step uninstall the oracleasm libraries
# rpm -e oracleasmlib-2.0.4-1.el5
# rpm -e oracleasm-2.6.18-308.el5-2.0.5-1.el5
# rpm -e oracleasm-support-2.1.3-1.el5
Related Posts
Migrating block devices using ASM instance to ASMLib
Moving ASMLib disk to block devices (Non-ASMLib) in 11gR2 Standalone

Useful Metalink Notes
How To Migrate ASMLIB Devices To Block Devices (Non-ASMLIB)? [ID 567508.1]

Tuesday, June 19, 2012

Moving ASMLib disk to block devices (Non-ASMLib) in 11gR2 Standalone

This post list steps to move ASMLib devices to block devices for a standalone 11gR2 installation with role separation created earlier.
1. As the grid user login to ASM instance and create a pfile from the ASM spfile
sqlplus  / as sysasm
SQL> create pfile='/home/grid/asmpfile.ora' from spfile;

File created.

$ more asmpfile.ora
+ASM.__oracle_base='/opt/app/oracle'#ORACLE_BASE set from in memory value
+ASM.asm_diskgroups='FLASH','DATA'#Manual Mount
*.asm_power_limit=1
*.diagnostic_dest='/opt/app/oracle'
*.instance_type='asm'
*.large_pool_size=12M
*.remote_login_passwordfile='EXCLUSIVE'
2. Current ASMLib disks are
SQL> select name,label,path from v$asm_disk;

NAME    LABEL   PATH
------- ------- -----------
DATA    DATA    ORCL:DATA
FLASH   FLASH   ORCL:FLASH
3. Before ASMLib disks could be moved the database must be stopped. This is required as ASM diskgroups will be dismounted later on.
srvctl stop database -d stdalone
4. Dismount and mount the flash diskgroup and monitor the ASM alert log which will show the ASM disk(s) used with the diskgroup. This information is useful to verify if the new block device disk assignment is working.
SQL> alter diskgroup flash dismount;

Diskgroup altered.
From the alert log
NOTE: cache deleting context for group FLASH 2/0x9328489f
Thu Jun 14 11:57:49 2012
NOTE: diskgroup resource ora.FLASH.dg is offline
SUCCESS: alter diskgroup flash dismount
NOTE: diskgroup resource ora.FLASH.dg is updated
When mounting
SQL> alter diskgroup flash mount
From alert log
NOTE: cache registered group FLASH number=2 incarn=0xed0848a6
NOTE: cache began mount (first) of group FLASH number=2 incarn=0xed0848a6
NOTE: Assigning number (2,0) to disk (ORCL:FLASH)
Thu Jun 14 11:58:18 2012
NOTE: GMON heartbeating for grp 2
5. Identify the block device used to create the ASMLib disk by querying the ASMLib disk
# /etc/init.d/oracleasm querydisk -d FLASH
Disk "FLASH" is a valid ASM disk on device [8, 33]
Block device could be identified with
ls -l /dev/sd*
brw-r----- 1 root disk 8,  0 Jun 14 11:47 /dev/sda
brw-r----- 1 root disk 8,  1 Jun 14 11:48 /dev/sda1
brw-r----- 1 root disk 8,  2 Jun 14 11:47 /dev/sda2
brw-r----- 1 root disk 8,  3 Jun 14 11:48 /dev/sda3
brw-r----- 1 root disk 8, 16 Jun 14 11:47 /dev/sdb
brw-r----- 1 root disk 8, 17 Jun 14 11:48 /dev/sdb1
brw-r----- 1 root disk 8, 32 Jun 14 11:47 /dev/sdc
brw-r----- 1 root disk 8, 33 Jun 14 11:48 /dev/sdc1
The block device used to create the FLASH ASMLib disk is /dev/sdc1.

6. Set the permission on the block device to grid:asmadmin
# chown grid:asmadmin /dev/sdc1
# ls -l /dev/sdc*
brw-r----- 1 root disk     8, 32 Jun 14 11:47 /dev/sdc
brw-r----- 1 grid asmadmin 8, 33 Jun 14 11:48 /dev/sdc1
7. Dismount the FLASH diskgroup again and set the asm_diskstring to include the block device. Still the DATA diskgroup is using ASMLib therefore ORCL* is also set. This mix setting is alright for testing purposes but it's better not to run production environments in the mix mode (both ASMLib and Block devices).
SQL> alter system set asm_diskstring ='ORCL:DATA','/dev/sdc1' SCOPE=MEMORY;
Mount the FLASH diskgroup and monitor the ASM alert log and verify block device is used
SQL> ALTER DISKGROUP flash mount
On the alert log
NOTE: cache registered group FLASH number=2 incarn=0x3d4848aa
NOTE: cache began mount (first) of group FLASH number=2 incarn=0x3d4848aa
NOTE: Assigning number (2,0) to disk (/dev/sdc1)
Thu Jun 14 12:01:15 2012
NOTE: GMON heartbeating for grp 2
GMON querying group 2 at 18 for pid 19, osid 3268
NOTE: cache opening disk 0 of grp 2: FLASH path:/dev/sdc1
Further verification from the v$asm_disk view
SQL> select name,label,path from v$asm_disk;

NAME    LABEL   PATH
------- ------- -----------
DATA    DATA    ORCL:DATA
FLASH   FLASH   /dev/sdc1
8. Do the same with other diskgroup
# /etc/init.d/oracleasm querydisk -d DATA
Disk "DATA" is a valid ASM disk on device [8, 17]
Set the block device permissions
# chown grid:asmadmin /dev/sdb1
# ls -l /dev/sdb*
brw-r----- 1 root disk     8, 16 Jun 14 11:47 /dev/sdb
brw-r----- 1 grid asmadmin 8, 17 Jun 14 11:48 /dev/sdb1
9. Dismount all diskgroups. Trying to change the asm_diskstring while diskgroups are mounted will throw the following error
SQL> alter system set asm_diskstring ='/dev/sdb1','/dev/sdc1' scope=memory;
alter system set asm_diskstring ='/dev/sdb1','/dev/sdc1' scope=memory
*
ERROR at line 1:
ORA-02097: parameter cannot be modified because specified value is invalid
ORA-15014: path 'ORCL:DATA' is not in the discovery set
Once all diskgroups are dismounted no issue.
SQL> alter diskgroup data dismount;
Diskgroup altered.

SQL> alter diskgroup flash dismount;
Diskgroup altered.

SQL> alter system set asm_diskstring ='/dev/sdb1','/dev/sdc1' scope=both;
Mount each diskgroup and verify the block device is used. Following from the alert log
Thu Jun 14 12:04:51 2012
SQL> alter diskgroup data mount
NOTE: cache registered group DATA number=1 incarn=0xa8c848b4
NOTE: cache began mount (first) of group DATA number=1 incarn=0xa8c848b4
NOTE: Assigning number (1,0) to disk (/dev/sdb1)
Further verification from v$asm_disk
SQL> select name,label,path from v$asm_disk;

NAME    LABEL   PATH
------- ------- -----------
DATA    DATA    /dev/sdb1
FLASH   FLASH   /dev/sdc1
10. Now that it's been verified that block devices are used for ASM create udev rules.
# ASM DATA
KERNEL=="sdb[1]", OWNER="grid", GROUP="asmadmin", MODE="660"

# ASM FLASH
KERNEL=="sdc[1]", OWNER="grid", GROUP="asmadmin", MODE="660"
11. Stop the high availability stack
crsctl stop has
and unload the oracleasm modules
# lsmod  | grep oracleasm
oracleasm              84136  1

# /etc/init.d/oracleasm stop
Dropping Oracle ASMLib disks:                              [  OK  ]
Shutting down the Oracle ASMLib driver:                    [  OK  ]

# lsmod  | grep oracleasm
disable loading of oracleasm when server is rebooted
# chkconfig oracleasm off
Restart the high availability service
crsctl start has
CRS-4123: Oracle High Availability Services has been started.
and check all ASM starting with block devices without an issue. When the restart is complete check the ASM alert log
SQL> ALTER DISKGROUP ALL MOUNT /* asm agent call crs *//* {0:0:2} */
NOTE: Diskgroups listed in ASM_DISKGROUPS are
         DATA
         FLASH
NOTE: cache registered group DATA number=1 incarn=0xa71a3a36
NOTE: cache began mount (first) of group DATA number=1 incarn=0xa71a3a36
NOTE: cache registered group FLASH number=2 incarn=0xa71a3a37
NOTE: cache began mount (first) of group FLASH number=2 incarn=0xa71a3a37
Errors in file /opt/app/oracle/diag/asm/+asm/+ASM/trace/+ASM_rbal_4010.trc:
ORA-15183: ASMLIB initialization error [driver/agent not installed]
WARNING: FAILED to load library: /opt/oracle/extapi/64/asm/orcl/1/libasm.so
NOTE: Assigning number (1,0) to disk (/dev/sdb1)
NOTE: Assigning number (2,0) to disk (/dev/sdc1)
and/or using kfod tool to confirm block devices are being used correctly
kfod asm_diskstring='/dev/sd*' disk=all
--------------------------------------------------------------------------------
 Disk          Size Path                                     User     Group
================================================================================
   1:      10236 Mb /dev/sdb1                                grid     asmadmin
   2:      10236 Mb /dev/sdc1                                grid     asmadmin
--------------------------------------------------------------------------------
ORACLE_SID ORACLE_HOME
================================================================================
      +ASM /opt/app/oracle/product/11.2.0/grid
12. If possible restart the server and check all changes (udev rules, disk permissions) and are working without issue and as the final step uninstall the ASMLib
# rpm -e oracleasmlib-2.0.4-1.el5
# rpm -e oracleasm-2.6.18-194.el5-2.0.5-1.el5
# rpm -e oracleasm-support-2.1.3-1.el5
Related Post
Migrating block devices using ASM instance to ASMLib
Moving ASMLib disk to block devices (Non-ASMLib) in 11gR1 RAC

Useful Metalink Notes
How To Migrate ASMLIB Devices To Block Devices (Non-ASMLIB)? [ID 567508.1]

Thursday, June 14, 2012

Installing 11gR2 (11.2.0.3) RAC with Role Separation on RHEL 6

Installing GI with role separation in RHEL 6 is identical to installing on OEL 6. This blog post is not meant to be an extensive how to install GI on RHEL 6, it will highlight the key aspects relating to RHEL 6 and role separation. There's also a post on installing standalone server with role separation on RHEL 6.
1. According to GI installation guide for RHEL 6 the kernel version must be 2.6.32-71.el6.x86_64 or later or if the RHEL 6 kernel is with unbreakable enterprise linux kernel then it must be 2.6.32-100.28.5.el6.x86_64 or later. In this case the kernel version is
uname -r
2.6.32-220.el6.x86_64
2. Create OS groups and users needed for role separation. Oracle user will own database software and grid user will own the clusterware.
create groups and users
groupadd -g 501 oinstall
groupadd -g 502 dba
groupadd -g 503 oper
groupadd -g 505 asmadmin
groupadd -g 506 asmdba
groupadd -g 507 asmoper

useradd -u 501 -g oinstall -G dba,oper,asmdba oracle
useradd -u 502 -g oinstall -G asmadmin,asmdba,asmoper,dba grid
grid user should be part of dba group as per metalink note 1084186.1

3. Create user equivalence for both grid user and oracle user.
$ /usr/bin/ssh-keygen -t rsa
Generating public/private rsa key pair.
Enter file in which to save the key (/home/grid/.ssh/id_rsa):
Created directory '/home/grid/.ssh'.
Enter passphrase (empty for no passphrase):
Enter same passphrase again:
Your identification has been saved in /home/grid/.ssh/id_rsa.
Your public key has been saved in /home/grid/.ssh/id_rsa.pub.
...
...
4. Create directory structure for clusterware and Oracle base and set with relevant ownership and permission.
# mkdir -p /opt/app/11.2.0/grid
# chown grid:oinstall /opt/app/11.2.0/grid
# chmod 775 /opt/app/11.2.0/grid

oracle base

# mkdir -p /opt/app/oracle
# chown oracle:oinstall /opt/app/oracle
# chmod 775 /opt/app/oracle

oraInventory

# mkdir -p /opt/app/oraInventory
# chown grid:oinstall /opt/app/oraInventory
# chmod 775 /opt/app/oraInventory
5. Verify pre-req package list is installed. Follow GI installation guide and metalink note Installing Oracle 11gR2 RDBMS on RHEL6 or OL6 64-bit (x86-64) [ID 1441282.1]

6.Set resource limit for both oracle and grid user in /etc/security/limits.conf file
oracle soft nproc 2047
oracle hard nproc 16384
oracle soft nofile 1024
oracle hard nofile 65536
oracle soft stack 10240
oracle hard stack 32768

grid soft nproc 2047
grid hard nproc 16384
grid soft nofile 1024
grid hard nofile 65536
grid soft stack 10240
grid hard stack 32768
7. Disable x11 forward for grid and oracle user by adding the following to each user's .ssh/config file
Host *
 ForwardX11 no
8. Disable tty on .bashrc for both grid and oracle user
if [ -t 0 ]; then
 stty intr ^C
fi
and also set umask value in .bashrc for grid and oracle user
umask 022
9. As per metalink note (1089399.1) for RHEL6/OEL6 to get oracleasm libraries it would require a UEL account . For this installation block devices will be used for ASM diskgroups. Setup udev rules for block devices with following ownerships and permissions.
#ASM OCR
KERNEL=="sdb[1]", OWNER="grid", GROUP="asmadmin", MODE="660"
KERNEL=="sdc[1]", OWNER="grid", GROUP="asmadmin", MODE="660"
KERNEL=="sdd[1]", OWNER="grid", GROUP="asmadmin", MODE="660"

# ASM DATA
KERNEL=="sde[1]", OWNER="grid", GROUP="asmadmin", MODE="660"

# ASM FLASH
KERNEL=="sdf[1]", OWNER="grid", GROUP="asmadmin", MODE="660"
If the permissions are incorrect this will be flagged during OUI pre-req check.

10. Edit the cvu_config file and set CV_ASSUME_DISTID=OEL6 (more on 1454982.1 and explained here). This will prevent pdksh package check being flagged as failed when running the installer using OUI. Still running the cluster verification will flag this as failed but this could be ignored.
./runcluvfy.sh stage -pre crsinst -n rhel6m1,rhel6m2  -fixup  -verbose

Performing pre-checks for cluster services setup

Checking node reachability...

Check: Node reachability from node "rhel6m1"
  Destination Node                      Reachable?
  ------------------------------------  ------------------------
  rhel6m1                               yes
  rhel6m2                               yes
Result: Node reachability check passed from node "rhel6m1"


Checking user equivalence...

Check: User equivalence for user "grid"
  Node Name                             Status
  ------------------------------------  ------------------------
  rhel6m2                               passed
  rhel6m1                               passed
Result: User equivalence check passed for user "grid"

Checking node connectivity...

Checking hosts config file...
  Node Name                             Status
  ------------------------------------  ------------------------
  rhel6m2                               passed
  rhel6m1                               passed

Verification of the hosts config file successful


Interface information for node "rhel6m2"
 Name   IP Address      Subnet          Gateway         Def. Gateway    HW Address        MTU
 ------ --------------- --------------- --------------- --------------- ----------------- ------
 eth0   192.168.0.86    192.168.0.0     0.0.0.0         192.168.0.100   08:00:27:8C:E5:85 1500
 eth0   192.168.0.90    192.168.0.0     0.0.0.0         192.168.0.100   08:00:27:8C:E5:85 1500
 eth1   192.168.1.88    192.168.1.0     0.0.0.0         192.168.0.100   08:00:27:E7:2E:BB 1500


Interface information for node "rhel6m1"
 Name   IP Address      Subnet          Gateway         Def. Gateway    HW Address        MTU
 ------ --------------- --------------- --------------- --------------- ----------------- ------
 eth0   192.168.0.85    192.168.0.0     0.0.0.0         192.168.0.100   08:00:27:FF:7A:38 1500
 eth0   192.168.0.89    192.168.0.0     0.0.0.0         192.168.0.100   08:00:27:FF:7A:38 1500
 eth1   192.168.1.87    192.168.1.0     0.0.0.0         192.168.0.100   08:00:27:F7:AB:A0 1500


Check: Node connectivity of subnet "192.168.0.0"
  Source                          Destination                     Connected?
  ------------------------------  ------------------------------  ----------------
  rhel6m2[192.168.0.86]           rhel6m2[192.168.0.90]           yes
  rhel6m2[192.168.0.86]           rhel6m1[192.168.0.85]           yes
  rhel6m2[192.168.0.86]           rhel6m1[192.168.0.89]           yes
  rhel6m2[192.168.0.90]           rhel6m1[192.168.0.85]           yes
  rhel6m2[192.168.0.90]           rhel6m1[192.168.0.89]           yes
  rhel6m1[192.168.0.85]           rhel6m1[192.168.0.89]           yes
Result: Node connectivity passed for subnet "192.168.0.0" with node(s) rhel6m2,rhel6m1


Check: TCP connectivity of subnet "192.168.0.0"
  Source                          Destination                     Connected?
  ------------------------------  ------------------------------  ----------------
  rhel6m1:192.168.0.85            rhel6m2:192.168.0.86            passed
  rhel6m1:192.168.0.85            rhel6m2:192.168.0.90            passed
  rhel6m1:192.168.0.85            rhel6m1:192.168.0.89            passed
Result: TCP connectivity check passed for subnet "192.168.0.0"


Check: Node connectivity of subnet "192.168.1.0"
  Source                          Destination                     Connected?
  ------------------------------  ------------------------------  ----------------
  rhel6m2[192.168.1.88]           rhel6m1[192.168.1.87]           yes
Result: Node connectivity passed for subnet "192.168.1.0" with node(s) rhel6m2,rhel6m1


Check: TCP connectivity of subnet "192.168.1.0"
  Source                          Destination                     Connected?
  ------------------------------  ------------------------------  ----------------
  rhel6m1:192.168.1.87            rhel6m2:192.168.1.88            passed
Result: TCP connectivity check passed for subnet "192.168.1.0"


Interfaces found on subnet "192.168.0.0" that are likely candidates for VIP are:
rhel6m2 eth0:192.168.0.86 eth0:192.168.0.90
rhel6m1 eth0:192.168.0.85 eth0:192.168.0.89

Interfaces found on subnet "192.168.1.0" that are likely candidates for a private interconnect are:
rhel6m2 eth1:192.168.1.88
rhel6m1 eth1:192.168.1.87
Checking subnet mask consistency...
Subnet mask consistency check passed for subnet "192.168.0.0".
Subnet mask consistency check passed for subnet "192.168.1.0".
Subnet mask consistency check passed.

Result: Node connectivity check passed

Checking multicast communication...

Checking subnet "192.168.0.0" for multicast communication with multicast group "230.0.1.0"...
Check of subnet "192.168.0.0" for multicast communication with multicast group "230.0.1.0" passed.

Checking subnet "192.168.1.0" for multicast communication with multicast group "230.0.1.0"...
Check of subnet "192.168.1.0" for multicast communication with multicast group "230.0.1.0" passed.

Check of multicast communication passed.

Checking ASMLib configuration.
  Node Name                             Status
  ------------------------------------  ------------------------
  rhel6m2                               passed
  rhel6m1                               passed
Result: Check for ASMLib configuration passed.

Check: Total memory
  Node Name     Available                 Required                  Status
  ------------  ------------------------  ------------------------  ----------
  rhel6m2       1.9597GB (2054884.0KB)    1.5GB (1572864.0KB)       passed
  rhel6m1       1.9597GB (2054884.0KB)    1.5GB (1572864.0KB)       passed
Result: Total memory check passed

Check: Available memory
  Node Name     Available                 Required                  Status
  ------------  ------------------------  ------------------------  ----------
  rhel6m2       1.8421GB (1931556.0KB)    50MB (51200.0KB)          passed
  rhel6m1       1.7056GB (1788460.0KB)    50MB (51200.0KB)          passed
Result: Available memory check passed

Check: Swap space
  Node Name     Available                 Required                  Status
  ------------  ------------------------  ------------------------  ----------
  rhel6m2       4GB (4194296.0KB)         2.9395GB (3082326.0KB)    passed
  rhel6m1       4GB (4194296.0KB)         2.9395GB (3082326.0KB)    passed
Result: Swap space check passed

Check: Free disk space for "rhel6m2:/tmp"
  Path              Node Name     Mount point   Available     Required      Status
  ----------------  ------------  ------------  ------------  ------------  ------------
  /tmp              rhel6m2       /             22.5312GB     1GB           passed
Result: Free disk space check passed for "rhel6m2:/tmp"

Check: Free disk space for "rhel6m1:/tmp"
  Path              Node Name     Mount point   Available     Required      Status
  ----------------  ------------  ------------  ------------  ------------  ------------
  /tmp              rhel6m1       /             21.4365GB     1GB           passed
Result: Free disk space check passed for "rhel6m1:/tmp"

Check: User existence for "grid"
  Node Name     Status                    Comment
  ------------  ------------------------  ------------------------
  rhel6m2       passed                    exists(502)
  rhel6m1       passed                    exists(502)

Checking for multiple users with UID value 502
Result: Check for multiple users with UID value 502 passed
Result: User existence check passed for "grid"

Check: Group existence for "oinstall"
  Node Name     Status                    Comment
  ------------  ------------------------  ------------------------
  rhel6m2       passed                    exists
  rhel6m1       passed                    exists
Result: Group existence check passed for "oinstall"

Check: Group existence for "dba"
  Node Name     Status                    Comment
  ------------  ------------------------  ------------------------
  rhel6m2       passed                    exists
  rhel6m1       passed                    exists
Result: Group existence check passed for "dba"

Check: Membership of user "grid" in group "oinstall" [as Primary]
  Node Name         User Exists   Group Exists  User in Group  Primary       Status
  ----------------  ------------  ------------  ------------  ------------  ------------
  rhel6m2           yes           yes           yes           yes           passed
  rhel6m1           yes           yes           yes           yes           passed
Result: Membership check for user "grid" in group "oinstall" [as Primary] passed

Check: Membership of user "grid" in group "dba"
  Node Name         User Exists   Group Exists  User in Group  Status
  ----------------  ------------  ------------  ------------  ----------------
  rhel6m2           yes           yes           yes           passed
  rhel6m1           yes           yes           yes           passed
Result: Membership check for user "grid" in group "dba" passed

Check: Run level
  Node Name     run level                 Required                  Status
  ------------  ------------------------  ------------------------  ----------
  rhel6m2       3                         3,5                       passed
  rhel6m1       3                         3,5                       passed
Result: Run level check passed

Check: Hard limits for "maximum open file descriptors"
  Node Name         Type          Available     Required      Status
  ----------------  ------------  ------------  ------------  ----------------
  rhel6m2           hard          65536         65536         passed
  rhel6m1           hard          65536         65536         passed
Result: Hard limits check passed for "maximum open file descriptors"

Check: Soft limits for "maximum open file descriptors"
  Node Name         Type          Available     Required      Status
  ----------------  ------------  ------------  ------------  ----------------
  rhel6m2           soft          1024          1024          passed
  rhel6m1           soft          1024          1024          passed
Result: Soft limits check passed for "maximum open file descriptors"

Check: Hard limits for "maximum user processes"
  Node Name         Type          Available     Required      Status
  ----------------  ------------  ------------  ------------  ----------------
  rhel6m2           hard          16384         16384         passed
  rhel6m1           hard          16384         16384         passed
Result: Hard limits check passed for "maximum user processes"

Check: Soft limits for "maximum user processes"
  Node Name         Type          Available     Required      Status
  ----------------  ------------  ------------  ------------  ----------------
  rhel6m2           soft          2047          2047          passed
  rhel6m1           soft          2047          2047          passed
Result: Soft limits check passed for "maximum user processes"

Check: System architecture
  Node Name     Available                 Required                  Status
  ------------  ------------------------  ------------------------  ----------
  rhel6m2       x86_64                    x86_64                    passed
  rhel6m1       x86_64                    x86_64                    passed
Result: System architecture check passed

Check: Kernel version
  Node Name     Available                 Required                  Status
  ------------  ------------------------  ------------------------  ----------
  rhel6m2       2.6.32-220.el6.x86_64     2.6.9                     passed
  rhel6m1       2.6.32-220.el6.x86_64     2.6.9                     passed
Result: Kernel version check passed

Check: Kernel parameter for "semmsl"
  Node Name         Current       Configured    Required      Status        Comment
  ----------------  ------------  ------------  ------------  ------------  ------------
  rhel6m2           3010          3010          250           passed
  rhel6m1           3010          3010          250           passed
Result: Kernel parameter check passed for "semmsl"

Check: Kernel parameter for "semmns"
  Node Name         Current       Configured    Required      Status        Comment
  ----------------  ------------  ------------  ------------  ------------  ------------
  rhel6m2           385280        385280        32000         passed
  rhel6m1           385280        385280        32000         passed
Result: Kernel parameter check passed for "semmns"

Check: Kernel parameter for "semopm"
  Node Name         Current       Configured    Required      Status        Comment
  ----------------  ------------  ------------  ------------  ------------  ------------
  rhel6m2           3010          3010          100           passed
  rhel6m1           3010          3010          100           passed
Result: Kernel parameter check passed for "semopm"

Check: Kernel parameter for "semmni"
  Node Name         Current       Configured    Required      Status        Comment
  ----------------  ------------  ------------  ------------  ------------  ------------
  rhel6m2           128           128           128           passed
  rhel6m1           128           128           128           passed
Result: Kernel parameter check passed for "semmni"

Check: Kernel parameter for "shmmax"
  Node Name         Current       Configured    Required      Status        Comment
  ----------------  ------------  ------------  ------------  ------------  ------------
  rhel6m2           68719476736   68719476736   1052100608    passed
  rhel6m1           68719476736   68719476736   1052100608    passed
Result: Kernel parameter check passed for "shmmax"

Check: Kernel parameter for "shmmni"
  Node Name         Current       Configured    Required      Status        Comment
  ----------------  ------------  ------------  ------------  ------------  ------------
  rhel6m2           4096          4096          4096          passed
  rhel6m1           4096          4096          4096          passed
Result: Kernel parameter check passed for "shmmni"

Check: Kernel parameter for "shmall"
  Node Name         Current       Configured    Required      Status        Comment
  ----------------  ------------  ------------  ------------  ------------  ------------
  rhel6m2           4294967296    4294967296    2097152       passed
  rhel6m1           4294967296    4294967296    2097152       passed
Result: Kernel parameter check passed for "shmall"

Check: Kernel parameter for "file-max"
  Node Name         Current       Configured    Required      Status        Comment
  ----------------  ------------  ------------  ------------  ------------  ------------
  rhel6m2           6815744       6815744       6815744       passed
  rhel6m1           6815744       6815744       6815744       passed
Result: Kernel parameter check passed for "file-max"

Check: Kernel parameter for "ip_local_port_range"
  Node Name         Current       Configured    Required      Status        Comment
  ----------------  ------------  ------------  ------------  ------------  ------------
  rhel6m2           between 9000.0 & 65500.0  between 9000.0 & 65500.0  between 9000.0 & 65500.0  passed
  rhel6m1           between 9000.0 & 65500.0  between 9000.0 & 65500.0  between 9000.0 & 65500.0  passed
Result: Kernel parameter check passed for "ip_local_port_range"

Check: Kernel parameter for "rmem_default"
  Node Name         Current       Configured    Required      Status        Comment
  ----------------  ------------  ------------  ------------  ------------  ------------
  rhel6m2           4194304       4194304       262144        passed
  rhel6m1           4194304       4194304       262144        passed
Result: Kernel parameter check passed for "rmem_default"

Check: Kernel parameter for "rmem_max"
  Node Name         Current       Configured    Required      Status        Comment
  ----------------  ------------  ------------  ------------  ------------  ------------
  rhel6m2           4194304       4194304       4194304       passed
  rhel6m1           4194304       4194304       4194304       passed
Result: Kernel parameter check passed for "rmem_max"

Check: Kernel parameter for "wmem_default"
  Node Name         Current       Configured    Required      Status        Comment
  ----------------  ------------  ------------  ------------  ------------  ------------
  rhel6m2           1048576       1048576       262144        passed
  rhel6m1           1048576       1048576       262144        passed
Result: Kernel parameter check passed for "wmem_default"

Check: Kernel parameter for "wmem_max"
  Node Name         Current       Configured    Required      Status        Comment
  ----------------  ------------  ------------  ------------  ------------  ------------
  rhel6m2           2097152       2097152       1048576       passed
  rhel6m1           2097152       2097152       1048576       passed
Result: Kernel parameter check passed for "wmem_max"

Check: Kernel parameter for "aio-max-nr"
  Node Name         Current       Configured    Required      Status        Comment
  ----------------  ------------  ------------  ------------  ------------  ------------
  rhel6m2           3145728       3145728       1048576       passed
  rhel6m1           3145728       3145728       1048576       passed
Result: Kernel parameter check passed for "aio-max-nr"

Check: Package existence for "make"
  Node Name     Available                 Required                  Status
  ------------  ------------------------  ------------------------  ----------
  rhel6m2       make-3.81-19.el6          make-3.80                 passed
  rhel6m1       make-3.81-19.el6          make-3.80                 passed
Result: Package existence check passed for "make"

Check: Package existence for "binutils"
  Node Name     Available                 Required                  Status
  ------------  ------------------------  ------------------------  ----------
  rhel6m2       binutils-2.20.51.0.2-5.28.el6  binutils-2.15.92.0.2      passed
  rhel6m1       binutils-2.20.51.0.2-5.28.el6  binutils-2.15.92.0.2      passed
Result: Package existence check passed for "binutils"

Check: Package existence for "gcc(x86_64)"
  Node Name     Available                 Required                  Status
  ------------  ------------------------  ------------------------  ----------
  rhel6m2       gcc(x86_64)-4.4.6-3.el6   gcc(x86_64)-3.4.6         passed
  rhel6m1       gcc(x86_64)-4.4.6-3.el6   gcc(x86_64)-3.4.6         passed
Result: Package existence check passed for "gcc(x86_64)"

Check: Package existence for "libaio(x86_64)"
  Node Name     Available                 Required                  Status
  ------------  ------------------------  ------------------------  ----------
  rhel6m2       libaio(x86_64)-0.3.107-10.el6  libaio(x86_64)-0.3.105    passed
  rhel6m1       libaio(x86_64)-0.3.107-10.el6  libaio(x86_64)-0.3.105    passed
Result: Package existence check passed for "libaio(x86_64)"

Check: Package existence for "glibc(x86_64)"
  Node Name     Available                 Required                  Status
  ------------  ------------------------  ------------------------  ----------
  rhel6m2       glibc(x86_64)-2.12-1.47.el6  glibc(x86_64)-2.3.4-2.41  passed
  rhel6m1       glibc(x86_64)-2.12-1.47.el6  glibc(x86_64)-2.3.4-2.41  passed
Result: Package existence check passed for "glibc(x86_64)"

Check: Package existence for "compat-libstdc++-33(x86_64)"
  Node Name     Available                 Required                  Status
  ------------  ------------------------  ------------------------  ----------
  rhel6m2       compat-libstdc++-33(x86_64)-3.2.3-69.el6  compat-libstdc++-33(x86_64)-3.2.3  passed
  rhel6m1       compat-libstdc++-33(x86_64)-3.2.3-69.el6  compat-libstdc++-33(x86_64)-3.2.3  passed
Result: Package existence check passed for "compat-libstdc++-33(x86_64)"

Check: Package existence for "elfutils-libelf(x86_64)"
  Node Name     Available                 Required                  Status
  ------------  ------------------------  ------------------------  ----------
  rhel6m2       elfutils-libelf(x86_64)-0.152-1.el6  elfutils-libelf(x86_64)-0.97  passed
  rhel6m1       elfutils-libelf(x86_64)-0.152-1.el6  elfutils-libelf(x86_64)-0.97  passed
Result: Package existence check passed for "elfutils-libelf(x86_64)"

Check: Package existence for "elfutils-libelf-devel"
  Node Name     Available                 Required                  Status
  ------------  ------------------------  ------------------------  ----------
  rhel6m2       elfutils-libelf-devel-0.152-1.el6  elfutils-libelf-devel-0.97  passed
  rhel6m1       elfutils-libelf-devel-0.152-1.el6  elfutils-libelf-devel-0.97  passed
Result: Package existence check passed for "elfutils-libelf-devel"

Check: Package existence for "glibc-common"
  Node Name     Available                 Required                  Status
  ------------  ------------------------  ------------------------  ----------
  rhel6m2       glibc-common-2.12-1.47.el6  glibc-common-2.3.4        passed
  rhel6m1       glibc-common-2.12-1.47.el6  glibc-common-2.3.4        passed
Result: Package existence check passed for "glibc-common"

Check: Package existence for "glibc-devel(x86_64)"
  Node Name     Available                 Required                  Status
  ------------  ------------------------  ------------------------  ----------
  rhel6m2       glibc-devel(x86_64)-2.12-1.47.el6  glibc-devel(x86_64)-2.3.4  passed
  rhel6m1       glibc-devel(x86_64)-2.12-1.47.el6  glibc-devel(x86_64)-2.3.4  passed
Result: Package existence check passed for "glibc-devel(x86_64)"

Check: Package existence for "glibc-headers"
  Node Name     Available                 Required                  Status
  ------------  ------------------------  ------------------------  ----------
  rhel6m2       glibc-headers-2.12-1.47.el6  glibc-headers-2.3.4       passed
  rhel6m1       glibc-headers-2.12-1.47.el6  glibc-headers-2.3.4       passed
Result: Package existence check passed for "glibc-headers"

Check: Package existence for "gcc-c++(x86_64)"
  Node Name     Available                 Required                  Status
  ------------  ------------------------  ------------------------  ----------
  rhel6m2       gcc-c++(x86_64)-4.4.6-3.el6  gcc-c++(x86_64)-3.4.6     passed
  rhel6m1       gcc-c++(x86_64)-4.4.6-3.el6  gcc-c++(x86_64)-3.4.6     passed
Result: Package existence check passed for "gcc-c++(x86_64)"

Check: Package existence for "libaio-devel(x86_64)"
  Node Name     Available                 Required                  Status
  ------------  ------------------------  ------------------------  ----------
  rhel6m2       libaio-devel(x86_64)-0.3.107-10.el6  libaio-devel(x86_64)-0.3.105  passed
  rhel6m1       libaio-devel(x86_64)-0.3.107-10.el6  libaio-devel(x86_64)-0.3.105  passed
Result: Package existence check passed for "libaio-devel(x86_64)"

Check: Package existence for "libgcc(x86_64)"
  Node Name     Available                 Required                  Status
  ------------  ------------------------  ------------------------  ----------
  rhel6m2       libgcc(x86_64)-4.4.6-3.el6  libgcc(x86_64)-3.4.6      passed
  rhel6m1       libgcc(x86_64)-4.4.6-3.el6  libgcc(x86_64)-3.4.6      passed
Result: Package existence check passed for "libgcc(x86_64)"

Check: Package existence for "libstdc++(x86_64)"
  Node Name     Available                 Required                  Status
  ------------  ------------------------  ------------------------  ----------
  rhel6m2       libstdc++(x86_64)-4.4.6-3.el6  libstdc++(x86_64)-3.4.6   passed
  rhel6m1       libstdc++(x86_64)-4.4.6-3.el6  libstdc++(x86_64)-3.4.6   passed
Result: Package existence check passed for "libstdc++(x86_64)"

Check: Package existence for "libstdc++-devel(x86_64)"
  Node Name     Available                 Required                  Status
  ------------  ------------------------  ------------------------  ----------
  rhel6m2       libstdc++-devel(x86_64)-4.4.6-3.el6  libstdc++-devel(x86_64)-3.4.6  passed
  rhel6m1       libstdc++-devel(x86_64)-4.4.6-3.el6  libstdc++-devel(x86_64)-3.4.6  passed
Result: Package existence check passed for "libstdc++-devel(x86_64)"

Check: Package existence for "sysstat"
  Node Name     Available                 Required                  Status
  ------------  ------------------------  ------------------------  ----------
  rhel6m2       sysstat-9.0.4-18.el6      sysstat-5.0.5             passed
  rhel6m1       sysstat-9.0.4-18.el6      sysstat-5.0.5             passed
Result: Package existence check passed for "sysstat"

Check: Package existence for "pdksh"
  Node Name     Available                 Required                  Status
  ------------  ------------------------  ------------------------  ----------
  rhel6m2       missing                   pdksh-5.2.14              failed
  rhel6m1       missing                   pdksh-5.2.14              failed
Result: Package existence check failed for "pdksh"

Check: Package existence for "expat(x86_64)"
  Node Name     Available                 Required                  Status
  ------------  ------------------------  ------------------------  ----------
  rhel6m2       expat(x86_64)-2.0.1-9.1.el6  expat(x86_64)-1.95.7      passed
  rhel6m1       expat(x86_64)-2.0.1-9.1.el6  expat(x86_64)-1.95.7      passed
Result: Package existence check passed for "expat(x86_64)"

Checking for multiple users with UID value 0
Result: Check for multiple users with UID value 0 passed

Check: Current group ID
Result: Current group ID check passed

Starting check for consistency of primary group of root user
  Node Name                             Status
  ------------------------------------  ------------------------
  rhel6m2                               passed
  rhel6m1                               passed

Check for consistency of root user's primary group passed

Starting Clock synchronization checks using Network Time Protocol(NTP)...

NTP Configuration file check started...
Network Time Protocol(NTP) configuration file not found on any of the nodes. Oracle Cluster Time Synchronization Service(CTSS) can be used instead of NTP for time synchronization on the cluster nodes
No NTP Daemons or Services were found to be running

Result: Clock synchronization check using Network Time Protocol(NTP) passed

Checking Core file name pattern consistency...
Core file name pattern consistency check passed.

Checking to make sure user "grid" is not in "root" group
  Node Name     Status                    Comment
  ------------  ------------------------  ------------------------
  rhel6m2       passed                    does not exist
  rhel6m1       passed                    does not exist
Result: User "grid" is not part of "root" group. Check passed

Check default user file creation mask
  Node Name     Available                 Required                  Comment
  ------------  ------------------------  ------------------------  ----------
  rhel6m2       0022                      0022                      passed
  rhel6m1       0022                      0022                      passed
Result: Default user file creation mask check passed
Checking consistency of file "/etc/resolv.conf" across nodes

Checking the file "/etc/resolv.conf" to make sure only one of domain and search entries is defined
File "/etc/resolv.conf" does not have both domain and search entries defined
Checking if domain entry in file "/etc/resolv.conf" is consistent across the nodes...
domain entry in file "/etc/resolv.conf" is consistent across nodes
Checking if search entry in file "/etc/resolv.conf" is consistent across the nodes...
search entry in file "/etc/resolv.conf" is consistent across nodes
Checking file "/etc/resolv.conf" to make sure that only one search entry is defined
All nodes have one search entry defined in file "/etc/resolv.conf"
Checking all nodes to make sure that search entry is "domain.net" as found on node "rhel6m2"
All nodes of the cluster have same value for 'search'
Checking DNS response time for an unreachable node
  Node Name                             Status
  ------------------------------------  ------------------------
  rhel6m2                               passed
  rhel6m1                               passed
The DNS response time for an unreachable node is within acceptable limit on all nodes

File "/etc/resolv.conf" is consistent across nodes

Check: Time zone consistency
Result: Time zone consistency check passed

Pre-check for cluster services setup was unsuccessful on all the nodes.
If only the pdksh has the status as failed this could be ignored and consider the pre-check is successful.

11. Start the clusterware installation.

Since block devices are used for ASM disks set the disk discovery path to block devices.


Set OS group chosen for ASM administration

When CV_ASSUME_DISTID=OEL6 is set the OUI doesn't flag the pdksh but correctly identify the ksh pre-check requirement.

Summary

Begin the installation

12. Run root scripts when prompted

Running root.sh on first node (rhel6m1)
# /opt/app/11.2.0/grid/root.sh
Performing root user operation for Oracle 11g

The following environment variables are set as:
    ORACLE_OWNER= grid
    ORACLE_HOME=  /opt/app/11.2.0/grid

Enter the full pathname of the local bin directory: [/usr/local/bin]:
   Copying dbhome to /usr/local/bin ...
   Copying oraenv to /usr/local/bin ...
   Copying coraenv to /usr/local/bin ...


Creating /etc/oratab file...
Entries will be added to the /etc/oratab file as needed by
Database Configuration Assistant when a database is created
Finished running generic part of root script.
Now product-specific root actions will be performed.
Using configuration parameter file: /opt/app/11.2.0/grid/crs/install/crsconfig_params
Creating trace directory
OLR initialization - successful
  root wallet
  root wallet cert
  root cert export
  peer wallet
  profile reader wallet
  pa wallet
  peer wallet keys
  pa wallet keys
  peer cert request
  pa cert request
  peer cert
  pa cert
  peer root cert TP
  profile reader root cert TP
  pa root cert TP
  peer pa cert TP
  pa peer cert TP
  profile reader pa cert TP
  profile reader peer cert TP
  peer user cert
  pa user cert
Adding Clusterware entries to upstart
CRS-2672: Attempting to start 'ora.mdnsd' on 'rhel6m1'
CRS-2676: Start of 'ora.mdnsd' on 'rhel6m1' succeeded
CRS-2672: Attempting to start 'ora.gpnpd' on 'rhel6m1'
CRS-2676: Start of 'ora.gpnpd' on 'rhel6m1' succeeded
CRS-2672: Attempting to start 'ora.cssdmonitor' on 'rhel6m1'
CRS-2672: Attempting to start 'ora.gipcd' on 'rhel6m1'
CRS-2676: Start of 'ora.cssdmonitor' on 'rhel6m1' succeeded
CRS-2676: Start of 'ora.gipcd' on 'rhel6m1' succeeded
CRS-2672: Attempting to start 'ora.cssd' on 'rhel6m1'
CRS-2672: Attempting to start 'ora.diskmon' on 'rhel6m1'
CRS-2676: Start of 'ora.diskmon' on 'rhel6m1' succeeded
CRS-2676: Start of 'ora.cssd' on 'rhel6m1' succeeded

ASM created and started successfully.

Disk Group CLUSTER_DG created successfully.

clscfg: -install mode specified
Successfully accumulated necessary OCR keys.
Creating OCR keys for user 'root', privgrp 'root'..
Operation successful.
CRS-4256: Updating the profile
Successful addition of voting disk 2c90c4f4a0fb4faebfb264244a836b96.
Successful addition of voting disk 1d7890a346284fa4bf0e278b2f3b1d3d.
Successful addition of voting disk 433f41571bf44fcebfe636a97d94777a.
Successfully replaced voting disk group with +CLUSTER_DG.
CRS-4256: Updating the profile
CRS-4266: Voting file(s) successfully replaced
##  STATE    File Universal Id                File Name Disk group
--  -----    -----------------                --------- ---------
 1. ONLINE   2c90c4f4a0fb4faebfb264244a836b96 (/dev/sdb1) [CLUSTER_DG]
 2. ONLINE   1d7890a346284fa4bf0e278b2f3b1d3d (/dev/sdc1) [CLUSTER_DG]
 3. ONLINE   433f41571bf44fcebfe636a97d94777a (/dev/sdd1) [CLUSTER_DG]
Located 3 voting disk(s).
CRS-2672: Attempting to start 'ora.asm' on 'rhel6m1'
CRS-2676: Start of 'ora.asm' on 'rhel6m1' succeeded
CRS-2672: Attempting to start 'ora.CLUSTER_DG.dg' on 'rhel6m1'
CRS-2676: Start of 'ora.CLUSTER_DG.dg' on 'rhel6m1' succeeded
Configure Oracle Grid Infrastructure for a Cluster ... succeeded
Running root.sh on the second node (rhel6m2)
# /opt/app/11.2.0/grid/root.sh
Performing root user operation for Oracle 11g

The following environment variables are set as:
    ORACLE_OWNER= grid
    ORACLE_HOME=  /opt/app/11.2.0/grid

Enter the full pathname of the local bin directory: [/usr/local/bin]:
   Copying dbhome to /usr/local/bin ...
   Copying oraenv to /usr/local/bin ...
   Copying coraenv to /usr/local/bin ...


Creating /etc/oratab file...
Entries will be added to the /etc/oratab file as needed by
Database Configuration Assistant when a database is created
Finished running generic part of root script.
Now product-specific root actions will be performed.
Using configuration parameter file: /opt/app/11.2.0/grid/crs/install/crsconfig_params
Creating trace directory
OLR initialization - successful
Adding Clusterware entries to upstart
CRS-4402: The CSS daemon was started in exclusive mode but found an active CSS daemon on node rhel6m1, number 1, and is terminating
An active cluster was found during exclusive startup, restarting to join the cluster
Configure Oracle Grid Infrastructure for a Cluster ... succeeded
CRS-4402 has been explained on (1212703.1), this is not an error on 11.2.0.3

13. Once the root.sh scripts have run on all nodes the clusterware installation will continue with running configuration assistants.

This will complete the clusterware installation as grid user. Check post installation configuration with
 cluvfy stage -post crsinst -n rhel6m1,rhel6m2
and using asmca created additional diskgroup that will be used when RAC DB is created.

14. Edit the cvu_config file in database software installation. Similar to clusterware check the cluvfy will flag pdksh as failed
cluvfy stage -pre dbinst -n rhel6m1,rhel6m2 -fixup -verbose
...
...
Check: Package existence for "pdksh"
  Node Name     Available                 Required                  Status
  ------------  ------------------------  ------------------------  ----------
  rhel6m2       missing                   pdksh-5.2.14              failed
  rhel6m1       missing                   pdksh-5.2.14              failed
Result: Package existence check failed for "pdksh"
...
...
But OUI will correctly identify the ksh package requirement.

Apart from this rest of the installation steps are same as in RHEL 5.
Select software only install. Database will be created after the installation using DBCA

Select all nodes part of the cluster

Installation path under Oracle base

OS group for database software management

Summary

Begin installation and execute the root.sh script when prompted.


15. Before beginning database created with dbca set read,write and execute permission for cfgtoollogs for Oracle user.
cd $ORACLE_BASE
#chmod 770 cfgtoollogs
Select all nodes the instance will run and complete DB creation.

Continue wizard to complete the remaining steps.


Related Posts
Installing 11gR2 (11.2.0.3) GI with Role Separation on OEL 6
Installing 11gR2 Standalone Server with ASM and Role Separation on RHEL 6
Installing 11.2.0.3 on RHEL 6