RAC to Single Instance Active Database Duplication

11g allows a database to be duplicated via Oracle Net without the use of backups. The source database must be open and in archive log mode or mounted after a clean shutdown.

This blog is about duplicating a RAC source database to a single instance in a different host. RAC is a 11.2 with two nodes

olsnodes -n
rac4    1
rac5    2

and two diskgroups for data (+DATA) and flash recovery (+FLASH).

1. Install Oracle software (Enterprise or Standard, same as source edition) on the host where duplicate database will be created or clone from an existing single instance Oracle Home. Also create locations for audit trail (adump directory) and location to store the control files.

2. Create a static listener entry on the listener.ora in duplicate database Oracle Home.

SID_LIST_LISTENER =
  (SID_LIST =
        (SID_DESC =
              (GLOBAL_DBNAME = stdaux)
              (SID_NAME = stdaux)
                (ORACLE_HOME = /opt/app/oracle/product/11.2.0/ent)
    )
  )

stdaux is single instance name.

3. Create a pfile that only contains the db_name entry

db_name='stdaux'

4. Create a password file in the duplicate Oracle Home with the same sys password as the source database sys password

orapwd file=orapwstdaux password=password ignorecase=y

5. Start the duplicate instance in nomount state

startup nomount;

6. Create a TNS entry in the tnsnames.ora file in the source database Oracle Home. Adding this to only one node's tnsnames.ora file is suffice, provided duplicate command is run from this node.

STDAUX =
  (DESCRIPTION =
    (ADDRESS = (PROTOCOL = TCP)(HOST = duplicate-hostname)(PORT = 1521))
    (CONNECT_DATA =
      (SERVER = DEDICATED)
      (SID = stdaux)
    )
  )

7. Connect to the target (source database) as well as the auxiliary (duplicate database) using rman

rman target / auxiliary sys/password@stdaux

Duplication will fail if connection to the auxiliary instance is not through Oracle Net.

8. The command used will copy the spfile from source to duplicate while overriding parameter values set using various clauses (set and parameter_value_convert). In 11gR2 RAC remote listener is set to scan listener and local listener is set to VIP TNS entry, which is something not required in a single instance. These parameters will be reset.
Since db_create_file_dest and db_recovery_file_dest will be used in command data files will be created as OMF in these locations. But control file location doesn't get created automatically, so it is necessary to create them manually as mentioned on step 1.
During this step if the duplication process terminates due to an error, delete the spfile on the duplicate Oracle Home (if it was created by the duplicate process) and shutdown and start up the instance in nomount state. Correct the error and try again.

RMAN> DUPLICATE TARGET DATABASE
2> TO stdaux
3> FROM ACTIVE DATABASE
4> SPFILE PARAMETER_VALUE_CONVERT 'rac11g2', 'stdaux', '+DATA', '/data/oradata', '+FLASH','/data/flash_recovery'
5> SET cluster_database='false'
6> reset REMOTE_LISTENER
7> reset local_listener
8> SET db_create_file_dest = '/data/oradata'
9> set db_recovery_file_dest = '/data/flash_recovery';

Starting Duplicate Db at 21-OCT-10
using target database control file instead of recovery catalog
allocated channel: ORA_AUX_DISK_1
channel ORA_AUX_DISK_1: SID=10 device type=DISK

contents of Memory Script:
{
   backup as copy reuse
   targetfile  '+DATA/rac11g2/spfilerac11g2.ora' auxiliary format
 '/opt/app/oracle/product/11.2.0/ent/dbs/spfilestdaux.ora'   ;
   sql clone "alter system set spfile= ''/opt/app/oracle/product/11.2.0/ent/dbs/spfilestdaux.ora''";
}
executing Memory Script
..
..
..
datafile 2 switched to datafile copy
input datafile copy RECID=1 STAMP=732970731 file name=/data/oradata/STDAUX/datafile/o1_mf_sysaux_12lr0eq9_.dbf
datafile 3 switched to datafile copy
input datafile copy RECID=2 STAMP=732970731 file name=/data/oradata/STDAUX/datafile/o1_mf_undotbs1_13lr0es0_.dbf
datafile 4 switched to datafile copy
input datafile copy RECID=3 STAMP=732970731 file name=/data/oradata/STDAUX/datafile/o1_mf_users_15lr0esj_.dbf
datafile 5 switched to datafile copy
input datafile copy RECID=4 STAMP=732970731 file name=/data/oradata/STDAUX/datafile/o1_mf_undotbs2_14lr0esf_.dbf

contents of Memory Script:
{
   Alter clone database open resetlogs;
}
executing Memory Script

database opened
Finished Duplicate Db at 21-OCT-10

rac11g2 is the RAC database name which will appear on various dump file paths and will be replaced by stdaux. Any reference paths to ASM diskgroup will be replaces with directory paths.

9. At the end of the successful execution of the command, duplicate process is complete. Single instance would have chosen one of the undo tablespace (UNDOTBS1) as the system undo tablespace, while other one (UNDOTBS2) will also be there as a tablespace. If needed some tablespaces could be skipped during the duplicate process but trying to skip undo tablespaces could result in the following error

RMAN-00571: ===========================================================
RMAN-00569: =============== ERROR MESSAGE STACK FOLLOWS ===============
RMAN-00571: ===========================================================
RMAN-03002: failure of Duplicate Db command at 10/21/2010 16:46:01
RMAN-05514: Tablespace UNDOTBS2 has undo information, cannot skip

10. Though the duplicate instance get created the spfile will still have parameters that were set at instance level.

rac11g21.__db_cache_size=369098752
rac11g22.__db_cache_size=339738624
stdaux.__db_cache_size=436207616
rac11g21.__java_pool_size=4194304
rac11g22.__java_pool_size=25165824
stdaux.__java_pool_size=4194304
rac11g21.__large_pool_size=8388608
rac11g22.__large_pool_size=8388608
stdaux.__large_pool_size=4194304
rac11g21.__oracle_base='/opt/app/oracle'#ORACLE_BASE set from environment
rac11g22.__oracle_base='/opt/app/orac
..
..
rac11g22.instance_number=2
rac11g21.instance_number=1
*.log_archive_format='%t_%s_%r.dbf'
*.open_cursors=300
*.pga_aggregate_target=209715200
*.processes=150
*.remote_login_passwordfile='exclusive'
*.sga_target=631242752
rac11g22.thread=2
rac11g21.thread=1
rac11g22.undo_tablespace='UNDOTBS2'
rac11g21.undo_tablespace='UNDOTBS1'

These instance level settings could be reset on the duplicate instance or create a pfile and delete the unwanted entries and create a spfile out of the modified pfile.

The way to create a duplicate instance without the unwanted parameter entries is to create a pfile with all the necessary parameter entries and use that to start the duplicate in nomount state. Then when running the duplicate command omit the spfile clause.

Replace the pfile created on step 3 with a pfile with following entries (or more if needed)

*.audit_file_dest='/opt/app/oracle/admin/stdaux/adump'
*.audit_trail='db'
*.compatible='11.2.0.0.0'
*.control_files='/data/oradata/stdaux/controlfile/o1_mf_62n35l5c_.ctl','/data/flash_recovery/stdaux/controlfile/o1_mf_62n35lcp_.ctl'
*.db_block_size=8192
*.db_create_file_dest='/data/oradata'
*.db_domain='domain.net'
*.db_name='stdaux'
*.db_recovery_file_dest='/data/flash_recovery'
*.db_recovery_file_dest_size=40705720320
*.diagnostic_dest='/opt/app/oracle'
*.dispatchers='(PROTOCOL=TCP) (SERVICE=stdauxXDB)'
*.log_archive_format='%t_%s_%r.dbf'
*.open_cursors=300
*.pga_aggregate_target=1326448640
*.processes=150
*.remote_login_passwordfile='EXCLUSIVE'
*.sga_target=3707764736
*.undo_tablespace='UNDOTBS1'

Start the duplicate instance in nomount state and follow the rest of the steps unitl the execution of duplicate command.
Replace the above given duplicate command with

RMAN> DUPLICATE TARGET DATABASE
2> TO stdaux
3> FROM ACTIVE DATABASE;

Starting Duplicate Db at 21-OCT-10
using target database control file instead of recovery catalog
allocated channel: ORA_AUX_DISK_1
channel ORA_AUX_DISK_1: SID=10 device type=DISK

contents of Memory Script:
{
   sql clone "create spfile from memory";
}
executing Memory Script

sql statement: create spfile from memory

contents of Memory Script:
{
   shutdown clone immediate;
   startup clone nomount;
}
executing Memory Script
..
..
contents of Memory Script:
{
   Alter clone database open resetlogs;
}
executing Memory Script

database opened
Finished Duplicate Db at 21-OCT-10

The duplicate instance's spfile will not contain any RAC instance specific entries, but unlike a spfile of a database created from the scratch it will show all the hidden Oracle parameters.

Related Post
Duplicate Database Without Target Connection or Catalog Connection

10gR2 RAC on RHEL 4 using VirtualBox

This blog is mainly about the pre-installation tasks (highlights) that is needed on VirtualBox to create 10gR2 RAC on RHEL 4. The oracle-base site has two articles on creating 11gR2 RAC using VirtualBox and VMware. These articles uses RHEL 5 on the virtual servers could be refered for installing a 11gR2 RAC.

1. Download Virtualbox rpm (VirtualBox-3.2-3.2.8_64453_rhel5-1.x86_64.rpm or later, chose the correct kernel architecture and host OS version). This version of virtualbox allows the creation of shared disks, which is a key requirment to create a RAC.

2. When installing RHEL 4 with a mixture of IDE controllers (for CD, iso image) and SATA controllers (for loacl hard disk) following error could be seen. (This is not seen when installing RHEL 5) Linux installation bootup seem to hang at this error but continues after a long wait but it wasn't possible to restart the virtual server after the Linux OS installation.

3. To fix this problem all the local disks were created using the IDE controller which seem to solve this issue.

4. Once Linux OS is installed in one virtual server, clone it with

VBoxManage clonehd /vm/rac1.vdi /vm/rac2.vdi --remember

When creating the second virtual server select "use existing harddisk" option to make use of the cloned disk. Once the clone (second rac node) is started change the hostname,IP and etc to make clone different from the source.
If the vdi file is copied instead of cloning generate a new uid with (tested on VirtualBox 4.1)

VBoxManage internalcommands sethduuid rac2.vdi
UUID changed to: ee7c49ae-2871-463b-a75c-a88135b57bde

5. Using IDE controllers only 3 local disks could be used at one time in the server (considering fourth IDE slot would be for CD). Create disks with shareable option

VBoxManage createhd --filename /vm/asm1.vdi --size 10240 --format VDI --variant Fixed --type shareable --remember

VBoxManage createhd --filename /vm/asm2.vdi --size 10240 --format VDI --variant Fixed --type shareable --remember 

and attach them to each virtual server

VBoxManage storageattach rac1 --storagectl "IDE Controller" --port 0 --device 1 --type hdd --medium /vm/asm1.vdi
VBoxManage storageattach rac1 --storagectl "IDE Controller" --port 1 --device 1 --type hdd --medium /vm/asm2.vdi

VBoxManage storageattach rac2 --storagectl "IDE Controller" --port 0 --device 1 --type hdd --medium /vm/asm1.vdi
VBoxManage storageattach rac2 --storagectl "IDE Controller" --port 1 --device 1 --type hdd --medium /vm/asm2.vdi

May need to assign IDE slots (primary salve/master, secondary slave/master) as appropriately.
6. These disks would appear as /dev/hdb and /dev/hdc (similar names) when listed with fdisk -l from the virtual server. To store the ocr,vote disks and asm spfile craeted partitons in one of them and use the reset of the space to create a ASM disk using oracleasm.

Disk /dev/hdb: 10.7 GB, 10737418240 bytes
16 heads, 63 sectors/track, 20805 cylinders
Units = cylinders of 1008 * 512 = 516096 bytes

Device Boot      Start         End      Blocks   Id  System
/dev/hdb1               1         214      107824+  83  Linux
/dev/hdb2             215         273       29736   83  Linux
/dev/hdb3             274         293       10080   83  Linux
/dev/hdb4             294       20805    10338048    5  Extended
/dev/hdb5             294       20805    10338016+  83  Linux

Disk /dev/hdc: 10.7 GB, 10737418240 bytes
16 heads, 63 sectors/track, 20805 cylinders
Units = cylinders of 1008 * 512 = 516096 bytes

Device Boot      Start         End      Blocks   Id  System
/dev/hdc1               1       20805    10485688+  83  Linux

Here hdb1 would be bound to a raw device and will be used as storage location for OCR, hdb2 for vote and hdb3 for asm spfile. Rest of it will be used for ASM (data). hdc1 would be used for another ASM disk (flash).

7. Provided other requriement are satisfied (two network cards) this is all that is required to get started with the RAC installation. (RAC related pre-installation ie. kernel parameters, required rpms and etc are needed, not covered here)

8. Use the following when starting virtual servers remotely and having a GUI interface is not important.

VBoxManage startvm rac1 --type headless
VBoxManage startvm rac2 --type headless

Restoring OCR & Vote disk due to ASM disk failures - 3

There are few changes to the setup that was used to test various ASM disk failure scenarios and restoring OCR and Vote disks. The main change is, setup used is no longer the RAC setup upgraded from 10gR2 to 11gR1 and then to 11gR2
It's a new installation of 11gR2 grid infrastructure (only GI was installed no RAC database was created. Installing GI was sufficient to test this scenario) and therefore the ASM spfile is also inside the same diskgroup as clusterware files. Metalink note 1082943.1 explains how to move it to another diskgroup. If not a pfile from the spfile should be created to re-create the spfile after the ocr and vote disk restore.

Scenario 3.
1. Both OCR and Vote disks are in ASM diskgroup
2. ASM diskgroup has normal redundancy with only three failure groups
3. All failure groups are affected
4. ASM Spfile is also located in the same diskgroup where clusterware files are located.

1. Current OCR, vote disk and ASM Spfile configuration

# ocrcheck
Status of Oracle Cluster Registry is as follows :
         Version                  :          3
         Total space (kbytes)     :     262120
         Used space (kbytes)      :       2272
         Available space (kbytes) :     259848
         ID                       : 1242190491
         Device/File Name         : +clusterdg
                                    Device/File integrity check succeeded
                                    Device/File not configured

         Cluster registry integrity check succeeded
         Logical corruption check succeeded 


$ crsctl query css votedisk
##  STATE    File Universal Id                File Name Disk group
--  -----    -----------------                --------- ---------
 1. ONLINE   7d49533611734f3bbf404d32f1759ed5 (ORCL:CLUS1) [CLUSTERDG]
 2. ONLINE   4a8c288d1ade4f8cbf6588c145b27489 (ORCL:CLUS2) [CLUSTERDG]
 3. ONLINE   ad241f9823cd4fb9bf3412ca67e591df (ORCL:CLUS3) [CLUSTERDG]
Located 3 voting disk(s).                                             

SQL> show parameter spfile

NAME        TYPE        VALUE
----------- ----------  -----
spfile      string      +CLUSTERDG/hpc-cluster/asmparameterfile/registry.253.730565167                            

srvctl config asm -a
ASM home: /opt/app/11.2.0/grid
ASM listener: LISTENER
ASM is enabled.

2. Identify the disks beloging to the ASM diskgroup using oracleasm query -p and corrupt them to simulate disk failure

# dd if=/dev/zero of=/dev/sdc10 count=204800 bs=8192
204800+0 records in
204800+0 records out
1677721600 bytes (1.7 GB) copied, 1.61087 seconds, 1.0 GB/s

# dd if=/dev/zero of=/dev/sdc3 count=204800 bs=8192
204800+0 records in
204800+0 records out
1677721600 bytes (1.7 GB) copied, 1.6883 seconds, 994 MB/s

# dd if=/dev/zero of=/dev/sdc2 count=204800 bs=8192
204800+0 records in
204800+0 records out
1677721600 bytes (1.7 GB) copied, 1.74684 seconds, 960 MB/s 

3. ocssd.log will show the detection of vote disk corruption

2010-09-24 15:46:02.893: [    CSSD][1136630080]clssgmDestroyProc: cleaning up proc(0x2aaab02131b0) con(0x8db0) skgpid  ospid 12671 with 0 clients, refcount 0
2010-09-24 15:46:02.893: [    CSSD][1136630080]clssgmDiscEndpcl: gipcDestroy 0x8db0
2010-09-24 15:46:03.009: [    CSSD][1231038784]clssnmvDiskKillCheck: voting disk corrupted (0x00000000,0x00000000) (ORCL:CLUS1)
2010-09-24 15:46:03.009: [    CSSD][1231038784]clssnmvDiskAvailabilityChange: voting file ORCL:CLUS1 now offline
2010-09-24 15:46:03.584: [    CLSF][1241528640]Closing handle:0x2aaab008cbe0
...
2010-09-24 15:46:03.584: [   SKGFD][1241528640]Lib :ASM:/opt/oracle/extapi/64/asm/orcl/1/libasm.so: closing handle 0x2aaab0197570 for disk :ORCL:CLUS1:
2010-09-24 15:46:13.269: [    CSSD][1168099648]clssnmvDiskKillCheck: voting disk corrupted (0x00000000,0x00000000) (ORCL:CLUS3)
2010-09-24 15:46:13.269: [    CSSD][1168099648]clssnmvDiskAvailabilityChange: voting file ORCL:CLUS3 now offline
2010-09-24 15:46:13.431: [    CLSF][1157609792]Closing handle:0x2aaab006ac20
2010-09-24 15:46:13.431: [   SKGFD][1157609792]Lib :ASM:/opt/oracle/extapi/64/asm/orcl/1/libasm.so: closing handle 0x2aaab0062830 for disk :ORCL:CLUS3:

Querying vote disks frequently will enable to spot the state change

$ crsctl query css votedisk
##  STATE    File Universal Id                File Name Disk group
--  -----    -----------------                --------- ---------
 1. PENDOFFL 7d49533611734f3bbf404d32f1759ed5 (ORCL:CLUS1) [CLUSTERDG]
 2. ONLINE   4a8c288d1ade4f8cbf6588c145b27489 (ORCL:CLUS2) [CLUSTERDG]
 3. ONLINE   ad241f9823cd4fb9bf3412ca67e591df (ORCL:CLUS3) [CLUSTERDG]
Located 3 voting disk(s).

$ crsctl query css votedisk
##  STATE    File Universal Id                File Name Disk group
--  -----    -----------------                --------- ---------
 1. ONLINE   7d49533611734f3bbf404d32f1759ed5 (ORCL:CLUS1) [CLUSTERDG]
 2. ONLINE   4a8c288d1ade4f8cbf6588c145b27489 (ORCL:CLUS2) [CLUSTERDG]
 3. ONLINE   ad241f9823cd4fb9bf3412ca67e591df (ORCL:CLUS3) [CLUSTERDG]
Located 3 voting disk(s).

crs_stat (deprecated in 11gR2) shows cluster applications are running but ocrcheck fails

crs_stat -t
Name           Type           Target    State     Host
------------------------------------------------------------
ora....ERDG.dg ora....up.type ONLINE    ONLINE    hpc1
ora....ER.lsnr ora....er.type ONLINE    ONLINE    hpc1
ora....N1.lsnr ora....er.type OFFLINE   OFFLINE
ora.asm        ora.asm.type   ONLINE    ONLINE    hpc1
ora.eons       ora.eons.type  ONLINE    ONLINE    hpc1
ora.gsd        ora.gsd.type   OFFLINE   OFFLINE
ora....SM1.asm application    ONLINE    ONLINE    hpc1
ora....C1.lsnr application    ONLINE    ONLINE    hpc1
ora.hpc1.gsd   application    OFFLINE   OFFLINE
ora.hpc1.ons   application    ONLINE    OFFLINE
ora.hpc1.vip   ora....t1.type ONLINE    ONLINE    hpc1
ora....network ora....rk.type ONLINE    ONLINE    hpc1
ora.oc4j       ora.oc4j.type  OFFLINE   OFFLINE
ora.ons        ora.ons.type   ONLINE    OFFLINE
ora....ry.acfs ora....fs.type ONLINE    ONLINE    hpc1
ora.scan1.vip  ora....ip.type OFFLINE   OFFLINE

ocrcheck
PROT-602: Failed to retrieve data from the cluster registry
PROC-26: Error while accessing the physical storage

4. Stop the crs on all nodes and start crs in exclusive mode in one node. Manual shutdown of ASM instance and database instance might be required if the stop command is unable complete these operations.

crsctl stop crs
CRS-2791: Starting shutdown of Oracle High Availability Services-managed resources on 'hpc1'
CRS-2673: Attempting to stop 'ora.crsd' on 'hpc1'
CRS-2790: Starting shutdown of Cluster Ready Services-managed resources on 'hpc1'
CRS-2673: Attempting to stop 'ora.CLUSTERDG.dg' on 'hpc1'
CRS-2673: Attempting to stop 'ora.registry.acfs' on 'hpc1'
CRS-2673: Attempting to stop 'ora.LISTENER.lsnr' on 'hpc1'
CRS-2677: Stop of 'ora.registry.acfs' on 'hpc1' succeeded
CRS-4549: Unexpected disconnect while executing shutdown request.
CRS-2677: Stop of 'ora.crsd' on 'hpc1' succeeded
CRS-2673: Attempting to stop 'ora.gpnpd' on 'hpc1'
CRS-2673: Attempting to stop 'ora.cssdmonitor' on 'hpc1'
CRS-2673: Attempting to stop 'ora.ctssd' on 'hpc1'
CRS-2673: Attempting to stop 'ora.evmd' on 'hpc1'
CRS-2673: Attempting to stop 'ora.asm' on 'hpc1'
CRS-2673: Attempting to stop 'ora.mdnsd' on 'hpc1'
CRS-2673: Attempting to stop 'ora.drivers.acfs' on 'hpc1'
CRS-2677: Stop of 'ora.cssdmonitor' on 'hpc1' succeeded
CRS-2677: Stop of 'ora.gpnpd' on 'hpc1' succeeded
CRS-2677: Stop of 'ora.evmd' on 'hpc1' succeeded
CRS-2677: Stop of 'ora.mdnsd' on 'hpc1' succeeded
CRS-2677: Stop of 'ora.ctssd' on 'hpc1' succeeded
CRS-2677: Stop of 'ora.asm' on 'hpc1' succeeded
CRS-2673: Attempting to stop 'ora.cssd' on 'hpc1'
CRS-2677: Stop of 'ora.cssd' on 'hpc1' succeeded
CRS-2673: Attempting to stop 'ora.diskmon' on 'hpc1'
CRS-2673: Attempting to stop 'ora.gipcd' on 'hpc1'
CRS-2677: Stop of 'ora.gipcd' on 'hpc1' succeeded
CRS-2677: Stop of 'ora.drivers.acfs' on 'hpc1' succeeded
CRS-2677: Stop of 'ora.diskmon' on 'hpc1' succeeded
CRS-2793: Shutdown of Oracle High Availability Services-managed resources on 'hpc1' has completed
CRS-4133: Oracle High Availability Services has been stopped.                                    

crsctl start crs -excl
CRS-4123: Oracle High Availability Services has been started.
CRS-2672: Attempting to start 'ora.gipcd' on 'hpc1'
CRS-2672: Attempting to start 'ora.mdnsd' on 'hpc1'
CRS-2676: Start of 'ora.gipcd' on 'hpc1' succeeded
CRS-2676: Start of 'ora.mdnsd' on 'hpc1' succeeded
CRS-2672: Attempting to start 'ora.gpnpd' on 'hpc1'
CRS-2676: Start of 'ora.gpnpd' on 'hpc1' succeeded
CRS-2672: Attempting to start 'ora.cssdmonitor' on 'hpc1'
CRS-2676: Start of 'ora.cssdmonitor' on 'hpc1' succeeded
CRS-2672: Attempting to start 'ora.cssd' on 'hpc1'
CRS-2679: Attempting to clean 'ora.diskmon' on 'hpc1'
CRS-2681: Clean of 'ora.diskmon' on 'hpc1' succeeded
CRS-2672: Attempting to start 'ora.diskmon' on 'hpc1'
CRS-2676: Start of 'ora.diskmon' on 'hpc1' succeeded
CRS-2676: Start of 'ora.cssd' on 'hpc1' succeeded
CRS-2672: Attempting to start 'ora.ctssd' on 'hpc1'
CRS-2672: Attempting to start 'ora.drivers.acfs' on 'hpc1'
CRS-2676: Start of 'ora.drivers.acfs' on 'hpc1' succeeded
CRS-2676: Start of 'ora.ctssd' on 'hpc1' succeeded
CRS-2672: Attempting to start 'ora.asm' on 'hpc1'
CRS-2676: Start of 'ora.asm' on 'hpc1' succeeded
CRS-2672: Attempting to start 'ora.crsd' on 'hpc1'
CRS-2676: Start of 'ora.crsd' on 'hpc1' succeeded

Various log files will show the status of vote disks and ocr disks. ocssd.log

2010-09-24 15:53:43.643: [    CSSD][1147734336]clssnmvDiskVerify: Successful discovery of 0 disks
2010-09-24 15:53:43.643: [    CSSD][1147734336]clssnmCompleteInitVFDiscovery: Completing initial voting file discovery
2010-09-24 15:53:43.643: [    CSSD][1147734336]clssnmvFindInitialConfigs: No voting files found
2010-09-24 15:53:43.644: [    CSSD][1147734336]clssnmCompleteVFDiscovery: Completing voting file discovery
2010-09-24 15:53:43.644: [    CSSD][1147734336]clssnmvVerifyCommittedConfigVFs: Insufficient voting files found, found 0 of 0 configured, needed 1 voting files 

crsd.log

2010-09-24 15:54:30.723: [  OCRASM][21660240]proprasmo: kgfoCheckMount returned [6]
2010-09-24 15:54:30.723: [  OCRASM][21660240]proprasmo: The ASM disk group clusterdg is not found or not mounted
2010-09-24 15:54:30.724: [  OCRRAW][21660240]proprioo: Failed to open [+clusterdg]. Returned proprasmo() with [26]. Marking loc
ion as UNAVAILABLE.
2010-09-24 15:54:30.724: [  OCRRAW][21660240]proprioo: No OCR/OLR devices are usable
2010-09-24 15:54:30.724: [  OCRASM][21660240]proprasmcl: asmhandle is NULL
2010-09-24 15:54:30.724: [  OCRRAW][21660240]proprinit: Could not open raw device
2010-09-24 15:54:30.724: [  OCRASM][21660240]proprasmcl: asmhandle is NULL
2010-09-24 15:54:30.724: [  OCRAPI][21660240]a_init:16!: Backend init unsuccessful : [26]
2010-09-24 15:54:30.724: [  CRSOCR][21660240] OCR context init failure.  Error: PROC-26: Error while accessing the physical sto
ge ASM error [SLOS: cat=8, opn=kgfoOpenFile01, dep=15056, loc=kgfokge
ORA-17503: ksfdopn:DGOpenFile05 Failed to open file +CLUSTERDG.255.4294967295
ORA-17503: ksfdopn:2 Failed to open file +CLUSTERDG.255.4294967295
ORA-15001: disk] [8]
2010-09-24 15:54:30.724: [    CRSD][21660240][PANIC] CRSD exiting: Could not init OCR, code: 26
2010-09-24 15:54:30.724: [    CRSD][21660240] Done.

ASM alert log shows why the diskgroup containign vote disks and ocr wasn't mounted

SQL> ALTER DISKGROUP ALL MOUNT /* asm agent */
Diskgroup used for OCR is:CLUSTERDG
NOTE: cache registered group CLUSTERDG number=1 incarn=0x6ed824a0
NOTE: cache began mount (first) of group CLUSTERDG number=1 incarn=0x6ed824a0
NOTE: Loaded library: /opt/oracle/extapi/64/asm/orcl/1/libasm.so
ERROR: no PST quorum in group: required 2, found 0

5. There's no SPfile for ASM but instance will be up

SQL> show parameter spfile

NAME    TYPE        VALUE
------- ---------   ------
spfile  string

6. To test various replace and repair scenarios, a diskgroup was created with a different name. Oracle Clusterware Admin Guide states "If the original OCR location does not exist, then you must create an empty (0 byte) OCR location before you run the ocrconfig -add or ocrconfig -replace commands. The OCR location that you are replacing can be either online or offline."

Replace and repair options could be used to replace the current location and to add, add and replace existing OCR locations. Cluster Admin guide also states "You cannot repair the OCR on a node on which Oracle Clusterware is running.If you run the ocrconfig -add | -repair | -replace command, then the device, file, or Oracle ASM disk group that you are adding must be accessible. This means that a device must exist. You must create an empty (0 byte) OCR location, or the Oracle ASM disk group must exist and be mounted."

There's the question of moutning ASM diskgroup while the crs down. If you try to manually mount the ASM instnace that is part of a cluster you'd get

sqlplus  / as sysasm

SQL> startup
ORA-01078: failure in processing system parameters
ORA-29701: unable to connect to Cluster Synchronization Service

All of the replace and repair option were useless and failed

# ocrconfig -restore /opt/app/11.2.0/grid/cdata/hpc-cluster/backup_20100924_142540.ocr
PROT-16: Internal Error 

# ocrconfig -replace +clusterdg -replacement +clusterdgbk
PROT-28: Cannot delete or replace the only configured Oracle Cluster Registry location

# ocrconfig -repair -replace +clusterdg -replacement +clusterdgbk
PROT-21: Invalid parameter

# ocrconfig -repair -replace +clusterdg -replacement +clusterdgbk
PROT-21: Invalid parameter

# ocrconfig -repair -add +clusterdgbk
PROT-21: Invalid parameter

# ocrconfig -add +clusterdgbk
PROT-1: Failed to initialize ocrconfig

Even creating diskgroup with the original name and another one with new name and trying to repair replace also failed

SQL> create diskgroup clusterdg disk 'ORCL:CLUS1' disk 'ORCL:CLUS2' disk 'ORCL:CLUS3' attribute 'compatible.asm'='11.2';
Diskgroup created.

SQL> create diskgroup clusterdg2 disk 'ORCL:RED1' disk 'ORCL:RED2' disk 'ORCL:RED3' attribute 'compatible.asm'='11.2';
Diskgroup created. 

# ocrconfig -repair -replace +clusterdg -replacement +clusterdg2
PROT-21: Invalid parameter

7. Finally created a diskgroup with the same name as original and did the OCR restore

SQL> create diskgroup clusterdg disk 'ORCL:RED1' disk 'ORCL:RED2' disk 'ORCL:RED3' attribute 'compatible.asm'='11.2';

ocrconfig -restore /opt/app/11.2.0/grid/cdata/hpc-cluster/backup_20100924_142540.ocr

Unlike the previous scenario there's no crs restart required to restore the vote disks, it was possible to restore the vote disks after restoring ocr

# crsctl query css votedisk
Located 0 voting disk(s).

# crsctl replace votedisk +clusterdg
Successful addition of voting disk 92814292a2ec4fa7bf2d6fe10960cc55
Successful addition of voting disk 57b152766d5f4f3dbf2935c93556b7f5
Successful addition of voting disk 6120792eb1284f64bf958f13a4947ece
Successfully replaced voting disk group with +clusterdg.
CRS-4266: Voting file(s) successfully replaced 

8. Stop the crs on the node and start crs on all nodes

crsctl stop crs -f
CRS-2791: Starting shutdown of Oracle High Availability Services-managed resources on 'hpc1'
CRS-2673: Attempting to stop 'ora.gpnpd' on 'hpc1'
CRS-2673: Attempting to stop 'ora.cssdmonitor' on 'hpc1'
CRS-2673: Attempting to stop 'ora.ctssd' on 'hpc1'
CRS-2673: Attempting to stop 'ora.asm' on 'hpc1'
CRS-2673: Attempting to stop 'ora.mdnsd' on 'hpc1'
CRS-2673: Attempting to stop 'ora.drivers.acfs' on 'hpc1'
CRS-2677: Stop of 'ora.cssdmonitor' on 'hpc1' succeeded
CRS-2677: Stop of 'ora.gpnpd' on 'hpc1' succeeded
CRS-2677: Stop of 'ora.mdnsd' on 'hpc1' succeeded
CRS-2677: Stop of 'ora.drivers.acfs' on 'hpc1' succeeded
CRS-2677: Stop of 'ora.asm' on 'hpc1' succeeded
CRS-2677: Stop of 'ora.ctssd' on 'hpc1' succeeded
CRS-2673: Attempting to stop 'ora.cssd' on 'hpc1'
CRS-2677: Stop of 'ora.cssd' on 'hpc1' succeeded
CRS-2673: Attempting to stop 'ora.diskmon' on 'hpc1'
CRS-2673: Attempting to stop 'ora.gipcd' on 'hpc1'
CRS-2677: Stop of 'ora.gipcd' on 'hpc1' succeeded
CRS-2677: Stop of 'ora.diskmon' on 'hpc1' succeeded
CRS-2793: Shutdown of Oracle High Availability Services-managed resources on 'hpc1' has completed
CRS-4133: Oracle High Availability Services has been stopped.

# crsctl start crs
CRS-4123: Oracle High Availability Services has been started.

9. The last step in this scenario is recreate the ASM SPfile. When ASM instance is started with a spfile or pfile, ASM alert log shows the file location. For spfile

Oracle Database 11g Enterprise Edition Release 11.2.0.1.0 - 64bit Production
With the Real Application Clusters and Automatic Storage Management options.
Using parameter settings in server-side spfile +CLUSTERDG/hpc-cluster/asmparameterfile/registry.253.730565167
System parameters with non-default values:
  large_pool_size          = 12M
  instance_type            = "asm"
  remote_login_passwordfile= "EXCLUSIVE"
  asm_diskstring           = "ORCL:CLUS*"
  asm_power_limit          = 1
  diagnostic_dest          = "/opt/app/oracle"
Cluster communication is configured to use the following interface(s) for this instance

For pfile

Oracle Database 11g Enterprise Edition Release 11.2.0.1.0 - 64bit Production
With the Real Application Clusters and Automatic Storage Management options.
Using parameter settings in client-side pfile /opt/app/11.2.0/grid/dbs/init+ASM1.ora on machine hpc1
System parameters with non-default values:
  large_pool_size          = 12M
  instance_type            = "asm"
  remote_login_passwordfile= "EXCLUSIVE"
  asm_diskstring           = "ORCL:CLUS*"
  asm_power_limit          = 1
  diagnostic_dest          = "/opt/app/oracle"
Cluster communication is configured to use the following interface(s) for this instance

But after the corruption of all the disks on the ASM diskgroup, there won't be a server side spfile and even when there's no client side pfile ASM instance starts. ASM alert log shows that it is using default parameter setting without a parameter file

Oracle Database 11g Enterprise Edition Release 11.2.0.1.0 - 64bit Production
With the Real Application Clusters and Automatic Storage Management options.
WARNING: using default parameter settings without any parameter file
Cluster communication is configured to use the following interface(s) for this instance

These default parameters cannot be considered as a spfile or a pfile trying to create one would fail

SQL> create pfile='/home/oracle/pfile.ora' from spfile;
create pfile='/home/oracle/pfile.ora' from spfile
*
ERROR at line 1:
ORA-01565: error in identifying file '?/dbs/spfile@.ora'
ORA-27037: unable to obtain file status
Linux-x86_64 Error: 2: No such file or directory
Additional information: 3

SQL> create spfile from pfile;
create spfile from pfile
*
ERROR at line 1:
ORA-17502: ksfdcre:4 Failed to create file
+CLUSTERDG/hpc-cluster/asmparameterfile/registry.253.730565167
ORA-15177: cannot operate on system aliases

But a pfile could be created from these values in memory using

SQL> create pfile='/home/oracle/pfile.ora' from memory;

File created.

Or if a pfile is available from the time before the diskgroup corruption it could be used to restore the ASM spfile

SQL> create spfile='+clusterdg' from pfile='/home/oracle/asmpfile.ora';

File created.

Last command above would create a new spfile with a different alias than the original and ASM instance would use than during startup.

Oracle Database 11g Enterprise Edition Release 11.2.0.1.0 - 64bit Production
With the Real Application Clusters and Automatic Storage Management options.
Using parameter settings in server-side spfile +CLUSTERDG/hpc-cluster/asmparameterfile/registry.253.730908391
System parameters with non-default values:
  large_pool_size          = 12M
  instance_type            = "asm"
  remote_login_passwordfile= "EXCLUSIVE"
  asm_diskstring           = "ORCL:CLUS*"
  asm_power_limit          = 1
  diagnostic_dest          = "/opt/app/oracle"
Cluster communication is configured to use the following interface(s) for this instance

Useful Metalink note
How to restore ASM based OCR after complete loss of the CRS diskgroup on Linux/Unix systems [ID 1062983.1]