"The Oracle ASM partnership and status table (PST) is replicated on multiple disks and is extended to store OCR. Consequently, OCR can tolerate the loss of the same number of disks as are in the underlying disk group and be relocated in response to disk failures.At least two OCR locations if the OCR is configured on an Oracle ASM disk group. You should configure the OCR in two independent disk groups. Typically this is the work area and the recovery area."(11gR2 Clusterware Admin Guide).
Scenario 1.
1. Only OCR in ASM diskgroup
2. ASM diskgroup has normal redundancy with only two failuregroups
3. Only one failure group is affected
4. Vote disk is located in a separate location (in another diskgroup or block device location - not supported by Oracle only valid during migration could be moved to after installation for testing purposes)
Considering the characteristics of the normal redundancy ASM diskgroups in this situation does not require any restoration of the OCR. When one failure group (or disk device) is affected, other will take over and cluster will function normal as long as one valid OCR copy is there. This is possible due to the two-way mirroring done in the normal redundancy ASM diskgroups. But there are few things to look out for to ensure smooth operation of the cluster.
1. Create an ASM diskgroup to store the OCR with normal redundancy containing only two failure groups
create diskgroup clusterdg normal redundancy failgroup failgp1 disk 'ORCL:CLUS1'2. Move the OCR to the ASM diskgroup
failgroup failgp2 disk 'ORCL:CLUS2' ATTRIBUTE 'compatible.asm' = '11.2';
3. Move vote disk to some other location (not in the same ASM diskgroup as OCR). In this case it was moved to a block device
Vote disk not located in the ASM diskgroup4. Identify the block device of a ASM disk in the diskgroup used to store the OCR and corrupt it to simulate disk failure
crsctl query css votedisk
## STATE File Universal Id File Name Disk group
-- ----- ----------------- --------- ---------
1. ONLINE 173f85c9944a4fc8bf01e31adf47ea62 (/dev/sdc6) []
Located 1 voting disk(s).
select g.name,d.name from v$asm_disk d, v$asm_diskgroup g5. As root user check ocr intergrity and confirm OCR integrity is intact
where g.name='CLUSTERDG' and g.group_number=d.group_number;
NAME NAME
---------- -----
CLUSTERDG CLUS1
CLUSTERDG CLUS3
# /etc/init.d/oracleasm querydisk -p CLUS1
Disk "CLUS1" is a valid ASM disk
/dev/sdc2: LABEL="CLUS1" TYPE="oracleasm"
dd if=/dev/zero of=/dev/sdc2 count=20240 bs=8192
20240+0 records in
20240+0 records out
165806080 bytes (166 MB) copied, 0.125701 seconds, 1.3 GB/s
# ocrcheck6. Eventually the ASM will detect the disk failure and it will drop it from the diskgroup.
Status of Oracle Cluster Registry is as follows :
Version : 3
Total space (kbytes) : 148348
Used space (kbytes) : 4580
Available space (kbytes) : 143768
ID : 552644455
Device/File Name : +CLUSTERDG
Device/File integrity check succeeded
Cluster registry integrity check succeeded
Logical corruption check succeeded
Fri Aug 13 14:05:13 20107. But if the cluster was restarted before ASM could clean itself following error messages could be seen on cluster alert log
WARNING: PST-initiated drop disk 1(1633509787).1(3916272010) (CLUS1)
NOTE: GroupBlock outside rolling migration privileged region
NOTE: requesting all-instance membership refresh for group=1
NOTE: initiating PST update: grp = 1
kfdp_update(): 26
Fri Aug 13 14:05:15 2010
kfdp_updateBg(): 26
NOTE: group CLUSTERDG: updated PST location: disk 0002 (PST copy 0)
NOTE: PST update grp = 1 completed successfully
Fri Aug 13 14:05:15 2010
NOTE: membership refresh pending for group 1/0x615d619b (CLUSTERDG)
kfdp_query(CLUSTERDG): 27
kfdp_queryBg(): 27
SUCCESS: refreshed membership for 1/0x615d619b (CLUSTERDG)
GMON SlaveB: Deferred DG Ops completed.
NOTE: starting rebalance of group 1/0x615d619b (CLUSTERDG) at power 1
GMON SlaveB: Deferred DG Ops completed.
Starting background process ARB0
Fri Aug 13 14:05:18 2010
ARB0 started with pid=35, OS id=14730
NOTE: assigning ARB0 to group 1/0x615d619b (CLUSTERDG)
NOTE: F1X0 copy 1 relocating from 1:2 to 2:2 for diskgroup 1 (CLUSTERDG)
NOTE: F1X0 copy 2 relocating from 2:2 to 1:2 for diskgroup 1 (CLUSTERDG)
NOTE: stopping process ARB0
SUCCESS: rebalance completed for group 1/0x615d619b (CLUSTERDG)
Fri Aug 13 14:05:21 2010
NOTE: GroupBlock outside rolling migration privileged region
NOTE: requesting all-instance membership refresh for group=1
NOTE: initiating PST update: grp = 1
kfdp_update(): 28
kfdp_updateBg(): 28
NOTE: group CLUSTERDG: updated PST location: disk 0002 (PST copy 0)
NOTE: PST update grp = 1 completed successfully
WARNING: offline disk number 1 has references (208 AUs)
NOTE: initiating PST update: grp = 1
kfdp_update(): 29
kfdp_updateBg(): 29
NOTE: group CLUSTERDG: updated PST location: disk 0002 (PST copy 0)
NOTE: PST update grp = 1 completed successfully
NOTE: membership refresh pending for group 1/0x615d619b (CLUSTERDG)
Fri Aug 13 14:05:27 2010
kfdp_query(CLUSTERDG): 30
Fri Aug 13 14:05:27 2010
kfdp_queryBg(): 30
SUCCESS: refreshed membership for 1/0x615d619b (CLUSTERDG)
SQL> select name from v$asm_disk;
NAME
-----------------------
_DROPPED_0001_CLUSTERDG
[ohasd(4494)]CRS-2765:Resource 'ora.crsd' has failed on server 'hpc1'.and on crsd.log
2010-08-13 13:30:56.569
[crsd(9550)]CRS-1013:The OCR location in an ASM disk group is inaccessible. Details in /opt/app/11.2.0/grid/log/hpc1/crsd/crsd.log.
2010-08-13 13:30:56.547: [ OCRASM][1796178512]proprasmo: Error in open/create file in dg [CLUSTERDG]This is because the diskgroup which contains the OCR does not automatically mounts due the failed disk in it. This could be confirmed by trying to mount it manually
[ OCRASM][1796178512]SLOS : SLOS: cat=8, opn=kgfoOpenFile01, dep=15056, loc=kgfokge
ORA-17503: ksfdopn:DGOpenFile05 Failed to open file +CLUSTERDG.255.4294967295
ORA-17503: ksfdopn:2 Failed to open file +CLUSTERDG.255.4294967295
ORA-15001: disk
2010-08-13 13:30:56.568: [ OCRASM][1796178512]proprasmo: kgfoCheckMount returned [6]
2010-08-13 13:30:56.568: [ OCRASM][1796178512]proprasmo: The ASM disk group CLUSTERDG is not found or not mounted
2010-08-13 13:30:56.569: [ OCRRAW][1796178512]proprioo: Failed to open [+CLUSTERDG]. Returned proprasmo() with [26]. Marking location as UNAVAILABLE.
2010-08-13 13:30:56.569: [ OCRRAW][1796178512]proprioo: No OCR/OLR devices are usable
2010-08-13 13:30:56.569: [ OCRASM][1796178512]proprasmcl: asmhandle is NULL
2010-08-13 13:30:56.569: [ OCRRAW][1796178512]proprinit: Could not open raw device
2010-08-13 13:30:56.569: [ OCRASM][1796178512]proprasmcl: asmhandle is NULL
2010-08-13 13:30:56.569: [ OCRAPI][1796178512]a_init:16!: Backend init unsuccessful : [26]
2010-08-13 13:30:56.569: [ CRSOCR][1796178512] OCR context init failure. Error: PROC-26: Error while accessing the physical storage ASM error [SLOS: cat=8, opn=kgfoOpenFile01, dep=15056, loc=kgfokge
ORA-17503: ksfdopn:DGOpenFile05 Failed to open file +CLUSTERDG.255.4294967295
ORA-17503: ksfdopn:2 Failed to open file +CLUSTERDG.255.4294967295
ORA-15001: disk
] [8]
2010-08-13 13:30:56.569: [ CRSD][1796178512][PANIC] CRSD exiting: Could not init OCR, code: 26
2010-08-13 13:30:56.569: [ CRSD][1796178512] Done.
alter diskgroup clusterdg mount;8. Mount the diskgroup with the force option and re-issue the cluster start command to start the cluster. Add another disk to the diskgroup to ensure two-way mirroring continues and drop the failed disk if ASM hasn't initiated the disk drop (how long ASM wait before initiating a disk drop is configurable)
alter diskgroup clusterdg mount
*
ERROR at line 1:
ORA-15032: not all alterations performed
ORA-15040: diskgroup is incomplete
ORA-15042: ASM disk "0" is missing from group number "1"
SQL> alter diskgroup clusterdg mount force;Useful Metalink note
Diskgroup altered.
# crsctl start cluster
CRS-2672: Attempting to start 'ora.crsd' on 'hpc1'
CRS-2676: Start of 'ora.crsd' on 'hpc1' succeeded
SQL> alter diskgroup clusterdg add failgroup fail3 disk 'ORCL:CLUS2';
SQL> alter diskgroup clusterdg drop disk CLUS1;
How to restore ASM based OCR after complete loss of the CRS diskgroup on Linux/Unix systems [ID 1062983.1]
