Restoring Vote disk due to ASM disk failures - 3

Previous blogs showed the restoring of vote disk due to one and two ASM disk failures in the diskgroup containing vote disks. This scenario is when all the disks in the ASM diskgroup containg vote disks are affected.

Scenario 3.
1. Only Vote disks are in ASM diskgroup
2. ASM diskgroup has normal redundancy with only three failure groups
3. All failure groups are affected
4. OCR is located in a separate location (in another diskgroup or block device location - not supported by Oracle only valid during migration could be moved to after installation for testing purposes)

1. Move the vote disks to ASM diskgroup if not done already. Below is the current configuration

crsctl query css votedisk
##  STATE    File Universal Id                File Name Disk group
--  -----    -----------------                --------- ---------
1. ONLINE   9da0c9d64b9a4fcabf3d143627e7f3e8 (ORCL:CLUS1) [CLUSTERDG]
2. ONLINE   380dba2e5b654f80bfe69d64c5d0c12c (ORCL:CLUS2) [CLUSTERDG]
3. ONLINE   95968700d82b4ff8bf88167a28b080a4 (ORCL:CLUS3) [CLUSTERDG]

2. Identify the disks belonging to the diskgroup and corrupt them to simulate disk failure. One disk being corrupted shown here

# /etc/init.d/oracleasm querydisk -p clus3
Disk "CLUS3" is a valid ASM disk
/dev/sdc10: LABEL="CLUS3" TYPE="oracleasm"

# dd if=/dev/zero of=/dev/sdc10 count=204800 bs=8192
204800+0 records in
204800+0 records out

3. ocssd.log will show the detection of vote disk corruption

2010-08-25 10:33:33.572: [    CSSD][1235470656]clssnmvDiskKillCheck: voting disk corrupted (0x00000000,0x00000000) (ORCL:CLUS1)
2010-08-25 10:33:33.572: [    CSSD][1235470656]clssnmvDiskAvailabilityChange: voting file ORCL:CLUS1 now offline
2010-08-25 10:33:34.263: [    CLSF][1193511232]Closing handle:0x68d6110
2010-08-25 10:33:34.263: [   SKGFD][1193511232]Lib :ASM:/opt/oracle/extapi/64/asm/orcl/1/libasm.so: closing handle 0x68ae880 for
disk :ORCL:CLUS1:

2010-08-25 10:33:34.263: [    CSSD][1193511232]clssnmvScanCompletions: completed 4 items
2010-08-25 10:33:34.270: [    CSSD][1141061952]clssnmvSchedDiskThreads: DiskPingThread for voting file ORCL:CLUS3 sched delay 1680 > margin 1500 cur_ms 1026610624 lastalive 1026608944
2010-08-25 10:33:34.270: [    CSSD][1141061952]clssnmvSchedDiskThreads: DiskPingThread for voting file ORCL:CLUS2 sched delay 1560 > margin 1500 cur_ms 1026610624 lastalive 1026609064
2010-08-25 10:33:34.270: [    CSSD][1141061952]clssnmvSchedDiskThreads: DiskPingThread for voting file ORCL:CLUS1 sched delay 1720 > margin 1500 cur_ms 1026610624 lastalive 1026608904
2010-08-25 10:33:34.572: [    CLSF][1235470656]Closing handle:0x2aaab4246560
2010-08-25 10:33:34.572: [   SKGFD][1235470656]Lib :ASM:/opt/oracle/extapi/64/asm/orcl/1/libasm.so: closing handle 0x2aaab423b080
for disk :ORCL:CLUS1:

2010-08-25 10:33:35.042: [    CLSF][1214490944]Closing handle:0x2aaab43557f0
2010-08-25 10:33:35.042: [   SKGFD][1214490944]Lib :ASM:/opt/oracle/extapi/64/asm/orcl/1/libasm.so: closing handle 0x2aaab43fd730
for disk :ORCL:CLUS1:

2010-08-25 10:33:36.268: [    CSSD][1298409792]clssnmSendingThread: sending status msg to all nodes
2010-08-25 10:33:36.268: [    CSSD][1298409792]clssnmSendingThread: sent 5 status msgs to all nodes
2010-08-25 10:33:36.272: [    CSSD][1141061952]clssnmvSchedDiskThreads: DiskPingThread for voting file ORCL:CLUS1 sched delay 3720 > margin 1500 cur_ms 1026612624 lastalive 1026608904
2010-08-25 10:33:37.044: [    CSSD][1214490944]clssnmvDiskOpen: Opening ORCL:CLUS1
2010-08-25 10:33:37.044: [   SKGFD][1214490944]Handle 0x2aaab423b080 from lib :ASM:/opt/oracle/extapi/64/asm/orcl/1/libasm.so: for disk :ORCL:CLUS1:

2010-08-25 10:33:37.044: [    CLSF][1214490944]Opened hdl:0x2aaab41e9170 for dev:ORCL:CLUS1:
2010-08-25 10:33:37.054: [    CSSD][1214490944]clssnmvStatusBlkInit: myinfo nodename hpc1, uniqueness 1282657936
2010-08-25 10:33:37.054: [    CSSD][1214490944]clssnmvDiskAvailabilityChange: voting file ORCL:CLUS1 now online
2010-08-25 10:33:37.054: [   SKGFD][1193511232]Handle 0x682fcc0 from lib :ASM:/opt/oracle/extapi/64/asm/orcl/1/libasm.so: for disk :ORCL:CLUS1:

2010-08-25 10:33:37.054: [    CLSF][1193511232]Opened hdl:0x6906de0 for dev:ORCL:CLUS1:
2010-08-25 10:33:37.575: [   SKGFD][1235470656]Handle 0x2aaab43fd730 from lib :ASM:/opt/oracle/extapi/64/asm/orcl/1/libasm.so: for disk :ORCL:CLUS1:

2010-08-25 10:33:37.575: [    CLSF][1235470656]Opened hdl:0x2aaab41ee180 for dev:ORCL:CLUS1:
2010-08-25 10:33:37.598: [    CSSD][1235470656]clssnmvDiskKillCheck: voting disk corrupted (0x00000000,0x00000000) (ORCL:CLUS1)
2010-08-25 10:33:37.598: [    CSSD][1235470656]clssnmvDiskAvailabilityChange: voting file ORCL:CLUS1 now offline
2010-08-25 10:33:38.055: [    CLSF][1193511232]Closing handle:0x6906de0
2010-08-25 10:33:38.055: [   SKGFD][1193511232]Lib :ASM:/opt/oracle/extapi/64/asm/orcl/1/libasm.so: closing handle 0x682fcc0 for
disk :ORCL:CLUS1:

2010-08-25 10:33:38.279: [    CSSD][1141061952]clssnmvSchedDiskThreads: DiskPingThread for voting file ORCL:CLUS1 sched delay 5730 > margin 1500 cur_ms 1026614634 lastalive 1026608904
2010-08-25 10:33:38.601: [    CLSF][1235470656]Closing handle:0x2aaab41ee180
2010-08-25 10:33:38.601: [   SKGFD][1235470656]Lib :ASM:/opt/oracle/extapi/64/asm/orcl/1/libasm.so: closing handle 0x2aaab43fd730
for disk :ORCL:CLUS1:

2010-08-25 10:33:38.658: [    CSSD][1214490944]clssnmvDiskAvailabilityChange: voting file ORCL:CLUS1 now online
2010-08-25 10:33:38.659: [   SKGFD][1193511232]Handle 0x682fcc0 from lib :ASM:/opt/oracle/extapi/64/asm/orcl/1/libasm.so: for disk :ORCL:CLUS1:

2010-08-25 10:33:38.659: [    CLSF][1193511232]Opened hdl:0x68b98f0 for dev:ORCL:CLUS1:
2010-08-25 10:33:39.600: [   SKGFD][1235470656]Handle 0x2aaab43fd730 from lib :ASM:/opt/oracle/extapi/64/asm/orcl/1/libasm.so: for disk :ORCL:CLUS1:

2010-08-25 10:33:39.600: [    CLSF][1235470656]Opened hdl:0x2aaab4276240 for dev:ORCL:CLUS1:
2010-08-25 10:33:39.689: [    CSSD][1172531520]clssnmvDiskKillCheck: voting disk corrupted (0x00000000,0x00000000) (ORCL:CLUS2)
2010-08-25 10:33:39.689: [    CSSD][1172531520]clssnmvDiskAvailabilityChange: voting file ORCL:CLUS2 now offline
2010-08-25 10:33:39.830: [    CLSF][1162041664]Closing handle:0x68a4730
2010-08-25 10:33:39.830: [   SKGFD][1162041664]Lib :ASM:/opt/oracle/extapi/64/asm/orcl/1/libasm.so: closing handle 0x68a4790 for
disk :ORCL:CLUS2:

2010-08-25 10:33:40.271: [    CLSF][1183021376]Closing handle:0x2aaab43fdd20
2010-08-25 10:33:40.271: [   SKGFD][1183021376]Lib :ASM:/opt/oracle/extapi/64/asm/orcl/1/libasm.so: closing handle 0x2aaab43fce20
for disk :ORCL:CLUS2:

2010-08-25 10:33:40.690: [    CLSF][1172531520]Closing handle:0x689e5d0
2010-08-25 10:33:40.690: [   SKGFD][1172531520]Lib :ASM:/opt/oracle/extapi/64/asm/orcl/1/libasm.so: closing handle 0x6c43870 for
disk :ORCL:CLUS2:

2010-08-25 10:33:41.278: [    CSSD][1298409792]clssnmSendingThread: sending status msg to all nodes
2010-08-25 10:33:41.278: [    CSSD][1298409792]clssnmSendingThread: sent 5 status msgs to all nodes
2010-08-25 10:33:41.281: [    CSSD][1141061952]clssnmvSchedDiskThreads: DiskPingThread for voting file ORCL:CLUS2 sched delay 2450 > margin 1500 cur_ms 1026617634 lastalive 1026615184
2010-08-25 10:33:43.289: [    CSSD][1141061952]clssnmvSchedDiskThreads: DiskPingThread for voting file ORCL:CLUS2 sched delay 4460 > margin 1500 cur_ms 1026619644 lastalive 1026615184
2010-08-25 10:33:43.832: [    CSSD][1162041664]clssnmvDiskOpen: Opening ORCL:CLUS2
2010-08-25 10:33:43.832: [   SKGFD][1162041664]Handle 0x6c43870 from lib :ASM:/opt/oracle/extapi/64/asm/orcl/1/libasm.so: for disk :ORCL:CLUS2:

2010-08-25 10:33:43.832: [    CLSF][1162041664]Opened hdl:0x68f9bc0 for dev:ORCL:CLUS2:
2010-08-25 10:33:43.876: [    CSSD][1162041664]clssnmvStatusBlkInit: myinfo nodename hpc1, uniqueness 1282657936
2010-08-25 10:33:43.876: [    CSSD][1162041664]clssnmvDiskAvailabilityChange: voting file ORCL:CLUS2 now online
2010-08-25 10:33:43.876: [   SKGFD][1183021376]Handle 0x2aaab43fce20 from lib :ASM:/opt/oracle/extapi/64/asm/orcl/1/libasm.so: for disk :ORCL:CLUS2:

2010-08-25 10:33:43.876: [    CLSF][1183021376]Opened hdl:0x2aaab41c1fc0 for dev:ORCL:CLUS2:
2010-08-25 10:33:44.698: [   SKGFD][1172531520]Handle 0x68a4790 from lib :ASM:/opt/oracle/extapi/64/asm/orcl/1/libasm.so: for disk :ORCL:CLUS2:

2010-08-25 10:33:44.698: [    CLSF][1172531520]Opened hdl:0x6c57440 for dev:ORCL:CLUS2:
2010-08-25 10:33:44.707: [    CSSD][1172531520]clssnmvDiskKillCheck: voting disk corrupted (0x00000000,0x00000000) (ORCL:CLUS2)
2010-08-25 10:33:44.707: [    CSSD][1172531520]clssnmvDiskAvailabilityChange: voting file ORCL:CLUS2 now offline
2010-08-25 10:33:44.820: [    CSSD][1162041664]clssnmvDiskAvailabilityChange: voting file ORCL:CLUS2 now online
2010-08-25 10:33:45.289: [    CSSD][1141061952]clssnmvSchedDiskThreads: DiskPingThread for voting file ORCL:CLUS1 sched delay 1750 > margin 1500 cur_ms 1026621644 lastalive 1026619894
2010-08-25 10:33:50.296: [    CSSD][1298409792]clssnmSendingThread: sent 4 status msgs to all nodes
2010-08-25 10:33:53.311: [    CSSD][1141061952]clssnmvSchedDiskThreads: DiskPingThread for voting file ORCL:CLUS1 sched delay 1660 > margin 1500 cur_ms 1026629664 lastalive 1026628004
2010-08-25 10:34:13.342: [    CSSD][1298409792]clssnmSendingThread: sent 5 status msgs to all nodes
2010-08-25 10:34:14.984: [    CSSD][1141061952]clssnmvSchedDiskThreads: DiskPingThread for voting file ORCL:CLUS2 sched delay 1560 > margin 1500 cur_ms 1026651334 lastalive 1026649774
2010-08-25 10:34:14.984: [    CSSD][1141061952]clssnmvSchedDiskThreads: DiskPingThread for voting file ORCL:CLUS1 sched delay 1580 > margin 1500 cur_ms 1026651334 lastalive 1026649754
2010-08-25 10:34:16.398: [    CSSD][1224980800]clssnmvDiskKillCheck: voting disk corrupted (0x00000000,0x00000000) (ORCL:CLUS3)
2010-08-25 10:34:16.398: [    CSSD][1224980800]clssnmvDiskAvailabilityChange: voting file ORCL:CLUS3 now offline
2010-08-25 10:34:16.830: [    CLSF][1204001088]Closing handle:0x68a5220
2010-08-25 10:34:16.830: [   SKGFD][1204001088]Lib :ASM:/opt/oracle/extapi/64/asm/orcl/1/libasm.so: closing handle 0x647b4f0 for
disk :ORCL:CLUS3:

2010-08-25 10:34:17.074: [    CSSD][1141061952]clssnmvSchedDiskThreads: DiskPingThread for voting file ORCL:CLUS1 sched delay 1700 > margin 1500 cur_ms 1026653424 lastalive 1026651724
2010-08-25 10:34:17.346: [    CLSF][1245960512]Closing handle:0x68f7160
2010-08-25 10:34:17.346: [   SKGFD][1245960512]Lib :ASM:/opt/oracle/extapi/64/asm/orcl/1/libasm.so: closing handle 0x6d04e50 for
disk :ORCL:CLUS3:

2010-08-25 10:34:17.350: [    CSSD][1298409792]clssnmSendingThread: sending status msg to all nodes
2010-08-25 10:34:17.350: [    CSSD][1298409792]clssnmSendingThread: sent 4 status msgs to all nodes
2010-08-25 10:34:17.395: [    CLSF][1224980800]Closing handle:0x2aaab019fce0
2010-08-25 10:34:17.395: [   SKGFD][1224980800]Lib :ASM:/opt/oracle/extapi/64/asm/orcl/1/libasm.so: closing handle 0x2aaab0115350
for disk :ORCL:CLUS3:

2010-08-25 10:34:19.164: [    CSSD][1141061952]clssnmvSchedDiskThreads: DiskPingThread for voting file ORCL:CLUS3 sched delay 3340 > margin 1500 cur_ms 1026655514 lastalive 1026652174
2010-08-25 10:34:19.164: [    CSSD][1141061952]clssnmvSchedDiskThreads: DiskPingThread for voting file ORCL:CLUS1 sched delay 2070 > margin 1500 cur_ms 1026655514 lastalive 1026653444
2010-08-25 10:34:20.828: [    CSSD][1204001088]clssnmvDiskOpen: Opening ORCL:CLUS3
2010-08-25 10:34:20.828: [   SKGFD][1204001088]Handle 0x647b4f0 from lib :ASM:/opt/oracle/extapi/64/asm/orcl/1/libasm.so: for disk :ORCL:CLUS3:

2010-08-25 10:34:20.828: [    CLSF][1204001088]Opened hdl:0x682fc30 for dev:ORCL:CLUS3:
2010-08-25 10:34:20.848: [    CSSD][1204001088]clssnmvStatusBlkInit: myinfo nodename hpc1, uniqueness 1282657936
2010-08-25 10:34:20.848: [    CSSD][1204001088]clssnmvDiskAvailabilityChange: voting file ORCL:CLUS3 now online
2010-08-25 10:34:20.848: [   SKGFD][1245960512]Handle 0x6c56ac0 from lib :ASM:/opt/oracle/extapi/64/asm/orcl/1/libasm.so: for disk :ORCL:CLUS3:

2010-08-25 10:34:20.848: [    CLSF][1245960512]Opened hdl:0x640ea10 for dev:ORCL:CLUS3:
2010-08-25 10:34:21.254: [    CSSD][1141061952]clssnmvSchedDiskThreads: DiskPingThread for voting file ORCL:CLUS3 sched delay 5430 > margin 1500 cur_ms 1026657604 lastalive 1026652174
2010-08-25 10:34:21.254: [   SKGFD][1224980800]Handle 0x2aaab02def70 from lib :ASM:/opt/oracle/extapi/64/asm/orcl/1/libasm.so: for disk :ORCL:CLUS3:

2010-08-25 10:34:21.254: [    CLSF][1224980800]Opened hdl:0x2aaab01a40e0 for dev:ORCL:CLUS3:
2010-08-25 10:34:21.284: [    CSSD][1224980800]clssnmvDiskKillCheck: voting disk corrupted (0x00000000,0x00000000) (ORCL:CLUS3)
2010-08-25 10:34:21.284: [    CSSD][1224980800]clssnmvDiskAvailabilityChange: voting file ORCL:CLUS3 now offline
2010-08-25 10:34:21.358: [    CSSD][1298409792]clssnmSendingThread: sending status msg to all nodes
2010-08-25 10:34:21.358: [    CSSD][1298409792]clssnmSendingThread: sent 4 status msgs to all nodes
2010-08-25 10:34:21.849: [    CLSF][1245960512]Closing handle:0x640ea10
2010-08-25 10:34:21.849: [   SKGFD][1245960512]Lib :ASM:/opt/oracle/extapi/64/asm/orcl/1/libasm.so: closing handle 0x6c56ac0 for
disk :ORCL:CLUS3:

2010-08-25 10:34:22.286: [    CLSF][1224980800]Closing handle:0x2aaab01a40e0
2010-08-25 10:34:22.286: [   SKGFD][1224980800]Lib :ASM:/opt/oracle/extapi/64/asm/orcl/1/libasm.so: closing handle 0x2aaab02def70
for disk :ORCL:CLUS3:

2010-08-25 10:34:22.313: [    CSSD][1204001088]clssnmvDiskAvailabilityChange: voting file ORCL:CLUS3 now online
2010-08-25 10:34:22.313: [   SKGFD][1245960512]Handle 0x6c56ac0 from lib :ASM:/opt/oracle/extapi/64/asm/orcl/1/libasm.so: for disk :ORCL:CLUS3:

2010-08-25 10:34:22.313: [    CLSF][1245960512]Opened hdl:0x641fd10 for dev:ORCL:CLUS3:
2010-08-25 10:34:23.285: [   SKGFD][1224980800]Handle 0x2aaab02def70 from lib :ASM:/opt/oracle/extapi/64/asm/orcl/1/libasm.so: for disk :ORCL:CLUS3:

2010-08-25 10:34:23.285: [    CLSF][1224980800]Opened hdl:0x2aaab019fce0 for dev:ORCL:CLUS3:
2010-08-25 10:34:25.366: [    CSSD][1298409792]clssnmSendingThread: sending status msg to all nodes
2010-08-25 10:34:25.366: [    CSSD][1298409792]clssnmSendingThread: sent 4 status msgs to all nodes

Even with all three vote disks gone cluster will be running and vote disks will show online status

crsctl query css votedisk
##  STATE    File Universal Id                File Name Disk group
--  -----    -----------------                --------- ---------
1. ONLINE   9da0c9d64b9a4fcabf3d143627e7f3e8 (ORCL:CLUS1) [CLUSTERDG]
2. ONLINE   380dba2e5b654f80bfe69d64c5d0c12c (ORCL:CLUS2) [CLUSTERDG]
3. ONLINE   95968700d82b4ff8bf88167a28b080a4 (ORCL:CLUS3) [CLUSTERDG]

4. Trying to stop and restart the cluster will result in a failure. Following message could be seen on occsd.log

2010-08-25 10:39:44.388: [    CSSD][1155574080]clssnmvDiskVerify: Successful discovery of 0 disks
2010-08-25 10:39:44.388: [    CSSD][1155574080]clssnmCompleteInitVFDiscovery: Completing initial voting file discovery
2010-08-25 10:39:44.388: [    CSSD][1155574080]clssnmvFindInitialConfigs: No voting files found
2010-08-25 10:39:44.388: [    CSSD][1155574080]###################################
2010-08-25 10:39:44.389: [    CSSD][1155574080]clssscExit: CSSD signal 11 in thread clssnmvDDiscThread
2010-08-25 10:39:44.389: [    CSSD][1155574080]###################################
2010-08-25 10:39:44.389: [    CSSD][1155574080]

To restore the vote disks stop crs on all nodes and start crs in exclusive mode on one node. Querying vote disks will show zero vote disks present.

crsctl stop crs
crsctl start crs -excl

crsctl query css votedisk
Located 0 voting disk(s).

5. Repair the ASM disks

# /etc/init.d/oracleasm  deletedisk CLUS1
Removing ASM disk "CLUS1":                                 [  OK  ]
# /etc/init.d/oracleasm  deletedisk CLUS2
Removing ASM disk "CLUS2":                                 [  OK  ]
# /etc/init.d/oracleasm  deletedisk CLUS3
Removing ASM disk "CLUS3":                                 [  OK  ]

# /etc/init.d/oracleasm  createdisk clus1 /dev/sdc2
Marking disk "clus1" as an ASM disk:                       [  OK  ]
# /etc/init.d/oracleasm  createdisk clus2 /dev/sdc3
Marking disk "clus2" as an ASM disk:                       [  OK  ]
# /etc/init.d/oracleasm  createdisk clus3 /dev/sdc10
Marking disk "clus3" as an ASM disk:                       [  OK  ]

6. Start the ASM instance, no diskgroup will be mounted and diskgroup used for storing vote disks will disappear from the diskgroup list.

sqlplus  / as sysasm

SQL> startup mount
ASM instance started

Total System Global Area  283930624 bytes
Fixed Size                  2212656 bytes
Variable Size             256552144 bytes
ASM Cache                  25165824 bytes
ORA-15110: no diskgroups mounted   

SQL> select name from v$asm_diskgroup;

NAME
-----
DATA
FLASH

7. Create a new diskgroup to store the vote disks. At the end of the command new diskgroup will be mounted.

SQL> create diskgroup clusterdg failgroup fail1 disk 'ORCL:CLUS1' failgroup fail2 disk 'ORCL:CLUS2' 
failgroup fail3 disk 'ORCL:CLUS3' attribute 'compatible.asm'='11.2';

SQL> select name,state from v$asm_diskgroup;

NAME       STATE
---------- ----------
DATA       DISMOUNTED
FLASH      DISMOUNTED
CLUSTERDG  MOUNTED

Fro ASM alert log

NOTE: Instance updated compatible.asm to 11.2.0.0.0 for grp 1
SUCCESS: diskgroup CLUSTERDG was mounted
SUCCESS: create diskgroup clusterdg failgroup fail1 disk 'ORCL:CLUS1' failgroup fail2 disk 'ORCL:CLUS2' failgroup fail3 disk 'ORCL:CLUS3' attribute 'compatible.asm'='11.2'
Wed Aug 25 10:44:55 2010
WARNING: failed to online diskgroup resource ora.CLUSTERDG.dg (unable to communicate with CRSD/OHASD)

8. Restore the vote disks to the newly created diskgroup

crsctl replace votedisk +clusterdg
Successful addition of voting disk 1e1e904b27c74f6abf4256e67c274824
Successful addition of voting disk 30e33d88487a4f4cbf68242bd8a06726
Successful addition of voting disk 3fa62d01b5c04fbcbf7160d3014e762c
Successfully replaced voting disk group with +clusterdg.
CRS-4266: Voting file(s) successfully replaced 

crsctl query css votedisk
##  STATE    File Universal Id                File Name Disk group
--  -----    -----------------                --------- ---------
 1. ONLINE   1e1e904b27c74f6abf4256e67c274824 (ORCL:CLUS1) [CLUSTERDG]
 2. ONLINE   30e33d88487a4f4cbf68242bd8a06726 (ORCL:CLUS2) [CLUSTERDG]
 3. ONLINE   3fa62d01b5c04fbcbf7160d3014e762c (ORCL:CLUS3) [CLUSTERDG]

From ASM alert log

Wed Aug 25 10:46:31 2010
NOTE: updated gpnp profile ASM diskstring: ORCL:*
Wed Aug 25 10:46:31 2010
NOTE: Creating voting files in diskgroup CLUSTERDG
Wed Aug 25 10:46:31 2010
NOTE: Voting File refresh pending for group 1/0xc488f8d5 (CLUSTERDG)
NOTE: Attempting voting file creation in diskgroup CLUSTERDG
NOTE: voting file allocation on grp 1 disk CLUS1
NOTE: voting file allocation on grp 1 disk CLUS2
NOTE: voting file allocation on grp 1 disk CLUS3

9. Stop the crs on the node stared in exclusive mode and start the crs on all nodes

crsctl stop crs
crsctl start crs

This shows that as long as a valid OCR is available even if all vote disks are corrupted they could be restored since vote disks are backed up as part of OCR.

"The dd commands used to back up and recover voting disks in previous versions of Oracle Clusterware are not supported in Oracle Clusterware 11g release 2 (11.2).

Voting disk management requires a valid and working OCR. Before you add, delete, replace, or restore voting disks, run the ocrcheck command as root. If OCR is not available or it is corrupt, then you must restore OCR.

The voting disk data is automatically backed up in OCR as part of any configuration change and is automatically restored to any voting disk added.

If all of the voting disks are corrupted, then Restore OCR. This step is necessary only if OCR is also corrupted or otherwise unavailable, such as if OCR is on Oracle ASM and the disk group is no longer available"(11gR2 Clusterware Admin Guide)

Whereas in 10g if vote disks are corrupted and no backup available then crs had to be reinstalled.

Useful Metalink note
How to restore ASM based OCR after complete loss of the CRS diskgroup on Linux/Unix systems [ID 1062983.1]