A! Help: Moving OCR, Vote and ASM from Raw Devices to Block Devices

Thursday, July 1, 2010

Moving OCR, Vote and ASM from Raw Devices to Block Devices

Two methods to move OCR from raw devices to block devices.
This blog uses the 10g cluster upgraded to 11g to move the raw device files (OCR,Vote, ASM files) to block devices.

Online method does not require any cluster downtime but require additional space, a new block device apart from the one already bound to the raw device.
Offline method requires a cluster downtime but does not require any additional space. It uses the existing block devices and removes the binding to the raw device.
This blog uses the Offline method. More on this and other methods on metalink note 428681.1

Moving OCR from raw device to block device with cluster downtime.

1. Shutdown all cluster services

crs_stop -all

and as root

crsctl stop crs

2. Check the status of the ocr

 ocrcheck
Status of Oracle Cluster Registry is as follows :
         Version                  :          2
         Total space (kbytes)     :     152364
         Used space (kbytes)      :       2780
         Available space (kbytes) :     149584
         ID                       :  552644455
         Device/File Name         : /dev/raw/raw1
                                    Device/File integrity check succeeded

                                    Device/File not configured

         Cluster registry integrity check succeeded

         Logical corruption check succeeded

3. As root run

ocrconfig -repair ocr /dev/sdc5
ocrconfig -overwrite

second command creates a file in $CRS_HOME/log/hostname/client/ocrconfig_pid.log. If there are multiple files then do a ls -lrt. The above command should have exited with a success status.

Oracle Database 11g CRS Release 11.1.0.7.0 - Production Copyright 1996, 2007 Oracle. All rights reserved.
2010-07-01 11:15:11.193: [ OCRCONF][2976058240]ocrconfig starts...
2010-07-01 11:15:12.883: [ OCRCONF][2976058240]Successfully overwrote OCR configuration on disk
2010-07-01 11:15:12.883: [ OCRCONF][2976058240]Exiting [status=success]...

since this only had on ocr location that's all required to move to block device.
Check the status of the ocr

ocrcheck
Status of Oracle Cluster Registry is as follows :
         Version                  :          2
         Total space (kbytes)     :     152364
         Used space (kbytes)      :       2780
         Available space (kbytes) :     149584
         ID                       :  552644455
         Device/File Name         :  /dev/sdc5
                                    Device/File integrity check succeeded

                                    Device/File not configured

         Cluster registry integrity check succeeded

         Logical corruption check succeeded

4. Edit the udev rules files connected with raw devices and remove any entries added to bind this block device to raw device and change the permissions of the raw device.

5. Create a new udev rules file to set the permission and ownership of the block device to 640 and root:ointall

vi /etc/udev/rules.d/99-sdc.rules
#OCR
KERNEL=="sdc5*", GROUP="oinstall", MODE="640"
start_udev

Refer Udev under what conditions should start_udev be run?

# /sbin/udevadm trigger --type=subsystems --action=add
# /sbin/udevadm trigger --type=devices --action=add
# /sbin/udevadm trigger --type=subsystems --action=change
# /sbin/udevadm trigger --type=devices --action=change

Verify if the raw device binding to the block device is gone. If not restart the server which will start the cluster and use the new ocr location.

Moving Vote disk from raw devices to block devices

1. Query the currently configured voting disks

crsctl query css votedisk
 0.     0    /dev/raw/raw2
Located 1 voting disk(s).

2. For 10g the cluster must be down and for 11.1 moving to a block device is an online operation and no cluster outage is required. 10g has a bug 3972986 which could corrupt the ocr if voting disk is added while cluster is up. More on this bug and workaround on metalink note 390880.1.
If the cluster only has one voting disks then a spare block device or raw device is needed, because cluster must have at least one voting disk inplace at all times, removal of all voting disks is not allowed. If there are multiple voting disks then moving to block device from raw device could be done one voting disk at a time.
Since this cluster only has one vote disk, a new partition was created of same size which is 150MB (10g only requires 20MB and 11g requires 280MB for new installations) and permissions and ownership is set. This spare device is called /dev/sdc10. raw2 is bound to /dev/sdc6

3. Backup the vote disk

dd if=/dev/raw/raw2 of=/home/oracle/votebackup

4. Add the vote disk to the spare block device vote disk, run the command as root

crsctl add css votedisk /dev/sdc10
Now formatting voting disk: /dev/sdc10.
Successful addition of voting disk /dev/sdc10.

Monitor the output on $CRS_HOME/log/hostname/alerthostname.log

2010-07-01 12:16:16.152
[cssd(6403)]CRS-1605:CSSD voting file is online: /dev/sdc10. Details in /opt/app/crs/product/10.2.0/crs/log/hpc1/cssd/ocssd.log.
2010-07-01 12:16:16.169
[cssd(6403)]CRS-1601:CSSD Reconfiguration complete. Active nodes are hpc1 .

Query the new vote disk

crsctl query css votedisk
 0.     0    /dev/raw/raw2
 1.     0    /dev/sdc10
Located 2 voting disk(s).

5. Remove the raw device vote disk

crsctl delete css votedisk /dev/raw/raw2
Successful deletion of voting disk /dev/raw/raw2.

$CRS_HOME/log/hostname/cssd/ocssd.log should have something similar to

[    CSSD]2010-07-01 12:22:44.983 [1220634944] >TRACE:   clssgmVotediskRemove: requested remove for votedisk 0 /dev/raw/raw2
[    CSSD]2010-07-01 12:22:45.801 [1189165376] >TRACE:   clssnmDoSyncUpdate: Wait for 0 vote ack(s)

Query the vote disk is removed from the cluster

crsctl query css votedisk
 1.     0    /dev/sdc10
Located 1 voting disk(s).

6. Add the block device that was used for the raw device as a vote disk, change the ownership and permissiosn on the block device before running the below command.

crsctl add css votedisk /dev/sdc6
Now formatting voting disk: /dev/sdc6.
Successful addition of voting disk /dev/sdc6.

From alert log

[cssd(6403)]CRS-1605:CSSD voting file is online: /dev/sdc6. Details in /opt/app/crs/product/10.2.0/crs/log/hpc1/cssd/ocssd.log.
2010-07-01 12:26:50.504
[cssd(6403)]CRS-1601:CSSD Reconfiguration complete. Active nodes are hpc1 .

From ocssd.log

[    CSSD]2010-07-01 12:26:50.489 [1220634944] >TRACE:   clssgmVotediskAdd: requested add for votedisk 0 /dev/sdc6
[    CSSD]2010-07-01 12:26:50.502 [1189165376] >TRACE:   clssnmDoSyncUpdate: Wait for 0 vote ack(s)

Query the vote disks

crsctl query css votedisk
 0.     0    /dev/sdc6
 1.     0    /dev/sdc10
Located 2 voting disk(s).

7. Remove raw device rules from udev rules files and add a new rule for the vote disk in the sdc rule file created in ocr section.

#vote disk
KERNEL=="sdc6", OWNER="oracle", GROUP="oinstall", MODE="660"

8. Remove the spare vote disk added. If following error

 crsctl delete css votedisk /dev/sdc10
Failure 8 with Cluster Synchronization Services while deleting voting disk.

is thrown then restart the cluster and retry

crsctl delete css votedisk /dev/sdc10
Successful deletion of voting disk /dev/sdc10.

9. All done vote disk is moved to use the block device

crsctl query css votedisk
 0.     0    /dev/sdc6
Located 1 voting disk(s).

Moving ASM Spfile to block device

If required the ASM spfile which was created in a shared storage raw device could be moved to a block device. Otherwise a init+ASM*.ora file must be created on each $ASM_HOME/dbs/ in the cluster.

1. Before moving create a pfile from the ASM spfile.

create pfile='/home/oracle/asmpfile.ora' from spfile;

Metalink note 973031.1 describes how to move ASM spfile to shared device. That theory is used here to move from raw dervices to block devices

2. Shutdown database instances and ASM instances

3. Remove ASM spfile entries from the udev rules files, and add a new rule on the udev block device rule file.

# asm spfile
KERNEL=="sdc7", OWNER="oracle", GROUP="oinstall", MODE="660"

4. Unbind the raw device from the block device which will be used to store the ASM spfile. (even if this step is skipped change will still work and raw devices won't be used for ASM spfile)

5. Startup the ASM instance with the pfile created earlier to nomount state.

startup nomount pfile='/home/oracle/asmpfile.ora';

and create the ASM spfile in the new block device location

SQL> create spfile='/dev/sdc7' from pfile='/home/oracle/asmpfile.ora';

6. Edit the $ORACLE_HOME/dbs/init+ASM*.ora file to contain the new spfile location replace

SPFILE='/dev/raw/raw3'

SPFILE='/dev/sdc7'

and restart the asm instance. Verify on the asm alert log that new spfile is used

Starting up ORACLE RDBMS Version: 11.1.0.7.0.
Using parameter settings in server-side pfile /opt/app/oracle/product/11.1.0/clusdb/dbs/init+ASM1.ora
System parameters with non-default values:
  large_pool_size          = 12M
  spfile                   = "/dev/sdc7"
  instance_type            = "asm"
  cluster_database         = TRUE
  instance_number          = 1
  asm_diskgroups           = "DATA"
  asm_diskgroups           = "FLASH"
  diagnostic_dest          = "/opt/app/oracle"

7. Shutdown the ASM instance and update the OCR ASM registry with the location of the new ASM spfile.
Usage: srvctl modify asm -n -i [-o ] [-p ]
-n Node name
-i ASM instance name
-o ORACLE_HOME path
-p Server parameter file path

srvctl modify asm -n hpc1 -i +ASM1 -o $ASM_HOME -p /dev/sdc7

Verify OCR reconfiguration on $CRS_HOME/log/hostname/alert*.log

[crsd(7207)]CRS-1005:The OCR upgrade was completed. Version has changed from 185599744 to 185599744. Details in /opt/app/crs/product/10.2.0/crs/log/hpc1/crsd/crsd.log.
2010-07-01 12:38:49.111
[crsd(7207)]CRS-1012:The OCR service started on node hpc1.
2010-07-01 12:38:49.812
[evmd(7208)]CRS-1401:EVMD started on node hpc1.
2010-07-01 12:38:50.776
[crsd(7207)]CRS-1201:CRSD started on node hpc1.
2010-07-01 12:40:05.470
[cssd(8011)]CRS-1601:CSSD Reconfiguration complete. Active nodes are hpc1 .

8. Start the ASM instances and the database instances

Moving ASM diskgroups from raw devices to block devices

1. View the currently used raw devices

select mount_status,state,name,path from v$asm_disk;
MOUNT_S STATE    NAME            PATH
------- -------- --------------- -------------
CACHED  NORMAL   DATA_0000       /dev/raw/raw4
CACHED  NORMAL   FLASH_0000      /dev/raw/raw5

2. Change the asm_diskstring in spfile to discover the block devices instead of the raw devices

alter system set asm_diskstring='/dev/sdc8','/dev/sdc9' scope=spfile;

3. Shutdown the database instances, ASM instances

4. Remove udev rules for asm disk and add new udev rules for block devices

KERNEL=="sdc[8-9]*", OWNER="oracle", GROUP="oinstall", MODE="660"

5. Restart the udev services and then the asm instnaces. On the ASM alert log verify the block devices are used

NOTE: Assigning number (1,0) to disk (/dev/sdc8)
NOTE: Assigning number (2,0) to disk (/dev/sdc9)

5. New paths on the asm disk view.

MOUNT_ST STATE   NAME            PATH
-------- ------- --------------- ---------
CACHED   NORMAL  DATA_0000       /dev/sdc8
CACHED   NORMAL  FLASH_0000      /dev/sdc9

A! Help

Labels

Thursday, July 1, 2010

Moving OCR, Vote and ASM from Raw Devices to Block Devices

About Me

Downloads

Quick Response

Popular

Blog Archive

Total Pageviews

Followers

Oracle Documentation