1. Create a pfile from the asm spfile. This is more for protection (to start ASM if there's a mistake in asm_string and can't get the ASM up with spfile) and is not part of the moving process.
SQL> create pfile='/home/oracle/asmpfile.ora' from spfile; File created. more asmpfile.ora +ASM1.__oracle_base='/opt/app/oracle'#ORACLE_BASE set from environment +ASM2.__oracle_base='/opt/app/oracle'#ORACLE_BASE set from environment +ASM1.asm_diskgroups='DATA','FLASH' +ASM2.asm_diskgroups='DATA','FLASH' *.cluster_database=true *.diagnostic_dest='/opt/app/oracle' +ASM2.instance_number=2 +ASM1.instance_number=1 *.instance_type='asm' *.large_pool_size=12M2. Current ASM disks
SQL> select name,label,path from v$asm_disk; NAME LABEL PATH ---------- ---------- ---------- DATA DATA ORCL:DATA FLASH FLASH ORCL:FLASH3. It is possible to dismount diskgroup in one node and test the procedure before applying the changes permanently.
SQL> alter diskgroup flash dismountFrom ASM alert log
SQL> alter diskgroup flash dismount NOTE: cache dismounting group 2/0x73784638 (FLASH) Fri Jun 22 13:29:49 2012 kjbdomdet send to node 1 detach from dom 2, sending detach message to node 1 Fri Jun 22 13:29:49 2012 NOTE: detached from domain 2 NOTE: cache dismounted group 2/0x73784638 (FLASH) kfdp_dismount(): 9 Fri Jun 22 13:29:49 2012 kfdp_dismountBg(): 9 NOTE: De-assigning number (2,0) from disk (ORCL:FLASH) SUCCESS: diskgroup FLASH was dismounted SUCCESS: alter diskgroup flash dismountStatus of ASM disks accross cluster which shows FLASH disgroup in node 1 dismounted
SQL> select inst_id,name,state from gv$asm_diskgroup;
INST_ID NAME STATE
------- --------- -----------
2 DATA MOUNTED
2 FLASH MOUNTED
1 DATA MOUNTED
1 FLASH DISMOUNTED
4. Query the ASMLib disk and find out the block device used for creating ASMLib disk# /etc/init.d/oracleasm querydisk -d FLASH Disk "FLASH" is a valid ASM disk on device [8, 49] ls -l /dev/sd* ... brw-r----- 1 root disk 8, 32 Jun 20 17:16 /dev/sdc brw-r----- 1 root disk 8, 33 Jun 20 17:17 /dev/sdc1 brw-r----- 1 root disk 8, 48 Jun 20 17:16 /dev/sdd brw-r----- 1 root disk 8, 49 Jun 20 17:17 /dev/sdd1Change the ownership of the block device
chown oracle:dba /dev/sdd1 ls -l /dev/sd* ... brw-r----- 1 root disk 8, 32 Jun 20 17:16 /dev/sdc brw-r----- 1 root disk 8, 33 Jun 20 17:17 /dev/sdc1 brw-r----- 1 root disk 8, 48 Jun 20 17:16 /dev/sdd brw-r----- 1 oracle dba 8, 49 Jun 20 17:17 /dev/sdd15. Change the asm_diskstring to include the block device and mount the flash diskgroup
SQL> alter system set asm_diskstring ='ORCL:DATA','/dev/sdd1' scope=memory; System altered.Mount the FLASH diskgroup and verify block device is used for ASM diskgroup
SQL> alter diskgroup flash mount NOTE: cache registered group FLASH number=2 incarn=0x73784644 NOTE: cache began mount (not first) of group FLASH number=2 incarn=0x73784644 NOTE: Assigning number (2,0) to disk (/dev/sdd1) kfdp_query(FLASH): 11 Fri Jun 22 13:35:46 2012 kfdp_queryBg(): 11 NOTE: cache opening disk 0 of grp 2: FLASH path:/dev/sdd1 NOTE: F1X0 found on disk 0 fcn 0.0 NOTE: cache mounting (not first) group 2/0x73784644 (FLASH) Fri Jun 22 13:35:46 2012 kjbdomatt send to node 1From the data views
SQL> select inst_id,name,label,path from gv$asm_disk;
INST_ID NAME LABEL PATH
---------- ---------- ---------- ----------
1 DATA DATA ORCL:DATA
1 FLASH /dev/sdd1
2 DATA DATA ORCL:DATA
2 FLASH FLASH ORCL:FLASH
6. Do the same with the other diskgroup. Query the disk to find out the block device and change the ownership of the block device so oracle process could write to it directly/etc/init.d/oracleasm querydisk -d data Disk "DATA" is a valid ASM disk on device [8, 33] # chown oracle:dba /dev/sdc1 ls /dev/sd* ... brw-r----- 1 root disk 8, 32 Jun 20 17:16 /dev/sdc brw-r----- 1 oracle dba 8, 33 Jun 20 17:17 /dev/sdc1 brw-r----- 1 root disk 8, 48 Jun 20 17:16 /dev/sdd brw-r----- 1 oracle dba 8, 49 Jun 22 13:39 /dev/sdd17. Unmount all the diskgroups in the node and set the asm_diskstring. Trying to set asm_diskstring while one is mounted would result in an error. See earlier post's step 9
SQL> alter diskgroup flash dismount; Diskgroup altered. SQL> alter diskgroup data dismount; Diskgroup altered. SQL> alter system set asm_diskstring ='/dev/sdc1','/dev/sdd1' scope=memory; System altered. SQL> alter diskgroup data mount; Diskgroup altered. SQL> alter diskgroup flash mount; Diskgroup altered.From ASM alert log
NOTE: cache registered group DATA number=1 incarn=0x7378464e NOTE: cache began mount (not first) of group DATA number=1 incarn=0x7378464e NOTE: Assigning number (1,0) to disk (/dev/sdc1) kfdp_query(DATA): 16 Fri Jun 22 13:41:12 2012 kfdp_queryBg(): 16 NOTE: cache opening disk 0 of grp 1: DATA path:/dev/sdc1 NOTE: F1X0 found on disk 0 fcn 0.0 NOTE: cache mounting (not first) group 1/0x7378464E (DATA) Fri Jun 22 13:41:12 2012 kjbdomatt send to node 1 ... ... SQL> alter diskgroup flash mount NOTE: cache registered group FLASH number=2 incarn=0x73784652 NOTE: cache began mount (not first) of group FLASH number=2 incarn=0x73784652 NOTE: Assigning number (2,0) to disk (/dev/sdd1) kfdp_query(FLASH): 19 kfdp_queryBg(): 19 NOTE: cache opening disk 0 of grp 2: FLASH path:/dev/sdd1 NOTE: F1X0 found on disk 0 fcn 0.0 NOTE: cache mounting (not first) group 2/0x73784652 (FLASH) kjbdomatt send to node 1Confirm first node's ASM diskgroups are now using block devices
SQL> select inst_id,name,label,path from gv$asm_disk;
INST_ID NAME LABEL PATH
---------- ---------- ---------- ----------
2 DATA DATA ORCL:DATA
2 FLASH FLASH ORCL:FLASH
1 DATA /dev/sdc1
1 FLASH /dev/sdd1
8. Create udev rules so the block device ownerships are persistent across reboots.# ASM DATA KERNEL=="sdc[1]", OWNER="oracle", GROUP="dba", MODE="660" # ASM FLASH KERNEL=="sdd[1]", OWNER="oracle", GROUP="dba", MODE="660"9. Set the new asm_diskstring values to ASM spfile
SQL> alter system set asm_diskstring ='/dev/sdc1','/dev/sdd1' scope=both;10. Unload the oracleasm module. To unload the oracleasm module the ASM instance must be shutdown.
lsmod | grep oracleasm oracleasm 84136 1 # /etc/init.d/oracleasm stop Dropping Oracle ASMLib disks: [ OK ] Shutting down the Oracle ASMLib driver: [FAILED] srvctl stop asm -n rac1 # /etc/init.d/oracleasm stop Dropping Oracle ASMLib disks: [ OK ] Shutting down the Oracle ASMLib driver: [ OK ] # /etc/init.d/oracleasm disable Writing Oracle ASM library driver configuration: done Dropping Oracle ASMLib disks: [ OK ] Shutting down the Oracle ASMLib driver: [ OK ] # chkconfig oracleasm off11. Start the ASM instance on the node
srvctl start asm -n rac1Verify the block devices are used by monitoring the ASM alert log
NOTE: cache opening disk 0 of grp 1: DATA path:/dev/sdc1 NOTE: F1X0 found on disk 0 fcn 0.0 NOTE: cache mounting (not first) group 1/0x737E1A2E (DATA) kjbdomatt send to node 1 NOTE: attached to recovery domain 1 NOTE: LGWR attempting to mount thread 1 for diskgroup 1 NOTE: LGWR mounted thread 1 for disk group 1 NOTE: opening chunk 1 at fcn 0.69396 ABA NOTE: seq=102 blk=4866 NOTE: cache mounting group 1/0x737E1A2E (DATA) succeeded NOTE: cache ending mount (success) of group DATA number=1 incarn=0x737e1a2e kfdp_query(FLASH): 4 kfdp_queryBg(): 4 NOTE: cache opening disk 0 of grp 2: FLASH path:/dev/sdd1 NOTE: F1X0 found on disk 0 fcn 0.0 NOTE: cache mounting (not first) group 2/0x737E1A2F (FLASH)Start the database instance on the node
12. Next step is to mount the asm diskgroup on the remaining node with the block devices. Stop the database instance running on the node and change the ownership of the block devices. These were identified earlier
# chown oracle:dba /dev/sdc1 # chown oracle:dba /dev/sdd113. Since asm_diskstring was set as both no need to make any changes simply dismount and mount one diskgroup to test the configuration
SQL> alter diskgroup flash dismount; Diskgroup altered. SQL> alter diskgroup flash mount; Diskgroup altered.14. Create a udev rules file in this node as well
# ASM DATA KERNEL=="sdc[1]", OWNER="oracle", GROUP="dba", MODE="660" # ASM FLASH KERNEL=="sdd[1]", OWNER="oracle", GROUP="dba", MODE="660"15. Stop the asm instance and unload the oracleasm module
srvctl stop asm -n rac2 # /etc/init.d/oracleasm stop Dropping Oracle ASMLib disks: [ OK ] Shutting down the Oracle ASMLib driver: [ OK ] # /etc/init.d/oracleasm disable Writing Oracle ASM library driver configuration: done Dropping Oracle ASMLib disks: [ OK ] Shutting down the Oracle ASMLib driver: [ OK ] #chkconfig oracleasm off15. Start asm instance and verify block devices being used through ASM alert log
srvctl start asm -n rac2 NOTE: cache opening disk 0 of grp 1: DATA path:/dev/sdc1 NOTE: F1X0 found on disk 0 fcn 0.0 NOTE: cache mounting (not first) group 1/0x73701F4C (DATA) kjbdomatt send to node 0 NOTE: attached to recovery domain 1 NOTE: LGWR attempting to mount thread 2 for diskgroup 1 NOTE: LGWR mounted thread 2 for disk group 1 NOTE: opening chunk 2 at fcn 0.69424 ABA NOTE: seq=86 blk=4843 NOTE: cache mounting group 1/0x73701F4C (DATA) succeeded NOTE: cache ending mount (success) of group DATA number=1 incarn=0x73701f4c kfdp_query(FLASH): 4 kfdp_queryBg(): 4 NOTE: cache opening disk 0 of grp 2: FLASH path:/dev/sdd1 NOTE: F1X0 found on disk 0 fcn 0.0 NOTE: cache mounting (not first) group 2/0x73701F4D (FLASH)Querying the data view
SQL> select inst_id,name,label,path from gv$asm_disk order by 1;
INST_ID NAME LABEL PATH
---------- ---------- ---------- ----------
1 DATA /dev/sdc1
1 FLASH /dev/sdd1
2 DATA /dev/sdc1
2 FLASH /dev/sdd1
16. As the last step uninstall the oracleasm libraries# rpm -e oracleasmlib-2.0.4-1.el5 # rpm -e oracleasm-2.6.18-308.el5-2.0.5-1.el5 # rpm -e oracleasm-support-2.1.3-1.el5Related Posts
Migrating block devices using ASM instance to ASMLib
Moving ASMLib disk to block devices (Non-ASMLib) in 11gR2 Standalone
Useful Metalink Notes
How To Migrate ASMLIB Devices To Block Devices (Non-ASMLIB)? [ID 567508.1]
