This post list steps specific to moving ASMLib disk to block devices in two nodes 11gR1 RAC environment. There's an earlier post which shows steps involved with
moving ASMLib to block devices in a 11gR2 standalone setup. Most of the steps are similar as the
earlier post but in the RAC environment the moving from ASMLib to block device could be done in a rolling fashion reducing down time.
1. Create a pfile from the asm spfile. This is more for protection (to start ASM if there's a mistake in asm_string and can't get the ASM up with spfile) and is not part of the moving process.
SQL> create pfile='/home/oracle/asmpfile.ora' from spfile;
File created.
more asmpfile.ora
+ASM1.__oracle_base='/opt/app/oracle'#ORACLE_BASE set from environment
+ASM2.__oracle_base='/opt/app/oracle'#ORACLE_BASE set from environment
+ASM1.asm_diskgroups='DATA','FLASH'
+ASM2.asm_diskgroups='DATA','FLASH'
*.cluster_database=true
*.diagnostic_dest='/opt/app/oracle'
+ASM2.instance_number=2
+ASM1.instance_number=1
*.instance_type='asm'
*.large_pool_size=12M
2. Current ASM disks
SQL> select name,label,path from v$asm_disk;
NAME LABEL PATH
---------- ---------- ----------
DATA DATA ORCL:DATA
FLASH FLASH ORCL:FLASH
3. It is possible to dismount diskgroup in one node and test the procedure before applying the changes permanently.
SQL> alter diskgroup flash dismount
From ASM alert log
SQL> alter diskgroup flash dismount
NOTE: cache dismounting group 2/0x73784638 (FLASH)
Fri Jun 22 13:29:49 2012
kjbdomdet send to node 1
detach from dom 2, sending detach message to node 1
Fri Jun 22 13:29:49 2012
NOTE: detached from domain 2
NOTE: cache dismounted group 2/0x73784638 (FLASH)
kfdp_dismount(): 9
Fri Jun 22 13:29:49 2012
kfdp_dismountBg(): 9
NOTE: De-assigning number (2,0) from disk (ORCL:FLASH)
SUCCESS: diskgroup FLASH was dismounted
SUCCESS: alter diskgroup flash dismount
Status of ASM disks accross cluster which shows FLASH disgroup in node 1 dismounted
SQL> select inst_id,name,state from gv$asm_diskgroup;
INST_ID NAME STATE
------- --------- -----------
2 DATA MOUNTED
2 FLASH MOUNTED
1 DATA MOUNTED
1 FLASH DISMOUNTED
4. Query the ASMLib disk and find out the block device used for creating ASMLib disk
# /etc/init.d/oracleasm querydisk -d FLASH
Disk "FLASH" is a valid ASM disk on device [8, 49]
ls -l /dev/sd*
...
brw-r----- 1 root disk 8, 32 Jun 20 17:16 /dev/sdc
brw-r----- 1 root disk 8, 33 Jun 20 17:17 /dev/sdc1
brw-r----- 1 root disk 8, 48 Jun 20 17:16 /dev/sdd
brw-r----- 1 root disk 8, 49 Jun 20 17:17 /dev/sdd1
Change the ownership of the block device
chown oracle:dba /dev/sdd1
ls -l /dev/sd*
...
brw-r----- 1 root disk 8, 32 Jun 20 17:16 /dev/sdc
brw-r----- 1 root disk 8, 33 Jun 20 17:17 /dev/sdc1
brw-r----- 1 root disk 8, 48 Jun 20 17:16 /dev/sdd
brw-r----- 1 oracle dba 8, 49 Jun 20 17:17 /dev/sdd1
5. Change the asm_diskstring to include the block device and mount the flash diskgroup
SQL> alter system set asm_diskstring ='ORCL:DATA','/dev/sdd1' scope=memory;
System altered.
Mount the FLASH diskgroup and verify block device is used for ASM diskgroup
SQL> alter diskgroup flash mount
NOTE: cache registered group FLASH number=2 incarn=0x73784644
NOTE: cache began mount (not first) of group FLASH number=2 incarn=0x73784644
NOTE: Assigning number (2,0) to disk (/dev/sdd1)
kfdp_query(FLASH): 11
Fri Jun 22 13:35:46 2012
kfdp_queryBg(): 11
NOTE: cache opening disk 0 of grp 2: FLASH path:/dev/sdd1
NOTE: F1X0 found on disk 0 fcn 0.0
NOTE: cache mounting (not first) group 2/0x73784644 (FLASH)
Fri Jun 22 13:35:46 2012
kjbdomatt send to node 1
From the data views
SQL> select inst_id,name,label,path from gv$asm_disk;
INST_ID NAME LABEL PATH
---------- ---------- ---------- ----------
1 DATA DATA ORCL:DATA
1 FLASH /dev/sdd1
2 DATA DATA ORCL:DATA
2 FLASH FLASH ORCL:FLASH
6. Do the same with the other diskgroup. Query the disk to find out the block device and change the ownership of the block device so oracle process could write to it directly
/etc/init.d/oracleasm querydisk -d data
Disk "DATA" is a valid ASM disk on device [8, 33]
# chown oracle:dba /dev/sdc1
ls /dev/sd*
...
brw-r----- 1 root disk 8, 32 Jun 20 17:16 /dev/sdc
brw-r----- 1 oracle dba 8, 33 Jun 20 17:17 /dev/sdc1
brw-r----- 1 root disk 8, 48 Jun 20 17:16 /dev/sdd
brw-r----- 1 oracle dba 8, 49 Jun 22 13:39 /dev/sdd1
7. Unmount all the diskgroups in the node and set the asm_diskstring. Trying to set asm_diskstring while one is mounted would result in an error.
See earlier post's step 9
SQL> alter diskgroup flash dismount;
Diskgroup altered.
SQL> alter diskgroup data dismount;
Diskgroup altered.
SQL> alter system set asm_diskstring ='/dev/sdc1','/dev/sdd1' scope=memory;
System altered.
SQL> alter diskgroup data mount;
Diskgroup altered.
SQL> alter diskgroup flash mount;
Diskgroup altered.
From ASM alert log
NOTE: cache registered group DATA number=1 incarn=0x7378464e
NOTE: cache began mount (not first) of group DATA number=1 incarn=0x7378464e
NOTE: Assigning number (1,0) to disk (/dev/sdc1)
kfdp_query(DATA): 16
Fri Jun 22 13:41:12 2012
kfdp_queryBg(): 16
NOTE: cache opening disk 0 of grp 1: DATA path:/dev/sdc1
NOTE: F1X0 found on disk 0 fcn 0.0
NOTE: cache mounting (not first) group 1/0x7378464E (DATA)
Fri Jun 22 13:41:12 2012
kjbdomatt send to node 1
...
...
SQL> alter diskgroup flash mount
NOTE: cache registered group FLASH number=2 incarn=0x73784652
NOTE: cache began mount (not first) of group FLASH number=2 incarn=0x73784652
NOTE: Assigning number (2,0) to disk (/dev/sdd1)
kfdp_query(FLASH): 19
kfdp_queryBg(): 19
NOTE: cache opening disk 0 of grp 2: FLASH path:/dev/sdd1
NOTE: F1X0 found on disk 0 fcn 0.0
NOTE: cache mounting (not first) group 2/0x73784652 (FLASH)
kjbdomatt send to node 1
Confirm first node's ASM diskgroups are now using block devices
SQL> select inst_id,name,label,path from gv$asm_disk;
INST_ID NAME LABEL PATH
---------- ---------- ---------- ----------
2 DATA DATA ORCL:DATA
2 FLASH FLASH ORCL:FLASH
1 DATA /dev/sdc1
1 FLASH /dev/sdd1
8.
Create udev rules so the block device ownerships are persistent across reboots.
# ASM DATA
KERNEL=="sdc[1]", OWNER="oracle", GROUP="dba", MODE="660"
# ASM FLASH
KERNEL=="sdd[1]", OWNER="oracle", GROUP="dba", MODE="660"
9. Set the new asm_diskstring values to ASM spfile
SQL> alter system set asm_diskstring ='/dev/sdc1','/dev/sdd1' scope=both;
10. Unload the oracleasm module. To unload the oracleasm module the ASM instance must be shutdown.
lsmod | grep oracleasm
oracleasm 84136 1
# /etc/init.d/oracleasm stop
Dropping Oracle ASMLib disks: [ OK ]
Shutting down the Oracle ASMLib driver: [FAILED]
srvctl stop asm -n rac1
# /etc/init.d/oracleasm stop
Dropping Oracle ASMLib disks: [ OK ]
Shutting down the Oracle ASMLib driver: [ OK ]
# /etc/init.d/oracleasm disable
Writing Oracle ASM library driver configuration: done
Dropping Oracle ASMLib disks: [ OK ]
Shutting down the Oracle ASMLib driver: [ OK ]
# chkconfig oracleasm off
11. Start the ASM instance on the node
srvctl start asm -n rac1
Verify the block devices are used by monitoring the ASM alert log
NOTE: cache opening disk 0 of grp 1: DATA path:/dev/sdc1
NOTE: F1X0 found on disk 0 fcn 0.0
NOTE: cache mounting (not first) group 1/0x737E1A2E (DATA)
kjbdomatt send to node 1
NOTE: attached to recovery domain 1
NOTE: LGWR attempting to mount thread 1 for diskgroup 1
NOTE: LGWR mounted thread 1 for disk group 1
NOTE: opening chunk 1 at fcn 0.69396 ABA
NOTE: seq=102 blk=4866
NOTE: cache mounting group 1/0x737E1A2E (DATA) succeeded
NOTE: cache ending mount (success) of group DATA number=1 incarn=0x737e1a2e
kfdp_query(FLASH): 4
kfdp_queryBg(): 4
NOTE: cache opening disk 0 of grp 2: FLASH path:/dev/sdd1
NOTE: F1X0 found on disk 0 fcn 0.0
NOTE: cache mounting (not first) group 2/0x737E1A2F (FLASH)
Start the database instance on the node
12. Next step is to mount the asm diskgroup on the remaining node with the block devices. Stop the database instance running on the node and change the ownership of the block devices. These were identified earlier
# chown oracle:dba /dev/sdc1
# chown oracle:dba /dev/sdd1
13. Since asm_diskstring was set as both no need to make any changes simply dismount and mount one diskgroup to test the configuration
SQL> alter diskgroup flash dismount;
Diskgroup altered.
SQL> alter diskgroup flash mount;
Diskgroup altered.
14. Create a udev rules file in this node as well
# ASM DATA
KERNEL=="sdc[1]", OWNER="oracle", GROUP="dba", MODE="660"
# ASM FLASH
KERNEL=="sdd[1]", OWNER="oracle", GROUP="dba", MODE="660"
15. Stop the asm instance and unload the oracleasm module
srvctl stop asm -n rac2
# /etc/init.d/oracleasm stop
Dropping Oracle ASMLib disks: [ OK ]
Shutting down the Oracle ASMLib driver: [ OK ]
# /etc/init.d/oracleasm disable
Writing Oracle ASM library driver configuration: done
Dropping Oracle ASMLib disks: [ OK ]
Shutting down the Oracle ASMLib driver: [ OK ]
#chkconfig oracleasm off
15. Start asm instance and verify block devices being used through ASM alert log
srvctl start asm -n rac2
NOTE: cache opening disk 0 of grp 1: DATA path:/dev/sdc1
NOTE: F1X0 found on disk 0 fcn 0.0
NOTE: cache mounting (not first) group 1/0x73701F4C (DATA)
kjbdomatt send to node 0
NOTE: attached to recovery domain 1
NOTE: LGWR attempting to mount thread 2 for diskgroup 1
NOTE: LGWR mounted thread 2 for disk group 1
NOTE: opening chunk 2 at fcn 0.69424 ABA
NOTE: seq=86 blk=4843
NOTE: cache mounting group 1/0x73701F4C (DATA) succeeded
NOTE: cache ending mount (success) of group DATA number=1 incarn=0x73701f4c
kfdp_query(FLASH): 4
kfdp_queryBg(): 4
NOTE: cache opening disk 0 of grp 2: FLASH path:/dev/sdd1
NOTE: F1X0 found on disk 0 fcn 0.0
NOTE: cache mounting (not first) group 2/0x73701F4D (FLASH)
Querying the data view
SQL> select inst_id,name,label,path from gv$asm_disk order by 1;
INST_ID NAME LABEL PATH
---------- ---------- ---------- ----------
1 DATA /dev/sdc1
1 FLASH /dev/sdd1
2 DATA /dev/sdc1
2 FLASH /dev/sdd1
16. As the last step uninstall the oracleasm libraries
# rpm -e oracleasmlib-2.0.4-1.el5
# rpm -e oracleasm-2.6.18-308.el5-2.0.5-1.el5
# rpm -e oracleasm-support-2.1.3-1.el5
Related Posts
Migrating block devices using ASM instance to ASMLib
Moving ASMLib disk to block devices (Non-ASMLib) in 11gR2 Standalone
Useful Metalink Notes
How To Migrate ASMLIB Devices To Block Devices (Non-ASMLIB)?
[ID 567508.1]
