Using Openfiler to Create iSCSI Targets on VirtualBox

This post list steps for installing openfiler and creating iscsi targets that could be used as storage for Oracle DBs. It list steps for creating single path iscsi targets and multipath iscsi targets. Lastly it shows how to attach iscsi targets directly to VirtualBox.

    Installing openfiler
    Creating iSCSI Targets
    Single Path iSCSI Target
    Multipath iSCSI Target
    Attach iSCSI Target to VirtualBox from Host

Installing openfiler
Download the latest openfiler software from openfiler.comThe Openfiler software is installed on a VirtualBox. To install openfiler on VirtulBox add the downloaded ISO image to the CD-ROM and set the boot order such that CD-ROM is higher than hard disk. Two virtual disks are created in the VirtualBox (see image below). One is to install the openfiler software (OS) (openfiler.vdi) . Then another disk that will be used to create iscsi targets (scsidisk.vdi).

Make sure the disk used for openfiler OS install is larger than 8GB even though openfiler states 8GB is the minimum size needed. Even when disks is slightly larger than 8GB installation still fails.

Once the disks are in order boot from the openfiler ISO image. Screenshots below give highlights (not all the steps).

Only select the disk designated for installing the OS. Uncheck the disk designated to create iscsi targets.

Set the IPs for the openfiler setup. In this case static IP configuration has been done as oppose to DHCP. As there's only one NIC on the openfiler server, the iscsi targets will be single path. If multipath iscsi targets are required, add another NIC to the openfiler server. Refer multipath iSCSI target step for more.

Once the installation is complete login to the web administration portal. The URL is listed at the end of the installation process.

Login to the web console (default username is "openfiler" and password is "password") and go to services tab and enable and start iSCSI target.

Openfiler installation is complete and ready to create iscsi targets.



Creating iSCSI Targets
To create iscsi targets login to the web console if already not logged in. The iscsi targets created on openfiler server is attached on a server with hostname city7. In this case the city7 is the iscsi initiator.

In order for city7 server to access the iscsi target add the city7 to network access configuration by selecting system tab.

Create the iscsi target using the disk designated for it. This could be done by selecting volume tag followed by the block device link and finally clicking on the actual disk. In this case the disk used for iscsi target is /dev/sdb.

This will open the create partition page. The edit partition section will show 100% storage. On the create partition section select the required amount of storage (in this case 10GB was chosen) and create a physical partition by clicking on the create button.

New partition will be shown on the edit partition section. Even though 10GB was selected, created partition was smaller than 10GB.

Create a volume group using the partition created earlier. In this case the volume group is named asmdg.

Once created new volume group will appear under volume group management section.

Next step is to create a volume within the volume group. For this select add volume link and create a volume of required size.

Once created the volume information will be listed under the associated volume group.

The next step is to create an iscsi target to the volume. This is done through the iscsi target link. Select target configuration tab and then creating a iSCSI Qualified Name (IQN) for the target. Below screenshot shows creating a IQN. Openfiler auto generate an IQN but it could be renamed as desired. In this case the IQN is iqn.2006-01.com.openfiler:asm.newdata

Select LUN mapping tap and map the earlier created IQN to the LUN (volume). The LUN name is the description of the volume created earlier (inside orange box), not the name of the volume.

Once the mapping is complete.

Finally update the network access control list (ACL) by allowing access to the iscsi target. As mentioned earlier, the server that will be accessing the iscsi target is called city7.

There's no need to setup CHAP authentication unless password protection is needed for the iscsi targets.
Creating iscsi target is now complete. Next step is to consume the iscsi target.

Single Path iSCSI Target
The city7 server that will consume the iscsi target is running RHEL 7.4 (3.10.0-693.el7.x86_64). In order to access iscsi target install the following RPM

iscsi-initiator-utils-6.2.0.874-4.el7.x86_64

Have the iscsid service enabled. If the service is down it's auto started during the target discovery. Nevertheless, it's best to have it enabled so auto starts across reboots.

# systemctl enable iscsid.service
# systemctl list-unit-files | grep iscsid | grep enable
iscsid.service                                enabled

systemctl status iscsid
â iscsid.service - Open-iSCSI
   Loaded: loaded (/usr/lib/systemd/system/iscsid.service; enabled; vendor preset: disabled)
   Active: active (running)

Next step is to discover the iscsi target

# iscsiadm -m discovery -t st -p 192.168.0.94 (or long form iscsiadm -m discovery -t sendtargets -p 192.168.0.94)
192.168.0.94:3260,1 iqn.2006-01.com.openfiler:asm.newdata

If for some reason the discovery fails as below then check the network access configuration and network ACL. The network/hostname must be considered in conjunction with the netmask.

#  iscsiadm -m discovery -t st -p 192.168.0.94
iscsiadm: No portals found

To confirm this is down to network issue comment out the /etc/initiators.allow and /etc/initiators.deny entries and try again (this is an auto generated file and not advised to manually edit). For example if the IP of the iscsi initiator (city7 in this case) is 192.168.0.85 and netmask was set to 255.255.255.0 (instead of 255.255.255.255 which will restrict access to all but just one IP) then /etc/initiators.allow entry would be iqn.2006-01.com.openfiler:asm.newdata 192.168.0.85/24. This is not what's expected and could result in iscsi discovery failure.
If the targets are discovered without any issue then manually login to the target

# iscsiadm -m node -T iqn.2006-01.com.openfiler:asm.newdata -p 192.168.0.94 -l
Logging in to [iface: default, target: iqn.2006-01.com.openfiler:asm.newdata, portal: 192.168.0.94,3260] (multiple)
Login to [iface: default, target: iqn.2006-01.com.openfiler:asm.newdata, portal: 192.168.0.94,3260] successful.

This will show a new disk in the scsi initiator server (city7).

 ls -l /dev/disk/by-path/
lrwxrwxrwx. 1 root root  9 Mar 22 15:23 ip-192.168.0.94:3260-iscsi-iqn.2006-01.com.openfiler:asm.newdata-lun-0 -> ../../sdd

fdisk will now show a new disk as a result of the iscsi target

fdisk -l

...
Disk /dev/sdd: 10.2 GB, 10234101760 bytes, 19988480 sectors
Units = sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes

Run the following so that iscsi initiator auto logins to the target accrsso reboots

iscsiadm -m node -T iqn.2006-01.com.openfiler:asm.newdata -p 192.168.0.94 --op update -n node.startup -v automatic

If for some reason the auto login need to be changed to manual login use the following command

iscsiadm -m node -T iqn.2006-01.com.openfiler:asm.newdata -p 192.168.0.88 --op update -n node.startup -v manaul

Since target discovery and login is working fine, create a partition on the new disk and a appropriate udev rule.

fdisk /dev/sdd

fdisk -l
Disk /dev/sdd: 10.2 GB, 10234101760 bytes, 19988480 sectors
...
   Device Boot      Start         End      Blocks   Id  System
/dev/sdd1            2048    19988479     9993216   83  Linux

To create the udev rule get the scsi id of the disk

/usr/lib/udev/scsi_id -g -u -d /dev/sdd
14f504e46494c4552484a535350652d4c774b6f2d5a597173

With the scsi id in hand create the udev rule for Oracle ASM with role separation as below

KERNEL=="sd?1",SUBSYSTEM=="block",PROGRAM=="/usr/lib/udev/scsi_id -g -u -d /dev/$parent",RESULT=="14f504e46494c4552484a535350652d4c774b6f2d5a597173",SYMLINK+="oracleasm/ndata1",OWNER="grid", GROUP="asmadmin", MODE="0660"

Run the udev rules and check new udev taking effect

ls -l /dev/oracleasm/*
lrwxrwxrwx. 1 root root 7 Mar 22 15:44 /dev/oracleasm/data1 -> ../sdb1
lrwxrwxrwx. 1 root root 7 Mar 22 15:44 /dev/oracleasm/fra1 -> ../sdc1

# udevadm trigger --type=devices --action=change

# ls -l /dev/oracleasm/*
lrwxrwxrwx. 1 root root 7 Mar 22 15:45 /dev/oracleasm/data1 -> ../sdb1
lrwxrwxrwx. 1 root root 7 Mar 22 15:45 /dev/oracleasm/fra1 -> ../sdc1
lrwxrwxrwx. 1 root root 7 Mar 22 15:45 /dev/oracleasm/ndata1 -> ../sdd1

Next step is to add the disk to ASM. This would conclude the iscsi single path setup. For completion of the post following two commands are also mentioned.
To logout of a target use the following command

# iscsiadm -m node -u -T iqn.2006-01.com.openfiler:asm.newdata -p 192.168.0.94
Logging out of session [sid: 2, target: iqn.2006-01.com.openfiler:asm.newdata, portal: 192.168.0.94,3260]
Logout of [sid: 2, target: iqn.2006-01.com.openfiler:asm.newdata, portal: 192.168.0.94,3260] successful.

To delete a target use the following command. This will only remove the entry on the iscsi initiator, not the actual iscsi target.

# iscsiadm -m node -o delete -T iqn.2006-01.com.openfiler:asm.newdata -p 192.168.0.94

Multipath iSCSI Target
Having a single network path to the iscsi target creates a single point of failure. The solution is to have multiple network paths (preferably from different network segments without having any common hardware. i.e NIC, switches etc) to the same iscsi target. Following diagram shows the multipath configuration between iscsi target (openfiler) and the iscsi initiator (city7).

The same iscsi target is exposed via IPs 192.168.0.94 and 192.168.1.88. These could be configured during the opefiler installation or later on.

The iscsi initiator also has two NICs with IPs 192.168.0.85 and 192.168.1.85 capable of accessing the iscsi target through different network path. Once two NICs are configured enable network access through open filer web console. This need to happen on network access configuration (under system tab)

and network ACL (volume -> iscsi targets).

Apart from these changes (having multiple NICs) there's no other changes on the iscsi target.
On the iscsi initiator install multipath software in addition to the iscsi software mentioned in singe path iscsi creation.

device-mapper-multipath-libs-0.4.9-111.el7.x86_64.rpm
device-mapper-multipath-0.4.9-111.el7.x86_64.rpm

Enable the multipath service and start it along with the iscsid service if iscsid is not already started.

systemctl enable multipathd.service
systemctl start multipathd.service
systemctl status multipathd.service

If /etc/multipath.conf file doesn't exists create a one from the provided template using the following command.

mpathconf --enable

This will create an multipath.conf file with all the section either commented or empty. Add following to default section and blacklist section. Among other things the default section tells multipathing to use user friendly names and round robin as the path policy.

cat /etc/multipath.conf | grep -A5 defaults
defaults {
        find_multipaths yes
        user_friendly_names yes
        failback immediate
        path_selector   "round-robin 0"
}

Add the following to the blacklist section

blacklist {
  devnode "^(ram|raw|loop|fd|md|dm-|sr|scd|st)[0-9]*"
  devnode "^hd[a-z]"
  devnode "^cciss!c[0-9]d[0-9]*[p[0-9]*]"
}

Next login to the iscsi targets using both paths.

iscsiadm -m node -T iqn.2006-01.com.openfiler:asm.newdata -p 192.168.0.94 -l
Logging in to [iface: default, target: iqn.2006-01.com.openfiler:asm.newdata, portal: 192.168.0.94,3260] (multiple)
Login to [iface: default, target: iqn.2006-01.com.openfiler:asm.newdata, portal: 192.168.0.94,3260] successful.

iscsiadm -m node -T iqn.2006-01.com.openfiler:asm.newdata -p 192.168.1.88 -l
Logging in to [iface: default, target: iqn.2006-01.com.openfiler:asm.newdata, portal: 192.168.1.88,3260] (multiple)
Login to [iface: default, target: iqn.2006-01.com.openfiler:asm.newdata, portal: 192.168.1.88,3260] successful.

If login is successful set auto login for both paths

# iscsiadm -m node -T iqn.2006-01.com.openfiler:asm.newdata -p 192.168.0.94 --op update -n node.startup -v automatic
# iscsiadm -m node -T iqn.2006-01.com.openfiler:asm.newdata -p 192.168.1.88 --op update -n node.startup -v automatic

Check multipath device are active

mpathf (14f504e46494c4552484a535350652d4c774b6f2d5a597173) dm-0 OPNFILER,VIRTUAL-DISK
size=9.5G features='0' hwhandler='0' wp=rw
|-+- policy='round-robin 0' prio=1 status=active
| `- 11:0:0:0 sde 8:64 active ready running
`-+- policy='round-robin 0' prio=1 status=enabled
  `- 10:0:0:0 sdd 8:48 active ready running

Using disk path it is possible to find out to which path the sd* device belong to

ls -l /dev/disk/by-path/
... ip-192.168.0.94:3260-iscsi-iqn.2006-01.com.openfiler:asm.newdata-lun-0 -> ../../sdd
... ip-192.168.1.88:3260-iscsi-iqn.2006-01.com.openfiler:asm.newdata-lun-0 -> ../../sde

Next step is to create a user friendly name for the multipath device. To do this use WWID (World Wide Identifier) of the mpath device (1365511.1).

/usr/lib/udev/scsi_id -g -u -d /dev/mapper/mpathf
14f504e46494c4552484a535350652d4c774b6f2d5a597173

Add a alias to the multipath device using the WWID obtained by adding following to the /etc/multipath.conf file.

multipaths {
  multipath {
    wwid                    14f504e46494c4552484a535350652d4c774b6f2d5a597173
    alias                   oradata
    path_grouping_policy    failover
  }

ls -l /dev/mapper
lrwxrwxrwx. 1 root root       7 Mar 29 10:54 mpathf -> ../dm-0

Restart the multipath service. Check the mapper directory again to see multipath device name being set to alias specified on the multipath.conf.

systemctl reload multipathd
ls -l /dev/mapper
lrwxrwxrwx. 1 root root       7 Mar 29 10:55 oradata -> ../dm-0

Run fdisk to partition the device (470913.1).This will create a partition in the mapper directory

fdisk /dev/mapper/oradata
/sbin/partprobe
ls -l /dev/mapper
lrwxrwxrwx. 1 root root       7 Mar 29 11:01 oradata -> ../dm-0
lrwxrwxrwx. 1 root root       7 Mar 29 11:01 oradata1 -> ../dm-1

To create a udev rule to set appropriate ownership for the partition use the DM_UUID value. This could be obtained by following command

udevadm info --query=all --name=/dev/mapper/oradata1
...
E: DM_UUID=part1-mpath-14f504e46494c4552484a535350652d4c774b6f2d5a597173
...

Create a udev rule as below using the obtained DM_UUID

KERNEL=="dm-*",ENV{DM_UUID}=="part1-mpath-14f504e46494c4552484a535350652d4c774b6f2d5a597173",OWNER="grid",GROUP="asmadmin",MODE="0660"

Reload the udev rules and verify permission change

ls -l /dev/dm-1
brw-rw----. 1 root disk 253, 1 Mar 29 11:01 /dev/dm-1
/sbin/udevadm control --reload-rules
/sbin/udevadm trigger --type=devices --action=change
ls -l /dev/dm-1
brw-rw----. 1 grid asmadmin 253, 1 Mar 29 11:09 /dev/dm-1

Finally verify multipath is working by downing one of the NICs at the iscsi target. As shown earlier eth0 has the IP 192.168.0.94 (sdd) and eth1 has 192.168.1.88 (sde). Current multipath status is as below

 multipath -ll
oradata (14f504e46494c4552484a535350652d4c774b6f2d5a597173) dm-0 OPNFILER,VIRTUAL-DISK
size=9.5G features='0' hwhandler='0' wp=rw
|-+- policy='round-robin 0' prio=1 status=active
| `- 12:0:0:0 sdd 8:48 active ready running
`-+- policy='round-robin 0' prio=1 status=enabled
  `- 13:0:0:0 sde 8:64 active ready running

To test one path stop the eth0 on iscsi target(opernfiler server)

ifdown eth0

Check the current multipath topology.

oradata (14f504e46494c4552484a535350652d4c774b6f2d5a597173) dm-0 OPNFILER,VIRTUAL-DISK
size=9.5G features='0' hwhandler='0' wp=rw
|-+- policy='round-robin 0' prio=0 status=enabled
| `- 12:0:0:0 sdd 8:48 failed faulty running
`-+- policy='round-robin 0' prio=1 status=active
  `- 13:0:0:0 sde 8:64 active ready running

The path through 192.168.0.94 (sdd) has become failed path through sde has become active. To check the other path start the downed NIC and wait for iscsi path to become active ready and then down the eth1.

openfiler ~]# ifup eth0

city7 ~]# multipath -ll
oradata (14f504e46494c4552484a535350652d4c774b6f2d5a597173) dm-0 OPNFILER,VIRTUAL-DISK
size=9.5G features='0' hwhandler='0' wp=rw
|-+- policy='round-robin 0' prio=1 status=active
| `- 12:0:0:0 sdd 8:48 active ready running
`-+- policy='round-robin 0' prio=1 status=enabled
  `- 13:0:0:0 sde 8:64 active ready running

openfiler ~]# ifdown eth1

city7 ~]# multipath -ll
oradata (14f504e46494c4552484a535350652d4c774b6f2d5a597173) dm-0 OPNFILER,VIRTUAL-DISK
size=9.5G features='0' hwhandler='0' wp=rw
|-+- policy='round-robin 0' prio=1 status=active
| `- 12:0:0:0 sdd 8:48 active ready running
`-+- policy='round-robin 0' prio=0 status=enabled
  `- 13:0:0:0 sde 8:64 failed faulty running

This concludes the using multipath iscsi targets.
If during testing and setup want to logout from all iscsi target paths at once use the following (this will stop multipath working)

iscsiadm -m node -u
Logging out of session [sid: 6, target: iqn.2006-01.com.openfiler:asm.newdata, portal: 192.168.0.94,3260]
Logging out of session [sid: 7, target: iqn.2006-01.com.openfiler:asm.newdata, portal: 192.168.1.88,3260]
Logout of [sid: 6, target: iqn.2006-01.com.openfiler:asm.newdata, portal: 192.168.0.94,3260] successful.
Logout of [sid: 7, target: iqn.2006-01.com.openfiler:asm.newdata, portal: 192.168.1.88,3260] successful.

Attach iSCSI Target to VirtualBox from Host
In the previous cases the iscsi target was attached to the Virtualbox while logged in to the VM. However it is possible to attach the iscsi target to the virtualbox VM from the host server. This is done using the VBoxManage command. Before running the command make sure host server is granted access to the iscsi target.

VBoxManage storageattach city7 --storagectl "SATA" --port 3 --device 0 --type hdd --medium iscsi --server 192.168.0.94 --target "iqn.2006-01.com.openfiler:asm.newdata" --tport 3260
iSCSI disk created. UUID: bc380eb5-e67b-49b7-9602-efbc5d6d8e87

This will be visible on the VirtualBox GUI.

When the VM is booted the iscsi target would be available as a standard block device (/dev/sd*).
To remove the iscsi from the VM run the following

VBoxManage storageattach city7 --storagectl "SATA" --port 3 --device 0 --medium none

This however doesn't remove the iscis target from virtual media manager only from the VM. It will remain an unattached device in the virtual media manager.

As such subsequent attaching of the iscsi target will fail with UUID already exists error.

VBoxManage storageattach city7 --storagectl "SATA" --port 3 --device 0 --type hdd --medium iscsi --server 192.168.0.94 --target "iqn.2006-01.com.openfiler:asm.newdata" --tport 3260
VBoxManage: error: Cannot register the hard disk '192.168.0.94|iqn.2006-01.com.openfiler:asm.newdata' {a866eaf1-f089-4a8d-b793-23028e3a1e14} because a hard disk '192.168.0.94|iqn.2006-01.com.openfiler:asm.newdata' 
with UUID {bc380eb5-e67b-49b7-9602-efbc5d6d8e87} already exists
VBoxManage: error: Details: code NS_ERROR_INVALID_ARG (0x80070057), component VirtualBoxWrap, interface IVirtualBox, callee nsISupports
VBoxManage: error: Context: "CreateMedium(Bstr("iSCSI").raw(), bstrISCSIMedium.raw(), AccessMode_ReadWrite, DeviceType_HardDisk, pMedium2Mount.asOutParam())" at line 608 of file VBoxManageStorageController.cpp

To fix this run the close medium command providing the UUID. This will remove the iscsi target from the virtual manager allowing the iscsi target to be attached again.

VBoxManage closemedium bc380eb5-e67b-49b7-9602-efbc5d6d8e87

This concludes using openfiler to create iSCSI targets on VirtualBox.

Parsing KFOD-00307: asmlib error!! function = [asm_open], error = [0], mesg = [Operation not permitted]

This was a new installation and storage vendor had setup the block devices as below.

ls -l /dev/eql/*
lrwxrwxrwx 1 root root 8 Jan 15 12:00 /dev/eql/clus-01 -> ../dm-16
lrwxrwxrwx 1 root root 8 Jan 15 12:00 /dev/eql/clus-02 -> ../dm-18
lrwxrwxrwx 1 root root 8 Jan 15 12:00 /dev/eql/clus-03 -> ../dm-20

(Showing only the block devices relevant to the post). This was using Dell EqualLogic PS4100x iSCSI SAN and was told to use /dev/eql/* for the database. Though asked "why the softlink and /dev/dm*" was told that's how it setup and have to use the /dev/eql/* softlink for "my work". At this point decided to use ASMLib since it was not possible to change the ownership of the softlink and ASMLib was able to use root owned disks without a problem. The Oracle documents that have read (605828.1,470913.1,602952.1) said not to use /dev/dm* so didn't want to directly use them and conveyed this to storage vendor before proceeding.
Installed and configured ASMLib

/usr/sbin/oracleasm configure
ORACLEASM_ENABLED=true
ORACLEASM_UID=grid
ORACLEASM_GID=asmadmin
ORACLEASM_SCANBOOT=true
ORACLEASM_SCANORDER="dm"
ORACLEASM_SCANEXCLUDE="sd"

and created the ASM disk that will be used in theASM diskgroup used for storing clusterware files using oracleasm

/etc/init.d/oracleasm listdisks
CLUS01
CLUS02
CLUS03

Went ahead with the installation and when at the ASM creation page, not ASM disks were visible when the discovery string ORCL:* was set. On the installer log following error message could be seen

INFO: ... discoveryString = ORCL:*
INFO: Executing [/tmp/OraInstall2013-01-10_02-28-18PM/ext/bin/kfod, nohdr=true, verbose=true, disks=all, status=true, op=disks, asm_diskstring='ORCL:*']
 with environment variables {XFILESEARCHPATH=/usr/dt/app-defaults/%L/Dt, LANG=en_GB.UTF-8, DISPLAY=:1, PWD=/home/grid/grid, _=/home/grid/grid/install/.oui, GS_LIB=/home/grid/.fonts, KDEDIR=/usr, KDE_MULTIHEAD=false, NLSPATH=/usr/dt/lib/nls/msg/%L/%N.cat, USER=grid, CLASSPATH=, XMODIFIERS=@im=none, G_BROKEN_FILENAMES=1, SSH_AGENT_PID=8475, COLORTERM=, HOME=/home/grid, HOSTNAME=RCL-DB-02, SESSION_MANAGER=local/RCL-DB-02:/tmp/.ICE-unix/8600, LESSOPEN=|/usr/bin/lesspipe.sh %s, LD_LIBRARY_PATH=/tmp/OraInstall2013-01-10_02-28-18PM/ext/bin/../lib, WINDOWID=4194311, ORACLE_HOME=/tmp/OraInstall2013-01-10_02-28-18PM/ext/bin, LOGNAME=grid, SSH_ASKPASS=/usr/libexec/openssh/gnome-ssh-askpass, SHELL=/bin/bash, SSH_AUTH_SOCK=/tmp/ssh-GSFGtX8450/agent.8450, CVS_RSH=ssh, DBUS_SESSION_BUS_ADDRESS=unix:abstract=/tmp/dbus-KbHCyuMD9K,guid=1f484b25202997262dd3e40050eecda6, INPUTRC=/etc/inputrc, SHLVL=5, KDE_IS_PRELINKED=1, KDE_FULL_SESSION=true, HISTTIMEFORMAT=%b %e - %H:%M:%S , MAIL=/var/spool/mail/grid, KDE_NO_IPV6=1, VNCDESKTOP=RCL-DB-02:1 (grid), KONSOLE_DCOP=DCOPRef(konsole-8643,konsole), JAVA_HOME=, LS_COLORS=no=00:fi=00:di=00;34:ln=00;36:pi=40;33:so=00;35:bd=40;33;01:cd=40;33;01:or=01;05;37;41:mi=01;05;37;41:ex=00;32:*.cmd=00;32:*.exe=00;32:*.com=00;32:*.btm=00;32:*.bat=00;32:*.sh=00;32:*.csh=00;32:*.tar=00;31:*.tgz=00;31:*.arj=00;31:*.taz=00;31:*.lzh=00;31:*.zip=00;31:*.z=00;31:*.Z=00;31:*.gz=00;31:*.bz2=00;31:*.bz=00;31:*.tz=00;31:*.rpm=00;31:*.cpio=00;31:*.jpg=00;35:*.gif=00;35:*.bmp=00;35:*.xbm=00;35:*.xpm=00;35:*.png=00;35:*.tif=00;35:, KONSOLE_DCOP_SESSION=DCOPRef(konsole-8643,session-1), LC_ALL=, HISTSIZE=1000, XDG_CONFIG_DIRS=/etc/kde/xdg:/etc/xdg, TERM=xterm, PATH=/usr/bin:/usr/ccs/bin:/usr/kerberos/bin:/usr/local/bin:/bin:/usr/bin:/home/grid/bin:/home/grid/grid/install}
INFO: Starting Output Reader Threads for process /tmp/OraInstall2013-01-10_02-28-18PM/ext/bin/kfod
INFO: Parsing KFOD-00307: asmlib error!! function = [asm_close], error = [0], mesg = [Invalid argument]
INFO: Parsing KFOD-00307: asmlib error!! function = [asm_open], error = [0], mesg = [Operation not permitted]
INFO: The process /tmp/OraInstall2013-01-10_02-28-18PM/ext/bin/kfod exited with code 0
INFO: Parsing KFOD-00307: asmlib error!! function = [asm_close], error = [0], mesg = [Invalid argument]
INFO: Waiting for output processor threads to exit.
INFO: Parsing KFOD-00307: asmlib error!! function = [asm_open], error = [0], mesg = [Operation not permitted]
INFO: Parsing KFOD-00307: asmlib error!! function = [asm_close], error = [0], mesg = [Invalid argument]
INFO: Parsing KFOD-00307: asmlib error!! function = [asm_open], error = [0], mesg = [Operation not permitted]
INFO: Output processor threads exited.

Closest metalink note that could be found was ORA-15186: ASMLIB error function = [asm_open], error = [1], mesg = [Operation not permitted] [ID 398622.1] but even that was not an exact match in terms of the error code and of no help as this was a new installation.
However it was confirmed that ASM disk are there by listing the /dev/oracleasm/disk and using that path in the discovery string.

It's at this point the tinkering work began. First place was the scan order. Try different combinations including eql/clus-*, eql/*, dm-* , dm* but still didn't discover any ASM disk using ORCL:*.
But when the exact device was specified in the scan order it was possible to get the ASM disks using the ORCL:*. It took a while to get the head around this

ORACLEASM_SCANORDER="dm" <---- DOES NOT WORK NO ASM DISK DISCOVERED
ORACLEASM_SCANORDER="dm-16 dm-18 dm-20" <--- WORK FINE SEE THE ASM DISK

At this point raise an SR. Understandably Oracle's initial effort revolved around checking if ASMLib setup is correct and it was confirmed to be setup correct and correct value for ORACLEASM_SCANORDER was dm. But still couldn't explain why dm doesn't work and dm-16 work (work = ASM disk visible in the installer).
After sometime an Oracle engineer spot the issue which was believed to do with the multipath setup used by the storage vendor and might even explain the oddity mentioned above. Storage vendor wasn't using the Linux multipathing software but a proprietary one that came with the SAN. Similar to SCANORDER and SCANEXCLUDE it had eql and ignore-eql.

ls -l /dev/ignore_eql/
lrwxrwxrwx 1 root root 8 Jan 18 19:32 0-1cb196-32b50460e-993001e8c7650e17_a -> ../dm-15
lrwxrwxrwx 1 root root 8 Jan 18 19:32 0-1cb196-33f50460e-197001e8c7950e17_a -> ../dm-17
lrwxrwxrwx 1 root root 8 Jan 18 19:32 0-1cb196-34f50460e-8a7001e8c7c50e17_a -> ../dm-19
lrwxrwxrwx 1 root root 7 Jan 18 19:32 0-1cb196-3a950460e-f17001e8c8550e17_a -> ../dm-9
lrwxrwxrwx 1 root root 8 Jan 18 19:32 0-1cb196-3ba50460e-ffb001e8c8850e17_a -> ../dm-13
lrwxrwxrwx 1 root root 8 Jan 18 19:32 0-1cb196-3c950460e-ce7001e8c8b50e17_a -> ../dm-11
lrwxrwxrwx 1 root root 7 Jan 18 19:32 0-1cb196-3e050460e-0a1001e8c8e50e17_a -> ../dm-5
lrwxrwxrwx 1 root root 7 Jan 18 19:32 0-1cb196-3f050460e-833001e8c9150e17_a -> ../dm-7
lrwxrwxrwx 1 root root 8 Jan 18 19:32 0-1cb196-3ff50460e-a3b001e8c9450e17_a -> ../dm-21

ls -l /dev/eql/
lrwxrwxrwx 1 root root 8 Jan 18 19:32 clus-01 -> ../dm-16
lrwxrwxrwx 1 root root 8 Jan 18 19:32 clus-02 -> ../dm-18
lrwxrwxrwx 1 root root 8 Jan 18 19:32 clus-03 -> ../dm-20

However looking at the /dev/mapper could see that there are multiple labels for the same device(only one set highlighted)

ls -l /dev/mapper/
brw-rw---- 1 root disk 253, 15 Jan 18 19:32 eql-0-1cb196-32b50460e-993001e8c7650e17_a <-- dm-15
brw-rw---- 1 root disk 253, 16 Jan 18 19:32 eql-0-1cb196-32b50460e-993001e8c7650e17-clus-01 <-- dm-16
brw-rw---- 1 root disk 253, 17 Jan 18 19:32 eql-0-1cb196-33f50460e-197001e8c7950e17_a
brw-rw---- 1 root disk 253, 18 Jan 18 19:32 eql-0-1cb196-33f50460e-197001e8c7950e17-clus-02
brw-rw---- 1 root disk 253, 19 Jan 18 19:32 eql-0-1cb196-34f50460e-8a7001e8c7c50e17_a
brw-rw---- 1 root disk 253, 20 Jan 18 19:32 eql-0-1cb196-34f50460e-8a7001e8c7c50e17-clus-03

and on the multipath list only one of these labels from each set appeared (only one highlighted)

 multipath -ll
eql-0-1cb196-34f50460e-8a7001e8c7c50e17_a () dm-19 EQLOGIC,100E-00
[size=1.0G][features=1 queue_if_no_path][hwhandler=0][rw]
\_ round-robin 0 [prio=2][active]
 \_ 6:0:0:0 sdf 8:80  [active][ready]
 \_ 14:0:0:0 sdm 8:192 [active][ready]
eql-0-1cb196-33f50460e-197001e8c7950e17_a () dm-17 EQLOGIC,100E-00
[size=1.0G][features=1 queue_if_no_path][hwhandler=0][rw]
\_ round-robin 0 [prio=2][active]
 \_ 1:0:0:0 sdj 8:144 [active][ready]
 \_ 15:0:0:0 sdn 8:208 [active][ready]
eql-0-1cb196-32b50460e-993001e8c7650e17_a () dm-15 EQLOGIC,100E-00
[size=1.0G][features=1 queue_if_no_path][hwhandler=0][rw]
\_ round-robin 0 [prio=2][active]
 \_ 16:0:0:0 sdo 8:224 [active][ready]
 \_ 17:0:0:0 sdp 8:240 [active][ready]

But the ASM is seeing all these paths.(only one set highlighted)

/etc/init.d/oracleasm querydisk -p CLUS01
Disk "CLUS01" is a valid ASM disk
/dev/sdj: LABEL="CLUS01" TYPE="oracleasm"
/dev/mapper/eql-0-1cb196-32b50460e-993001e8c7650e17_a: LABEL="CLUS01" TYPE="oracleasm" <-- dm-15
/dev/mapper/eql-0-1cb196-32b50460e-993001e8c7650e17-clus-01: LABEL="CLUS01" TYPE="oracleasm" <-- dm-16
/dev/sdp: LABEL="CLUS01" TYPE="oracleasm"

/etc/init.d/oracleasm querydisk -p CLUS02
Disk "CLUS02" is a valid ASM disk
/dev/sdf: LABEL="CLUS02" TYPE="oracleasm"
/dev/mapper/eql-0-1cb196-33f50460e-197001e8c7950e17_a: LABEL="CLUS02" TYPE="oracleasm"
/dev/mapper/eql-0-1cb196-33f50460e-197001e8c7950e17-clus-02: LABEL="CLUS02" TYPE="oracleasm"
/dev/sdo: LABEL="CLUS02" TYPE="oracleasm"

/etc/init.d/oracleasm querydisk -p CLUS03
Disk "CLUS03" is a valid ASM disk
/dev/mapper/eql-0-1cb196-34f50460e-8a7001e8c7c50e17_a: LABEL="CLUS03" TYPE="oracleasm"
/dev/mapper/eql-0-1cb196-34f50460e-8a7001e8c7c50e17-clus-03: LABEL="CLUS03" TYPE="oracleasm"
/dev/sdm: LABEL="CLUS03" TYPE="oracleasm"
/dev/sdn: LABEL="CLUS03" TYPE="oracleasm"

It was thought that this could explain why dm in the scan order would not discover any ASM disks while dm-16 (and others) could discover them.
Unfortunately that's where this post ends. Going to the storage vendor with this information didn't yield anything. "That's how it is setup" was repeated and due the time critical nature of the work decided to drop using ASMLib altogether and decided to go for block devices (events on this post happened before the post on the link) after letting the storage vendor know what Oracle says about using /dev/dm*.

Useful metalink notes
New ASMLib / oracleasm Disk Gets "header_status=Unknown" - Cannot be Added to ASM Diskgroup [ID 391136.1]
ORA-15186: ASMLIB error function = [asm_open], error = [1], mesg = [Operation not permitted] [ID 398622.1]

ASM Disk Creation Fails with EMC PowerPath

Creating an ASM disk with oracleasm createdisk fails with

Marking disk "/dev/emcpowera1" as an ASM disk: asmtool: Device "/dev/emcpowera1" is not a partition [FAILED]

even though multipathing is setup correctly.

According to metalink note 469163.1 ASMLib: oracleasm createdisk command fails: Device '/dev/emcpowera1 is not a partition this is due to the PowerPath version is less than 5.3.0.

In spite this error ASM disks could be created using asmtool by specifying the force option.

# /usr/sbin/asmtool -C -l /dev/oracleasm -n DATA01 -s /dev/emcpowera1 -a force=yes

Once created the disk will be visible through listdisks command and running a scandisks on other nodes in the cluster will make it discoverable on these nodes as well.

Another related metalink note is 566676.1 How To Determinate If An EMCPOWER Partition Is Valid For ASMLIB? which list a different cause for the problem but error message shown is same as above.

ASMLib on Multipath Disks

The configuration file contains many configuration variables. The ORACLEASM_SCANORDER variable specifies disks to be scanned first.

The ORACLEASM_SCANEXCLUDE variable specifies the disks that are to be ignored.The variables take a whitespace-seperated list of prefix strings to match. In other words, if a disk starts with one of the prefix strings, it matches. For example, the prefix string sd will match all SCSI devices. The disk sda starts with sd. Note that these are not globs. They do not use wildcards. They are simple prefixes. Also note that the /dev/ path is not part of the prefix.


# ORACLEASM_SCANORDER: Matching patterns to order disk scanning
#ORACLEASM_SCANORDER=${ORACLEASM_SCANORDER}ORACLEASM_SCANORDER=emcpowera


# ORACLEASM_SCANEXCLUDE: Matching patterns to exclude disks from scan
#ORACLEASM_SCANEXCLUDE=${ORACLEASM_SCANEXCLUDE}ORACLEASM_SCANEXCLUDE="sda sdb sdc sdd"

more on metalink note 309815.1



Useful metalink notes
Configuring Oracle ASMLib on Multipath Disks [ID 309815.1]
How to Partition DM-Multipath Pseudo Devices [ID 470913.1]
Configuring Oracle ASMLib on Multipath Disks [ID 309815.1]
How To Setup ASM & ASMLIB On Native Linux Multipath Mapper disks? [ID 602952.1]
Configuring non-raw multipath devices for Oracle Clusterware 11g (11.1.0, 11.2.0) on RHEL5/OL5 [ID 605828.1]
Configuring raw devices (multipath) for Oracle Clusterware 10g Release 2 (10.2.0) on RHEL5/OEL5 [ID 564580.1]
How to Configure LUNs for ASM Disks using WWID, DM-Multipathing, and ASMLIB on RHEL 5/OEL 5 [ID 1365511.1]