Using Openfiler to Create iSCSI Targets on VirtualBox

This post list steps for installing openfiler and creating iscsi targets that could be used as storage for Oracle DBs. It list steps for creating single path iscsi targets and multipath iscsi targets. Lastly it shows how to attach iscsi targets directly to VirtualBox.

    Installing openfiler
    Creating iSCSI Targets
    Single Path iSCSI Target
    Multipath iSCSI Target
    Attach iSCSI Target to VirtualBox from Host

Installing openfiler
Download the latest openfiler software from openfiler.comThe Openfiler software is installed on a VirtualBox. To install openfiler on VirtulBox add the downloaded ISO image to the CD-ROM and set the boot order such that CD-ROM is higher than hard disk. Two virtual disks are created in the VirtualBox (see image below). One is to install the openfiler software (OS) (openfiler.vdi) . Then another disk that will be used to create iscsi targets (scsidisk.vdi).

Make sure the disk used for openfiler OS install is larger than 8GB even though openfiler states 8GB is the minimum size needed. Even when disks is slightly larger than 8GB installation still fails.

Once the disks are in order boot from the openfiler ISO image. Screenshots below give highlights (not all the steps).

Only select the disk designated for installing the OS. Uncheck the disk designated to create iscsi targets.

Set the IPs for the openfiler setup. In this case static IP configuration has been done as oppose to DHCP. As there's only one NIC on the openfiler server, the iscsi targets will be single path. If multipath iscsi targets are required, add another NIC to the openfiler server. Refer multipath iSCSI target step for more.

Once the installation is complete login to the web administration portal. The URL is listed at the end of the installation process.

Login to the web console (default username is "openfiler" and password is "password") and go to services tab and enable and start iSCSI target.

Openfiler installation is complete and ready to create iscsi targets.



Creating iSCSI Targets
To create iscsi targets login to the web console if already not logged in. The iscsi targets created on openfiler server is attached on a server with hostname city7. In this case the city7 is the iscsi initiator.

In order for city7 server to access the iscsi target add the city7 to network access configuration by selecting system tab.

Create the iscsi target using the disk designated for it. This could be done by selecting volume tag followed by the block device link and finally clicking on the actual disk. In this case the disk used for iscsi target is /dev/sdb.

This will open the create partition page. The edit partition section will show 100% storage. On the create partition section select the required amount of storage (in this case 10GB was chosen) and create a physical partition by clicking on the create button.

New partition will be shown on the edit partition section. Even though 10GB was selected, created partition was smaller than 10GB.

Create a volume group using the partition created earlier. In this case the volume group is named asmdg.

Once created new volume group will appear under volume group management section.

Next step is to create a volume within the volume group. For this select add volume link and create a volume of required size.

Once created the volume information will be listed under the associated volume group.

The next step is to create an iscsi target to the volume. This is done through the iscsi target link. Select target configuration tab and then creating a iSCSI Qualified Name (IQN) for the target. Below screenshot shows creating a IQN. Openfiler auto generate an IQN but it could be renamed as desired. In this case the IQN is iqn.2006-01.com.openfiler:asm.newdata

Select LUN mapping tap and map the earlier created IQN to the LUN (volume). The LUN name is the description of the volume created earlier (inside orange box), not the name of the volume.

Once the mapping is complete.

Finally update the network access control list (ACL) by allowing access to the iscsi target. As mentioned earlier, the server that will be accessing the iscsi target is called city7.

There's no need to setup CHAP authentication unless password protection is needed for the iscsi targets.
Creating iscsi target is now complete. Next step is to consume the iscsi target.

Single Path iSCSI Target
The city7 server that will consume the iscsi target is running RHEL 7.4 (3.10.0-693.el7.x86_64). In order to access iscsi target install the following RPM

iscsi-initiator-utils-6.2.0.874-4.el7.x86_64

Have the iscsid service enabled. If the service is down it's auto started during the target discovery. Nevertheless, it's best to have it enabled so auto starts across reboots.

# systemctl enable iscsid.service
# systemctl list-unit-files | grep iscsid | grep enable
iscsid.service                                enabled

systemctl status iscsid
â iscsid.service - Open-iSCSI
   Loaded: loaded (/usr/lib/systemd/system/iscsid.service; enabled; vendor preset: disabled)
   Active: active (running)

Next step is to discover the iscsi target

# iscsiadm -m discovery -t st -p 192.168.0.94 (or long form iscsiadm -m discovery -t sendtargets -p 192.168.0.94)
192.168.0.94:3260,1 iqn.2006-01.com.openfiler:asm.newdata

If for some reason the discovery fails as below then check the network access configuration and network ACL. The network/hostname must be considered in conjunction with the netmask.

#  iscsiadm -m discovery -t st -p 192.168.0.94
iscsiadm: No portals found

To confirm this is down to network issue comment out the /etc/initiators.allow and /etc/initiators.deny entries and try again (this is an auto generated file and not advised to manually edit). For example if the IP of the iscsi initiator (city7 in this case) is 192.168.0.85 and netmask was set to 255.255.255.0 (instead of 255.255.255.255 which will restrict access to all but just one IP) then /etc/initiators.allow entry would be iqn.2006-01.com.openfiler:asm.newdata 192.168.0.85/24. This is not what's expected and could result in iscsi discovery failure.
If the targets are discovered without any issue then manually login to the target

# iscsiadm -m node -T iqn.2006-01.com.openfiler:asm.newdata -p 192.168.0.94 -l
Logging in to [iface: default, target: iqn.2006-01.com.openfiler:asm.newdata, portal: 192.168.0.94,3260] (multiple)
Login to [iface: default, target: iqn.2006-01.com.openfiler:asm.newdata, portal: 192.168.0.94,3260] successful.

This will show a new disk in the scsi initiator server (city7).

 ls -l /dev/disk/by-path/
lrwxrwxrwx. 1 root root  9 Mar 22 15:23 ip-192.168.0.94:3260-iscsi-iqn.2006-01.com.openfiler:asm.newdata-lun-0 -> ../../sdd

fdisk will now show a new disk as a result of the iscsi target

fdisk -l

...
Disk /dev/sdd: 10.2 GB, 10234101760 bytes, 19988480 sectors
Units = sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes

Run the following so that iscsi initiator auto logins to the target accrsso reboots

iscsiadm -m node -T iqn.2006-01.com.openfiler:asm.newdata -p 192.168.0.94 --op update -n node.startup -v automatic

If for some reason the auto login need to be changed to manual login use the following command

iscsiadm -m node -T iqn.2006-01.com.openfiler:asm.newdata -p 192.168.0.88 --op update -n node.startup -v manaul

Since target discovery and login is working fine, create a partition on the new disk and a appropriate udev rule.

fdisk /dev/sdd

fdisk -l
Disk /dev/sdd: 10.2 GB, 10234101760 bytes, 19988480 sectors
...
   Device Boot      Start         End      Blocks   Id  System
/dev/sdd1            2048    19988479     9993216   83  Linux

To create the udev rule get the scsi id of the disk

/usr/lib/udev/scsi_id -g -u -d /dev/sdd
14f504e46494c4552484a535350652d4c774b6f2d5a597173

With the scsi id in hand create the udev rule for Oracle ASM with role separation as below

KERNEL=="sd?1",SUBSYSTEM=="block",PROGRAM=="/usr/lib/udev/scsi_id -g -u -d /dev/$parent",RESULT=="14f504e46494c4552484a535350652d4c774b6f2d5a597173",SYMLINK+="oracleasm/ndata1",OWNER="grid", GROUP="asmadmin", MODE="0660"

Run the udev rules and check new udev taking effect

ls -l /dev/oracleasm/*
lrwxrwxrwx. 1 root root 7 Mar 22 15:44 /dev/oracleasm/data1 -> ../sdb1
lrwxrwxrwx. 1 root root 7 Mar 22 15:44 /dev/oracleasm/fra1 -> ../sdc1

# udevadm trigger --type=devices --action=change

# ls -l /dev/oracleasm/*
lrwxrwxrwx. 1 root root 7 Mar 22 15:45 /dev/oracleasm/data1 -> ../sdb1
lrwxrwxrwx. 1 root root 7 Mar 22 15:45 /dev/oracleasm/fra1 -> ../sdc1
lrwxrwxrwx. 1 root root 7 Mar 22 15:45 /dev/oracleasm/ndata1 -> ../sdd1

Next step is to add the disk to ASM. This would conclude the iscsi single path setup. For completion of the post following two commands are also mentioned.
To logout of a target use the following command

# iscsiadm -m node -u -T iqn.2006-01.com.openfiler:asm.newdata -p 192.168.0.94
Logging out of session [sid: 2, target: iqn.2006-01.com.openfiler:asm.newdata, portal: 192.168.0.94,3260]
Logout of [sid: 2, target: iqn.2006-01.com.openfiler:asm.newdata, portal: 192.168.0.94,3260] successful.

To delete a target use the following command. This will only remove the entry on the iscsi initiator, not the actual iscsi target.

# iscsiadm -m node -o delete -T iqn.2006-01.com.openfiler:asm.newdata -p 192.168.0.94

Multipath iSCSI Target
Having a single network path to the iscsi target creates a single point of failure. The solution is to have multiple network paths (preferably from different network segments without having any common hardware. i.e NIC, switches etc) to the same iscsi target. Following diagram shows the multipath configuration between iscsi target (openfiler) and the iscsi initiator (city7).

The same iscsi target is exposed via IPs 192.168.0.94 and 192.168.1.88. These could be configured during the opefiler installation or later on.

The iscsi initiator also has two NICs with IPs 192.168.0.85 and 192.168.1.85 capable of accessing the iscsi target through different network path. Once two NICs are configured enable network access through open filer web console. This need to happen on network access configuration (under system tab)

and network ACL (volume -> iscsi targets).

Apart from these changes (having multiple NICs) there's no other changes on the iscsi target.
On the iscsi initiator install multipath software in addition to the iscsi software mentioned in singe path iscsi creation.

device-mapper-multipath-libs-0.4.9-111.el7.x86_64.rpm
device-mapper-multipath-0.4.9-111.el7.x86_64.rpm

Enable the multipath service and start it along with the iscsid service if iscsid is not already started.

systemctl enable multipathd.service
systemctl start multipathd.service
systemctl status multipathd.service

If /etc/multipath.conf file doesn't exists create a one from the provided template using the following command.

mpathconf --enable

This will create an multipath.conf file with all the section either commented or empty. Add following to default section and blacklist section. Among other things the default section tells multipathing to use user friendly names and round robin as the path policy.

cat /etc/multipath.conf | grep -A5 defaults
defaults {
        find_multipaths yes
        user_friendly_names yes
        failback immediate
        path_selector   "round-robin 0"
}

Add the following to the blacklist section

blacklist {
  devnode "^(ram|raw|loop|fd|md|dm-|sr|scd|st)[0-9]*"
  devnode "^hd[a-z]"
  devnode "^cciss!c[0-9]d[0-9]*[p[0-9]*]"
}

Next login to the iscsi targets using both paths.

iscsiadm -m node -T iqn.2006-01.com.openfiler:asm.newdata -p 192.168.0.94 -l
Logging in to [iface: default, target: iqn.2006-01.com.openfiler:asm.newdata, portal: 192.168.0.94,3260] (multiple)
Login to [iface: default, target: iqn.2006-01.com.openfiler:asm.newdata, portal: 192.168.0.94,3260] successful.

iscsiadm -m node -T iqn.2006-01.com.openfiler:asm.newdata -p 192.168.1.88 -l
Logging in to [iface: default, target: iqn.2006-01.com.openfiler:asm.newdata, portal: 192.168.1.88,3260] (multiple)
Login to [iface: default, target: iqn.2006-01.com.openfiler:asm.newdata, portal: 192.168.1.88,3260] successful.

If login is successful set auto login for both paths

# iscsiadm -m node -T iqn.2006-01.com.openfiler:asm.newdata -p 192.168.0.94 --op update -n node.startup -v automatic
# iscsiadm -m node -T iqn.2006-01.com.openfiler:asm.newdata -p 192.168.1.88 --op update -n node.startup -v automatic

Check multipath device are active

mpathf (14f504e46494c4552484a535350652d4c774b6f2d5a597173) dm-0 OPNFILER,VIRTUAL-DISK
size=9.5G features='0' hwhandler='0' wp=rw
|-+- policy='round-robin 0' prio=1 status=active
| `- 11:0:0:0 sde 8:64 active ready running
`-+- policy='round-robin 0' prio=1 status=enabled
  `- 10:0:0:0 sdd 8:48 active ready running

Using disk path it is possible to find out to which path the sd* device belong to

ls -l /dev/disk/by-path/
... ip-192.168.0.94:3260-iscsi-iqn.2006-01.com.openfiler:asm.newdata-lun-0 -> ../../sdd
... ip-192.168.1.88:3260-iscsi-iqn.2006-01.com.openfiler:asm.newdata-lun-0 -> ../../sde

Next step is to create a user friendly name for the multipath device. To do this use WWID (World Wide Identifier) of the mpath device (1365511.1).

/usr/lib/udev/scsi_id -g -u -d /dev/mapper/mpathf
14f504e46494c4552484a535350652d4c774b6f2d5a597173

Add a alias to the multipath device using the WWID obtained by adding following to the /etc/multipath.conf file.

multipaths {
  multipath {
    wwid                    14f504e46494c4552484a535350652d4c774b6f2d5a597173
    alias                   oradata
    path_grouping_policy    failover
  }

ls -l /dev/mapper
lrwxrwxrwx. 1 root root       7 Mar 29 10:54 mpathf -> ../dm-0

Restart the multipath service. Check the mapper directory again to see multipath device name being set to alias specified on the multipath.conf.

systemctl reload multipathd
ls -l /dev/mapper
lrwxrwxrwx. 1 root root       7 Mar 29 10:55 oradata -> ../dm-0

Run fdisk to partition the device (470913.1).This will create a partition in the mapper directory

fdisk /dev/mapper/oradata
/sbin/partprobe
ls -l /dev/mapper
lrwxrwxrwx. 1 root root       7 Mar 29 11:01 oradata -> ../dm-0
lrwxrwxrwx. 1 root root       7 Mar 29 11:01 oradata1 -> ../dm-1

To create a udev rule to set appropriate ownership for the partition use the DM_UUID value. This could be obtained by following command

udevadm info --query=all --name=/dev/mapper/oradata1
...
E: DM_UUID=part1-mpath-14f504e46494c4552484a535350652d4c774b6f2d5a597173
...

Create a udev rule as below using the obtained DM_UUID

KERNEL=="dm-*",ENV{DM_UUID}=="part1-mpath-14f504e46494c4552484a535350652d4c774b6f2d5a597173",OWNER="grid",GROUP="asmadmin",MODE="0660"

Reload the udev rules and verify permission change

ls -l /dev/dm-1
brw-rw----. 1 root disk 253, 1 Mar 29 11:01 /dev/dm-1
/sbin/udevadm control --reload-rules
/sbin/udevadm trigger --type=devices --action=change
ls -l /dev/dm-1
brw-rw----. 1 grid asmadmin 253, 1 Mar 29 11:09 /dev/dm-1

Finally verify multipath is working by downing one of the NICs at the iscsi target. As shown earlier eth0 has the IP 192.168.0.94 (sdd) and eth1 has 192.168.1.88 (sde). Current multipath status is as below

 multipath -ll
oradata (14f504e46494c4552484a535350652d4c774b6f2d5a597173) dm-0 OPNFILER,VIRTUAL-DISK
size=9.5G features='0' hwhandler='0' wp=rw
|-+- policy='round-robin 0' prio=1 status=active
| `- 12:0:0:0 sdd 8:48 active ready running
`-+- policy='round-robin 0' prio=1 status=enabled
  `- 13:0:0:0 sde 8:64 active ready running

To test one path stop the eth0 on iscsi target(opernfiler server)

ifdown eth0

Check the current multipath topology.

oradata (14f504e46494c4552484a535350652d4c774b6f2d5a597173) dm-0 OPNFILER,VIRTUAL-DISK
size=9.5G features='0' hwhandler='0' wp=rw
|-+- policy='round-robin 0' prio=0 status=enabled
| `- 12:0:0:0 sdd 8:48 failed faulty running
`-+- policy='round-robin 0' prio=1 status=active
  `- 13:0:0:0 sde 8:64 active ready running

The path through 192.168.0.94 (sdd) has become failed path through sde has become active. To check the other path start the downed NIC and wait for iscsi path to become active ready and then down the eth1.

openfiler ~]# ifup eth0

city7 ~]# multipath -ll
oradata (14f504e46494c4552484a535350652d4c774b6f2d5a597173) dm-0 OPNFILER,VIRTUAL-DISK
size=9.5G features='0' hwhandler='0' wp=rw
|-+- policy='round-robin 0' prio=1 status=active
| `- 12:0:0:0 sdd 8:48 active ready running
`-+- policy='round-robin 0' prio=1 status=enabled
  `- 13:0:0:0 sde 8:64 active ready running

openfiler ~]# ifdown eth1

city7 ~]# multipath -ll
oradata (14f504e46494c4552484a535350652d4c774b6f2d5a597173) dm-0 OPNFILER,VIRTUAL-DISK
size=9.5G features='0' hwhandler='0' wp=rw
|-+- policy='round-robin 0' prio=1 status=active
| `- 12:0:0:0 sdd 8:48 active ready running
`-+- policy='round-robin 0' prio=0 status=enabled
  `- 13:0:0:0 sde 8:64 failed faulty running

This concludes the using multipath iscsi targets.
If during testing and setup want to logout from all iscsi target paths at once use the following (this will stop multipath working)

iscsiadm -m node -u
Logging out of session [sid: 6, target: iqn.2006-01.com.openfiler:asm.newdata, portal: 192.168.0.94,3260]
Logging out of session [sid: 7, target: iqn.2006-01.com.openfiler:asm.newdata, portal: 192.168.1.88,3260]
Logout of [sid: 6, target: iqn.2006-01.com.openfiler:asm.newdata, portal: 192.168.0.94,3260] successful.
Logout of [sid: 7, target: iqn.2006-01.com.openfiler:asm.newdata, portal: 192.168.1.88,3260] successful.

Attach iSCSI Target to VirtualBox from Host
In the previous cases the iscsi target was attached to the Virtualbox while logged in to the VM. However it is possible to attach the iscsi target to the virtualbox VM from the host server. This is done using the VBoxManage command. Before running the command make sure host server is granted access to the iscsi target.

VBoxManage storageattach city7 --storagectl "SATA" --port 3 --device 0 --type hdd --medium iscsi --server 192.168.0.94 --target "iqn.2006-01.com.openfiler:asm.newdata" --tport 3260
iSCSI disk created. UUID: bc380eb5-e67b-49b7-9602-efbc5d6d8e87

This will be visible on the VirtualBox GUI.

When the VM is booted the iscsi target would be available as a standard block device (/dev/sd*).
To remove the iscsi from the VM run the following

VBoxManage storageattach city7 --storagectl "SATA" --port 3 --device 0 --medium none

This however doesn't remove the iscis target from virtual media manager only from the VM. It will remain an unattached device in the virtual media manager.

As such subsequent attaching of the iscsi target will fail with UUID already exists error.

VBoxManage storageattach city7 --storagectl "SATA" --port 3 --device 0 --type hdd --medium iscsi --server 192.168.0.94 --target "iqn.2006-01.com.openfiler:asm.newdata" --tport 3260
VBoxManage: error: Cannot register the hard disk '192.168.0.94|iqn.2006-01.com.openfiler:asm.newdata' {a866eaf1-f089-4a8d-b793-23028e3a1e14} because a hard disk '192.168.0.94|iqn.2006-01.com.openfiler:asm.newdata' 
with UUID {bc380eb5-e67b-49b7-9602-efbc5d6d8e87} already exists
VBoxManage: error: Details: code NS_ERROR_INVALID_ARG (0x80070057), component VirtualBoxWrap, interface IVirtualBox, callee nsISupports
VBoxManage: error: Context: "CreateMedium(Bstr("iSCSI").raw(), bstrISCSIMedium.raw(), AccessMode_ReadWrite, DeviceType_HardDisk, pMedium2Mount.asOutParam())" at line 608 of file VBoxManageStorageController.cpp

To fix this run the close medium command providing the UUID. This will remove the iscsi target from the virtual manager allowing the iscsi target to be attached again.

VBoxManage closemedium bc380eb5-e67b-49b7-9602-efbc5d6d8e87

This concludes using openfiler to create iSCSI targets on VirtualBox.