ORA-15081: failed to submit an I/O operation to a disk

DBCA creation fails with ORA-15081.

This is a new 11gR2 (11.2.0.3) RAC installation with role separation. Oracle user and grid user both has correct user group setup and permission of oracle binaries are also correct.
This RAC setup does not use ASMLib instead uses direct block devices. When block devices are used their permission must be 660 and ownership must be grid:asmadmin. The storage vendor setup the block devices as below

ls -l /dev/eql/*
lrwxrwxrwx 1 root root 7 Jan 15 12:00 /dev/eql/data-01 -> ../dm-6
lrwxrwxrwx 1 root root 7 Jan 15 12:00 /dev/eql/data-02 -> ../dm-8
lrwxrwxrwx 1 root root 8 Jan 15 12:00 /dev/eql/clus-01 -> ../dm-16
lrwxrwxrwx 1 root root 8 Jan 15 12:00 /dev/eql/clus-02 -> ../dm-18
lrwxrwxrwx 1 root root 8 Jan 15 12:00 /dev/eql/clus-03 -> ../dm-20
lrwxrwxrwx 1 root root 8 Jan 15 12:00 /dev/eql/data-03 -> ../dm-22
lrwxrwxrwx 1 root root 8 Jan 15 12:00 /dev/eql/flash-01 -> ../dm-10
lrwxrwxrwx 1 root root 8 Jan 15 12:00 /dev/eql/flash-02 -> ../dm-14
lrwxrwxrwx 1 root root 8 Jan 15 12:00 /dev/eql/flash-03 -> ../dm-12

and asked to use /dev/eql/* to creat ASM diskgroups. This was using Dell EqualLogic iSCSI SAN. Even though the softlink had permission 777 the block device pointed by these only had 640 with grid:asmadmin set. (Storage vendor was asked to set 660 on these). 640 permission was fine (though wrong) for installing clusterware but the erroneous permission setup was only noticed during the dbca execution as oracle user.

brw-r----- 1 grid asmadmin 253,  6 Jan 15 14:22 /dev/dm-6
brw-r----- 1 grid asmadmin 253,  8 Jan 15 14:11 /dev/dm-8
brw-r----- 1 grid asmadmin 253, 10 Jan 15 14:22 /dev/dm-10
brw-r----- 1 grid asmadmin 253, 12 Jan 15 14:22 /dev/dm-12
brw-r----- 1 grid asmadmin 253, 14 Jan 15 13:12 /dev/dm-14
brw-r----- 1 grid asmadmin 253, 16 Jan 15 14:24 /dev/dm-16
brw-r----- 1 grid asmadmin 253, 18 Jan 15 14:24 /dev/dm-18
brw-r----- 1 grid asmadmin 253, 20 Jan 15 14:24 /dev/dm-20
brw-r----- 1 grid asmadmin 253, 22 Jan 15 14:22 /dev/dm-22

After asking the storage vendor to properly setup the permission (to 660) was able to create the database.

brw-rw---- 1 grid asmadmin 253,  6 Jan 16 13:55 /dev/dm-6
brw-rw---- 1 grid asmadmin 253,  8 Jan 16 13:38 /dev/dm-8
brw-rw---- 1 grid asmadmin 253, 10 Jan 16 13:55 /dev/dm-10
brw-rw---- 1 grid asmadmin 253, 12 Jan 16 13:38 /dev/dm-12
brw-rw---- 1 grid asmadmin 253, 14 Jan 16 13:38 /dev/dm-14
brw-rw---- 1 grid asmadmin 253, 16 Jan 16 13:55 /dev/dm-16
brw-rw---- 1 grid asmadmin 253, 18 Jan 16 13:55 /dev/dm-18
brw-rw---- 1 grid asmadmin 253, 20 Jan 16 13:55 /dev/dm-20
brw-rw---- 1 grid asmadmin 253, 22 Jan 16 13:38 /dev/dm-22

On a separate note: 605828.1,470913.1,602952.1 all say not to use /dev/dm-*.



Useful metalink notes
ORA-15081: failed to submit an I/O operation to a disk [ID 1297099.1]
Bug 11695285 - ORA-15081 I/O write error but disk is online [ID 11695285.8]
ORA-17502, OSD-4002 and ORA-15081 when creating a datafile on a ASM diskgroup [ID 369898.1]
RAC Database Terminates: ORA-27041, ORA-15080, ORA-63999, ORA-01114, ORA-15081 [ID 1476707.1]
10gR2 Database Creation Fails with 11gR2 ASM storage: ORA-15045, ORA-17502, ORA-15081 [ID 1384180.1]
Database Creation on 11.2 Grid Infrastructure with Role Separation ( ORA-15025, KFSG-00312, ORA-15081 ) [ID 1084186.1]