Udev Rules for AWS EBS Volumes

AWS EBS volumes could be used as ASM disks. When ASMLib or AFD is not used, udev rules must be used to set the correct permissions on the EBS block volumes. It's generally good practice to use a unique identifier (UUID) to identify the partitions that require permission being set. It's not a guarantee that partitions or the blocks will be attached to the server in the same order and getting the same name.
Depending on the tool/option used a unique identifier may not be always available. For example the default dos partition table type created using fdisk would not generate any unique ID.

Model: Xen Virtual Block Device (xvd)
Disk /dev/xvdb: 268GB
Sector size (logical/physical): 512B/512B
Partition Table: msdos
Disk Flags:

Number  Start   End    Size   Type     File system  Flags
 1      1049kB  268GB  268GB  primary  ext4

udevadm info --query=property /dev/xvdd1
DEVNAME=/dev/xvdd1
DEVPATH=/devices/vbd-51760/block/xvdd/xvdd1
DEVTYPE=partition
ID_PART_ENTRY_DISK=202:48
ID_PART_ENTRY_NUMBER=1
ID_PART_ENTRY_OFFSET=2048
ID_PART_ENTRY_SCHEME=dos
ID_PART_ENTRY_SIZE=209713152
ID_PART_ENTRY_TYPE=0x83
ID_PART_TABLE_TYPE=dos
MAJOR=202
MINOR=49
SUBSYSTEM=block
TAGS=:systemd:
USEC_INITIALIZED=634695673

On the other-hand if the partition table type was GPT, this could generate several unique IDs which could be used in the udev rule to identify the partition.

fdisk /dev/sdd

Command (m for help): g
Building a new GPT disklabel (GUID: EC7F1589-8BD2-4C94-8F8F-D22013D40406)


Command (m for help): n
Partition number (1-128, default 1):
First sector (2048-20971486, default 2048):
Last sector, +sectors or +size{K,M,G,T,P} (2048-20971486, default 20971486):
Created partition 1


Command (m for help): w
The partition table has been altered!

Calling ioctl() to re-read partition table.
Syncing disks.

 fdisk -l /dev/sdd
WARNING: fdisk GPT support is currently new, and therefore in an experimental phase. Use at your own discretion.

Disk /dev/sdd: 10.7 GB, 10737418240 bytes, 20971520 sectors
Units = sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disk label type: gpt
Disk identifier: EC7F1589-8BD2-4C94-8F8F-D22013D40406


#         Start          End    Size  Type            Name
 1         2048     20971486     10G  Linux filesyste

udevadm info --query=property /dev/sdd1
...
ID_PART_ENTRY_DISK=8:48
ID_PART_ENTRY_NUMBER=1
ID_PART_ENTRY_OFFSET=2048
ID_PART_ENTRY_SCHEME=gpt
ID_PART_ENTRY_SIZE=20969439
ID_PART_ENTRY_TYPE=0fc63daf-8483-4772-8e79-3d69d8477de4
ID_PART_ENTRY_UUID=573dded4-21f1-48ce-925f-e02c5d94dace
ID_PART_TABLE_TYPE=gpt
ID_PATH=pci-0000:00:0d.0-ata-4.0
ID_PATH_TAG=pci-0000_00_0d_0-ata-4_0
...
SUBSYSTEM=block
TAGS=:systemd:

The partition UUID could be found out using blkid as well.

blkid /dev/sdd1
/dev/sdd1: PARTLABEL="data" PARTUUID="573dded4-21f1-48ce-925f-e02c5d94dace"

The ID_PART_ENTRY_UUID could be used in the udev rule.

KERNEL=="sd?1",ENV{ID_PART_ENTRY_UUID}=="573dded4-21f1-48ce-925f-e02c5d94dace", SYMLINK+="oracleasm/cgdata1", OWNER="oracle", GROUP="asmadmin", MODE="0660"

Similar to fdisk with g (gpt) option, parted could be used to achieve the same.

parted /dev/xvdd
GNU Parted 3.1
Using /dev/xvdd
Welcome to GNU Parted! Type 'help' to view a list of commands.
(parted) mktable gpt
Warning: The existing disk label on /dev/xvdd will be destroyed and all data on this disk will be lost. Do you want to continue?
Yes/No? yes
(parted) mkpart data 0% 100%

(parted) print all
Model: Xen Virtual Block Device (xvd)
Disk /dev/xvdd: 107GB
Sector size (logical/physical): 512B/512B
Partition Table: gpt
Disk Flags:

Number  Start   End    Size   File system  Name  Flags
 1      1049kB  107GB  107GB               data

udevadm info --query=property /dev/xvdd1
DEVLINKS=/dev/disk/by-partlabel/data /dev/disk/by-partuuid/5a081ffa-56e2-467b-82a9-16e3a4f441bd
DEVNAME=/dev/xvdd1
DEVPATH=/devices/vbd-51760/block/xvdd/xvdd1
DEVTYPE=partition
ID_PART_ENTRY_DISK=202:48
ID_PART_ENTRY_NAME=data
ID_PART_ENTRY_NUMBER=1
ID_PART_ENTRY_OFFSET=2048
ID_PART_ENTRY_SCHEME=gpt
ID_PART_ENTRY_SIZE=209711104
ID_PART_ENTRY_TYPE=ebd0a0a2-b9e5-4433-87c0-68b6b72699c7
ID_PART_ENTRY_UUID=5a081ffa-56e2-467b-82a9-16e3a4f441bd
ID_PART_TABLE_TYPE=gpt
MAJOR=202
MINOR=49
SUBSYSTEM=block
TAGS=:systemd:
USEC_INITIALIZED=685327395

Installing 10gR2 clusterware on RHEL 5

In RHEL/OEL 5 and onwards raw devices are deprecated (raw devices was initially deprecated in EL5 (GA-U3), but later undeprecated from EL5 U4). Therefore OCFS2 would be the only option to store OCR, VOTE disks and even ASM spfile. But there's a way to enable raw devices on RHEL/OEL 5 and process is explained in metalink note 465001.1 for singlepath raw devices and 564580.1 for multipath raw devices.

This blog is for singlepath.

1. Edit the scsi_id.config to return the device ID. This is done by adding option=-g to the /etc/scsi_id.config file.

# some libata drives require vpd page 0x80
vendor="ATA",options=-b 0x80
options=-g 

2. Get the device IDs with

# /sbin/scsi_id -g -s /block/sdc/sdc1
360a98000686f6959684a45124114174

Note down these IDs they will be used in the next step

3. Create a udev rules file in

vi /etc/udev/rules.d/61-oracleraw.rules

and add the following lines

ACTION=="add", KERNEL=="sd*", PROGRAM=="/sbin/scsi_id", RESULT=="360a98000686f6959684a45124114174", RUN+="/bin/raw /dev/raw/raw1 %N"
ACTION=="add", KERNEL=="sd*", PROGRAM=="/sbin/scsi_id", RESULT=="360a98000686f6959684a451241131151", RUN+="/bin/raw /dev/raw/raw2 %N"

For each rule, if all specified keys (KERNEL, PROGRAM, RESULT) are matched and raw device created if one or more keys are unmatched, the rule is completely ignored and the default (arbitrary) kernel-assigned device file name is assigned to devices.
The 'RUN+=' directive, is always the very last directive to execute in a rule file, therefore a separate file is used to permission resultant raw devices.
4. Create another udev rules files to set the permission on the raw devices.

 vi /etc/udev/rules.d/65-raw-permissions.rules

and add the following entries

KERNEL=="raw1", OWNER="root", GROUP="oinstall", MODE="640"
KERNEL=="raw2", OWNER="oracle", GROUP="oinstall", MODE="640"

5.Check the udev rules with

udevtest /block/sdc/sdc1

The device path should be the one relative to sysfs directory /sys/block. Restart the udev service with

start_udev

and verify the raw devices are enabled with

raw -qa

and permission on raw devices with

ls -l /dev/raw/* 

Metalink note also describes a way to create named devices. This is not related to raw device bindings.

 vi etc/udev/rules.d/55-oracle-naming.rules

and add the following lines

KERNEL=="sd*", BUS=="scsi", PROGRAM=="/sbin/scsi_id", RESULT=="360a98000686f6959684a45124114174", NAME="ocr1", OWNER="root", GROUP="oinstall", MODE="0640"
KERNEL=="sd*", BUS=="scsi", PROGRAM=="/sbin/scsi_id", RESULT=="360a98000686f6959684a451241131151", NAME="vote1", OWNER="oracle", GROUP="oinstall", MODE="0640"

For each rule, if all specified keys (KERNEL, BUS, PROGRAM, RESULT) are matched, the rule is applied and the specified assignments (NAME, OWNER, GROUP, MODE) are made and applied to the device. If, however, one or more keys are unmatched, the rule is completely ignored and the default (arbitrary) kernel-assigned device file name is assigned to devices.
Restart the udev service.

For system that are running on low-end hardware for demonstration or proof-of-concept purposes this might not work. The maxtor external device that was used to test this installation didn't give the device ID as the above command.

/sbin/scsi_id -g -s /block/sdc/sdc5
1Maxtor  OneTouch        2HA1BNAG

In this case udev rules file could be created applying rules for each individual partition.

vi 61-oracleraw.rules
ACTION=="add", KERNEL=="sdc5", RUN+="/bin/raw /dev/raw/raw1 %N"
ACTION=="add", KERNEL=="sdc6", RUN+="/bin/raw /dev/raw/raw2 %N"
ACTION=="add", KERNEL=="sdc7", RUN+="/bin/raw /dev/raw/raw3 %N"
ACTION=="add", KERNEL=="sdc8", RUN+="/bin/raw /dev/raw/raw4 %N"
ACTION=="add", KERNEL=="sdc9", RUN+="/bin/raw /dev/raw/raw5 %N"

and another rules file for permissions.

vi 65-raw-permissions.rules
KERNEL=="raw1", OWNER="root", GROUP="oinstall", MODE="640"
KERNEL=="raw2", OWNER="oracle", GROUP="oinstall", MODE="660"
KERNEL=="raw3", OWNER="oracle", GROUP="oinstall", MODE="660"
KERNEL=="raw4", OWNER="oracle", GROUP="dba", MODE="660"
KERNEL=="raw5", OWNER="oracle", GROUP="dba", MODE="660"

Restart the udev service and verify raw devices are enabled.

There are several other issues when installing on RHEL5 and these are documented on metalin note 414163.1
1. To run the installater ignoreSysPrereqs option is needed

2. Virtual IP Configuration Assistant (VIPCA) failes at the end of root.sh. To solve this before running the root.sh edit

$CRS_HOME/bin/vipca

and unset the LD_ASSUME_KERNEL variable.

if [ "$arch" = "i686" -o "$arch" = "ia64" -o "$arch" = "x86_64" ]
then
  LD_ASSUME_KERNEL=2.4.19
  export LD_ASSUME_KERNEL
fi

unset LD_ASSUME_KERNEL #add this line

Also edit the following files

$CRS_HOME/bin/srvctl
$RDBMS_HOME/bin/srvctl
$ASM_HOME/bin/srvctl

and unset the same variable as above

LD_ASSUME_KERNEL=2.4.19
export LD_ASSUME_KERNEL

unset LD_ASSUME_KERNEL # add this line 

These changes may be necessary after applying the 10.2.0.2 or 10.2.0.3 patchsets, as these patchset will still include those settings unnecessary for OEL5 or RHEL5 or SLES10. It is fixed in the 10.2.0.4 and 10.2.0.5 patchsets.

3. VIP will again fail if the VIP IP's are in a non-routable range eg. 10.x.x.x, 172.[16-31].x.x, 192.168.x.x range. If the OUI window is open, click OK and it will create the "oifcfg" information. Run the VIPCA manually as root and once done retry the cluvfy part at the end of OUI, and it should succeed.
If not then use

oifcfg setif -global eth*

to set public and cluster_interconnect interfaces.

Installing RAC on RHEL 5 after the clusterware did not throw any errors.

Block device permissions on 11g Clusterware

According to 11gR1 clusterware installation guide, the way to preserve the permissions of block devices used for OCR and Vote disk is to add file to /etc/udev/permissions.d directory

But this does not work for RHEL 5 as there's no permission.d folder. On RHEL5 have to create a file in /etc/udev/rules.d/99-raw.rules or /etc/udev/rules.d/99-sdb.rules (if the block device used for ocr and vote disk is a partion of sdb) with the following content

KERNEL=="raw[1-2]*", GROUP="oinstall", MODE="640"
KERNEL=="raw[3-5]*", OWNER="oracle", GROUP="oinstall", MODE="660"

Or

#OCR
KERNEL=="sdb[1]*", GROUP="oinstall", MODE="640"
#VOTE and ASM spfile
KERNEL=="sdb[2-3]*", OWNER="oracle", GROUP="oinstall", MODE="660"
#ASM
KERNEL=="sdb[5-6]*", OWNER="oracle", GROUP="dba", MODE="660"

Read more from here

Useful metalink note
How to Setup UDEV Rules for RAC OCR & Voting devices on SLES10, RHEL5, OEL5 [ID 414897.1]
Configuring raw devices (singlepath) for Oracle Clusterware 10g Release 2 (10.2.0) on RHEL5/OEL5 [ID 465001.1]
Configuring raw devices (multipath) for Oracle Clusterware 10g Release 2 (10.2.0) on RHEL5/OEL5 [ID 564580.1]

Creating a Udev Permissions File for Oracle Clusterware

The procedure to create a permissions file to grant oinstall group members write
privileges to block devices is as follows:

1. Log in as root.

2. Change to the /etc/udev/permissions.d directory:
# cd /etc/udev/permissions.d

3. Start a text editor, such as vi, and enter the partition information where you want
to place the OCR and voting disk files, using the syntax

device[partitions]:root:oinstall:0640.

Note that Oracle recommends that you place the OCR and the voting disk files on separate physical disks. For example, to grant oinstall members access to SCSI disks to place OCR files on sda1 and sdb2, and to grant the Oracle Clusterware owner (in this example crs) permissions to place voting disks on sdb3, sdc1 and sda2, add the following information to the file:

# OCR disks
sda1:root:oinstall:0640
sdb2:root:oinstall:0640

# Voting disks
sda2:crs:oinstall:0640
sdb3:crs:oinstall:0640
sdc1:crs:oinstall:0640

4. Save the file:
On Asianux 2, Enterprise Linux 4, and Red Hat Enterprise Linux 4 systems, save
the file as 49-oracle.permissions.
On Asianux 3, Enterprise Linux 5, Red Hat Enterprise Linux 5, and SUSE
Enterprise Server 10 systems, save the file as 51-oracle.permissions.

Eg. Setting raw devices permissions

#vote disks
raw/raw1:oracle:oinstall:0644
raw/raw2:oracle:oinstall:0644
raw/raw3:oracle:oinstall:0644

# ASM SPfile
raw/raw4:oracle:oinstall:0640

#OCR
raw/raw5:root:oinstall:0640
raw/raw6:root:oinstall:0640