Installing 10gR2 clusterware on RHEL 5

In RHEL/OEL 5 and onwards raw devices are deprecated (raw devices was initially deprecated in EL5 (GA-U3), but later undeprecated from EL5 U4). Therefore OCFS2 would be the only option to store OCR, VOTE disks and even ASM spfile. But there's a way to enable raw devices on RHEL/OEL 5 and process is explained in metalink note 465001.1 for singlepath raw devices and 564580.1 for multipath raw devices.

This blog is for singlepath.

1. Edit the scsi_id.config to return the device ID. This is done by adding option=-g to the /etc/scsi_id.config file.

# some libata drives require vpd page 0x80
vendor="ATA",options=-b 0x80
options=-g 

2. Get the device IDs with

# /sbin/scsi_id -g -s /block/sdc/sdc1
360a98000686f6959684a45124114174

Note down these IDs they will be used in the next step

3. Create a udev rules file in

vi /etc/udev/rules.d/61-oracleraw.rules

and add the following lines

ACTION=="add", KERNEL=="sd*", PROGRAM=="/sbin/scsi_id", RESULT=="360a98000686f6959684a45124114174", RUN+="/bin/raw /dev/raw/raw1 %N"
ACTION=="add", KERNEL=="sd*", PROGRAM=="/sbin/scsi_id", RESULT=="360a98000686f6959684a451241131151", RUN+="/bin/raw /dev/raw/raw2 %N"

For each rule, if all specified keys (KERNEL, PROGRAM, RESULT) are matched and raw device created if one or more keys are unmatched, the rule is completely ignored and the default (arbitrary) kernel-assigned device file name is assigned to devices.
The 'RUN+=' directive, is always the very last directive to execute in a rule file, therefore a separate file is used to permission resultant raw devices.
4. Create another udev rules files to set the permission on the raw devices.

 vi /etc/udev/rules.d/65-raw-permissions.rules

and add the following entries

KERNEL=="raw1", OWNER="root", GROUP="oinstall", MODE="640"
KERNEL=="raw2", OWNER="oracle", GROUP="oinstall", MODE="640"

5.Check the udev rules with

udevtest /block/sdc/sdc1

The device path should be the one relative to sysfs directory /sys/block. Restart the udev service with

start_udev

and verify the raw devices are enabled with

raw -qa

and permission on raw devices with

ls -l /dev/raw/* 

Metalink note also describes a way to create named devices. This is not related to raw device bindings.

 vi etc/udev/rules.d/55-oracle-naming.rules

and add the following lines

KERNEL=="sd*", BUS=="scsi", PROGRAM=="/sbin/scsi_id", RESULT=="360a98000686f6959684a45124114174", NAME="ocr1", OWNER="root", GROUP="oinstall", MODE="0640"
KERNEL=="sd*", BUS=="scsi", PROGRAM=="/sbin/scsi_id", RESULT=="360a98000686f6959684a451241131151", NAME="vote1", OWNER="oracle", GROUP="oinstall", MODE="0640"

For each rule, if all specified keys (KERNEL, BUS, PROGRAM, RESULT) are matched, the rule is applied and the specified assignments (NAME, OWNER, GROUP, MODE) are made and applied to the device. If, however, one or more keys are unmatched, the rule is completely ignored and the default (arbitrary) kernel-assigned device file name is assigned to devices.
Restart the udev service.

For system that are running on low-end hardware for demonstration or proof-of-concept purposes this might not work. The maxtor external device that was used to test this installation didn't give the device ID as the above command.

/sbin/scsi_id -g -s /block/sdc/sdc5
1Maxtor  OneTouch        2HA1BNAG

In this case udev rules file could be created applying rules for each individual partition.

vi 61-oracleraw.rules
ACTION=="add", KERNEL=="sdc5", RUN+="/bin/raw /dev/raw/raw1 %N"
ACTION=="add", KERNEL=="sdc6", RUN+="/bin/raw /dev/raw/raw2 %N"
ACTION=="add", KERNEL=="sdc7", RUN+="/bin/raw /dev/raw/raw3 %N"
ACTION=="add", KERNEL=="sdc8", RUN+="/bin/raw /dev/raw/raw4 %N"
ACTION=="add", KERNEL=="sdc9", RUN+="/bin/raw /dev/raw/raw5 %N"

and another rules file for permissions.

vi 65-raw-permissions.rules
KERNEL=="raw1", OWNER="root", GROUP="oinstall", MODE="640"
KERNEL=="raw2", OWNER="oracle", GROUP="oinstall", MODE="660"
KERNEL=="raw3", OWNER="oracle", GROUP="oinstall", MODE="660"
KERNEL=="raw4", OWNER="oracle", GROUP="dba", MODE="660"
KERNEL=="raw5", OWNER="oracle", GROUP="dba", MODE="660"

Restart the udev service and verify raw devices are enabled.

There are several other issues when installing on RHEL5 and these are documented on metalin note 414163.1
1. To run the installater ignoreSysPrereqs option is needed

2. Virtual IP Configuration Assistant (VIPCA) failes at the end of root.sh. To solve this before running the root.sh edit

$CRS_HOME/bin/vipca

and unset the LD_ASSUME_KERNEL variable.

if [ "$arch" = "i686" -o "$arch" = "ia64" -o "$arch" = "x86_64" ]
then
  LD_ASSUME_KERNEL=2.4.19
  export LD_ASSUME_KERNEL
fi

unset LD_ASSUME_KERNEL #add this line

Also edit the following files

$CRS_HOME/bin/srvctl
$RDBMS_HOME/bin/srvctl
$ASM_HOME/bin/srvctl

and unset the same variable as above

LD_ASSUME_KERNEL=2.4.19
export LD_ASSUME_KERNEL

unset LD_ASSUME_KERNEL # add this line 

These changes may be necessary after applying the 10.2.0.2 or 10.2.0.3 patchsets, as these patchset will still include those settings unnecessary for OEL5 or RHEL5 or SLES10. It is fixed in the 10.2.0.4 and 10.2.0.5 patchsets.

3. VIP will again fail if the VIP IP's are in a non-routable range eg. 10.x.x.x, 172.[16-31].x.x, 192.168.x.x range. If the OUI window is open, click OK and it will create the "oifcfg" information. Run the VIPCA manually as root and once done retry the cluvfy part at the end of OUI, and it should succeed.
If not then use

oifcfg setif -global eth*

to set public and cluster_interconnect interfaces.

Installing RAC on RHEL 5 after the clusterware did not throw any errors.