Wednesday, June 30, 2010

Installing 10gR2 clusterware on RHEL 5

In RHEL/OEL 5 and onwards raw devices are deprecated (raw devices was initially deprecated in EL5 (GA-U3), but later undeprecated from EL5 U4). Therefore OCFS2 would be the only option to store OCR, VOTE disks and even ASM spfile. But there's a way to enable raw devices on RHEL/OEL 5 and process is explained in metalink note 465001.1 for singlepath raw devices and 564580.1 for multipath raw devices.

This blog is for singlepath.

1. Edit the scsi_id.config to return the device ID. This is done by adding option=-g to the /etc/scsi_id.config file.
# some libata drives require vpd page 0x80
vendor="ATA",options=-b 0x80
2. Get the device IDs with
# /sbin/scsi_id -g -s /block/sdc/sdc1
Note down these IDs they will be used in the next step

3. Create a udev rules file in
vi /etc/udev/rules.d/61-oracleraw.rules
and add the following lines
ACTION=="add", KERNEL=="sd*", PROGRAM=="/sbin/scsi_id", RESULT=="360a98000686f6959684a45124114174", RUN+="/bin/raw /dev/raw/raw1 %N"
ACTION=="add", KERNEL=="sd*", PROGRAM=="/sbin/scsi_id", RESULT=="360a98000686f6959684a451241131151", RUN+="/bin/raw /dev/raw/raw2 %N"
For each rule, if all specified keys (KERNEL, PROGRAM, RESULT) are matched and raw device created if one or more keys are unmatched, the rule is completely ignored and the default (arbitrary) kernel-assigned device file name is assigned to devices.
The 'RUN+=' directive, is always the very last directive to execute in a rule file, therefore a separate file is used to permission resultant raw devices.
4. Create another udev rules files to set the permission on the raw devices.
 vi /etc/udev/rules.d/65-raw-permissions.rules
and add the following entries
KERNEL=="raw1", OWNER="root", GROUP="oinstall", MODE="640"
KERNEL=="raw2", OWNER="oracle", GROUP="oinstall", MODE="640"
5.Check the udev rules with
udevtest /block/sdc/sdc1
The device path should be the one relative to sysfs directory /sys/block. Restart the udev service with
and verify the raw devices are enabled with
raw -qa
and permission on raw devices with
ls -l /dev/raw/* 

Metalink note also describes a way to create named devices. This is not related to raw device bindings.
 vi etc/udev/rules.d/55-oracle-naming.rules
and add the following lines
KERNEL=="sd*", BUS=="scsi", PROGRAM=="/sbin/scsi_id", RESULT=="360a98000686f6959684a45124114174", NAME="ocr1", OWNER="root", GROUP="oinstall", MODE="0640"
KERNEL=="sd*", BUS=="scsi", PROGRAM=="/sbin/scsi_id", RESULT=="360a98000686f6959684a451241131151", NAME="vote1", OWNER="oracle", GROUP="oinstall", MODE="0640"
For each rule, if all specified keys (KERNEL, BUS, PROGRAM, RESULT) are matched, the rule is applied and the specified assignments (NAME, OWNER, GROUP, MODE) are made and applied to the device. If, however, one or more keys are unmatched, the rule is completely ignored and the default (arbitrary) kernel-assigned device file name is assigned to devices.
Restart the udev service.

For system that are running on low-end hardware for demonstration or proof-of-concept purposes this might not work. The maxtor external device that was used to test this installation didn't give the device ID as the above command.
/sbin/scsi_id -g -s /block/sdc/sdc5
1Maxtor OneTouch 2HA1BNAG
In this case udev rules file could be created applying rules for each individual partition.
vi 61-oracleraw.rules
ACTION=="add", KERNEL=="sdc5", RUN+="/bin/raw /dev/raw/raw1 %N"
ACTION=="add", KERNEL=="sdc6", RUN+="/bin/raw /dev/raw/raw2 %N"
ACTION=="add", KERNEL=="sdc7", RUN+="/bin/raw /dev/raw/raw3 %N"
ACTION=="add", KERNEL=="sdc8", RUN+="/bin/raw /dev/raw/raw4 %N"
ACTION=="add", KERNEL=="sdc9", RUN+="/bin/raw /dev/raw/raw5 %N"
and another rules file for permissions.
vi 65-raw-permissions.rules
KERNEL=="raw1", OWNER="root", GROUP="oinstall", MODE="640"
KERNEL=="raw2", OWNER="oracle", GROUP="oinstall", MODE="660"
KERNEL=="raw3", OWNER="oracle", GROUP="oinstall", MODE="660"
KERNEL=="raw4", OWNER="oracle", GROUP="dba", MODE="660"
KERNEL=="raw5", OWNER="oracle", GROUP="dba", MODE="660"
Restart the udev service and verify raw devices are enabled.

There are several other issues when installing on RHEL5 and these are documented on metalin note 414163.1
1. To run the installater ignoreSysPrereqs option is needed

2. Virtual IP Configuration Assistant (VIPCA) failes at the end of To solve this before running the edit
and unset the LD_ASSUME_KERNEL variable.
if [ "$arch" = "i686" -o "$arch" = "ia64" -o "$arch" = "x86_64" ]

unset LD_ASSUME_KERNEL #add this line

Also edit the following files
and unset the same variable as above

unset LD_ASSUME_KERNEL # add this line
These changes may be necessary after applying the or patchsets, as these patchset will still include those settings unnecessary for OEL5 or RHEL5 or SLES10. It is fixed in the and patchsets.

3. VIP will again fail if the VIP IP's are in a non-routable range eg. 10.x.x.x, 172.[16-31].x.x, 192.168.x.x range. If the OUI window is open, click OK and it will create the "oifcfg" information. Run the VIPCA manually as root and once done retry the cluvfy part at the end of OUI, and it should succeed.
If not then use
oifcfg setif -global eth*
to set public and cluster_interconnect interfaces.

Installing RAC on RHEL 5 after the clusterware did not throw any errors.