Upgrading RHEL 7 OS in a 11gR2 (11.2.0.4) and 12cR1 (12.1.0.2) Oracle Restart Environments

As per Oracle documentation "You must relink the Oracle Clusterware and Oracle ASM binaries every time you apply an operating system patch or after an operating system upgrade." There are two earlier posts which shows upgrading of RHEL5 and RHEL6 OS in a 11gR2 RAC environment. This post shows relinking of GI and Oracle homes after upgrading of RHEL 7 in 11.2.0.4 and 12.1.0.2 single instance (Oracle restart) environments.
1. The current RHEL version is 7.1 and the kernel is

uname -r
3.10.0-123.el7.x86_64

2. Before the OS upgrade stop the HAS and disable the auto start on reboot.

crsctl stop has
crsctl disable has
CRS-4621: Oracle High Availability Services autostart is disabled.

3. Comment the entries in oracle-ohasd.service

cat /etc/systemd/system/oracle-ohasd.service
# Copyright (c) 2014, Oracle and/or its affiliates. All rights reserved.
#
# Oracle OHASD startup

[Unit]
#Description=Oracle High Availability Services
#After=syslog.target

[Service]
#ExecStart=/etc/init.d/init.ohasd run >/dev/null 2>&1 </dev/null
#Type=simple
#Restart=always

[Install]
#WantedBy=multi-user.target graphical.target

4. Upgrade the RHEL 7 OS. Kernel and RHEL7 version after the upgrade are

uname -r
3.10.0-327.el7.x86_64
cat /etc/redhat-release
Red Hat Enterprise Linux Server release 7.2 (Maipo)

5. Uncomment the entires in oracle-ohasd.service file and enable auto start of HAS

crsctl enable has
CRS-4622: Oracle High Availability Services autostart is enabled.

Until this point the steps are common for both 11.2.0.4 and 12.1.0.2.

Relinking 11.2.0.4 GI and Oracle Homes
1.Relinking Oracle Home as oracle user. Make sure the relink binary in the PATH is from Oracle home

which relink
/opt/app/oracle/product/11.2.0/dbhome_4/bin/relink

Run relink all

cd $ORACLE_HOME/bin
./relink all
writing relink log to: /opt/app/oracle/product/11.2.0/dbhome_4/install/relink.log

At the end of the relink the log file have the following

test ! -f /opt/app/oracle/product/11.2.0/dbhome_4/bin/oracle ||\
   mv -f /opt/app/oracle/product/11.2.0/dbhome_4/bin/oracle /opt/app/oracle/product/11.2.0/dbhome_4/bin/oracleO
mv /opt/app/oracle/product/11.2.0/dbhome_4/rdbms/lib/oracle /opt/app/oracle/product/11.2.0/dbhome_4/bin/oracle
chmod 6751 /opt/app/oracle/product/11.2.0/dbhome_4/bin/oracle

2. Relinkning GI Home
Set ORACLE_HOME=$GI_HOME
As root unlock the GI Home. Using the RHEL 7 provided perl version will result in following error

# perl $ORACLE_HOME/crs/install/roothas.pl -unlock
Can't locate Env.pm in @INC (@INC contains: /usr/local/lib64/perl5 /usr/local/share/perl5 /usr/lib64/perl5/vendor_perl /usr/share/perl5/vendor_perl /usr/lib64/perl5 /usr/share/perl5 . /opt/app/oracle/product/11.2.0/grid/crs/install) at /opt/app/oracle/product/11.2.0/grid/crs/install/crsconfig_lib.pm line 710.
BEGIN failed--compilation aborted at /opt/app/oracle/product/11.2.0/grid/crs/install/crsconfig_lib.pm line 710.
Compilation failed in require at /opt/app/oracle/product/11.2.0/grid/crs/install/roothas.pl line 171.
BEGIN failed--compilation aborted at /opt/app/oracle/product/11.2.0/grid/crs/install/roothas.pl line 171.

Solution for this was mentioned on the changing of hostname post which is to use oracle provided perl

# $ORACLE_HOME/perl/bin/perl $ORACLE_HOME/crs/install/roothas.pl -unlock
Using configuration parameter file: /opt/app/oracle/product/11.2.0/grid/crs/install/crsconfig_params
Successfully unlock /opt/app/oracle/product/11.2.0/grid

3. As grid user run the relink

$ORACLE_HOME/bin/relink
writing relink log to: /opt/app/oracle/product/11.2.0/grid/install/relink.log

The log file will have the following entry at the end

tail /opt/app/oracle/product/11.2.0/grid/install/relink.log

log tail
test ! -f /opt/app/oracle/product/11.2.0/grid/bin/oracle ||\
   mv -f /opt/app/oracle/product/11.2.0/grid/bin/oracle /opt/app/oracle/product/11.2.0/grid/bin/oracleO
mv /opt/app/oracle/product/11.2.0/grid/rdbms/lib/oracle /opt/app/oracle/product/11.2.0/grid/bin/oracle
chmod 6751 /opt/app/oracle/product/11.2.0/grid/bin/oracle

4. As root patch the GI Home

# cd $ORACLE_HOME/rdbms/install
# ./rootadd_rdbms.sh

$ORACLE_HOME/perl/bin/perl $ORACLE_HOME/crs/install/roothas.pl -patch
Using configuration parameter file: /opt/app/oracle/product/11.2.0/grid/crs/install/crsconfig_params
Broadcast message from systemd-journald@rhel7s.domain.net (Thu 2016-02-18 16:50:14 GMT):
dracut[16456]: dracut: creation of /boot/initramfs-3.10.0-327.el7.x86_64.tmp failed

Message from syslogd@rhel7s at Feb 18 16:50:14 ...
 dracut:dracut: creation of /boot/initramfs-3.10.0-327.el7.x86_64.tmp failed
ACFS driver install actions failed
CRS-4124: Oracle High Availability Services startup failed.
CRS-4000: Command Start failed, or completed with errors.
Timed out waiting for ohasd to start.

During the patching the HAS stack start will fail, this is expected.
5. Reboot the host after uncommenting the lines in oracle-ohasd.service together with enabling HAS service (crsctl enable has). After the reboot HAS stack will be up and running as before

Resource Name                       Type                           Target             State              Host
-------------                       ------                         -------            --------           ----------
ora.DATA.dg                         ora.diskgroup.type             ONLINE             ONLINE             rhel7s
ora.FRA.dg                          ora.diskgroup.type             ONLINE             ONLINE             rhel7s
ora.LISTENER.lsnr                   ora.listener.type              ONLINE             ONLINE             rhel7s
ora.asm                             ora.asm.type                   ONLINE             ONLINE             rhel7s
ora.cssd                            ora.cssd.type                  ONLINE             ONLINE             rhel7s
ora.diskmon                         ora.diskmon.type               OFFLINE            OFFLINE
ora.evmd                            ora.evm.type                   ONLINE             ONLINE             rhel7s
ora.ons                             ora.ons.type                   OFFLINE            OFFLINE
ora.std11g2.db                      ora.database.type              ONLINE             ONLINE             rhel7s

Relinking 12.1.0.2 GI and Oracle Homes
1. Relinking Oracle home as oracle user. Make sure the relink binary in PATH is from Oracle home. Run relink all to relink the oracle home.

relink all
writing relink log to: /opt/app/oracle/product/12.1.0/dbhome_2/install/relink.log

The log file end with the following

tail /opt/app/oracle/product/12.1.0/dbhome_2/install/relink.log
test ! -f /opt/app/oracle/product/12.1.0/dbhome_2/bin/oracle ||\
   mv -f /opt/app/oracle/product/12.1.0/dbhome_2/bin/oracle /opt/app/oracle/product/12.1.0/dbhome_2/bin/oracleO
mv /opt/app/oracle/product/12.1.0/dbhome_2/rdbms/lib/oracle /opt/app/oracle/product/12.1.0/dbhome_2/bin/oracle
chmod 6751 /opt/app/oracle/product/12.1.0/dbhome_2/bin/oracle

2. As root unlock the GI Home by using the $GI_HOME/crs/install/roothas.sh. Unlike 11.2.0.4 there are no issues in unlocking GI home in 12c

./roothas.sh -unlock
Using configuration parameter file: /opt/app/oracle/product/12.1.0/grid/crs/install/crsconfig_params
2016/02/19 18:52:17 CLSRSC-347: Successfully unlock /opt/app/oracle/product/12.1.0/grid

3. As grid user run the relink.

which relink
/opt/app/oracle/product/12.1.0/grid/bin/relink
$ relink
writing relink log to: /opt/app/oracle/product/12.1.0/grid/install/relink.log

The log file will end with the following

tail /opt/app/oracle/product/12.1.0/grid/install/relink.log

test ! -f /opt/app/oracle/product/12.1.0/grid/bin/oracle ||\
   mv -f /opt/app/oracle/product/12.1.0/grid/bin/oracle /opt/app/oracle/product/12.1.0/grid/bin/oracleO
mv /opt/app/oracle/product/12.1.0/grid/rdbms/lib/oracle /opt/app/oracle/product/12.1.0/grid/bin/oracle
chmod 6751 /opt/app/oracle/product/12.1.0/grid/bin/oracle

4. As root patch the GI Home

# $GI_HOME/rdbms/install/rootadd_rdbms.sh

./roothas.sh -patch
Using configuration parameter file: /opt/app/oracle/product/12.1.0/grid/crs/install/crsconfig_params
CRS-4124: Oracle High Availability Services startup failed.
CRS-4000: Command Start failed, or completed with errors.
2016/02/19 18:59:50 CLSRSC-199: Timed out waiting for OHASD to start

The command '/opt/app/oracle/product/12.1.0/grid/perl/bin/perl -I/opt/app/oracle/product/12.1.0/grid/perl/lib -I/opt/app/oracle/product/12.1.0/grid/crs/install /opt/app/oracle/product/12.1.0/grid/crs/install/roothas.pl -patch' execution failed

5. Similar to 11.2.0.4, start of the HAS stack will fail, this is expected. Reboot the host after uncommenting the lines in oracle-ohasd.service together with enabling HAS service (crsctl enable has). After the reboot HAS will be up and running.

Resource Name             Type                      Target             State              Host
-------------             ------                    -------            --------           ----------
ora.DATA.dg               ora.diskgroup.type        ONLINE             ONLINE             rhel7
ora.FRA.dg                ora.diskgroup.type        ONLINE             ONLINE             rhel7
ora.LISTENER.lsnr         ora.listener.type         ONLINE             ONLINE             rhel7
ora.asm                   ora.asm.type              ONLINE             ONLINE             rhel7
ora.cssd                  ora.cssd.type             ONLINE             ONLINE             rhel7
ora.diskmon               ora.diskmon.type          OFFLINE            OFFLINE
ora.evmd                  ora.evm.type              ONLINE             ONLINE             rhel7
ora.ons                   ora.ons.type              OFFLINE            OFFLINE
ora.se2db.db              ora.database.type         ONLINE             ONLINE             rhel7

This concludes the steps for relinking GI and Oracle homes after RHEL 7 OS upgrade.

Related Posts
Upgrading RHEL 6 OS in a 11gR2 RAC Environment
Upgrading OS in 11gR2 RAC Environment
Upgrading ASMLib and OS in 11gR1 RAC Environment

Update on 2017-07-10
As per 2282371.1 and 2281492.1 certain RHEL 7 kernel versions (3.10.0-514.21.2.el7.x86_64) cause issues with regard to starting of OHASD in 11.2.0.4. Avoid these kernel versions when installing or upgrading OS when 11.2.0.4 is installed.

Useful metalink notes
OHASD Fails to Start With Kernel Version 3.10.0-514.21.2.el7.x86_64 [ID 2281492.1]
Grid Infrastructure Fails to Start OHASD With RedHat Linux or Oracle Linux with RedHat Compatible Kernel (RHCK) Version 3.10.0-514.21.2.EL7.X86_64 or Higher [ID 2282371.1]

Upgrading RHEL 6 OS in a 11gR2 RAC Environment

This post upgrade the RHEL 6 OS on severs which are running 11gR2 RAC. Current kernel is

$ uname -r
2.6.32-220.el6.x86_64

and the database patch level is

opatch lsinventory -local | grep Patch
Oracle Interim Patch Installer version 11.2.0.3.0
OPatch version    : 11.2.0.3.0
Patch  14275605     : applied on Thu Nov 15 14:21:27 GMT 2012
Unique Patch ID:  15379762
Patch description:  "Database Patch Set Update : 11.2.0.3.4 (14275605)"
Sub-patch  13923374; "Database Patch Set Update : 11.2.0.3.3 (13923374)"
Sub-patch  13696216; "Database Patch Set Update : 11.2.0.3.2 (13696216)"
Sub-patch  13343438; "Database Patch Set Update : 11.2.0.3.1 (13343438)"
Patch  14275572     : applied on Thu Nov 15 14:20:32 GMT 2012
Unique Patch ID:  15379762
Patch description:  "Grid Infrastructure Patch Set Update : 11.2.0.3.4 (14275572)"
OPatch succeeded.

The database software is running standard edition.
The steps are similar to that of upgrading RHEL 5 OS running 11gR2 RAC with few minor differences. This RAC system does not use ASMLibs.
1. Current kernel version

$ uname -r
2.6.32-220.el6.x86_64

2. Stop all the cluster components

crsctl stop cluster -all

and stop disable start of crs after reboot

crsctl stop crs
crsctl disable crs

3. In RHEL 6 inittab is depreciated and the spawning process file is located in /etc/init with the name oracle-ohasd.conf
More on this is available on Troubleshoot Grid Infrastructure Startup Issues [ID 1050908.1] If Oracle is 11.2.0.4 comment out the trace file analyzer script as well oracle-tfa.conf.

cat oracle-ohasd.conf
# Copyright (c) 2001, 2011, Oracle and/or its affiliates. All rights reserved.
#
# Oracle OHASD startup

start on runlevel [35]
stop  on runlevel [!35]
respawn
exec /etc/init.d/init.ohasd run >/dev/null 2>&1 </dev/null

As root comment last two lines

#respawn
#exec /etc/init.d/init.ohasd run >/dev/null 2>&1 </dev/null

4. Upgrade the OS (only few screenshots shown)

5. New kernel version after upgrade

uname -r
2.6.32-279.el6.x86_64

6. Stop oracle-ohasd

initctl stop oracle-ohasd
oracle-ohasd stop/waiting

7. Un-comment the last two lines on /etc/init/oracle-ohasd that was commented earlier

start on runlevel [35]
stop  on runlevel [!35]
respawn
exec /etc/init.d/init.ohasd run >/dev/null 2>&1 </dev/null

8. Enable crs on startup

crsctl enable crs
CRS-4622: Oracle High Availability Services autostart is enabled.

9. Relink the Oracle Home binaries as the oracle software owner

which relink
/opt/app/oracle/product/11.2.0/dbhome_1/bin/relink
$ cd $ORACLE_HOME/bin
$ ./relink all
writing relink log to: /opt/app/oracle/product/11.2.0/dbhome_1/install/relink.log

The relink will end with following on the log

test ! -f /opt/app/oracle/product/11.2.0/dbhome_1/bin/oracle ||\
           mv -f /opt/app/oracle/product/11.2.0/dbhome_1/bin/oracle /opt/app/oracle/product/11.2.0/dbhome_1/bin/oracleO
mv /opt/app/oracle/product/11.2.0/dbhome_1/rdbms/lib/oracle /opt/app/oracle/product/11.2.0/dbhome_1/bin/oracle
chmod 6751 /opt/app/oracle/product/11.2.0/dbhome_1/bin/oracle

9. Unlock the permission on the grid software as root

# cd $GI_HOME/crs/install

# perl rootcrs.pl -unlock
Using configuration parameter file: ./crsconfig_params
CRS-4544: Unable to connect to OHAS
CRS-4000: Command Stop failed, or completed with errors.
Successfully unlock /opt/app/11.2.0/grid

10. As the grid software owner (gird in this case) relink the grid software

[grid@rhel6m1 ~]$ export ORACLE_HOME=$GI_HOME
[grid@rhel6m1 ~]$ export PATH=$ORACLE_HOME/bin:$PATH
[grid@rhel6m1 ~]$ which relink
/opt/app/11.2.0/grid/bin/relink
[grid@rhel6m1 ~]$ $GI_HOME/bin/relink
writing relink log to: /opt/app/11.2.0/grid/install/relink.log

The relink ends with

test ! -f /opt/app/11.2.0/grid/bin/oracle ||\
           mv -f /opt/app/11.2.0/grid/bin/oracle /opt/app/11.2.0/grid/bin/oracleO
mv /opt/app/11.2.0/grid/rdbms/lib/oracle /opt/app/11.2.0/grid/bin/oracle
chmod 6751 /opt/app/11.2.0/grid/bin/oracle

11. Run the post relink script. rootcrs.pl -patch takes a while to complete and ends with an error as shown below

# cd $GI_HOME/rdbms/install
# ./rootadd_rdbms.sh

# cd $GI_HOME/crs/install
# perl rootcrs.pl -patch
Using configuration parameter file: ./crsconfig_params
CRS-4124: Oracle High Availability Services startup failed.
CRS-4000: Command Start failed, or completed with errors.
Failed to write the checkpoint:'' with status:FAIL.Error code is 256
Oracle Grid Infrastructure stack start initiated but failed to complete at crsconfig_lib.pm line 11545.

After an SR it was confirmed that this error is ignorable and crs could be started with "init 3" or by rebooting the server. Reason that above error happen is that since "ohasd run" line was commented on oracle-ohasd file there's no "ohasd run" process active when the kernel is upgraded and server is rebooted. Without "ohasd run" active the crsctl start crs (issued during the rootcrs.pl -patch) will fail to start the clusterware stack. This has no effect on the actual relink. This is also mentioned in 1050908.1
For metalink notes related to Upgrading OS and relinking clusterware and oracle binaries after OS upgrade refer the related post given below.

Related Posts
Upgrading OS in 11gR2 RAC Environment (RHEL 5)
Upgrading ASMLib and OS in 11gR1 RAC Environment

Useful metalink notes
Executing "relink all" resets permission of extjob, jssu, oradism, externaljob.ora [ID 1555453.1]
Relinking Oracle Home FAQ ( Frequently Asked Questions) [ID 1467060.1]

Upgrading OS in 11gR2 RAC Environment

Oracle clusterware 11gR2 requires relinking whenever an OS patch or OS is upgraded. According to GI Installation guide "You must relink the Oracle Clusterware and Oracle ASM binaries every time you apply an operating system patch or after an operating system upgrade. Before relinking executables, you must shut down all executables that run in the Oracle home directory that you are relinking. In addition, shut down applications linked with Oracle shared libraries."
Relink is done with the following steps, again from the GI installation guide

As root:
# cd Grid_home/crs/install
# perl rootcrs.pl -unlock

As the Oracle Grid Infrastructure for a Cluster owner:
$ export ORACLE_HOME=Grid_home
$ Grid_home/bin/relink

As root again:
# cd Grid_home/rdbms/install/
# ./rootadd_rdbms.sh
# cd Grid_home/crs/install
# perl rootcrs.pl -patch

As per (743649.1) Oralce support rolling upgrade of OS when both OS are certified on the database that's running. This is supported only for the duration of the upgrade. But according to (1391807.1) using multiple versions of ASMLib across the cluster may not be compatible. In this case ASM disks were migrated from ASMLib disk to block devices before the upgrade therefore an ASMLib upgrade was not required. But if ASMLib was in use then ASMLib should be upgraded as part of the OS upgrade.
1. Current kernel version

uname -r
2.6.18-194.el5

2. This upgrade is done across the cluster (not a rolling upgrade) therefore stop all the cluster components

crsctl stop cluster -all

and stop disable start of crs

crsctl stop crs
crsctl disable crs

Also comment spawning of high availability processes

#h1:35:respawn:/etc/init.d/init.ohasd run >/dev/null 2>&1 </dev/null

# init q

3. Upgrade OS. New kernel version

uname -r
2.6.18-308.el5

4. Remove the comment of ohas from inittab and enable crs

h1:35:respawn:/etc/init.d/init.ohasd run >/dev/null 2>&1 </dev/null

crsctl enable crs

5. Relink ORACLE_HOME binaries

 which relink
/opt/app/oracle/product/11.2.0/db_2/bin/relink

cd $ORACLE_HOME/bin
$ relink all
writing relink log to: /opt/app/oracle/product/11.2.0/db_2/install/relink.log

Relink ends with following in the log file

test ! -f /opt/app/oracle/product/11.2.0/db_2/bin/oracle ||\
           mv -f /opt/app/oracle/product/11.2.0/db_2/bin/oracle /opt/app/oracle/product/11.2.0/db_2/bin/oracleO
mv /opt/app/oracle/product/11.2.0/db_2/rdbms/lib/oracle /opt/app/oracle/product/11.2.0/db_2/bin/oracle
chmod 6751 /opt/app/oracle/product/11.2.0/db_2/bin/oracle

5. Relink the GI HOME binaries

cd $GI_HOME/crs/install

# perl rootcrs.pl -unlock
Using configuration parameter file: ./crsconfig_params
CRS-4544: Unable to connect to OHAS
CRS-4000: Command Stop failed, or completed with errors.
Successfully unlock /opt/app/11.2.0/grid3

$ export ORACLE_HOME=$GI_HOME
$ export PATH=$ORACLE_HOME/bin:$PATH
$ which relink
/opt/app/11.2.0/grid3/bin/relink

$GI_HOME/bin/relink

Relink ends with the following in the log file

test ! -f /opt/app/11.2.0/grid3/bin/oracle ||\
           mv -f /opt/app/11.2.0/grid3/bin/oracle /opt/app/11.2.0/grid3/bin/oracleO
mv /opt/app/11.2.0/grid3/rdbms/lib/oracle /opt/app/11.2.0/grid3/bin/oracle
chmod 6751 /opt/app/11.2.0/grid3/bin/oracle

Complete the relink process

# cd $GI_HOME/rdbms/install

# ./rootadd_rdbms.sh

# cd $GI_HOME/crs/install
# perl rootcrs.pl -patch

Using configuration parameter file: ./crsconfig_params
CRS-4124: Oracle High Availability Services startup failed.
CRS-4000: Command Start failed, or completed with errors.

Execute

/sbin/init q

to start the cluster stack.

Related Post
Upgrading ASMLib and OS in 11gR1 RAC Environment
Upgrading RHEL 6 OS in a 11gR2 RAC Environment

Useful Metalink notes
Oracle 11gR2 Relink New Feature [ID 883299.1]

Upgrading ASMLib and OS in 11gR1 RAC Environment

Upgrading operating system (OS) requires several consideration and tasks.

1. If ASMLib is used then ASMLib libraries must be upgraded as part of the OS upgrade.
2. After the upgrade Oracle/clusterware binaries should be relinked.

In this case the two nodes 11gR1 (11.1.0.7) RAC environment running on RHEL 5 (2.6.18-194.el5) will be upgraded to RHEL 5 (2.6.18-308.el5).
As per (743649.1) Oralce support rolling upgrade of OS when both OS are certified on the database that's running. This is supported only for the duration of the upgrade. But according to (1391807.1) using multiple versions of ASMLib across the cluster may not be compatible. To avoid these incompatible issues and to minimize the total downtime one node could be upgraded first and then shutting down the other node before the upgraded one with new version of ASMLib is started. This way only one node will be active in the cluster when two nodes are having different ASMLib versions. In this case all nodes are upgraded at the same time incurring total system outage.

1. Find out if ASMLibs are available for the kernel version to which the system is being upgraded to. If not move asmlib devices to block devices before the upgrade (also true if upgrading to RHEL 6 where to get asmlibs require ULN account) or see if (462618.1) could be of any help.

2. Make a note of the ownership and permission of the following files in $CRS_HOME/bin directory using the script below. Some files might be missing this is not a problem.

#!/bin/sh

if [ "$CRS_HOME" != "" ]; then
                echo "crs home exist"

        for i in clntsh.map clntst_1.lis clntst_2.lis clntst.lis libclntsh.so libclntsh.so.10.1 libclntst10.so libclntsh.so.11.1 libclntst11.so clntst
        do
                if [ -f $CRS_HOME/lib/$i ]; then
                        echo "file exists"
                        ls -l $CRS_HOME/lib/$i

                        if [ -f $CRS_HOME/lib32/$i ]; then
                                echo "file in lib32"
                                ls -l $CRS_HOME/lib32/$i
                        else
                                echo "file not exists"
                        fi
                else
                        if [ -f $CRS_HOME/lib32/$i ]; then
                                echo "file in lib32"
                                ls -l $CRS_HOME/lib32/$i
                        else
                                echo "file not exists"
                        fi
                fi
        done
else
                echo "Please set CRS_HOME and run again"
fi

./cluster_file_check.sh
crs home exist
file exists
-rwxr--r-- 1 root root 4765562 Feb  8  2011 /opt/crs/oracle/product/11.1.0/crs/lib/clntsh.map
file in lib32
-rw-r--r-- 1 root root 3985432 Feb  8  2011 /opt/crs/oracle/product/11.1.0/crs/lib32/clntsh.map
file not exists
file not exists
file not exists
file exists
lrwxrwxrwx 1 root root 17 Feb  8  2011 /opt/crs/oracle/product/11.1.0/crs/lib/libclntsh.so -> libclntsh.so.11.1
file in lib32
lrwxrwxrwx 1 root root 17 Feb  8  2011 /opt/crs/oracle/product/11.1.0/crs/lib32/libclntsh.so -> libclntsh.so.11.1
file exists
lrwxrwxrwx 1 oracle oinstall 51 Feb  8  2011 /opt/crs/oracle/product/11.1.0/crs/lib/libclntsh.so.10.1 -> /opt/crs/oracle/product/11.1.0/crs/lib/libclntsh.so
file in lib32
lrwxrwxrwx 1 oracle oinstall 53 Feb  8  2011 /opt/crs/oracle/product/11.1.0/crs/lib32/libclntsh.so.10.1 -> /opt/crs/oracle/product/11.1.0/crs/lib32/libclntsh.so
file not exists
file exists
-rwxr-xr-x 1 root root 48316147 Feb  8  2011 /opt/crs/oracle/product/11.1.0/crs/lib/libclntsh.so.11.1
file in lib32
-rwxr-xr-x 1 root root 37081093 Feb  8  2011 /opt/crs/oracle/product/11.1.0/crs/lib32/libclntsh.so.11.1
file not exists
file not exists

3. Make a note of the asm disk's scsi_ids , this setup is a test setup using Virtualbox. The scsi_ids will be compared after the OS upgrade and must be the same as before the upgrade.

# blkid|grep sd.*oracleasm|while read a b;do echo -n $a$b" scsi_id=";(echo $a|tr -d [:digit:]|tr -d [:]|cut -d"/" -f3|xargs -i scsi_id -g -s /block/{})done;
/dev/sdc1:LABEL="DATA" TYPE="oracleasm" scsi_id=SATA     VBOX HARDDISK  VB35fd6dbb-14d87b49
/dev/sdd1:LABEL="FLASH" TYPE="oracleasm" scsi_id=SATA     VBOX HARDDISK  VB4b88ed44-eff71eb1

4. Shutdown the cluster stack across the cluster. If one node at a time is upgraded then do this only on one node. In this case since all nodes are upgraded at the same time entire cluster is brought down

crs_stop -all

5. Stop the crs and disbale auto start of the crs. Do this on all ndoes.

# crsctl stop crs
Stopping resources.
This could take several minutes.
Successfully stopped Oracle Clusterware resources
Stopping Cluster Synchronization Services.
Shutting down the Cluster Synchronization Services daemon.
Shutdown request successfully issued.

# crsctl disable crs
Oracle Clusterware is disabled for start-up after a reboot.

6. Comment out the crs spwaning processes entries in /etc/inittab. Do this on all nodes

#h1:35:respawn:/etc/init.d/init.evmd run >/dev/null 2>&1 </dev/null
#h2:35:respawn:/etc/init.d/init.cssd fatal >/dev/null 2>&1 </dev/null
#h3:35:respawn:/etc/init.d/init.crsd run >/dev/null 2>&1 </dev/null

Run init on the systme and verify that no crs related processes are running.

/sbin/init q

ps -ef|grep css
ps -ef|grep crs
ps -ef|grep evm
ps -ef|grep init

7. Stop oracleasm running and disable and prevent it loading on reboot. Do this on all nodes.

# /etc/init.d/oracleasm stop
Dropping Oracle ASMLib disks:                              [  OK  ]
Shutting down the Oracle ASMLib driver:                    [  OK  ]

# /etc/init.d/oracleasm disable
Writing Oracle ASM library driver configuration: done
Dropping Oracle ASMLib disks:                              [  OK  ]
Shutting down the Oracle ASMLib driver:                    [  OK  ]

# /sbin/chkconfig --list | grep oracleasm
oracleasm       0:off   1:off   2:on    3:on    4:on    5:on    6:off
# /sbin/chkconfig oracleasm off
# /sbin/chkconfig --list | grep oracleasm
oracleasm       0:off   1:off   2:off   3:off   4:off   5:off   6:off

8. Make a backup of /etc/sysconfig/oracleasm-_dev_oracleasm on all nodes.

# cp /etc/sysconfig/oracleasm-_dev_oracleasm /etc/sysconfig/oracleasm-_dev_oracleasm.bak

At this point the system is ready for the OS upgrade.

9. Carry out the OS ugprade. All Oracle Home locations (ORACLE_HOME,ASM_HOME and CRS_HOME) are to be same before and after the upgrade

10. Install the new version of oracleasm and remove the old version. Old version could only be removed after installing the new version due to dependencies.

rpm -e oracleasm-2.6.18-194.el5-2.0.5-1.el5
error: Failed dependencies:
        oracleasm >= 1.0.4 is needed by (installed) oracleasmlib-2.0.4-1.el5.x86_64

# rpm -ivh oracleasm-2.6.18-308.el5-2.0.5-1.el5.x86_64.rpm
warning: oracleasm-2.6.18-308.el5-2.0.5-1.el5.x86_64.rpm: Header V3 DSA signature: NOKEY, key ID 1e5e0159
Preparing...                ########################################### [100%]
   1:oracleasm-2.6.18-308.el########################################### [100%]

# rpm -e oracleasm-2.6.18-194.el5-2.0.5-1.el5

# rpm -qa | grep oracleasm
oracleasmlib-2.0.4-1.el5
oracleasm-2.6.18-308.el5-2.0.5-1.el5
oracleasm-support-2.1.3-1.el5

11. Configure oracleasm and make sure content on oracleasm-_dev_oracleasm.bak and oracleasm-_dev_oracleasm are the same.

/etc/init.d/oracleasm configure
Configuring the Oracle ASM library driver.

This will configure the on-boot properties of the Oracle ASM library
driver.  The following questions will determine whether the driver is
loaded on boot and what permissions it will have.  The current values
will be shown in brackets ('[]').  Hitting  without typing an
answer will keep that current value.  Ctrl-C will abort.

Default user to own the driver interface [oracle]:
Default group to own the driver interface [dba]:
Start Oracle ASM library driver on boot (y/n) [n]: y
Scan for Oracle ASM disks on boot (y/n) [y]:
Writing Oracle ASM library driver configuration: done
Initializing the Oracle ASMLib driver:                     [  OK  ]
Scanning the system for Oracle ASMLib disks:               [  OK  ]

12. Enable oracleasm and set it up to start on reboot.

/etc/init.d/oracleasm enable
Writing Oracle ASM library driver configuration: done
Initializing the Oracle ASMLib driver:                     [  OK  ]
Scanning the system for Oracle ASMLib disks:               [  OK  ]

chkconfig --list | grep oracleasm
oracleasm       0:off   1:off   2:off   3:off   4:off   5:off   6:off

# chkconfig oracleasm on

# chkconfig --list | grep oracleasm
oracleasm       0:off   1:off   2:on    3:on    4:on    5:on    6:off

13. Verify scsi_id are same as before the upgrade

# blkid|grep sd.*oracleasm|while read a b;do echo -n $a$b" scsi_id=";(echo $a|tr -d [:digit:]|tr -d [:]|cut -d"/" -f3|xargs -i scsi_id -g -s /block/{})done;
/dev/sdc1:LABEL="DATA" TYPE="oracleasm" scsi_id=SATA     VBOX HARDDISK  VB35fd6dbb-14d87b49
/dev/sdd1:LABEL="FLASH" TYPE="oracleasm" scsi_id=SATA     VBOX HARDDISK  VB4b88ed44-eff71eb1

With this the ASMLib upgrade is complete. Next is to relink the Oracle binaries.

14. As oracle user relink both ASM_HOME and ORACLE_HOME. Verify which relink is in the path variable using

which relink

Relink ORACLE_HOME

$ cd $ORACLE_HOME/bin
$ relink all

Reset the PATH and ORACLE_HOME before running relink for ASM_HOME

export PATH=$ASM_HOME/bin:$PATH
export ORACLE_HOME=$ASM_HOME

Verify the relink binary from ASM_HOME is in path

which relink

Run the relink of ASM_HOME

$ cd $ASM_HOME/bin
$ relink all

15. As per (743649.1 step 10) clusterware client shared libraries should be relinked as the clusterware software owner. But running this as software owner result in permission issue

export ORACLE_HOME=$CRS_HOME (important to set ORACLE_HOME variable to CRS_HOME location)
cd $ORACLE_HOME/bin

./genclntsh
/bin/rm: cannot remove `/opt/crs/oracle/product/11.1.0/crs/lib/libclntsh.so.11.1': Permission denied
/bin/rm: cannot remove `/opt/crs/oracle/product/11.1.0/crs/lib/clntsh.map': Permission denied
./genclntsh: line 309: /opt/crs/oracle/product/11.1.0/crs/lib/clntsh.map: Permission denied
genclntsh: Failed to link libclntsh.so.11.1

Permission issue persist even after setting the permission to oracle:oinstall. Running ./genclntsh as root results in files get created with new timestamp permission same as before (metalink note could be wrong!??). When genclntsh runs without any issue no message is shown and return to shell prompt. After run check the file permissions are same as before (step 2 above).

./cluster_file_check.sh
crs home exist
file exists
-rw-r--r-- 1 root root 4764652 Jun 20 16:49 /opt/crs/oracle/product/11.1.0/crs/lib/clntsh.map
file in lib32
-rw-r--r-- 1 root root 3984490 Jun 20 16:49 /opt/crs/oracle/product/11.1.0/crs/lib32/clntsh.map
file not exists
file not exists
file not exists
file exists
lrwxrwxrwx 1 root root 17 Jun 20 16:49 /opt/crs/oracle/product/11.1.0/crs/lib/libclntsh.so -> libclntsh.so.11.1
file in lib32
lrwxrwxrwx 1 root root 17 Jun 20 16:49 /opt/crs/oracle/product/11.1.0/crs/lib32/libclntsh.so -> libclntsh.so.11.1
file exists
lrwxrwxrwx 1 oracle oinstall 51 Feb  8  2011 /opt/crs/oracle/product/11.1.0/crs/lib/libclntsh.so.10.1 -> /opt/crs/oracle/product/11.1.0/crs/lib/libclntsh.so
file in lib32
lrwxrwxrwx 1 oracle oinstall 53 Feb  8  2011 /opt/crs/oracle/product/11.1.0/crs/lib32/libclntsh.so.10.1 -> /opt/crs/oracle/product/11.1.0/crs/lib32/libclntsh.so
file not exists
file exists
-rwxr-xr-x 1 root root 48316147 Jun 20 16:49 /opt/crs/oracle/product/11.1.0/crs/lib/libclntsh.so.11.1
file in lib32
-rwxr-xr-x 1 root root 37081093 Jun 20 16:49 /opt/crs/oracle/product/11.1.0/crs/lib32/libclntsh.so.11.1
file not exists
file not exists

16. Enable crs and remove the comments on /etc/inittab and start crs

# crsctl enable crs
Oracle Clusterware is enabled for start-up after a reboot.

h1:35:respawn:/etc/init.d/init.evmd run >/dev/null 2>&1 </dev/null
h2:35:respawn:/etc/init.d/init.cssd fatal >/dev/null 2>&1 </dev/null
h3:35:respawn:/etc/init.d/init.crsd run >/dev/null 2>&1 </dev/null

17. Start crs with crsctl or with OS init

crsctl start crs
Attempting to start Oracle Clusterware stack
The CRS stack will be started shortly

or

/sbin/init q

This conclude the ASMLib and OS upgrade.

Useful Metalink Notes
How to Relink Oracle Database Software on UNIX [ID 131321.1]
Is It Necessary To Relink Oracle Following OS Upgrade? [ID 444595.1]
Will an Operating System Upgrade Affect Oracle Clusterware? [ID 743649.1]
How To Upgrade ASMLib Kernel Driver as Part of Kernel Upgrade? [ID 1391807.1]
Do You Need to Relink Oracle Clusterware When Upgrading the Operating System? [ID 743649.1]
Is It Required To Relink The Oracle Binaries When Booting With The Old Kernel? [ID 461138.1]
Cannot Find Exact Kernel Version Match For ASMLib (Workaround using oracleasm_debug_link tool) [ID 462618.1]
RAC: Frequently Asked Questions [ID 220970.1] (Is a relink required for the clusterware home after an OS upgrade?)

Useful White Paper
Best Practices for Optimizing Availability During Planned Maintenance Using Oracle Clusterware and Oracle Real Application Clusters. Oracle Maximum Availability Architecture White Paper September 2007

Related Posts
Upgrading RHEL 6 OS in a 11gR2 RAC Environment
Upgrading OS in 11gR2 RAC Environment (RHEL 5)