Thursday, July 5, 2012

Upgrading OS in 11gR2 RAC Environment

Oracle clusterware 11gR2 requires relinking whenever an OS patch or OS is upgraded. According to GI Installation guide "You must relink the Oracle Clusterware and Oracle ASM binaries every time you apply an operating system patch or after an operating system upgrade. Before relinking executables, you must shut down all executables that run in the Oracle home directory that you are relinking. In addition, shut down applications linked with Oracle shared libraries."
Relink is done with the following steps, again from the GI installation guide
As root:
# cd Grid_home/crs/install
# perl -unlock

As the Oracle Grid Infrastructure for a Cluster owner:
$ export ORACLE_HOME=Grid_home
$ Grid_home/bin/relink

As root again:
# cd Grid_home/rdbms/install/
# ./
# cd Grid_home/crs/install
# perl -patch
As per (743649.1) Oralce support rolling upgrade of OS when both OS are certified on the database that's running. This is supported only for the duration of the upgrade. But according to (1391807.1) using multiple versions of ASMLib across the cluster may not be compatible. In this case ASM disks were migrated from ASMLib disk to block devices before the upgrade therefore an ASMLib upgrade was not required. But if ASMLib was in use then ASMLib should be upgraded as part of the OS upgrade.
1. Current kernel version
uname -r
2. This upgrade is done across the cluster (not a rolling upgrade) therefore stop all the cluster components
crsctl stop cluster -all
and stop disable start of crs
crsctl stop crs
crsctl disable crs
Also comment spawning of high availability processes
#h1:35:respawn:/etc/init.d/init.ohasd run >/dev/null 2>&1 </dev/null

# init q
3. Upgrade OS. New kernel version
uname -r
4. Remove the comment of ohas from inittab and enable crs
h1:35:respawn:/etc/init.d/init.ohasd run >/dev/null 2>&1 </dev/null

crsctl enable crs
5. Relink ORACLE_HOME binaries
 which relink

$ relink all
writing relink log to: /opt/app/oracle/product/11.2.0/db_2/install/relink.log
Relink ends with following in the log file
test ! -f /opt/app/oracle/product/11.2.0/db_2/bin/oracle ||\
           mv -f /opt/app/oracle/product/11.2.0/db_2/bin/oracle /opt/app/oracle/product/11.2.0/db_2/bin/oracleO
mv /opt/app/oracle/product/11.2.0/db_2/rdbms/lib/oracle /opt/app/oracle/product/11.2.0/db_2/bin/oracle
chmod 6751 /opt/app/oracle/product/11.2.0/db_2/bin/oracle
5. Relink the GI HOME binaries
cd $GI_HOME/crs/install

# perl -unlock
Using configuration parameter file: ./crsconfig_params
CRS-4544: Unable to connect to OHAS
CRS-4000: Command Stop failed, or completed with errors.
Successfully unlock /opt/app/11.2.0/grid3

$ export PATH=$ORACLE_HOME/bin:$PATH
$ which relink

Relink ends with the following in the log file
test ! -f /opt/app/11.2.0/grid3/bin/oracle ||\
           mv -f /opt/app/11.2.0/grid3/bin/oracle /opt/app/11.2.0/grid3/bin/oracleO
mv /opt/app/11.2.0/grid3/rdbms/lib/oracle /opt/app/11.2.0/grid3/bin/oracle
chmod 6751 /opt/app/11.2.0/grid3/bin/oracle
Complete the relink process
# cd $GI_HOME/rdbms/install

# ./

# cd $GI_HOME/crs/install
# perl -patch

Using configuration parameter file: ./crsconfig_params
CRS-4124: Oracle High Availability Services startup failed.
CRS-4000: Command Start failed, or completed with errors.
/sbin/init q
to start the cluster stack.

Related Post
Upgrading ASMLib and OS in 11gR1 RAC Environment
Upgrading RHEL 6 OS in a 11gR2 RAC Environment

Useful Metalink notes
Oracle 11gR2 Relink New Feature [ID 883299.1]