Unified Audit Relink is looking for gcc after applying RU 19.29

Following error was observed while relinking for unified audit option after applying RU 19.29.

echo $ORACLE_HOME
/opt/app/oracle/product/19.29.0/dbhome_1

cd $ORACLE_HOME/rdbms/lib
make -f ins_rdbms.mk uniaud_on ioracle ORACLE_HOME=$ORACLE_HOME
/bin/sh: /usr/bin/gcc: No such file or directory
expr: missing operand
Try 'expr --help' for more information.
/bin/sh: line 0: test: -gt: unary operator expected
/usr/bin/ar cr /opt/app/oracle/product/19.29.0/dbhome_1/rdbms/lib/libknlopt.a /opt/app/oracle/product/19.29.0/dbhome_1/rdbms/lib/kzaiang.o
chmod 755 /opt/app/oracle/product/19.29.0/dbhome_1/bin
cd /opt/app/oracle/product/19.29.0/dbhome_1/rdbms/lib/;\
/usr/bin/ar r  /opt/app/oracle/product/19.29.0/dbhome_1/rdbms/lib/libknlopt.a `/usr/bin/ar t /opt/app/oracle/product/19.29.0/dbhome_1/rdbms/lib/libknlopt.a` ;

 - Linking Oracle
rm -f /opt/app/oracle/product/19.29.0/dbhome_1/rdbms/lib/oracle
/opt/app/oracle/product/19.29.0/dbhome_1/bin/orald  -o /opt/app/oracle/product/19.29.0/dbhome_1/rdbms/lib/oracle -m64 -z noexecstack -Wl,--disable-new-dtags -L/opt/app/oracle/product/19.29.0/dbhome_1/rdbms/lib/ -L/opt/app/oracle/product/19.29.0/dbhome_1/lib/ -L/opt/app/oracle/product/19.29.0/dbhome_1/lib/stubs/   -Wl,-E /opt/app/oracle/product/19.29.0/dbhome_1/rdbms/lib/opimai.o /opt/app/oracle/product/19.29.0/dbhome_1/rdbms/lib/ssoraed.o /opt/app/oracle/product/19.29.0/dbhome_1/rdbms/lib/ttcsoi.o -Wl,--whole-archive -lperfsrv19 -Wl,--no-whole-archive /opt/app/oracle/product/19.29.0/dbhome_1/lib/nautab.o /opt/app/oracle/product/19.29.0/dbhome_1/lib/naeet.o /opt/app/oracle/product/19.29.0/dbhome_1/lib/naect.o /opt/app/oracle/product/19.29.0/dbhome_1/lib/naedhs.o /opt/app/oracle/product/19.29.0/dbhome_1/rdbms/lib/config.o  -ldmext -lserver19 -lodm19 -lofs -lcell19 -lnnet19 -lskgxp19 -lsnls19 -lnls19  -lcore19 -lsnls19 -lnls19 -lcore19 -lsnls19 -lnls19 -lxml19 -lcore19 -lunls19 -lsnls19 -lnls19 -lcore19 -lnls19 -lclient19  -lvsnst19 -lcommon19 -lgeneric19 -lknlopt -loraolap19 -lskjcx19 -lslax19 -lpls19  -lrt -lplp19 -ldmext -lserver19 -lclient19  -lvsnst19 -lcommon19 -lgeneric19 `if [ -f /opt/app/oracle/product/19.29.0/dbhome_1/lib/libavserver19.a ] ; then echo "-lavserver19" ; else echo "-lavstub19"; fi` `if [ -f /opt/app/oracle/product/19.29.0/dbhome_1/lib/libavclient19.a ] ; then echo "-lavclient19" ; fi` -lknlopt -lslax19 -lpls19  -lrt -lplp19 -ljavavm19 -lserver19  -lwwg  `cat /opt/app/oracle/product/19.29.0/dbhome_1/lib/ldflags`    -lncrypt19 -lnsgr19 -lnzjs19 -ln19 -lnl19 -lngsmshd19 -lnro19 `cat /opt/app/oracle/product/19.29.0/dbhome_1/lib/ldflags`    -lncrypt19 -lnsgr19 -lnzjs19 -ln19 -lnl19 -lngsmshd19 -lnnzst19 -lzt19 -lztkg19 -lmm -lsnls19 -lnls19  -lcore19 -lsnls19 -lnls19 -lcore19 -lsnls19 -lnls19 -lxml19 -lcore19 -lunls19 -lsnls19 -lnls19 -lcore19 -lnls19 -lztkg19 `cat /opt/app/oracle/product/19.29.0/dbhome_1/lib/ldflags`    -lncrypt19 -lnsgr19 -lnzjs19 -ln19 -lnl19 -lngsmshd19 -lnro19 `cat /opt/app/oracle/product/19.29.0/dbhome_1/lib/ldflags`    -lncrypt19 -lnsgr19 -lnzjs19 -ln19 -lnl19 -lngsmshd19 -lnnzst19 -lzt19 -lztkg19   -lsnls19 -lnls19  -lcore19 -lsnls19 -lnls19 -lcore19 -lsnls19 -lnls19 -lxml19 -lcore19 -lunls19 -lsnls19 -lnls19 -lcore19 -lnls19 `if /usr/bin/ar tv /opt/app/oracle/product/19.29.0/dbhome_1/rdbms/lib/libknlopt.a | grep "kxmnsd.o" > /dev/null 2>&1 ; then echo " " ; else echo "-lordsdo19 -lserver19"; fi` -L/opt/app/oracle/product/19.29.0/dbhome_1/ctx/lib/ -lctxc19 -lctx19 -lzx19 -lgx19 -lctx19 -lzx19 -lgx19 -lclscest19 -loevm -lclsra19 -ldbcfg19 -lhasgen19 -lskgxn2 -lnnzst19 -lzt19 -lxml19 -lgeneric19 -locr19 -locrb19 -locrutl19 -lhasgen19 -lskgxn2 -lnnzst19 -lzt19 -lxml19 -lgeneric19  -lgeneric19 -lorazip -loraz -llzopro5 -lorabz2 -lorazstd -loralz4 -lipp_z -lipp_bz2 -lippdc -lipps -lippcore  -lippcp -lsnls19 -lnls19  -lcore19 -lsnls19 -lnls19 -lcore19 -lsnls19 -lnls19 -lxml19 -lcore19 -lunls19 -lsnls19 -lnls19 -lcore19 -lnls19 -lsnls19 -lunls19  -lsnls19 -lnls19  -lcore19 -lsnls19 -lnls19 -lcore19 -lsnls19 -lnls19 -lxml19 -lcore19 -lunls19 -lsnls19 -lnls19 -lcore19 -lnls19 -lasmclnt19 -lcommon19 -lcore19  -ledtn19 -laio -lons  -lmql1 -lipc1    -lfthread19    `cat /opt/app/oracle/product/19.29.0/dbhome_1/lib/sysliblist` -Wl,-rpath,/opt/app/oracle/product/19.29.0/dbhome_1/lib -lm    `cat /opt/app/oracle/product/19.29.0/dbhome_1/lib/sysliblist` -ldl -lm   -L/opt/app/oracle/product/19.29.0/dbhome_1/lib `test -x /usr/bin/hugeedit -a -r /usr/lib64/libhugetlbfs.so && test -r /opt/app/oracle/product/19.29.0/dbhome_1/rdbms/lib/shugetlbfs.o && echo -Wl,-zcommon-page-size=2097152 -Wl,-zmax-page-size=2097152 -lhugetlbfs`
rm -f /opt/app/oracle/product/19.29.0/dbhome_1/bin/oracle
mv /opt/app/oracle/product/19.29.0/dbhome_1/rdbms/lib/oracle /opt/app/oracle/product/19.29.0/dbhome_1/bin/oracle
chmod 6751 /opt/app/oracle/product/19.29.0/dbhome_1/bin/oracle
(if [ ! -f /opt/app/oracle/product/19.29.0/dbhome_1/bin/crsd.bin ]; then \
    getcrshome="/opt/app/oracle/product/19.29.0/dbhome_1/srvm/admin/getcrshome" ; \
    if [ -f "$getcrshome" ]; then \
        crshome="`$getcrshome`"; \
        if [ -n "$crshome" ]; then \
            if [ $crshome != /opt/app/oracle/product/19.29.0/dbhome_1 ]; then \
                oracle="/opt/app/oracle/product/19.29.0/dbhome_1/bin/oracle"; \
                $crshome/bin/setasmgidwrap oracle_binary_path=$oracle; \
            fi \
        fi \
    fi \
fi\
);

gcc and gcc-c++ are not required on Oracle Linux and Red Hat Enterprise Linux (refer 2668780.1 and 2982833.1).


This appears to be an issue introduced with 19.29. No such issue with RU 19.28.

echo $ORACLE_HOME
/opt/app/oracle/product/19.28.0/dbhome_1

cd $ORACLE_HOME/rdbms/lib
make -f ins_rdbms.mk uniaud_on ioracle ORACLE_HOME=$ORACLE_HOME
/usr/bin/ar cr /opt/app/oracle/product/19.28.0/dbhome_1/rdbms/lib/libknlopt.a /opt/app/oracle/product/19.28.0/dbhome_1/rdbms/lib/kzaiang.o
chmod 755 /opt/app/oracle/product/19.28.0/dbhome_1/bin
cd /opt/app/oracle/product/19.28.0/dbhome_1/rdbms/lib/;\
/usr/bin/ar r  /opt/app/oracle/product/19.28.0/dbhome_1/rdbms/lib/libknlopt.a `/usr/bin/ar t /opt/app/oracle/product/19.28.0/dbhome_1/rdbms/lib/libknlopt.a` ;

 - Linking Oracle
rm -f /opt/app/oracle/product/19.28.0/dbhome_1/rdbms/lib/oracle
/opt/app/oracle/product/19.28.0/dbhome_1/bin/orald  -o /opt/app/oracle/product/19.28.0/dbhome_1/rdbms/lib/oracle -m64 -z noexecstack -Wl,--disable-new-dtags -L/opt/app/oracle/product/19.28.0/dbhome_1/rdbms/lib/ -L/opt/app/oracle/product/19.28.0/dbhome_1/lib/ -L/opt/app/oracle/product/19.28.0/dbhome_1/lib/stubs/   -Wl,-E /opt/app/oracle/product/19.28.0/dbhome_1/rdbms/lib/opimai.o /opt/app/oracle/product/19.28.0/dbhome_1/rdbms/lib/ssoraed.o /opt/app/oracle/product/19.28.0/dbhome_1/rdbms/lib/ttcsoi.o -Wl,--whole-archive -lperfsrv19 -Wl,--no-whole-archive /opt/app/oracle/product/19.28.0/dbhome_1/lib/nautab.o /opt/app/oracle/product/19.28.0/dbhome_1/lib/naeet.o /opt/app/oracle/product/19.28.0/dbhome_1/lib/naect.o /opt/app/oracle/product/19.28.0/dbhome_1/lib/naedhs.o /opt/app/oracle/product/19.28.0/dbhome_1/rdbms/lib/config.o  -ldmext -lserver19 -lodm19 -lofs -lcell19 -lnnet19 -lskgxp19 -lsnls19 -lnls19  -lcore19 -lsnls19 -lnls19 -lcore19 -lsnls19 -lnls19 -lxml19 -lcore19 -lunls19 -lsnls19 -lnls19 -lcore19 -lnls19 -lclient19  -lvsnst19 -lcommon19 -lgeneric19 -lknlopt -loraolap19 -lskjcx19 -lslax19 -lpls19  -lrt -lplp19 -ldmext -lserver19 -lclient19  -lvsnst19 -lcommon19 -lgeneric19 `if [ -f /opt/app/oracle/product/19.28.0/dbhome_1/lib/libavserver19.a ] ; then echo "-lavserver19" ; else echo "-lavstub19"; fi` `if [ -f /opt/app/oracle/product/19.28.0/dbhome_1/lib/libavclient19.a ] ; then echo "-lavclient19" ; fi` -lknlopt -lslax19 -lpls19  -lrt -lplp19 -ljavavm19 -lserver19  -lwwg  `cat /opt/app/oracle/product/19.28.0/dbhome_1/lib/ldflags`    -lncrypt19 -lnsgr19 -lnzjs19 -ln19 -lnl19 -lngsmshd19 -lnro19 `cat /opt/app/oracle/product/19.28.0/dbhome_1/lib/ldflags`    -lncrypt19 -lnsgr19 -lnzjs19 -ln19 -lnl19 -lngsmshd19 -lnnzst19 -lzt19 -lztkg19 -lmm -lsnls19 -lnls19  -lcore19 -lsnls19 -lnls19 -lcore19 -lsnls19 -lnls19 -lxml19 -lcore19 -lunls19 -lsnls19 -lnls19 -lcore19 -lnls19 -lztkg19 `cat /opt/app/oracle/product/19.28.0/dbhome_1/lib/ldflags`    -lncrypt19 -lnsgr19 -lnzjs19 -ln19 -lnl19 -lngsmshd19 -lnro19 `cat /opt/app/oracle/product/19.28.0/dbhome_1/lib/ldflags`    -lncrypt19 -lnsgr19 -lnzjs19 -ln19 -lnl19 -lngsmshd19 -lnnzst19 -lzt19 -lztkg19   -lsnls19 -lnls19  -lcore19 -lsnls19 -lnls19 -lcore19 -lsnls19 -lnls19 -lxml19 -lcore19 -lunls19 -lsnls19 -lnls19 -lcore19 -lnls19 `if /usr/bin/ar tv /opt/app/oracle/product/19.28.0/dbhome_1/rdbms/lib/libknlopt.a | grep "kxmnsd.o" > /dev/null 2>&1 ; then echo " " ; else echo "-lordsdo19 -lserver19"; fi` -L/opt/app/oracle/product/19.28.0/dbhome_1/ctx/lib/ -lctxc19 -lctx19 -lzx19 -lgx19 -lctx19 -lzx19 -lgx19 -lclscest19 -loevm -lclsra19 -ldbcfg19 -lhasgen19 -lskgxn2 -lnnzst19 -lzt19 -lxml19 -lgeneric19 -locr19 -locrb19 -locrutl19 -lhasgen19 -lskgxn2 -lnnzst19 -lzt19 -lxml19 -lgeneric19  -lgeneric19 -lorazip -loraz -llzopro5 -lorabz2 -lorazstd -loralz4 -lipp_z -lipp_bz2 -lippdc -lipps -lippcore  -lippcp -lsnls19 -lnls19  -lcore19 -lsnls19 -lnls19 -lcore19 -lsnls19 -lnls19 -lxml19 -lcore19 -lunls19 -lsnls19 -lnls19 -lcore19 -lnls19 -lsnls19 -lunls19  -lsnls19 -lnls19  -lcore19 -lsnls19 -lnls19 -lcore19 -lsnls19 -lnls19 -lxml19 -lcore19 -lunls19 -lsnls19 -lnls19 -lcore19 -lnls19 -lasmclnt19 -lcommon19 -lcore19  -ledtn19 -laio -lons  -lmql1 -lipc1    -lfthread19    `cat /opt/app/oracle/product/19.28.0/dbhome_1/lib/sysliblist` -Wl,-rpath,/opt/app/oracle/product/19.28.0/dbhome_1/lib -lm    `cat /opt/app/oracle/product/19.28.0/dbhome_1/lib/sysliblist` -ldl -lm   -L/opt/app/oracle/product/19.28.0/dbhome_1/lib `test -x /usr/bin/hugeedit -a -r /usr/lib64/libhugetlbfs.so && test -r /opt/app/oracle/product/19.28.0/dbhome_1/rdbms/lib/shugetlbfs.o && echo -Wl,-zcommon-page-size=2097152 -Wl,-zmax-page-size=2097152 -lhugetlbfs`
rm -f /opt/app/oracle/product/19.28.0/dbhome_1/bin/oracle
mv /opt/app/oracle/product/19.28.0/dbhome_1/rdbms/lib/oracle /opt/app/oracle/product/19.28.0/dbhome_1/bin/oracle
chmod 6751 /opt/app/oracle/product/19.28.0/dbhome_1/bin/oracle
(if [ ! -f /opt/app/oracle/product/19.28.0/dbhome_1/bin/crsd.bin ]; then \
    getcrshome="/opt/app/oracle/product/19.28.0/dbhome_1/srvm/admin/getcrshome" ; \
    if [ -f "$getcrshome" ]; then \
        crshome="`$getcrshome`"; \
        if [ -n "$crshome" ]; then \
            if [ $crshome != /opt/app/oracle/product/19.28.0/dbhome_1 ]; then \
                oracle="/opt/app/oracle/product/19.28.0/dbhome_1/bin/oracle"; \
                $crshome/bin/setasmgidwrap oracle_binary_path=$oracle; \
            fi \
        fi \
    fi \
fi\
);

Upgrading RHEL 7 OS in a 11gR2 (11.2.0.4) and 12cR1 (12.1.0.2) Oracle Restart Environments

As per Oracle documentation "You must relink the Oracle Clusterware and Oracle ASM binaries every time you apply an operating system patch or after an operating system upgrade." There are two earlier posts which shows upgrading of RHEL5 and RHEL6 OS in a 11gR2 RAC environment. This post shows relinking of GI and Oracle homes after upgrading of RHEL 7 in 11.2.0.4 and 12.1.0.2 single instance (Oracle restart) environments.
1. The current RHEL version is 7.1 and the kernel is

uname -r
3.10.0-123.el7.x86_64

2. Before the OS upgrade stop the HAS and disable the auto start on reboot.

crsctl stop has
crsctl disable has
CRS-4621: Oracle High Availability Services autostart is disabled.

3. Comment the entries in oracle-ohasd.service

cat /etc/systemd/system/oracle-ohasd.service
# Copyright (c) 2014, Oracle and/or its affiliates. All rights reserved.
#
# Oracle OHASD startup

[Unit]
#Description=Oracle High Availability Services
#After=syslog.target

[Service]
#ExecStart=/etc/init.d/init.ohasd run >/dev/null 2>&1 </dev/null
#Type=simple
#Restart=always

[Install]
#WantedBy=multi-user.target graphical.target

4. Upgrade the RHEL 7 OS. Kernel and RHEL7 version after the upgrade are

uname -r
3.10.0-327.el7.x86_64
cat /etc/redhat-release
Red Hat Enterprise Linux Server release 7.2 (Maipo)

5. Uncomment the entires in oracle-ohasd.service file and enable auto start of HAS

crsctl enable has
CRS-4622: Oracle High Availability Services autostart is enabled.

Until this point the steps are common for both 11.2.0.4 and 12.1.0.2.

Relinking 11.2.0.4 GI and Oracle Homes
1.Relinking Oracle Home as oracle user. Make sure the relink binary in the PATH is from Oracle home

which relink
/opt/app/oracle/product/11.2.0/dbhome_4/bin/relink

Run relink all

cd $ORACLE_HOME/bin
./relink all
writing relink log to: /opt/app/oracle/product/11.2.0/dbhome_4/install/relink.log

At the end of the relink the log file have the following

test ! -f /opt/app/oracle/product/11.2.0/dbhome_4/bin/oracle ||\
   mv -f /opt/app/oracle/product/11.2.0/dbhome_4/bin/oracle /opt/app/oracle/product/11.2.0/dbhome_4/bin/oracleO
mv /opt/app/oracle/product/11.2.0/dbhome_4/rdbms/lib/oracle /opt/app/oracle/product/11.2.0/dbhome_4/bin/oracle
chmod 6751 /opt/app/oracle/product/11.2.0/dbhome_4/bin/oracle

2. Relinkning GI Home
Set ORACLE_HOME=$GI_HOME
As root unlock the GI Home. Using the RHEL 7 provided perl version will result in following error

# perl $ORACLE_HOME/crs/install/roothas.pl -unlock
Can't locate Env.pm in @INC (@INC contains: /usr/local/lib64/perl5 /usr/local/share/perl5 /usr/lib64/perl5/vendor_perl /usr/share/perl5/vendor_perl /usr/lib64/perl5 /usr/share/perl5 . /opt/app/oracle/product/11.2.0/grid/crs/install) at /opt/app/oracle/product/11.2.0/grid/crs/install/crsconfig_lib.pm line 710.
BEGIN failed--compilation aborted at /opt/app/oracle/product/11.2.0/grid/crs/install/crsconfig_lib.pm line 710.
Compilation failed in require at /opt/app/oracle/product/11.2.0/grid/crs/install/roothas.pl line 171.
BEGIN failed--compilation aborted at /opt/app/oracle/product/11.2.0/grid/crs/install/roothas.pl line 171.

Solution for this was mentioned on the changing of hostname post which is to use oracle provided perl

# $ORACLE_HOME/perl/bin/perl $ORACLE_HOME/crs/install/roothas.pl -unlock
Using configuration parameter file: /opt/app/oracle/product/11.2.0/grid/crs/install/crsconfig_params
Successfully unlock /opt/app/oracle/product/11.2.0/grid

3. As grid user run the relink

$ORACLE_HOME/bin/relink
writing relink log to: /opt/app/oracle/product/11.2.0/grid/install/relink.log

The log file will have the following entry at the end

tail /opt/app/oracle/product/11.2.0/grid/install/relink.log

log tail
test ! -f /opt/app/oracle/product/11.2.0/grid/bin/oracle ||\
   mv -f /opt/app/oracle/product/11.2.0/grid/bin/oracle /opt/app/oracle/product/11.2.0/grid/bin/oracleO
mv /opt/app/oracle/product/11.2.0/grid/rdbms/lib/oracle /opt/app/oracle/product/11.2.0/grid/bin/oracle
chmod 6751 /opt/app/oracle/product/11.2.0/grid/bin/oracle

4. As root patch the GI Home

# cd $ORACLE_HOME/rdbms/install
# ./rootadd_rdbms.sh

$ORACLE_HOME/perl/bin/perl $ORACLE_HOME/crs/install/roothas.pl -patch
Using configuration parameter file: /opt/app/oracle/product/11.2.0/grid/crs/install/crsconfig_params
Broadcast message from systemd-journald@rhel7s.domain.net (Thu 2016-02-18 16:50:14 GMT):
dracut[16456]: dracut: creation of /boot/initramfs-3.10.0-327.el7.x86_64.tmp failed

Message from syslogd@rhel7s at Feb 18 16:50:14 ...
 dracut:dracut: creation of /boot/initramfs-3.10.0-327.el7.x86_64.tmp failed
ACFS driver install actions failed
CRS-4124: Oracle High Availability Services startup failed.
CRS-4000: Command Start failed, or completed with errors.
Timed out waiting for ohasd to start.

During the patching the HAS stack start will fail, this is expected.
5. Reboot the host after uncommenting the lines in oracle-ohasd.service together with enabling HAS service (crsctl enable has). After the reboot HAS stack will be up and running as before

Resource Name                       Type                           Target             State              Host
-------------                       ------                         -------            --------           ----------
ora.DATA.dg                         ora.diskgroup.type             ONLINE             ONLINE             rhel7s
ora.FRA.dg                          ora.diskgroup.type             ONLINE             ONLINE             rhel7s
ora.LISTENER.lsnr                   ora.listener.type              ONLINE             ONLINE             rhel7s
ora.asm                             ora.asm.type                   ONLINE             ONLINE             rhel7s
ora.cssd                            ora.cssd.type                  ONLINE             ONLINE             rhel7s
ora.diskmon                         ora.diskmon.type               OFFLINE            OFFLINE
ora.evmd                            ora.evm.type                   ONLINE             ONLINE             rhel7s
ora.ons                             ora.ons.type                   OFFLINE            OFFLINE
ora.std11g2.db                      ora.database.type              ONLINE             ONLINE             rhel7s

Relinking 12.1.0.2 GI and Oracle Homes
1. Relinking Oracle home as oracle user. Make sure the relink binary in PATH is from Oracle home. Run relink all to relink the oracle home.

relink all
writing relink log to: /opt/app/oracle/product/12.1.0/dbhome_2/install/relink.log

The log file end with the following

tail /opt/app/oracle/product/12.1.0/dbhome_2/install/relink.log
test ! -f /opt/app/oracle/product/12.1.0/dbhome_2/bin/oracle ||\
   mv -f /opt/app/oracle/product/12.1.0/dbhome_2/bin/oracle /opt/app/oracle/product/12.1.0/dbhome_2/bin/oracleO
mv /opt/app/oracle/product/12.1.0/dbhome_2/rdbms/lib/oracle /opt/app/oracle/product/12.1.0/dbhome_2/bin/oracle
chmod 6751 /opt/app/oracle/product/12.1.0/dbhome_2/bin/oracle

2. As root unlock the GI Home by using the $GI_HOME/crs/install/roothas.sh. Unlike 11.2.0.4 there are no issues in unlocking GI home in 12c

./roothas.sh -unlock
Using configuration parameter file: /opt/app/oracle/product/12.1.0/grid/crs/install/crsconfig_params
2016/02/19 18:52:17 CLSRSC-347: Successfully unlock /opt/app/oracle/product/12.1.0/grid

3. As grid user run the relink.

which relink
/opt/app/oracle/product/12.1.0/grid/bin/relink
$ relink
writing relink log to: /opt/app/oracle/product/12.1.0/grid/install/relink.log

The log file will end with the following

tail /opt/app/oracle/product/12.1.0/grid/install/relink.log

test ! -f /opt/app/oracle/product/12.1.0/grid/bin/oracle ||\
   mv -f /opt/app/oracle/product/12.1.0/grid/bin/oracle /opt/app/oracle/product/12.1.0/grid/bin/oracleO
mv /opt/app/oracle/product/12.1.0/grid/rdbms/lib/oracle /opt/app/oracle/product/12.1.0/grid/bin/oracle
chmod 6751 /opt/app/oracle/product/12.1.0/grid/bin/oracle

4. As root patch the GI Home

# $GI_HOME/rdbms/install/rootadd_rdbms.sh

./roothas.sh -patch
Using configuration parameter file: /opt/app/oracle/product/12.1.0/grid/crs/install/crsconfig_params
CRS-4124: Oracle High Availability Services startup failed.
CRS-4000: Command Start failed, or completed with errors.
2016/02/19 18:59:50 CLSRSC-199: Timed out waiting for OHASD to start

The command '/opt/app/oracle/product/12.1.0/grid/perl/bin/perl -I/opt/app/oracle/product/12.1.0/grid/perl/lib -I/opt/app/oracle/product/12.1.0/grid/crs/install /opt/app/oracle/product/12.1.0/grid/crs/install/roothas.pl -patch' execution failed

5. Similar to 11.2.0.4, start of the HAS stack will fail, this is expected. Reboot the host after uncommenting the lines in oracle-ohasd.service together with enabling HAS service (crsctl enable has). After the reboot HAS will be up and running.

Resource Name             Type                      Target             State              Host
-------------             ------                    -------            --------           ----------
ora.DATA.dg               ora.diskgroup.type        ONLINE             ONLINE             rhel7
ora.FRA.dg                ora.diskgroup.type        ONLINE             ONLINE             rhel7
ora.LISTENER.lsnr         ora.listener.type         ONLINE             ONLINE             rhel7
ora.asm                   ora.asm.type              ONLINE             ONLINE             rhel7
ora.cssd                  ora.cssd.type             ONLINE             ONLINE             rhel7
ora.diskmon               ora.diskmon.type          OFFLINE            OFFLINE
ora.evmd                  ora.evm.type              ONLINE             ONLINE             rhel7
ora.ons                   ora.ons.type              OFFLINE            OFFLINE
ora.se2db.db              ora.database.type         ONLINE             ONLINE             rhel7

This concludes the steps for relinking GI and Oracle homes after RHEL 7 OS upgrade.

Related Posts
Upgrading RHEL 6 OS in a 11gR2 RAC Environment
Upgrading OS in 11gR2 RAC Environment
Upgrading ASMLib and OS in 11gR1 RAC Environment

Update on 2017-07-10
As per 2282371.1 and 2281492.1 certain RHEL 7 kernel versions (3.10.0-514.21.2.el7.x86_64) cause issues with regard to starting of OHASD in 11.2.0.4. Avoid these kernel versions when installing or upgrading OS when 11.2.0.4 is installed.

Useful metalink notes
OHASD Fails to Start With Kernel Version 3.10.0-514.21.2.el7.x86_64 [ID 2281492.1]
Grid Infrastructure Fails to Start OHASD With RedHat Linux or Oracle Linux with RedHat Compatible Kernel (RHCK) Version 3.10.0-514.21.2.EL7.X86_64 or Higher [ID 2282371.1]

Relink Fails due to Requirement Checks Failure

Oracle recommends re-linking oracle binaries after OS upgrade. There are few earlier posts on this topic. In this case the server in question was a Amazon EC2 server and didn't have any swap configured.

$ cat /proc/meminfo | grep "Swap"
SwapCached:          0 kB
SwapTotal:           0 kB
SwapFree:            0 kB

After the OS upgrade , relinking of oracle binaries failed due to insufficient swap space.

$ ./relink all
writing relink log to: /opt/app/oracle/product/testbox/11.2.0/install/relink.log

$ more ../install/relink.log
Starting Oracle Universal Installer...

Checking swap space: 0 MB available, 500 MB required.    Failed <<<<

Some requirement checks failed. You must fulfill these requirements before

continuing with the installation,

Exiting Oracle Universal Installer, log for this session can be found at /opt/app/oracle/oraInventory/logs/installActions2015-09-25_12-11-54PM.log

Only parameters that could be passed on to relink is all and as_installed. However looking inside relink script it could be seen that it is using runInstaller.

#-----------------------------------
# location of runInstaller executable
RUNINSTALLER=$ORACLE_HOME/oui/bin/runInstaller

#--------------------------

Runinstaller command gets set of arguments.

#-----------------------------------
# full argument list for runInstaller
#
ARGS="-relink -waitForCompletion -maketargetsxml $MAKEORDER $LOGDIR_ARG ORACLE_HOME=$ORACLE_HOME"

#--------------------------

Adding -ignoreSysPrereqs as the first argument resolved the issue.

ARGS="-ignoreSysPrereqs -relink -waitForCompletion -maketargetsxml $MAKEORDER $LOGDIR_ARG ORACLE_HOME=$ORACLE_HOME"

Relink completes successfully afterwards and relink.log will end with the following for oracle home relinks

test ! -f /opt/app/oracle/product/testbox/11.2.0/bin/oracle ||\
           mv -f /opt/app/oracle/product/testbox/11.2.0/bin/oracle /opt/app/oracle/product/testbox/11.2.0/bin/oracleO
mv /opt/app/oracle/product/testbox/11.2.0/rdbms/lib/oracle /opt/app/oracle/product/testbox/11.2.0/bin/oracle
chmod 6751 /opt/app/oracle/product/testbox/11.2.0/bin/oracle

and with the following for GI home relinking

test ! -f /grid/oracle/product/11.2.0/bin/oracle ||\
           mv -f /grid/oracle/product/11.2.0/bin/oracle /grid/oracle/product/11.2.0/bin/oracleO
mv /grid/oracle/product/11.2.0/rdbms/lib/oracle /grid/oracle/product/11.2.0/bin/oracle
chmod 6751 /grid/oracle/product/11.2.0/bin/oracle

Once the relinking is completed remove the ignoreSysPrereqs parameter from the relink file.
If the server already has swap created and want to recreate the same scenario as above for testing, use

swapoff -a
swapon -a

to disable and enable swapping
Also if the environment uses role separation then after the relink it may be necessary to set the correct permission on the oracle binary using setasmgidwrap.

Related Posts
Upgrading RHEL 6 OS in a 11gR2 RAC Environment
Upgrading OS in 11gR2 RAC Environment (RHEL 5)
Upgrading ASMLib and OS in 11gR1 RAC Environment

Upgrading RHEL 6 OS in a 11gR2 RAC Environment

This post upgrade the RHEL 6 OS on severs which are running 11gR2 RAC. Current kernel is

$ uname -r
2.6.32-220.el6.x86_64

and the database patch level is

opatch lsinventory -local | grep Patch
Oracle Interim Patch Installer version 11.2.0.3.0
OPatch version    : 11.2.0.3.0
Patch  14275605     : applied on Thu Nov 15 14:21:27 GMT 2012
Unique Patch ID:  15379762
Patch description:  "Database Patch Set Update : 11.2.0.3.4 (14275605)"
Sub-patch  13923374; "Database Patch Set Update : 11.2.0.3.3 (13923374)"
Sub-patch  13696216; "Database Patch Set Update : 11.2.0.3.2 (13696216)"
Sub-patch  13343438; "Database Patch Set Update : 11.2.0.3.1 (13343438)"
Patch  14275572     : applied on Thu Nov 15 14:20:32 GMT 2012
Unique Patch ID:  15379762
Patch description:  "Grid Infrastructure Patch Set Update : 11.2.0.3.4 (14275572)"
OPatch succeeded.

The database software is running standard edition.
The steps are similar to that of upgrading RHEL 5 OS running 11gR2 RAC with few minor differences. This RAC system does not use ASMLibs.
1. Current kernel version

$ uname -r
2.6.32-220.el6.x86_64

2. Stop all the cluster components

crsctl stop cluster -all

and stop disable start of crs after reboot

crsctl stop crs
crsctl disable crs

3. In RHEL 6 inittab is depreciated and the spawning process file is located in /etc/init with the name oracle-ohasd.conf
More on this is available on Troubleshoot Grid Infrastructure Startup Issues [ID 1050908.1] If Oracle is 11.2.0.4 comment out the trace file analyzer script as well oracle-tfa.conf.

cat oracle-ohasd.conf
# Copyright (c) 2001, 2011, Oracle and/or its affiliates. All rights reserved.
#
# Oracle OHASD startup

start on runlevel [35]
stop  on runlevel [!35]
respawn
exec /etc/init.d/init.ohasd run >/dev/null 2>&1 </dev/null

As root comment last two lines

#respawn
#exec /etc/init.d/init.ohasd run >/dev/null 2>&1 </dev/null

4. Upgrade the OS (only few screenshots shown)

5. New kernel version after upgrade

uname -r
2.6.32-279.el6.x86_64

6. Stop oracle-ohasd

initctl stop oracle-ohasd
oracle-ohasd stop/waiting

7. Un-comment the last two lines on /etc/init/oracle-ohasd that was commented earlier

start on runlevel [35]
stop  on runlevel [!35]
respawn
exec /etc/init.d/init.ohasd run >/dev/null 2>&1 </dev/null

8. Enable crs on startup

crsctl enable crs
CRS-4622: Oracle High Availability Services autostart is enabled.

9. Relink the Oracle Home binaries as the oracle software owner

which relink
/opt/app/oracle/product/11.2.0/dbhome_1/bin/relink
$ cd $ORACLE_HOME/bin
$ ./relink all
writing relink log to: /opt/app/oracle/product/11.2.0/dbhome_1/install/relink.log

The relink will end with following on the log

test ! -f /opt/app/oracle/product/11.2.0/dbhome_1/bin/oracle ||\
           mv -f /opt/app/oracle/product/11.2.0/dbhome_1/bin/oracle /opt/app/oracle/product/11.2.0/dbhome_1/bin/oracleO
mv /opt/app/oracle/product/11.2.0/dbhome_1/rdbms/lib/oracle /opt/app/oracle/product/11.2.0/dbhome_1/bin/oracle
chmod 6751 /opt/app/oracle/product/11.2.0/dbhome_1/bin/oracle

9. Unlock the permission on the grid software as root

# cd $GI_HOME/crs/install

# perl rootcrs.pl -unlock
Using configuration parameter file: ./crsconfig_params
CRS-4544: Unable to connect to OHAS
CRS-4000: Command Stop failed, or completed with errors.
Successfully unlock /opt/app/11.2.0/grid

10. As the grid software owner (gird in this case) relink the grid software

[grid@rhel6m1 ~]$ export ORACLE_HOME=$GI_HOME
[grid@rhel6m1 ~]$ export PATH=$ORACLE_HOME/bin:$PATH
[grid@rhel6m1 ~]$ which relink
/opt/app/11.2.0/grid/bin/relink
[grid@rhel6m1 ~]$ $GI_HOME/bin/relink
writing relink log to: /opt/app/11.2.0/grid/install/relink.log

The relink ends with

test ! -f /opt/app/11.2.0/grid/bin/oracle ||\
           mv -f /opt/app/11.2.0/grid/bin/oracle /opt/app/11.2.0/grid/bin/oracleO
mv /opt/app/11.2.0/grid/rdbms/lib/oracle /opt/app/11.2.0/grid/bin/oracle
chmod 6751 /opt/app/11.2.0/grid/bin/oracle

11. Run the post relink script. rootcrs.pl -patch takes a while to complete and ends with an error as shown below

# cd $GI_HOME/rdbms/install
# ./rootadd_rdbms.sh

# cd $GI_HOME/crs/install
# perl rootcrs.pl -patch
Using configuration parameter file: ./crsconfig_params
CRS-4124: Oracle High Availability Services startup failed.
CRS-4000: Command Start failed, or completed with errors.
Failed to write the checkpoint:'' with status:FAIL.Error code is 256
Oracle Grid Infrastructure stack start initiated but failed to complete at crsconfig_lib.pm line 11545.

After an SR it was confirmed that this error is ignorable and crs could be started with "init 3" or by rebooting the server. Reason that above error happen is that since "ohasd run" line was commented on oracle-ohasd file there's no "ohasd run" process active when the kernel is upgraded and server is rebooted. Without "ohasd run" active the crsctl start crs (issued during the rootcrs.pl -patch) will fail to start the clusterware stack. This has no effect on the actual relink. This is also mentioned in 1050908.1
For metalink notes related to Upgrading OS and relinking clusterware and oracle binaries after OS upgrade refer the related post given below.

Related Posts
Upgrading OS in 11gR2 RAC Environment (RHEL 5)
Upgrading ASMLib and OS in 11gR1 RAC Environment

Useful metalink notes
Executing "relink all" resets permission of extjob, jssu, oradism, externaljob.ora [ID 1555453.1]
Relinking Oracle Home FAQ ( Frequently Asked Questions) [ID 1467060.1]

Upgrading OS in 11gR2 RAC Environment

Oracle clusterware 11gR2 requires relinking whenever an OS patch or OS is upgraded. According to GI Installation guide "You must relink the Oracle Clusterware and Oracle ASM binaries every time you apply an operating system patch or after an operating system upgrade. Before relinking executables, you must shut down all executables that run in the Oracle home directory that you are relinking. In addition, shut down applications linked with Oracle shared libraries."
Relink is done with the following steps, again from the GI installation guide

As root:
# cd Grid_home/crs/install
# perl rootcrs.pl -unlock

As the Oracle Grid Infrastructure for a Cluster owner:
$ export ORACLE_HOME=Grid_home
$ Grid_home/bin/relink

As root again:
# cd Grid_home/rdbms/install/
# ./rootadd_rdbms.sh
# cd Grid_home/crs/install
# perl rootcrs.pl -patch

As per (743649.1) Oralce support rolling upgrade of OS when both OS are certified on the database that's running. This is supported only for the duration of the upgrade. But according to (1391807.1) using multiple versions of ASMLib across the cluster may not be compatible. In this case ASM disks were migrated from ASMLib disk to block devices before the upgrade therefore an ASMLib upgrade was not required. But if ASMLib was in use then ASMLib should be upgraded as part of the OS upgrade.
1. Current kernel version

uname -r
2.6.18-194.el5

2. This upgrade is done across the cluster (not a rolling upgrade) therefore stop all the cluster components

crsctl stop cluster -all

and stop disable start of crs

crsctl stop crs
crsctl disable crs

Also comment spawning of high availability processes

#h1:35:respawn:/etc/init.d/init.ohasd run >/dev/null 2>&1 </dev/null

# init q

3. Upgrade OS. New kernel version

uname -r
2.6.18-308.el5

4. Remove the comment of ohas from inittab and enable crs

h1:35:respawn:/etc/init.d/init.ohasd run >/dev/null 2>&1 </dev/null

crsctl enable crs

5. Relink ORACLE_HOME binaries

 which relink
/opt/app/oracle/product/11.2.0/db_2/bin/relink

cd $ORACLE_HOME/bin
$ relink all
writing relink log to: /opt/app/oracle/product/11.2.0/db_2/install/relink.log

Relink ends with following in the log file

test ! -f /opt/app/oracle/product/11.2.0/db_2/bin/oracle ||\
           mv -f /opt/app/oracle/product/11.2.0/db_2/bin/oracle /opt/app/oracle/product/11.2.0/db_2/bin/oracleO
mv /opt/app/oracle/product/11.2.0/db_2/rdbms/lib/oracle /opt/app/oracle/product/11.2.0/db_2/bin/oracle
chmod 6751 /opt/app/oracle/product/11.2.0/db_2/bin/oracle

5. Relink the GI HOME binaries

cd $GI_HOME/crs/install

# perl rootcrs.pl -unlock
Using configuration parameter file: ./crsconfig_params
CRS-4544: Unable to connect to OHAS
CRS-4000: Command Stop failed, or completed with errors.
Successfully unlock /opt/app/11.2.0/grid3

$ export ORACLE_HOME=$GI_HOME
$ export PATH=$ORACLE_HOME/bin:$PATH
$ which relink
/opt/app/11.2.0/grid3/bin/relink

$GI_HOME/bin/relink

Relink ends with the following in the log file

test ! -f /opt/app/11.2.0/grid3/bin/oracle ||\
           mv -f /opt/app/11.2.0/grid3/bin/oracle /opt/app/11.2.0/grid3/bin/oracleO
mv /opt/app/11.2.0/grid3/rdbms/lib/oracle /opt/app/11.2.0/grid3/bin/oracle
chmod 6751 /opt/app/11.2.0/grid3/bin/oracle

Complete the relink process

# cd $GI_HOME/rdbms/install

# ./rootadd_rdbms.sh

# cd $GI_HOME/crs/install
# perl rootcrs.pl -patch

Using configuration parameter file: ./crsconfig_params
CRS-4124: Oracle High Availability Services startup failed.
CRS-4000: Command Start failed, or completed with errors.

Execute

/sbin/init q

to start the cluster stack.

Related Post
Upgrading ASMLib and OS in 11gR1 RAC Environment
Upgrading RHEL 6 OS in a 11gR2 RAC Environment

Useful Metalink notes
Oracle 11gR2 Relink New Feature [ID 883299.1]

Upgrading ASMLib and OS in 11gR1 RAC Environment

Upgrading operating system (OS) requires several consideration and tasks.

1. If ASMLib is used then ASMLib libraries must be upgraded as part of the OS upgrade.
2. After the upgrade Oracle/clusterware binaries should be relinked.

In this case the two nodes 11gR1 (11.1.0.7) RAC environment running on RHEL 5 (2.6.18-194.el5) will be upgraded to RHEL 5 (2.6.18-308.el5).
As per (743649.1) Oralce support rolling upgrade of OS when both OS are certified on the database that's running. This is supported only for the duration of the upgrade. But according to (1391807.1) using multiple versions of ASMLib across the cluster may not be compatible. To avoid these incompatible issues and to minimize the total downtime one node could be upgraded first and then shutting down the other node before the upgraded one with new version of ASMLib is started. This way only one node will be active in the cluster when two nodes are having different ASMLib versions. In this case all nodes are upgraded at the same time incurring total system outage.

1. Find out if ASMLibs are available for the kernel version to which the system is being upgraded to. If not move asmlib devices to block devices before the upgrade (also true if upgrading to RHEL 6 where to get asmlibs require ULN account) or see if (462618.1) could be of any help.

2. Make a note of the ownership and permission of the following files in $CRS_HOME/bin directory using the script below. Some files might be missing this is not a problem.

#!/bin/sh

if [ "$CRS_HOME" != "" ]; then
                echo "crs home exist"

        for i in clntsh.map clntst_1.lis clntst_2.lis clntst.lis libclntsh.so libclntsh.so.10.1 libclntst10.so libclntsh.so.11.1 libclntst11.so clntst
        do
                if [ -f $CRS_HOME/lib/$i ]; then
                        echo "file exists"
                        ls -l $CRS_HOME/lib/$i

                        if [ -f $CRS_HOME/lib32/$i ]; then
                                echo "file in lib32"
                                ls -l $CRS_HOME/lib32/$i
                        else
                                echo "file not exists"
                        fi
                else
                        if [ -f $CRS_HOME/lib32/$i ]; then
                                echo "file in lib32"
                                ls -l $CRS_HOME/lib32/$i
                        else
                                echo "file not exists"
                        fi
                fi
        done
else
                echo "Please set CRS_HOME and run again"
fi

./cluster_file_check.sh
crs home exist
file exists
-rwxr--r-- 1 root root 4765562 Feb  8  2011 /opt/crs/oracle/product/11.1.0/crs/lib/clntsh.map
file in lib32
-rw-r--r-- 1 root root 3985432 Feb  8  2011 /opt/crs/oracle/product/11.1.0/crs/lib32/clntsh.map
file not exists
file not exists
file not exists
file exists
lrwxrwxrwx 1 root root 17 Feb  8  2011 /opt/crs/oracle/product/11.1.0/crs/lib/libclntsh.so -> libclntsh.so.11.1
file in lib32
lrwxrwxrwx 1 root root 17 Feb  8  2011 /opt/crs/oracle/product/11.1.0/crs/lib32/libclntsh.so -> libclntsh.so.11.1
file exists
lrwxrwxrwx 1 oracle oinstall 51 Feb  8  2011 /opt/crs/oracle/product/11.1.0/crs/lib/libclntsh.so.10.1 -> /opt/crs/oracle/product/11.1.0/crs/lib/libclntsh.so
file in lib32
lrwxrwxrwx 1 oracle oinstall 53 Feb  8  2011 /opt/crs/oracle/product/11.1.0/crs/lib32/libclntsh.so.10.1 -> /opt/crs/oracle/product/11.1.0/crs/lib32/libclntsh.so
file not exists
file exists
-rwxr-xr-x 1 root root 48316147 Feb  8  2011 /opt/crs/oracle/product/11.1.0/crs/lib/libclntsh.so.11.1
file in lib32
-rwxr-xr-x 1 root root 37081093 Feb  8  2011 /opt/crs/oracle/product/11.1.0/crs/lib32/libclntsh.so.11.1
file not exists
file not exists

3. Make a note of the asm disk's scsi_ids , this setup is a test setup using Virtualbox. The scsi_ids will be compared after the OS upgrade and must be the same as before the upgrade.

# blkid|grep sd.*oracleasm|while read a b;do echo -n $a$b" scsi_id=";(echo $a|tr -d [:digit:]|tr -d [:]|cut -d"/" -f3|xargs -i scsi_id -g -s /block/{})done;
/dev/sdc1:LABEL="DATA" TYPE="oracleasm" scsi_id=SATA     VBOX HARDDISK  VB35fd6dbb-14d87b49
/dev/sdd1:LABEL="FLASH" TYPE="oracleasm" scsi_id=SATA     VBOX HARDDISK  VB4b88ed44-eff71eb1

4. Shutdown the cluster stack across the cluster. If one node at a time is upgraded then do this only on one node. In this case since all nodes are upgraded at the same time entire cluster is brought down

crs_stop -all

5. Stop the crs and disbale auto start of the crs. Do this on all ndoes.

# crsctl stop crs
Stopping resources.
This could take several minutes.
Successfully stopped Oracle Clusterware resources
Stopping Cluster Synchronization Services.
Shutting down the Cluster Synchronization Services daemon.
Shutdown request successfully issued.

# crsctl disable crs
Oracle Clusterware is disabled for start-up after a reboot.

6. Comment out the crs spwaning processes entries in /etc/inittab. Do this on all nodes

#h1:35:respawn:/etc/init.d/init.evmd run >/dev/null 2>&1 </dev/null
#h2:35:respawn:/etc/init.d/init.cssd fatal >/dev/null 2>&1 </dev/null
#h3:35:respawn:/etc/init.d/init.crsd run >/dev/null 2>&1 </dev/null

Run init on the systme and verify that no crs related processes are running.

/sbin/init q

ps -ef|grep css
ps -ef|grep crs
ps -ef|grep evm
ps -ef|grep init

7. Stop oracleasm running and disable and prevent it loading on reboot. Do this on all nodes.

# /etc/init.d/oracleasm stop
Dropping Oracle ASMLib disks:                              [  OK  ]
Shutting down the Oracle ASMLib driver:                    [  OK  ]

# /etc/init.d/oracleasm disable
Writing Oracle ASM library driver configuration: done
Dropping Oracle ASMLib disks:                              [  OK  ]
Shutting down the Oracle ASMLib driver:                    [  OK  ]

# /sbin/chkconfig --list | grep oracleasm
oracleasm       0:off   1:off   2:on    3:on    4:on    5:on    6:off
# /sbin/chkconfig oracleasm off
# /sbin/chkconfig --list | grep oracleasm
oracleasm       0:off   1:off   2:off   3:off   4:off   5:off   6:off

8. Make a backup of /etc/sysconfig/oracleasm-_dev_oracleasm on all nodes.

# cp /etc/sysconfig/oracleasm-_dev_oracleasm /etc/sysconfig/oracleasm-_dev_oracleasm.bak

At this point the system is ready for the OS upgrade.

9. Carry out the OS ugprade. All Oracle Home locations (ORACLE_HOME,ASM_HOME and CRS_HOME) are to be same before and after the upgrade

10. Install the new version of oracleasm and remove the old version. Old version could only be removed after installing the new version due to dependencies.

rpm -e oracleasm-2.6.18-194.el5-2.0.5-1.el5
error: Failed dependencies:
        oracleasm >= 1.0.4 is needed by (installed) oracleasmlib-2.0.4-1.el5.x86_64

# rpm -ivh oracleasm-2.6.18-308.el5-2.0.5-1.el5.x86_64.rpm
warning: oracleasm-2.6.18-308.el5-2.0.5-1.el5.x86_64.rpm: Header V3 DSA signature: NOKEY, key ID 1e5e0159
Preparing...                ########################################### [100%]
   1:oracleasm-2.6.18-308.el########################################### [100%]

# rpm -e oracleasm-2.6.18-194.el5-2.0.5-1.el5

# rpm -qa | grep oracleasm
oracleasmlib-2.0.4-1.el5
oracleasm-2.6.18-308.el5-2.0.5-1.el5
oracleasm-support-2.1.3-1.el5

11. Configure oracleasm and make sure content on oracleasm-_dev_oracleasm.bak and oracleasm-_dev_oracleasm are the same.

/etc/init.d/oracleasm configure
Configuring the Oracle ASM library driver.

This will configure the on-boot properties of the Oracle ASM library
driver.  The following questions will determine whether the driver is
loaded on boot and what permissions it will have.  The current values
will be shown in brackets ('[]').  Hitting  without typing an
answer will keep that current value.  Ctrl-C will abort.

Default user to own the driver interface [oracle]:
Default group to own the driver interface [dba]:
Start Oracle ASM library driver on boot (y/n) [n]: y
Scan for Oracle ASM disks on boot (y/n) [y]:
Writing Oracle ASM library driver configuration: done
Initializing the Oracle ASMLib driver:                     [  OK  ]
Scanning the system for Oracle ASMLib disks:               [  OK  ]

12. Enable oracleasm and set it up to start on reboot.

/etc/init.d/oracleasm enable
Writing Oracle ASM library driver configuration: done
Initializing the Oracle ASMLib driver:                     [  OK  ]
Scanning the system for Oracle ASMLib disks:               [  OK  ]

chkconfig --list | grep oracleasm
oracleasm       0:off   1:off   2:off   3:off   4:off   5:off   6:off

# chkconfig oracleasm on

# chkconfig --list | grep oracleasm
oracleasm       0:off   1:off   2:on    3:on    4:on    5:on    6:off

13. Verify scsi_id are same as before the upgrade

# blkid|grep sd.*oracleasm|while read a b;do echo -n $a$b" scsi_id=";(echo $a|tr -d [:digit:]|tr -d [:]|cut -d"/" -f3|xargs -i scsi_id -g -s /block/{})done;
/dev/sdc1:LABEL="DATA" TYPE="oracleasm" scsi_id=SATA     VBOX HARDDISK  VB35fd6dbb-14d87b49
/dev/sdd1:LABEL="FLASH" TYPE="oracleasm" scsi_id=SATA     VBOX HARDDISK  VB4b88ed44-eff71eb1

With this the ASMLib upgrade is complete. Next is to relink the Oracle binaries.

14. As oracle user relink both ASM_HOME and ORACLE_HOME. Verify which relink is in the path variable using

which relink

Relink ORACLE_HOME

$ cd $ORACLE_HOME/bin
$ relink all

Reset the PATH and ORACLE_HOME before running relink for ASM_HOME

export PATH=$ASM_HOME/bin:$PATH
export ORACLE_HOME=$ASM_HOME

Verify the relink binary from ASM_HOME is in path

which relink

Run the relink of ASM_HOME

$ cd $ASM_HOME/bin
$ relink all

15. As per (743649.1 step 10) clusterware client shared libraries should be relinked as the clusterware software owner. But running this as software owner result in permission issue

export ORACLE_HOME=$CRS_HOME (important to set ORACLE_HOME variable to CRS_HOME location)
cd $ORACLE_HOME/bin

./genclntsh
/bin/rm: cannot remove `/opt/crs/oracle/product/11.1.0/crs/lib/libclntsh.so.11.1': Permission denied
/bin/rm: cannot remove `/opt/crs/oracle/product/11.1.0/crs/lib/clntsh.map': Permission denied
./genclntsh: line 309: /opt/crs/oracle/product/11.1.0/crs/lib/clntsh.map: Permission denied
genclntsh: Failed to link libclntsh.so.11.1

Permission issue persist even after setting the permission to oracle:oinstall. Running ./genclntsh as root results in files get created with new timestamp permission same as before (metalink note could be wrong!??). When genclntsh runs without any issue no message is shown and return to shell prompt. After run check the file permissions are same as before (step 2 above).

./cluster_file_check.sh
crs home exist
file exists
-rw-r--r-- 1 root root 4764652 Jun 20 16:49 /opt/crs/oracle/product/11.1.0/crs/lib/clntsh.map
file in lib32
-rw-r--r-- 1 root root 3984490 Jun 20 16:49 /opt/crs/oracle/product/11.1.0/crs/lib32/clntsh.map
file not exists
file not exists
file not exists
file exists
lrwxrwxrwx 1 root root 17 Jun 20 16:49 /opt/crs/oracle/product/11.1.0/crs/lib/libclntsh.so -> libclntsh.so.11.1
file in lib32
lrwxrwxrwx 1 root root 17 Jun 20 16:49 /opt/crs/oracle/product/11.1.0/crs/lib32/libclntsh.so -> libclntsh.so.11.1
file exists
lrwxrwxrwx 1 oracle oinstall 51 Feb  8  2011 /opt/crs/oracle/product/11.1.0/crs/lib/libclntsh.so.10.1 -> /opt/crs/oracle/product/11.1.0/crs/lib/libclntsh.so
file in lib32
lrwxrwxrwx 1 oracle oinstall 53 Feb  8  2011 /opt/crs/oracle/product/11.1.0/crs/lib32/libclntsh.so.10.1 -> /opt/crs/oracle/product/11.1.0/crs/lib32/libclntsh.so
file not exists
file exists
-rwxr-xr-x 1 root root 48316147 Jun 20 16:49 /opt/crs/oracle/product/11.1.0/crs/lib/libclntsh.so.11.1
file in lib32
-rwxr-xr-x 1 root root 37081093 Jun 20 16:49 /opt/crs/oracle/product/11.1.0/crs/lib32/libclntsh.so.11.1
file not exists
file not exists

16. Enable crs and remove the comments on /etc/inittab and start crs

# crsctl enable crs
Oracle Clusterware is enabled for start-up after a reboot.

h1:35:respawn:/etc/init.d/init.evmd run >/dev/null 2>&1 </dev/null
h2:35:respawn:/etc/init.d/init.cssd fatal >/dev/null 2>&1 </dev/null
h3:35:respawn:/etc/init.d/init.crsd run >/dev/null 2>&1 </dev/null

17. Start crs with crsctl or with OS init

crsctl start crs
Attempting to start Oracle Clusterware stack
The CRS stack will be started shortly

or

/sbin/init q

This conclude the ASMLib and OS upgrade.

Useful Metalink Notes
How to Relink Oracle Database Software on UNIX [ID 131321.1]
Is It Necessary To Relink Oracle Following OS Upgrade? [ID 444595.1]
Will an Operating System Upgrade Affect Oracle Clusterware? [ID 743649.1]
How To Upgrade ASMLib Kernel Driver as Part of Kernel Upgrade? [ID 1391807.1]
Do You Need to Relink Oracle Clusterware When Upgrading the Operating System? [ID 743649.1]
Is It Required To Relink The Oracle Binaries When Booting With The Old Kernel? [ID 461138.1]
Cannot Find Exact Kernel Version Match For ASMLib (Workaround using oracleasm_debug_link tool) [ID 462618.1]
RAC: Frequently Asked Questions [ID 220970.1] (Is a relink required for the clusterware home after an OS upgrade?)

Useful White Paper
Best Practices for Optimizing Availability During Planned Maintenance Using Oracle Clusterware and Oracle Real Application Clusters. Oracle Maximum Availability Architecture White Paper September 2007

Related Posts
Upgrading RHEL 6 OS in a 11gR2 RAC Environment
Upgrading OS in 11gR2 RAC Environment (RHEL 5)

Implementing Class of Secure Transport (COST) to Restrict Instance Registration in Oracle 11gR2 SE RAC (Solution mentioned in Oracle Security Alert for CVE-2012-1675)

This post is related to implementing the solution for security vulnerability mentioned in Oracle Security Alert for CVE-2012-1675. There is another post but that is related to version 11gR1 and this one is specific to 11gR2.
The metalink note related to applying the solution in a RAC environment is Using Class of Secure Transport (COST) to Restrict Instance Registration in Oracle RAC [ID 1340831.1]
However this solution fails when tried on a standard edition RAC environment as standard edition Oracle binaries are not linked with TCPS protocol. The standard edition RAC environment created for RHEL 6 is used in this case.
All the steps in section 1 in (1340831.1) up to step 1.4 work same on SE as on EE edition and problem will be encountered trying to implement step 1.5.
As oracle user (since this is a role separated environment) try to set TCPS protocol for the remote listener using scan ips.

show parameter remote_listener

NAME                                 TYPE        VALUE
------------------------------------ ----------- ------------------------------
remote_listener                      string      rhel6m-scan:1521

This environment only used one scan ip (not recommended by Oracle)

srvctl config scan
SCAN name: rhel6m-scan, Network: 1/192.168.0.0/255.255.255.0/eth0
SCAN VIP name: scan1, IP: /rhel6m-scan/192.168.0.91

Trying to set TCPS will give the following error

SQL> alter system set remote_listener='(ADDRESS_LIST=(ADDRESS=(PROTOCOL=TCPS)(HOST=192.168.0.91)(PORT=1523)))' scope=both sid='*';
alter system set remote_listener='(ADDRESS_LIST=(ADDRESS=(PROTOCOL=TCPS)(HOST=192.168.0.91)(PORT=1523)))' scope=both sid='*'
*
ERROR at line 1:
ORA-02097: parameter cannot be modified because specified value is invalid
ORA-00119: invalid specification for system parameter REMOTE_LISTENER
ORA-00130: invalid listener address
'(ADDRESS=(PROTOCOL=TCPS)(HOST=192.168.0.91)(PORT=1523))'

Error messages here doesn't help much as it doesn't specifically say which parameter is invalid unlike in 11gR1 which mentioned invalid protocol. It could be tested indeed it's the protocol that is causing the problem by testing the same command with TCP instead of TCPS, in this case it would work without an error

SQL> alter system set remote_listener='(ADDRESS_LIST=(ADDRESS=(PROTOCOL=TCP)(HOST=192.168.0.91)(PORT=1523)))' scope=both sid='*';

System altered.

Solution is to relink the binaries with tcps same as before. Stop all processes running out of the ORACLE_HOME

srvctl stop home -o $ORACLE_HOME -s status -n `hostname -s`

Change into $ORACLE_HOME/lib and make a backup of the existing libntcps11.a file and copy the libntcps11_ee.a.dbl

cp libntcps11.a libntcps11.a.orig
cp libntcps11_ee.a.dbl libntcps11.a

Execute relink all and monitor the log file

$ relink all
writing relink log to: /opt/app/oracle/product/11.2.0/dbhome_1/install/relink.log

Relinking ends with following text

-L/opt/app/oracle/product/11.2.0/dbhome_1/lib
test ! -f /opt/app/oracle/product/11.2.0/dbhome_1/bin/oracle ||\
           mv -f /opt/app/oracle/product/11.2.0/dbhome_1/bin/oracle /opt/app/oracle/product/11.2.0/dbhome_1/bin/oracleO
mv /opt/app/oracle/product/11.2.0/dbhome_1/rdbms/lib/oracle /opt/app/oracle/product/11.2.0/dbhome_1/bin/oracle
chmod 6751 /opt/app/oracle/product/11.2.0/dbhome_1/bin/oracle

It is important to relink all the oracle homes in the cluster if not the error will still persist

SQL> alter system set remote_listener='(ADDRESS_LIST=(ADDRESS=(PROTOCOL=TCPS)(HOST=192.168.0.91)(PORT=1523)))' scope=both sid='*';
alter system set remote_listener='(ADDRESS_LIST=(ADDRESS=(PROTOCOL=TCPS)(HOST=192.168.0.91)(PORT=1523)))' scope=both sid='*'
*
ERROR at line 1:
ORA-32008: error while processing parameter update at instance std11g21
ORA-02097: parameter cannot be modified because specified value is invalid
ORA-00119: invalid specification for system parameter REMOTE_LISTENER
ORA-00130: invalid listener address
'(ADDRESS=(PROTOCOL=TCPS)(HOST=192.168.0.91)(PORT=1523))'

But to confirm relinking has worked the above command could be executed at instance level

SQL> alter system set remote_listener='(ADDRESS_LIST=(ADDRESS=(PROTOCOL=TCPS)(HOST=192.168.0.91)(PORT=1523)))' scope=both sid='std11g22';

System altered.

Once all oracle homes are relinked execute the original command.

SQL> alter system set remote_listener='(ADDRESS_LIST=(ADDRESS=(PROTOCOL=TCPS)(HOST=192.168.0.91)(PORT=1523)))' scope=both sid='*';

System altered.

Rest of the steps in (1340831.1) could be executed without any additional work.
If the libntcps11_ee.a.dbl file is missing it could obtained by extracting the

database/stage/Components/oracle.network.rsf/11.2.0.3.0/1/DataFiles/filegroup5.jar

Related Post
Using Class of Secure Transport (COST) to Restrict Instance Registration in Oracle SE RAC Fails (Solution mentioned in Oracle Security Alert for CVE-2012-1675)




Update on 11th November 2015
For database version 11.2.0.4 Oracle has introduced "Valid Node Checking for Registration (VNCR)" as an alternative for implementing COST, if the sole reason for implementing cost is preventing remote registration. MOS note 1340831.1 has been updated with this information. VNCR related information could be found on following docs.
How to Enable VNCR on RAC Database to Register only Local Instances [ID 1914282.1]
Valid Node Checking For Registration (VNCR) [ID 1600630.1]

Check which Binary Options are Installed

Oracle binary get linked with many of the database options.
RAC - Real Application Cluster
RAT - Real Application Testing
OLS - Oracle Label Security
DV - Database Vault
ASM - Automated Storage Management
OLAP - Oracle OLAP
PART - Oracle Partitioning
CTX - Context Management Text

Following script could be used to find out which options are already relinked with Oracle binary. (It is assumed ORACLE_HOME is set)

component=(RAC RAT OLS DV ASM OLAP PART CTX)
compfile=(kcsm.o  kecwr.o kzlilbac.o  kzvidv.o kfon.o xsyeolap.o kkpoban.o kciwcx.o)
i=0
for comp in ${component[@]}
  do
    x=`ar -t $ORACLE_HOME/rdbms/lib/libknlopt.a | grep -c ${compfile[i]}`
    if [[ "$x" == "0" ]]; then
      printf "%-15s %s \033[31m %8s \033[0m %2s\n" "$comp" "[" "NOT INSTALLED" "]"
    else
      printf "%-15s %s \033[32m %11s \033[0m %4s\n" "$comp" "["   "INSTALLED"   "]"
    fi
    let "i++"
  done

RAC             [  NOT INSTALLED   ]
RAT             [    INSTALLED     ]
OLS             [  NOT INSTALLED   ]
DV              [  NOT INSTALLED   ]
ASM             [  NOT INSTALLED   ]
OLAP            [    INSTALLED     ]
PART            [    INSTALLED     ]
CTX             [    INSTALLED     ]

More on metalink note
How to Check and Enable/Disable Oracle Binary Options [ID 948061.1]
How to Check Whether Oracle Binary/Instance is RAC Enabled and Relink Oracle Binary in RAC [ID 284785.1]

Using Class of Secure Transport (COST) to Restrict Instance Registration in Oracle SE RAC Fails (Solution mentioned in Oracle Security Alert for CVE-2012-1675)

Oracle Security Alert CVE-2012-1675 mentions two metalink notes.
Using Class of Secure Transport (COST) to Restrict Instance Registration in Oracle RAC [ID 1340831.1] for RAC systems and Using Class of Secure Transport (COST) to Restrict Instance Registration [ID 1453883.1] for non-RAC systems. Non-RAC systme works fine in Oracle Enterprise and Standard Edition systems. But the solution given in 1340831.1 for "Oracle RAC versions prior to 11.2 (10.2.0.3 - 11.1)" cannot be implemented in Oracle Standard Edition RAC. After the wallet and the patch 12880299 is applied the metalink notes says to add a line to listener.ora for TCPS protocol and restart the listener. (This environment has seperate homes for ASM and DB and listener runs out of ASM home). Oracle database version is 11.1.0.7.11

LISTENER_RAC1 =
  (DESCRIPTION_LIST =
    (DESCRIPTION =
      (ADDRESS = (PROTOCOL = TCP)(HOST = rac1-vip)(PORT = 1521)(IP = FIRST))
      (ADDRESS = (PROTOCOL = TCPS)(HOST = rac1-vip)(PORT = 1523)(IP = FIRST))
      (ADDRESS = (PROTOCOL = TCP)(HOST = 192.168.0.85)(PORT = 1521)(IP = FIRST))
    )
  )

WALLET_LOCATION =
  (SOURCE =
   (METHOD = FILE)
    (METHOD_DATA =
     (DIRECTORY = /opt/app/oracle/product/11.1.0/asm_1/network/admin/cost)
    )
  )

#SECURE_REGISTER_LISTENER_RAC1 = (TCP,TCPS)

It then ask to stop and start the listener on that node. During the start of the listener following error is shown on the command line.

$ASM_HOME/bin/srvctl start listener -n `hostname -s`
rac1:ora.rac1.LISTENER_RAC1.lsnr:
rac1:ora.rac1.LISTENER_RAC1.lsnr:LSNRCTL for Linux: Version 11.1.0.7.0 - Production on 09-MAY-2012 15:01:38
rac1:ora.rac1.LISTENER_RAC1.lsnr:
rac1:ora.rac1.LISTENER_RAC1.lsnr:Copyright (c) 1991, 2008, Oracle.  All rights reserved.
rac1:ora.rac1.LISTENER_RAC1.lsnr:
rac1:ora.rac1.LISTENER_RAC1.lsnr:Starting /opt/app/oracle/product/11.1.0/asm_1/bin/tnslsnr: please wait...
rac1:ora.rac1.LISTENER_RAC1.lsnr:
rac1:ora.rac1.LISTENER_RAC1.lsnr:TNSLSNR for Linux: Version 11.1.0.7.0 - Production
rac1:ora.rac1.LISTENER_RAC1.lsnr:System parameter file is /opt/app/oracle/product/11.1.0/asm_1/network/admin/listener.ora
rac1:ora.rac1.LISTENER_RAC1.lsnr:Log messages written to /opt/app/oracle/diag/tnslsnr/rac1/listener_rac1/alert/log.xml
rac1:ora.rac1.LISTENER_RAC1.lsnr:Listening on: (DESCRIPTION=(ADDRESS=(PROTOCOL=tcp)(HOST=192.168.0.89)(PORT=1521)))
rac1:ora.rac1.LISTENER_RAC1.lsnr:Error listening on: (DESCRIPTION=(ADDRESS=(PROTOCOL=TCPS)(HOST=rac1-vip)(PORT=1523)(IP=FIRST)))
rac1:ora.rac1.LISTENER_RAC1.lsnr:TNS-12557: TNS:protocol adapter not loadable
rac1:ora.rac1.LISTENER_RAC1.lsnr: TNS-12560: TNS:protocol adapter error
rac1:ora.rac1.LISTENER_RAC1.lsnr:  TNS-00527: Protocol Adapter not loadable
rac1:ora.rac1.LISTENER_RAC1.lsnr:
rac1:ora.rac1.LISTENER_RAC1.lsnr:
rac1:ora.rac1.LISTENER_RAC1.lsnr:Listener failed to start. See the error message(s) above...
rac1:ora.rac1.LISTENER_RAC1.lsnr:
rac1:ora.rac1.LISTENER_RAC1.lsnr:
rac1:ora.rac1.LISTENER_RAC1.lsnr:LSNRCTL for Linux: Version 11.1.0.7.0 - Production on 09-MAY-2012 15:01:38
rac1:ora.rac1.LISTENER_RAC1.lsnr:
rac1:ora.rac1.LISTENER_RAC1.lsnr:Copyright (c) 1991, 2008, Oracle.  All rights reserved.
rac1:ora.rac1.LISTENER_RAC1.lsnr:
rac1:ora.rac1.LISTENER_RAC1.lsnr:Connecting to (DESCRIPTION=(ADDRESS=(PROTOCOL=TCP)(HOST=rac1-vip)(PORT=1521)(IP=FIRST)))
rac1:ora.rac1.LISTENER_RAC1.lsnr:TNS-12541: TNS:no listener
rac1:ora.rac1.LISTENER_RAC1.lsnr: TNS-12560: TNS:protocol adapter error
rac1:ora.rac1.LISTENER_RAC1.lsnr:  TNS-00511: No listener
rac1:ora.rac1.LISTENER_RAC1.lsnr:   Linux Error: 111: Connection refused
rac1:ora.rac1.LISTENER_RAC1.lsnr:Connecting to (DESCRIPTION=(ADDRESS=(PROTOCOL=TCPS)(HOST=rac1-vip)(PORT=1523)(IP=FIRST)))
rac1:ora.rac1.LISTENER_RAC1.lsnr:TNS-12557: TNS:protocol adapter not loadable
rac1:ora.rac1.LISTENER_RAC1.lsnr: TNS-12560: TNS:protocol adapter error
rac1:ora.rac1.LISTENER_RAC1.lsnr:  TNS-00527: Protocol Adapter not loadable
rac1:ora.rac1.LISTENER_RAC1.lsnr:
rac1:ora.rac1.LISTENER_RAC1.lsnr:Connecting to (DESCRIPTION=(ADDRESS=(PROTOCOL=TCP)(HOST=192.168.0.85)(PORT=1521)(IP=FIRST)))
rac1:ora.rac1.LISTENER_RAC1.lsnr:TNS-12541: TNS:no listener
rac1:ora.rac1.LISTENER_RAC1.lsnr: TNS-12560: TNS:protocol adapter error
rac1:ora.rac1.LISTENER_RAC1.lsnr:  TNS-00511: No listener
rac1:ora.rac1.LISTENER_RAC1.lsnr:   Linux Error: 111: Connection refused
CRS-0215: Could not start resource 'ora.rac1.LISTENER_RAC1.lsnr'.

On listener.log

System parameter file is /opt/app/oracle/product/11.1.0/asm_1/network/admin/listener.ora
Log messages written to /opt/app/oracle/diag/tnslsnr/rac1/listener_rac1/alert/log.xml
Trace information written to /opt/app/oracle/diag/tnslsnr/rac1/listener_rac1/trace/ora_15792_47190667617024.trc
Trace level is currently 0

Started with pid=15792
Listening on: (DESCRIPTION=(ADDRESS=(PROTOCOL=tcp)(HOST=192.168.0.89)(PORT=1521)))
Error listening on: (DESCRIPTION=(ADDRESS=(PROTOCOL=TCPS)(HOST=rac1-vip)(PORT=1523)(IP=FIRST)))
TNS-12557: TNS:protocol adapter not loadable
 TNS-12560: TNS:protocol adapter error
  TNS-00527: Protocol Adapter not loadable

No longer listening on: (DESCRIPTION=(ADDRESS=(PROTOCOL=tcp)(HOST=192.168.0.89)(PORT=1521)))

On crsd.log

2012-05-09 15:01:37.891: [  CRSRES][1510025536] startRunnable: setting CLI values
2012-05-09 15:01:37.895: [  CRSRES][1510025536] Attempting to start `ora.rac1.LISTENER_RAC1.lsnr` on member `rac1`
2012-05-09 15:01:38.920: [  CRSAPP][1510025536] StartResource error for ora.rac1.LISTENER_RAC1.lsnr error code = 1
2012-05-09 15:01:39.240: [  CRSRES][1510025536] Start of `ora.rac1.LISTENER_RAC1.lsnr` on member `rac1` failed.
2012-05-09 15:01:39.255: [  CRSRES][1510025536] rac2 : CRS-1019: Resource ora.rac1.LISTENER_RAC1.lsnr (application) cannot run on rac2

It could be that in SE the Advance Security Option needed to support TCPS is not there.

In the security alert Oracle mentions "Please note that Oracle has added Oracle Advanced Security SSL/TLS to the Oracle Database Standard Edition license when used with the Real Application Clusters and Oracle has added Oracle Advanced Security SSL/TLS to the Enterprise Edition Real Application Clusters (Oracle RAC) and RAC One Node options so that the directions provided in the Support Notes referenced above can be applied by all Oracle customers without additional cost."

The steps works in a 11.1.0.7.11 Enterprise Edition (EE) RAC without any issue. This could be explained using How To Check if Oracle Advanced Security Option is Installed ? [ID 549989.1] in which it is mentioned "Oracle product development has decided that starting with Oracle 11g Oracle Advanced Security Option will be installed by default when installing Enterprise Edition . There is no way to uninstall or to install it."

Update 31 May 2012
For the steps mentioned in 1340831.1 to work on Oracle SE it requires Oracle binaries to be relinked with the tcps. In this example the listener is running out of ASM home therefore oracle binaries in ASM Home will be relinked with tcps. It is not necessary to relink the Oracle Home if listener is not running out of it
1. cd $ASM_HOME/lib (or $ORACLE_HOME/lib if a separate home for ASM is not used).

2. Make a backup of the current tcps binary

$  cp libntcps11.a libntcps11.a.orig

3. Find libntcps11_ee.a.dbl file in the same lib directory and make a copy of it with the file name libntcps11.a

$ cp libntcps11_ee.a.dbl libntcps11.a

4. To make sure the relink is happening in ASM_HOME set the ORACLE_HOME to ASM_HOME and set relink binary in PATH variable

export ORACLE_HOME=$ASM_HOME
which relink
/opt/app/oracle/product/11.1.0/db_1/bin/relink <-- still using the ORACLE_HOME relink

export PATH=$ORACLE_HOME/bin:$PATH
/opt/app/oracle/product/11.1.0/asm_1/bin/relink

5. Stop all processes running out of ASM_HOME. Since database instance is dependent on asm instance this would mean database instance should be stopped as well. Once all running processes are stopped (except clusterware processes) start the relink with

relink all

and the relink process end with following lines (this is on 11.1.0.7 , on other versions it may be different)

..
..
/app/oracle/product/11.1.0/asm_1/lib
mv -f /opt/app/oracle/product/11.1.0/asm_1/bin/ctxkbtc /opt/app/oracle/product/11.1.0/asm_1/bin/ctxkbtc0
mv ctxkbtc /opt/app/oracle/product/11.1.0/asm_1/bin/ctxkbtc
chmod 751 /opt/app/oracle/product/11.1.0/asm_1/bin/ctxkbtc

Once the relink is complete continue with the steps in the metalink document and the listener will be able to start with TCPS.
If the file libntcps11_ee.a.dbl is missing in $ORACLE_HOME/lib then it could be obtained by extracting the

database/stage/Components/oracle.network.rsf/11.1.0.6.0/1/DataFiles/filegroup7.jar

file in the 11.1.0.6 base installation. Even though the system is upgraded to 11.1.0.7 there's no copy of it in 11.1.0.7 patchset files which mean it is not part of the upgraded processes thus only place to get it is on 11.1.0.6 base instllation.