Upgrading RHEL 6 OS in a 11gR2 RAC Environment

This post upgrade the RHEL 6 OS on severs which are running 11gR2 RAC. Current kernel is

$ uname -r
2.6.32-220.el6.x86_64

and the database patch level is

opatch lsinventory -local | grep Patch
Oracle Interim Patch Installer version 11.2.0.3.0
OPatch version    : 11.2.0.3.0
Patch  14275605     : applied on Thu Nov 15 14:21:27 GMT 2012
Unique Patch ID:  15379762
Patch description:  "Database Patch Set Update : 11.2.0.3.4 (14275605)"
Sub-patch  13923374; "Database Patch Set Update : 11.2.0.3.3 (13923374)"
Sub-patch  13696216; "Database Patch Set Update : 11.2.0.3.2 (13696216)"
Sub-patch  13343438; "Database Patch Set Update : 11.2.0.3.1 (13343438)"
Patch  14275572     : applied on Thu Nov 15 14:20:32 GMT 2012
Unique Patch ID:  15379762
Patch description:  "Grid Infrastructure Patch Set Update : 11.2.0.3.4 (14275572)"
OPatch succeeded.

The database software is running standard edition.
The steps are similar to that of upgrading RHEL 5 OS running 11gR2 RAC with few minor differences. This RAC system does not use ASMLibs.
1. Current kernel version

$ uname -r
2.6.32-220.el6.x86_64

2. Stop all the cluster components

crsctl stop cluster -all

and stop disable start of crs after reboot

crsctl stop crs
crsctl disable crs

3. In RHEL 6 inittab is depreciated and the spawning process file is located in /etc/init with the name oracle-ohasd.conf
More on this is available on Troubleshoot Grid Infrastructure Startup Issues [ID 1050908.1] If Oracle is 11.2.0.4 comment out the trace file analyzer script as well oracle-tfa.conf.

cat oracle-ohasd.conf
# Copyright (c) 2001, 2011, Oracle and/or its affiliates. All rights reserved.
#
# Oracle OHASD startup

start on runlevel [35]
stop  on runlevel [!35]
respawn
exec /etc/init.d/init.ohasd run >/dev/null 2>&1 </dev/null

As root comment last two lines

#respawn
#exec /etc/init.d/init.ohasd run >/dev/null 2>&1 </dev/null

4. Upgrade the OS (only few screenshots shown)

5. New kernel version after upgrade

uname -r
2.6.32-279.el6.x86_64

6. Stop oracle-ohasd

initctl stop oracle-ohasd
oracle-ohasd stop/waiting

7. Un-comment the last two lines on /etc/init/oracle-ohasd that was commented earlier

start on runlevel [35]
stop  on runlevel [!35]
respawn
exec /etc/init.d/init.ohasd run >/dev/null 2>&1 </dev/null

8. Enable crs on startup

crsctl enable crs
CRS-4622: Oracle High Availability Services autostart is enabled.

9. Relink the Oracle Home binaries as the oracle software owner

which relink
/opt/app/oracle/product/11.2.0/dbhome_1/bin/relink
$ cd $ORACLE_HOME/bin
$ ./relink all
writing relink log to: /opt/app/oracle/product/11.2.0/dbhome_1/install/relink.log

The relink will end with following on the log

test ! -f /opt/app/oracle/product/11.2.0/dbhome_1/bin/oracle ||\
           mv -f /opt/app/oracle/product/11.2.0/dbhome_1/bin/oracle /opt/app/oracle/product/11.2.0/dbhome_1/bin/oracleO
mv /opt/app/oracle/product/11.2.0/dbhome_1/rdbms/lib/oracle /opt/app/oracle/product/11.2.0/dbhome_1/bin/oracle
chmod 6751 /opt/app/oracle/product/11.2.0/dbhome_1/bin/oracle

9. Unlock the permission on the grid software as root

# cd $GI_HOME/crs/install

# perl rootcrs.pl -unlock
Using configuration parameter file: ./crsconfig_params
CRS-4544: Unable to connect to OHAS
CRS-4000: Command Stop failed, or completed with errors.
Successfully unlock /opt/app/11.2.0/grid

10. As the grid software owner (gird in this case) relink the grid software

[grid@rhel6m1 ~]$ export ORACLE_HOME=$GI_HOME
[grid@rhel6m1 ~]$ export PATH=$ORACLE_HOME/bin:$PATH
[grid@rhel6m1 ~]$ which relink
/opt/app/11.2.0/grid/bin/relink
[grid@rhel6m1 ~]$ $GI_HOME/bin/relink
writing relink log to: /opt/app/11.2.0/grid/install/relink.log

The relink ends with

test ! -f /opt/app/11.2.0/grid/bin/oracle ||\
           mv -f /opt/app/11.2.0/grid/bin/oracle /opt/app/11.2.0/grid/bin/oracleO
mv /opt/app/11.2.0/grid/rdbms/lib/oracle /opt/app/11.2.0/grid/bin/oracle
chmod 6751 /opt/app/11.2.0/grid/bin/oracle

11. Run the post relink script. rootcrs.pl -patch takes a while to complete and ends with an error as shown below

# cd $GI_HOME/rdbms/install
# ./rootadd_rdbms.sh

# cd $GI_HOME/crs/install
# perl rootcrs.pl -patch
Using configuration parameter file: ./crsconfig_params
CRS-4124: Oracle High Availability Services startup failed.
CRS-4000: Command Start failed, or completed with errors.
Failed to write the checkpoint:'' with status:FAIL.Error code is 256
Oracle Grid Infrastructure stack start initiated but failed to complete at crsconfig_lib.pm line 11545.

After an SR it was confirmed that this error is ignorable and crs could be started with "init 3" or by rebooting the server. Reason that above error happen is that since "ohasd run" line was commented on oracle-ohasd file there's no "ohasd run" process active when the kernel is upgraded and server is rebooted. Without "ohasd run" active the crsctl start crs (issued during the rootcrs.pl -patch) will fail to start the clusterware stack. This has no effect on the actual relink. This is also mentioned in 1050908.1
For metalink notes related to Upgrading OS and relinking clusterware and oracle binaries after OS upgrade refer the related post given below.

Related Posts
Upgrading OS in 11gR2 RAC Environment (RHEL 5)
Upgrading ASMLib and OS in 11gR1 RAC Environment

Useful metalink notes
Executing "relink all" resets permission of extjob, jssu, oradism, externaljob.ora [ID 1555453.1]
Relinking Oracle Home FAQ ( Frequently Asked Questions) [ID 1467060.1]

ORA-04063: package body "XDB.DBMS_CSX_INT" has errors

Following could be observed in the database alert log (11.2.0.3) during a schema export.

Errors in file /opt/app/oracle/diag/rdbms/db/db/trace/db_dw00_5669.trc:
ORA-00604: error occurred at recursive SQL level 3
ORA-04063: package body "XDB.DBMS_CSX_INT" has errors
XDB SGA reset to NULL.

This is a consequence of revoking execute privilege on utl_file from public. More on revoking execute privilege from public on some packages is available on 247093.1,797706.1,1165830.1.
To resolve the issue with the expdp grant the revoke execute privilege back to public or explicitly to the xdb user.


Useful metalink notes
Data Pump Export (expdp) = ORA-39126 ORA-39127 ORA-4063 (on XDB.DBMS_XDBUTIL_INT & XDB.DBMS_XDBZ0) ORA-6508 RESOURCE_VIEW Query = ORA-4063 [ID 1526443.1]
ORA-20011 Approximate NDV failed ORA-04063 package body XDB.DBMS_CSX_INT has errors [ID 1356250.1]

Related to revoking execute privilege on certain packages from Public
Problems After Revoking Execute On DBMS_SQL, DBMS_JOB, DBMS_LOB, DBMS_RANDOM, and DBMS_OBFUSCATION_TOOLKIT From PUBLIC [ID 1165830.1]
Be Cautious When Revoking Privileges Granted to PUBLIC [ID 247093.1]
Invalid Objects After Revoking EXECUTE Privilege From PUBLIC [ID 797706.1]

ORA-06512: ORACLE_OCM.MGMT_DB_LL_METRICS on Alert Log

On a new install 11.2.0.3 following could be observed on the alert log

Errors in file /opt/app/oracle/diag/rdbms/db/db1/trace/db1_j004_5824.trc:
ORA-12012: error on auto execute of job "ORACLE_OCM"."MGMT_CONFIG_JOB_2_1"
ORA-29280: invalid directory path
ORA-06512: at "ORACLE_OCM.MGMT_DB_LL_METRICS", line 2436
ORA-06512: at line 1

This is because the OCM is trying to write a directory that doesn't exists and to resolve this the out of the box OCM that comes with installer need further configuration. If OCM is not used drop the OCM user and the default directory setup. This does not affect any other database components.

SQL> drop user ORACLE_OCM cascade;
User dropped.

SQL> drop directory ORACLE_OCM_CONFIG_DIR;
Directory dropped.

If OCM is used then configure it by following "Oracle Configuration Manager Installation and Administration Guide".



Useful metalink notes
How to deinstall OCM [ID 761313.1]
Oracle Configuration Manager FAQ [ID 369111.5]
How To Manually Remove ORACLE_OCM From Database [ID 859113.1]
"ORA-12012: error on auto execute of job ORACLE_OCM.MGMT_CONFIG_JOB_2_1" And "ORA-29280: invalid directory path" In Database AlertLog [ID 1453959.1]

PRVF-5150 : Path /dev/oracleasm/disks/DISK is not a valid path on all nodes

In a new clusterware installation

PRVF-5150 : Path /dev/oracleasm/disks/DISK_NAME is not a valid path on all nodes

appears in the OUI but running cluvfy -pre crsinst shows that all pre-reqs are successful.

If the ASM setup is validated as per 1210863.1 then this pre-req failure on OUI could be ignored and installation could continue.


Useful metalink notes
Grid installation reports error on "Prerequisite checks" PRVF -5150 [ID 1498115.1]
Device Checks for ASM Fails with PRVF-5150: Path ORCL: is not a valid path [ID 1210863.1]
Bug 14048512 - PRVF-5150 : Path ORCL:DISK1 is not a valid path on all nodes [ID 14048512.8]
GI 11.2.0.3 Installation On Errors PRVF-5150 saying ASMLIB path is not a valid path [ID 1474961.1]

PRVF-5637 : DNS response time could not be checked on following nodes

Running cluvfy (pre crsinst) showed the following error for /etc/resolve.conf

Checking consistency of file "/etc/resolv.conf" across nodes
Checking the file "/etc/resolv.conf" to make sure only one of domain and search entries is defined
File "/etc/resolv.conf" does not have both domain and search entries defined
Checking if domain entry in file "/etc/resolv.conf" is consistent across the nodes...
domain entry in file "/etc/resolv.conf" is consistent across nodes
Checking file "/etc/resolv.conf" to make sure that only one domain entry is defined
All nodes have one domain entry defined in file "/etc/resolv.conf"
Checking all nodes to make sure that domain is "oracle.private" as found on node "DB-01"
All nodes of the cluster have same value for 'domain'
Checking if search entry in file "/etc/resolv.conf" is consistent across the nodes...
search entry in file "/etc/resolv.conf" is consistent across nodes
Checking DNS response time for an unreachable node
  Node Name                             Status
  ------------------------------------  ------------------------
  DB-01                             failed
  DB-02                             failed
PRVF-5637 : DNS response time could not be checked on following nodes: DB-01,DB-02
File "/etc/resolv.conf" is not consistent across nodes

1480242.1 and 1356975.1 list several bugs and other reasons as to why this check failed. But in this case the reason was none of those specified in the metalink notes. Beside the reasons listed on the above mentioned metalink notes, another reason it seems is the missing nslookup utility. Cluvfy must be able to find the nslookup if not it will erroneously state that File "/etc/resolv.conf" is not consistent across nodes when in fact it has not done the test at all.



Install nslookup utility with

yum install bind-utils

Afterwards this step would be successful. (if none of the bugs/reasons mentioned on 1480242.1 and 1356975.1 are there)

Checking consistency of file "/etc/resolv.conf" across nodes
Checking the file "/etc/resolv.conf" to make sure only one of domain and search entries is defined
File "/etc/resolv.conf" does not have both domain and search entries defined
Checking if domain entry in file "/etc/resolv.conf" is consistent across the nodes...
domain entry in file "/etc/resolv.conf" is consistent across nodes
Checking file "/etc/resolv.conf" to make sure that only one domain entry is defined
All nodes have one domain entry defined in file "/etc/resolv.conf"
Checking all nodes to make sure that domain is "oracle.private" as found on node "DB-01"
All nodes of the cluster have same value for 'domain'
Checking if search entry in file "/etc/resolv.conf" is consistent across the nodes...
search entry in file "/etc/resolv.conf" is consistent across nodes
Checking DNS response time for an unreachable node
  Node Name                             Status
  ------------------------------------  ------------------------
  DB-01                             passed
  DB-02                             passed
The DNS response time for an unreachable node is within acceptable limit on all nodes
File "/etc/resolv.conf" is consistent across nodes

Useful metalink notes
PRVF-5637 : DNS response time could not be checked on following nodes [ID 1480242.1]
PRVF-5636 : The DNS response time for an unreachable node exceeded "15000" ms on following nodes [ID 1356975.1]


Update 12 March 2013
As mentioned on 1480242.1 RHEL 6 the nslookup for unknown host would return 1 instead of 0. Metalink note says this was confirmed from 6.3 but wasn't the case on the tested 6.3 as it returned 0

[root@rhel6m1 ~]# uname -r
2.6.32-279.el6.x86_64 <-- (rhel-server-6.3-x86_64)

[root@rhel6m1 ~]#  nslookup unknown-not-reachable-node
Server:         1.6.9.2
Address:        1.6.9.2#53

Non-authoritative answer:
Name:   unknown-not-reachable-node.code.net
Address: 23.11.26.18

[root@rhel6m1 ~]# echo $?
0

Same was observed on 6.2

uname -r
2.6.32-220.el6.x86_64  <-- (rhel-server-6.2-x86_64)

[root@rhel6m1 ~]# nslookup unknown-not-reachable-node
Server:         11.7.9.2
Address:        11.7.9.2#53

Non-authoritative answer:
Name:   unknown-not-reachable-node.code.net
Address: 23.11.26.18

[root@rhel6m1 ~]# echo $?
0

However on 6.4 the nslookup returned 1 instead of 0 (matching the description on metalink note)

[grid@db1 grid]$ uname -r
2.6.32-358.0.1.el6.x86_64  <-- (rhel-server-6.4-x86_64)

[grid@db1 grid]$ nslookup unknown-not-reachable-node
Server:         13.20.9.5
Address:        13.20.9.5#53

** server can't find unknown-not-reachable-node: SERVFAIL

[grid@db1 grid]$  echo $?
1

Therefore pre-req would fail on RHEL 6.4 with PRVF-5637 due to the change in the return value of the nslookup. Only way to verify that pre-req is passed is through manual inspection of /etc/resolve.conf

Changing Listener and SCAN Listener Name in 11gR2 RAC

When the 11gR2 clusterware is installed the listener and scan listeners are created with default names. Default listener is named "LISTENER" while the scan listeners have the following naming convention "LISTENER_SCAN#" (# scan listener number).
At times it may be required to setup the cluster with all non-default values thus requiring to rename the default listener names. This post uses the 11gR2 RAC installation with role separation (with COST setup) and already has the listener and scan listener default ports changed.

Changing the scan listener name (read end of the post for known issue)
1. Current scan listener configuration is as follows

cat /opt/app/11.2.0/grid/network/admin/listener.ora
LISTENER=(DESCRIPTION=(ADDRESS_LIST=(ADDRESS=(PROTOCOL=IPC)(KEY=LISTENER))))            # line added by Agent
LISTENER_SCAN1=(DESCRIPTION=(ADDRESS_LIST=(ADDRESS=(PROTOCOL=IPC)(KEY=LISTENER_SCAN1))))                # line added by Agent
ENABLE_GLOBAL_DYNAMIC_ENDPOINT_LISTENER_SCAN1=ON                # line added by Agent
ENABLE_GLOBAL_DYNAMIC_ENDPOINT_LISTENER=ON              # line added by Agent

WALLET_LOCATION =
  (SOURCE =
   (METHOD = FILE)
    (METHOD_DATA =
     (DIRECTORY = /opt/app/11.2.0/grid/network/admin/cost)
    )
  )

SECURE_REGISTER_LISTENER = (IPC,TCP)
SECURE_REGISTER_LISTENER_SCAN1 = (IPC,TCPS) <---- part of COST

2. New scan listener cannot be added while there are scan listeners already configured.

srvctl config scan_listener
SCAN Listener LISTENER_SCAN1 exists. Port: TCP:9120/TCPS:1523

srvctl add scan_listener -l myscanlistener
PRCS-1028 : Single Client Access Name listeners already exist

3. To add a new scan listener remove the default scan listener first. When the new scan listener name is specified it's give only with the prefix of the scan listener name. There's no need to include the "SCAN", this will be added by the clusteware.

srvctl stop scan_listener

srvctl remove scan_listener
Remove scan listener? (y/[n]) y

srvctl add scan_listener -l myscanlistener -p TCP:9120/TCPS:1523

srvctl config scan_listener
SCAN Listener MYSCANLISTENER_SCAN1 exists. Port: TCP:9120/TCPS:1523

Listener.ora file won't be modified at this stage. Entries related to new scan listener name will be added when the scan listener is started. The old scan listener entries will still be there on the listener.ora and must be manually removed. Listener.ora files across all nodes will be updated when the scan listener is started. The COST related scan listener entries must be manually changed

cat /opt/app/11.2.0/grid/network/admin/listener.ora   # before scan listener start
LISTENER=(DESCRIPTION=(ADDRESS_LIST=(ADDRESS=(PROTOCOL=IPC)(KEY=LISTENER))))            # line added by Agent
LISTENER_SCAN1=(DESCRIPTION=(ADDRESS_LIST=(ADDRESS=(PROTOCOL=IPC)(KEY=LISTENER_SCAN1))))                # line added by Agent
ENABLE_GLOBAL_DYNAMIC_ENDPOINT_LISTENER_SCAN1=ON                # line added by Agent
ENABLE_GLOBAL_DYNAMIC_ENDPOINT_LISTENER=ON              # line added by Agent

WALLET_LOCATION =
  (SOURCE =
   (METHOD = FILE)
    (METHOD_DATA =
     (DIRECTORY = /opt/app/11.2.0/grid/network/admin/cost)
    )
  )

SECURE_REGISTER_LISTENER = (IPC,TCP)
SECURE_REGISTER_LISTENER_SCAN1 = (IPC,TCPS)

srvctl start scan_listener

srvctl config scan_listener
SCAN Listener MYSCANLISTENER_SCAN1 exists. Port: TCP:9120/TCPS:1523

srvctl status scan_listener
SCAN Listener MYSCANLISTENER_SCAN1 is enabled
SCAN listener MYSCANLISTENER_SCAN1 is running on node rhel6m1

cat /opt/app/11.2.0/grid/network/admin/listener.ora   # after scan listener start
MYSCANLISTENER_SCAN1=(DESCRIPTION=(ADDRESS_LIST=(ADDRESS=(PROTOCOL=IPC)(KEY=MYSCANLISTENER_SCAN1))))            # line added by Agent
LISTENER=(DESCRIPTION=(ADDRESS_LIST=(ADDRESS=(PROTOCOL=IPC)(KEY=LISTENER))))            # line added by Agent
LISTENER_SCAN1=(DESCRIPTION=(ADDRESS_LIST=(ADDRESS=(PROTOCOL=IPC)(KEY=LISTENER_SCAN1))))                # line added by Agent
ENABLE_GLOBAL_DYNAMIC_ENDPOINT_LISTENER_SCAN1=ON                # line added by Agent
ENABLE_GLOBAL_DYNAMIC_ENDPOINT_LISTENER=ON              # line added by Agent

WALLET_LOCATION =
  (SOURCE =
   (METHOD = FILE)
    (METHOD_DATA =
     (DIRECTORY = /opt/app/11.2.0/grid/network/admin/cost)
    )
  )

SECURE_REGISTER_LISTENER = (IPC,TCP)
SECURE_REGISTER_MYSCANLISTENER_SCAN1 = (IPC,TCPS) <--- must be changed manually

ENABLE_GLOBAL_DYNAMIC_ENDPOINT_MYSCANLISTENER_SCAN1=ON          # line added by Agent

4. Check the scan listener status

lsnrctl status MYSCANLISTENER_SCAN1

LSNRCTL for Linux: Version 11.2.0.3.0 - Production on 24-JAN-2013 10:35:24

Copyright (c) 1991, 2011, Oracle.  All rights reserved.

Connecting to (DESCRIPTION=(ADDRESS=(PROTOCOL=IPC)(KEY=MYSCANLISTENER_SCAN1)))
STATUS of the LISTENER
------------------------
Alias                     MYSCANLISTENER_SCAN1
Version                   TNSLSNR for Linux: Version 11.2.0.3.0 - Production
Start Date                24-JAN-2013 10:34:57
Uptime                    0 days 0 hr. 0 min. 27 sec
Trace Level               off
Security                  ON: Local OS Authentication
SNMP                      OFF
Listener Parameter File   /opt/app/11.2.0/grid/network/admin/listener.ora
Listener Log File         /opt/app/11.2.0/grid/log/diag/tnslsnr/rhel6m1/myscanlistener_scan1/alert/log.xml
Listening Endpoints Summary...
  (DESCRIPTION=(ADDRESS=(PROTOCOL=ipc)(KEY=MYSCANLISTENER_SCAN1)))
  (DESCRIPTION=(ADDRESS=(PROTOCOL=tcps)(HOST=192.168.0.91)(PORT=1523)))
  (DESCRIPTION=(ADDRESS=(PROTOCOL=tcp)(HOST=192.168.0.91)(PORT=9120)))
Services Summary...
Service "std11g2" has 1 instance(s).
  Instance "std11g21", status READY, has 1 handler(s) for this service...
Service "std11g2XDB" has 1 instance(s).
  Instance "std11g21", status READY, has 1 handler(s) for this service...
The command completed successfully

Changing the listener name
1. To add the new listener name on the same port as the current default listener name -s option must be used. This will skip the port check if not an error will be thrown.

srvctl status listener
Listener LISTENER is enabled
Listener LISTENER is running on node(s): rhel6m2,rhel6m1

srvctl add listener -l mylistener -p 9120   # without -s results in an error
PRCN-2061 : Failed to add listener ora.MYLISTENER.lsnr
PRCN-2065 : Port(s) 9120 are not available on the nodes given
PRCN-2067 : Port 9120 is not available across node(s) "rhel6m1-vip,rhel6m2-vip"

srvctl add listener -l mylistener -s -p 9120

srvctl config listener
Name: MYLISTENER
Network: 1, Owner: grid
Home: <CRS home>
End points: TCP:9120

Before removing the default listener modify the asm configuration with the new listener. If not asm configuration will continue to use the old listener name and will report PRCA-1032: ASM Listener LISTENER Does Not Exist when srvctl config asm is run. It's important that this is done before the default listener is dropped if not the default listener need to be added and then new listener name is specified for the asm configuration. More on this is available on 1428284.1

srvctl modify asm -l MYLISTENER
srvctl config asm
ASM home: /opt/app/11.2.0/grid
ASM listener: MYLISTENER

2. Stop the default listener and remove it from the configuration.

srvctl stop listener -l listener
srvctl remove listener -l listener

3. The new listener name related entries are not added to the listener.ora file until the listener is restarted and update will be done across all nodes of the cluster. Similar to scan listener the old entries will still remain the listener.ora file and must be manually removed. At the same time also edit the COST related listener entry.

cat /opt/app/11.2.0/grid/network/admin/listener.ora  # before listener start
MYSCANLISTENER_SCAN1=(DESCRIPTION=(ADDRESS_LIST=(ADDRESS=(PROTOCOL=IPC)(KEY=MYSCANLISTENER_SCAN1))))            # line added by Agent
LISTENER=(DESCRIPTION=(ADDRESS_LIST=(ADDRESS=(PROTOCOL=IPC)(KEY=LISTENER))))            # line added by Agent
ENABLE_GLOBAL_DYNAMIC_ENDPOINT_LISTENER=ON              # line added by Agent

WALLET_LOCATION =
  (SOURCE =
   (METHOD = FILE)
    (METHOD_DATA =
     (DIRECTORY = /opt/app/11.2.0/grid/network/admin/cost)
    )
  )

SECURE_REGISTER_LISTENER = (IPC,TCP)
SECURE_REGISTER_MYSCANLISTENER_SCAN1 = (IPC,TCPS)
ENABLE_GLOBAL_DYNAMIC_ENDPOINT_MYSCANLISTENER_SCAN1=ON          # line added by Agent

srvctl start listener -l mylistener

cat /opt/app/11.2.0/grid/network/admin/listener.ora  # after listener start
MYLISTENER=(DESCRIPTION=(ADDRESS_LIST=(ADDRESS=(PROTOCOL=IPC)(KEY=MYLISTENER))))                # line added by Agent
MYSCANLISTENER_SCAN1=(DESCRIPTION=(ADDRESS_LIST=(ADDRESS=(PROTOCOL=IPC)(KEY=MYSCANLISTENER_SCAN1))))            # line added by Agent
LISTENER=(DESCRIPTION=(ADDRESS_LIST=(ADDRESS=(PROTOCOL=IPC)(KEY=LISTENER))))            # line added by Agent
ENABLE_GLOBAL_DYNAMIC_ENDPOINT_LISTENER=ON              # line added by Agent

WALLET_LOCATION =
  (SOURCE =
   (METHOD = FILE)
    (METHOD_DATA =
     (DIRECTORY = /opt/app/11.2.0/grid/network/admin/cost)
    )
  )

SECURE_REGISTER_MYLISTENER = (IPC,TCP) <-- must be changed manually
SECURE_REGISTER_MYSCANLISTENER_SCAN1 = (IPC,TCPS)
ENABLE_GLOBAL_DYNAMIC_ENDPOINT_MYSCANLISTENER_SCAN1=ON          # line added by Agent
ENABLE_GLOBAL_DYNAMIC_ENDPOINT_MYLISTENER=ON            # line added by Agent

4. Listener status

lsnrctl status mylistener

LSNRCTL for Linux: Version 11.2.0.3.0 - Production on 24-JAN-2013 11:57:18

Copyright (c) 1991, 2011, Oracle.  All rights reserved.

Connecting to (DESCRIPTION=(ADDRESS=(PROTOCOL=IPC)(KEY=MYLISTENER)))
STATUS of the LISTENER
------------------------
Alias                     MYLISTENER
Version                   TNSLSNR for Linux: Version 11.2.0.3.0 - Production
Start Date                24-JAN-2013 11:51:49
Uptime                    0 days 0 hr. 5 min. 29 sec
Trace Level               off
Security                  ON: Local OS Authentication
SNMP                      OFF
Listener Parameter File   /opt/app/11.2.0/grid/network/admin/listener.ora
Listener Log File         /opt/app/11.2.0/grid/log/diag/tnslsnr/rhel6m2/mylistener/alert/log.xml
Listening Endpoints Summary...
  (DESCRIPTION=(ADDRESS=(PROTOCOL=ipc)(KEY=MYLISTENER)))
  (DESCRIPTION=(ADDRESS=(PROTOCOL=tcp)(HOST=192.168.0.86)(PORT=9120)))
  (DESCRIPTION=(ADDRESS=(PROTOCOL=tcp)(HOST=192.168.0.90)(PORT=9120)))
Services Summary...
Service "+ASM" has 1 instance(s).
  Instance "+ASM2", status READY, has 1 handler(s) for this service...
Service "std11g2" has 1 instance(s).
  Instance "std11g22", status READY, has 1 handler(s) for this service...
Service "std11g2XDB" has 1 instance(s).
  Instance "std11g22", status READY, has 1 handler(s) for this service...
The command completed successfully

This concludes the changing of listener and scan listener names.

Known Issues
There were no known application related issues after this listener name change. Application was able to connect (JDBC) using the scan ip. However running cluvfy comp healthcheck resulted in following error

ERROR:
PRVG-0222 : Error establishing connection to database "std11g2". Verification will be skipped for this database.
PRCQ-1000 : An error occurred while establishing connection to database with user name "cvusys" and connect descriptor:
(DESCRIPTION = (LOAD_BALANCE=on)  (ADDRESS = (PROTOCOL = TCP)(HOST = rhel6m-scan)(PORT = null)) (CONNECT_DATA =(SERVER = DEDICATED)(SERVICE_NAME = std11g2)))
IO Error: The Network Adapter could not establish the connection

It could be seen that it's trying to use the scan-ip but the port value is null. When the scan listener name is reverted back to the default value (LISTENER_SCAN1) then the cluvfy comp healthcheck continue without any errors. It cluvfy comp healthcheck depends on the default scan listener name. SR was raised regarding this. However Oracle couldn't confirm or deny that there are components depending on the scan listener name to be of the default format. Final reply for this SR was "I think you are the first person did this change. Since you have followed the steps to change the SCAN Listener using srvctl then it is fine." With a phone call Oracle confirmed the steps are indeed correct and there are no documents available (yet) detailing how to change scan listener name. For the time being the RAC setup work fine and had no adverse effect other than the cluvfy issue.

Related Post
Changing Listener and SCAN Listener Port in 11gR2 RAC

Useful metalink notes
PRCA-1032: ASM Listener LISTENER Does Not Exist On 11.2 SIHA/OHAS Configuration. [ID 1428284.1]

Parsing KFOD-00307: asmlib error!! function = [asm_open], error = [0], mesg = [Operation not permitted]

This was a new installation and storage vendor had setup the block devices as below.

ls -l /dev/eql/*
lrwxrwxrwx 1 root root 8 Jan 15 12:00 /dev/eql/clus-01 -> ../dm-16
lrwxrwxrwx 1 root root 8 Jan 15 12:00 /dev/eql/clus-02 -> ../dm-18
lrwxrwxrwx 1 root root 8 Jan 15 12:00 /dev/eql/clus-03 -> ../dm-20

(Showing only the block devices relevant to the post). This was using Dell EqualLogic PS4100x iSCSI SAN and was told to use /dev/eql/* for the database. Though asked "why the softlink and /dev/dm*" was told that's how it setup and have to use the /dev/eql/* softlink for "my work". At this point decided to use ASMLib since it was not possible to change the ownership of the softlink and ASMLib was able to use root owned disks without a problem. The Oracle documents that have read (605828.1,470913.1,602952.1) said not to use /dev/dm* so didn't want to directly use them and conveyed this to storage vendor before proceeding.
Installed and configured ASMLib

/usr/sbin/oracleasm configure
ORACLEASM_ENABLED=true
ORACLEASM_UID=grid
ORACLEASM_GID=asmadmin
ORACLEASM_SCANBOOT=true
ORACLEASM_SCANORDER="dm"
ORACLEASM_SCANEXCLUDE="sd"

and created the ASM disk that will be used in theASM diskgroup used for storing clusterware files using oracleasm

/etc/init.d/oracleasm listdisks
CLUS01
CLUS02
CLUS03

Went ahead with the installation and when at the ASM creation page, not ASM disks were visible when the discovery string ORCL:* was set. On the installer log following error message could be seen

INFO: ... discoveryString = ORCL:*
INFO: Executing [/tmp/OraInstall2013-01-10_02-28-18PM/ext/bin/kfod, nohdr=true, verbose=true, disks=all, status=true, op=disks, asm_diskstring='ORCL:*']
 with environment variables {XFILESEARCHPATH=/usr/dt/app-defaults/%L/Dt, LANG=en_GB.UTF-8, DISPLAY=:1, PWD=/home/grid/grid, _=/home/grid/grid/install/.oui, GS_LIB=/home/grid/.fonts, KDEDIR=/usr, KDE_MULTIHEAD=false, NLSPATH=/usr/dt/lib/nls/msg/%L/%N.cat, USER=grid, CLASSPATH=, XMODIFIERS=@im=none, G_BROKEN_FILENAMES=1, SSH_AGENT_PID=8475, COLORTERM=, HOME=/home/grid, HOSTNAME=RCL-DB-02, SESSION_MANAGER=local/RCL-DB-02:/tmp/.ICE-unix/8600, LESSOPEN=|/usr/bin/lesspipe.sh %s, LD_LIBRARY_PATH=/tmp/OraInstall2013-01-10_02-28-18PM/ext/bin/../lib, WINDOWID=4194311, ORACLE_HOME=/tmp/OraInstall2013-01-10_02-28-18PM/ext/bin, LOGNAME=grid, SSH_ASKPASS=/usr/libexec/openssh/gnome-ssh-askpass, SHELL=/bin/bash, SSH_AUTH_SOCK=/tmp/ssh-GSFGtX8450/agent.8450, CVS_RSH=ssh, DBUS_SESSION_BUS_ADDRESS=unix:abstract=/tmp/dbus-KbHCyuMD9K,guid=1f484b25202997262dd3e40050eecda6, INPUTRC=/etc/inputrc, SHLVL=5, KDE_IS_PRELINKED=1, KDE_FULL_SESSION=true, HISTTIMEFORMAT=%b %e - %H:%M:%S , MAIL=/var/spool/mail/grid, KDE_NO_IPV6=1, VNCDESKTOP=RCL-DB-02:1 (grid), KONSOLE_DCOP=DCOPRef(konsole-8643,konsole), JAVA_HOME=, LS_COLORS=no=00:fi=00:di=00;34:ln=00;36:pi=40;33:so=00;35:bd=40;33;01:cd=40;33;01:or=01;05;37;41:mi=01;05;37;41:ex=00;32:*.cmd=00;32:*.exe=00;32:*.com=00;32:*.btm=00;32:*.bat=00;32:*.sh=00;32:*.csh=00;32:*.tar=00;31:*.tgz=00;31:*.arj=00;31:*.taz=00;31:*.lzh=00;31:*.zip=00;31:*.z=00;31:*.Z=00;31:*.gz=00;31:*.bz2=00;31:*.bz=00;31:*.tz=00;31:*.rpm=00;31:*.cpio=00;31:*.jpg=00;35:*.gif=00;35:*.bmp=00;35:*.xbm=00;35:*.xpm=00;35:*.png=00;35:*.tif=00;35:, KONSOLE_DCOP_SESSION=DCOPRef(konsole-8643,session-1), LC_ALL=, HISTSIZE=1000, XDG_CONFIG_DIRS=/etc/kde/xdg:/etc/xdg, TERM=xterm, PATH=/usr/bin:/usr/ccs/bin:/usr/kerberos/bin:/usr/local/bin:/bin:/usr/bin:/home/grid/bin:/home/grid/grid/install}
INFO: Starting Output Reader Threads for process /tmp/OraInstall2013-01-10_02-28-18PM/ext/bin/kfod
INFO: Parsing KFOD-00307: asmlib error!! function = [asm_close], error = [0], mesg = [Invalid argument]
INFO: Parsing KFOD-00307: asmlib error!! function = [asm_open], error = [0], mesg = [Operation not permitted]
INFO: The process /tmp/OraInstall2013-01-10_02-28-18PM/ext/bin/kfod exited with code 0
INFO: Parsing KFOD-00307: asmlib error!! function = [asm_close], error = [0], mesg = [Invalid argument]
INFO: Waiting for output processor threads to exit.
INFO: Parsing KFOD-00307: asmlib error!! function = [asm_open], error = [0], mesg = [Operation not permitted]
INFO: Parsing KFOD-00307: asmlib error!! function = [asm_close], error = [0], mesg = [Invalid argument]
INFO: Parsing KFOD-00307: asmlib error!! function = [asm_open], error = [0], mesg = [Operation not permitted]
INFO: Output processor threads exited.

Closest metalink note that could be found was ORA-15186: ASMLIB error function = [asm_open], error = [1], mesg = [Operation not permitted] [ID 398622.1] but even that was not an exact match in terms of the error code and of no help as this was a new installation.
However it was confirmed that ASM disk are there by listing the /dev/oracleasm/disk and using that path in the discovery string.

It's at this point the tinkering work began. First place was the scan order. Try different combinations including eql/clus-*, eql/*, dm-* , dm* but still didn't discover any ASM disk using ORCL:*.
But when the exact device was specified in the scan order it was possible to get the ASM disks using the ORCL:*. It took a while to get the head around this

ORACLEASM_SCANORDER="dm" <---- DOES NOT WORK NO ASM DISK DISCOVERED
ORACLEASM_SCANORDER="dm-16 dm-18 dm-20" <--- WORK FINE SEE THE ASM DISK

At this point raise an SR. Understandably Oracle's initial effort revolved around checking if ASMLib setup is correct and it was confirmed to be setup correct and correct value for ORACLEASM_SCANORDER was dm. But still couldn't explain why dm doesn't work and dm-16 work (work = ASM disk visible in the installer).
After sometime an Oracle engineer spot the issue which was believed to do with the multipath setup used by the storage vendor and might even explain the oddity mentioned above. Storage vendor wasn't using the Linux multipathing software but a proprietary one that came with the SAN. Similar to SCANORDER and SCANEXCLUDE it had eql and ignore-eql.

ls -l /dev/ignore_eql/
lrwxrwxrwx 1 root root 8 Jan 18 19:32 0-1cb196-32b50460e-993001e8c7650e17_a -> ../dm-15
lrwxrwxrwx 1 root root 8 Jan 18 19:32 0-1cb196-33f50460e-197001e8c7950e17_a -> ../dm-17
lrwxrwxrwx 1 root root 8 Jan 18 19:32 0-1cb196-34f50460e-8a7001e8c7c50e17_a -> ../dm-19
lrwxrwxrwx 1 root root 7 Jan 18 19:32 0-1cb196-3a950460e-f17001e8c8550e17_a -> ../dm-9
lrwxrwxrwx 1 root root 8 Jan 18 19:32 0-1cb196-3ba50460e-ffb001e8c8850e17_a -> ../dm-13
lrwxrwxrwx 1 root root 8 Jan 18 19:32 0-1cb196-3c950460e-ce7001e8c8b50e17_a -> ../dm-11
lrwxrwxrwx 1 root root 7 Jan 18 19:32 0-1cb196-3e050460e-0a1001e8c8e50e17_a -> ../dm-5
lrwxrwxrwx 1 root root 7 Jan 18 19:32 0-1cb196-3f050460e-833001e8c9150e17_a -> ../dm-7
lrwxrwxrwx 1 root root 8 Jan 18 19:32 0-1cb196-3ff50460e-a3b001e8c9450e17_a -> ../dm-21

ls -l /dev/eql/
lrwxrwxrwx 1 root root 8 Jan 18 19:32 clus-01 -> ../dm-16
lrwxrwxrwx 1 root root 8 Jan 18 19:32 clus-02 -> ../dm-18
lrwxrwxrwx 1 root root 8 Jan 18 19:32 clus-03 -> ../dm-20

However looking at the /dev/mapper could see that there are multiple labels for the same device(only one set highlighted)

ls -l /dev/mapper/
brw-rw---- 1 root disk 253, 15 Jan 18 19:32 eql-0-1cb196-32b50460e-993001e8c7650e17_a <-- dm-15
brw-rw---- 1 root disk 253, 16 Jan 18 19:32 eql-0-1cb196-32b50460e-993001e8c7650e17-clus-01 <-- dm-16
brw-rw---- 1 root disk 253, 17 Jan 18 19:32 eql-0-1cb196-33f50460e-197001e8c7950e17_a
brw-rw---- 1 root disk 253, 18 Jan 18 19:32 eql-0-1cb196-33f50460e-197001e8c7950e17-clus-02
brw-rw---- 1 root disk 253, 19 Jan 18 19:32 eql-0-1cb196-34f50460e-8a7001e8c7c50e17_a
brw-rw---- 1 root disk 253, 20 Jan 18 19:32 eql-0-1cb196-34f50460e-8a7001e8c7c50e17-clus-03

and on the multipath list only one of these labels from each set appeared (only one highlighted)

 multipath -ll
eql-0-1cb196-34f50460e-8a7001e8c7c50e17_a () dm-19 EQLOGIC,100E-00
[size=1.0G][features=1 queue_if_no_path][hwhandler=0][rw]
\_ round-robin 0 [prio=2][active]
 \_ 6:0:0:0 sdf 8:80  [active][ready]
 \_ 14:0:0:0 sdm 8:192 [active][ready]
eql-0-1cb196-33f50460e-197001e8c7950e17_a () dm-17 EQLOGIC,100E-00
[size=1.0G][features=1 queue_if_no_path][hwhandler=0][rw]
\_ round-robin 0 [prio=2][active]
 \_ 1:0:0:0 sdj 8:144 [active][ready]
 \_ 15:0:0:0 sdn 8:208 [active][ready]
eql-0-1cb196-32b50460e-993001e8c7650e17_a () dm-15 EQLOGIC,100E-00
[size=1.0G][features=1 queue_if_no_path][hwhandler=0][rw]
\_ round-robin 0 [prio=2][active]
 \_ 16:0:0:0 sdo 8:224 [active][ready]
 \_ 17:0:0:0 sdp 8:240 [active][ready]

But the ASM is seeing all these paths.(only one set highlighted)

/etc/init.d/oracleasm querydisk -p CLUS01
Disk "CLUS01" is a valid ASM disk
/dev/sdj: LABEL="CLUS01" TYPE="oracleasm"
/dev/mapper/eql-0-1cb196-32b50460e-993001e8c7650e17_a: LABEL="CLUS01" TYPE="oracleasm" <-- dm-15
/dev/mapper/eql-0-1cb196-32b50460e-993001e8c7650e17-clus-01: LABEL="CLUS01" TYPE="oracleasm" <-- dm-16
/dev/sdp: LABEL="CLUS01" TYPE="oracleasm"

/etc/init.d/oracleasm querydisk -p CLUS02
Disk "CLUS02" is a valid ASM disk
/dev/sdf: LABEL="CLUS02" TYPE="oracleasm"
/dev/mapper/eql-0-1cb196-33f50460e-197001e8c7950e17_a: LABEL="CLUS02" TYPE="oracleasm"
/dev/mapper/eql-0-1cb196-33f50460e-197001e8c7950e17-clus-02: LABEL="CLUS02" TYPE="oracleasm"
/dev/sdo: LABEL="CLUS02" TYPE="oracleasm"

/etc/init.d/oracleasm querydisk -p CLUS03
Disk "CLUS03" is a valid ASM disk
/dev/mapper/eql-0-1cb196-34f50460e-8a7001e8c7c50e17_a: LABEL="CLUS03" TYPE="oracleasm"
/dev/mapper/eql-0-1cb196-34f50460e-8a7001e8c7c50e17-clus-03: LABEL="CLUS03" TYPE="oracleasm"
/dev/sdm: LABEL="CLUS03" TYPE="oracleasm"
/dev/sdn: LABEL="CLUS03" TYPE="oracleasm"

It was thought that this could explain why dm in the scan order would not discover any ASM disks while dm-16 (and others) could discover them.
Unfortunately that's where this post ends. Going to the storage vendor with this information didn't yield anything. "That's how it is setup" was repeated and due the time critical nature of the work decided to drop using ASMLib altogether and decided to go for block devices (events on this post happened before the post on the link) after letting the storage vendor know what Oracle says about using /dev/dm*.

Useful metalink notes
New ASMLib / oracleasm Disk Gets "header_status=Unknown" - Cannot be Added to ASM Diskgroup [ID 391136.1]
ORA-15186: ASMLIB error function = [asm_open], error = [1], mesg = [Operation not permitted] [ID 398622.1]