Showing posts with label listener. Show all posts
Showing posts with label listener. Show all posts

Saturday, November 20, 2021

Using UR=A to Connect to Databases in Nomount Mode

Time to time there are situation where connecting to a database in nomount mode is needed. Most noteably is the data guard standby creation. When a database is started in nomount mode
startup nomount;
the service on the listener would have a blocke status.
Service "devcdb" has 1 instance(s).
  Instance "devcdb", status BLOCKED, has 1 handler(s) for this service...
Any attempt to connect to the instance using service name will fail.
sqlplus sys@devcdb as sysdba

SQL*Plus: Release 19.0.0.0.0 - Production on Thu Oct 14 20:27:16 2021
Version 19.12.0.0.0

Copyright (c) 1982, 2021, Oracle.  All rights reserved.

ERROR:
ORA-12528: TNS:listener: all appropriate instances are blocking new connections
The TNS entry used is shown below.
DEVCDB =
  (DESCRIPTION =
    (ADDRESS = (PROTOCOL = TCP)(HOST = test-4)(PORT = 1521))
    (CONNECT_DATA =
      (SERVER = DEDICATED)
      (SERVICE_NAME = devcdb)
    )
  )
One solution is to create a static listener entries as done when creating data guard configurations.



Other options is to use UR=A in the TNS entry. With the use of UR=A in TNS entry there's no need to create a static listener entry. Dynamic listener would allow connection without any issue.
DEVCDBUR =
  (DESCRIPTION =
    (ADDRESS = (PROTOCOL = TCP)(HOST = test-4-254.company.net)(PORT = 1521))
    (CONNECT_DATA =
      (SERVER = DEDICATED)
      (SERVICE_NAME = devcdb)
      (UR = A)
    )
  )
  
sqlplus sys@devcdbur as sysdba

SQL*Plus: Release 19.0.0.0.0 - Production on Thu Oct 14 20:47:13 2021
Version 19.12.0.0.0

Copyright (c) 1982, 2021, Oracle.  All rights reserved.

Enter password:

Connected to:
Oracle Database 19c Enterprise Edition Release 19.0.0.0.0 - Production
Version 19.12.0.0.0

SQL>
Even during normal DG operation use of UR=A in TNS entries used for DG traffic could be useful. During switchover primary could attmept to connect to standby which is in the nomount state (progressing to mount or read only). Without UR=A the primary connection would fail (and succeed later after reattempt, once standby has transitioned out of nomount). To reduce the false alerts occuring in these situation UR=A could be used.
For example without UR=A entry it took nearly 15s for DG broker to connect to the new standby during a switchover (output on a 19.13 system).
DGMGRL> switchover to dgtest
Performing switchover NOW, please wait...
Operation requires a connection to database "dgtest"
Connecting ...
Connected to "dgtest"
Connected as SYSDBA.
New primary database "dgtest" is opening...
Oracle Clusterware is restarting database "dgtest3" ...
[W000 2021-10-25T13:49:46.227+00:00] Failed to attach to dgtest3tns.
[W000 2021-10-25T13:49:47.231+00:00] Failed to attach to dgtest3tns.
[W000 2021-10-25T13:49:48.234+00:00] Failed to attach to dgtest3tns.
[W000 2021-10-25T13:49:49.237+00:00] Failed to attach to dgtest3tns.
[W000 2021-10-25T13:49:50.240+00:00] Failed to attach to dgtest3tns.
[W000 2021-10-25T13:49:51.242+00:00] Failed to attach to dgtest3tns.
[W000 2021-10-25T13:49:52.245+00:00] Failed to attach to dgtest3tns.
[W000 2021-10-25T13:49:53.248+00:00] Failed to attach to dgtest3tns.
[W000 2021-10-25T13:49:54.250+00:00] Failed to attach to dgtest3tns.
[W000 2021-10-25T13:49:55.252+00:00] Failed to attach to dgtest3tns.
[W000 2021-10-25T13:49:56.255+00:00] Failed to attach to dgtest3tns.
[W000 2021-10-25T13:49:57.257+00:00] Failed to attach to dgtest3tns.
[W000 2021-10-25T13:49:58.260+00:00] Failed to attach to dgtest3tns.
[W000 2021-10-25T13:49:59.262+00:00] Failed to attach to dgtest3tns.
[W000 2021-10-25T13:50:00.265+00:00] Failed to attach to dgtest3tns.
[W000 2021-10-25T13:50:01.274+00:00] Failed to attach to dgtest3tns.
Connected to "dgtest3"
Connected to "dgtest3"
Switchover succeeded, new primary is "dgtest"
However, with UR=A in the TNS entry it was able to connect sooner.
DGMGRL> switchover to dgtest
Performing switchover NOW, please wait...
Operation requires a connection to database "dgtest"
Connecting ...
Connected to "dgtest"
Connected as SYSDBA.
New primary database "dgtest" is opening...
Oracle Clusterware is restarting database "dgtest3" ...
[W000 2021-10-25T13:55:24.705+00:00] Failed to attach to dgtest3tns.
[W000 2021-10-25T13:55:25.708+00:00] Failed to attach to dgtest3tns.
[W000 2021-10-25T13:55:26.710+00:00] Failed to attach to dgtest3tns.
[W000 2021-10-25T13:55:27.714+00:00] Failed to attach to dgtest3tns.
[W000 2021-10-25T13:55:28.716+00:00] Failed to attach to dgtest3tns.
[W000 2021-10-25T13:55:29.718+00:00] Failed to attach to dgtest3tns.
[W000 2021-10-25T13:55:30.722+00:00] Failed to attach to dgtest3tns.
Connected to "dgtest3"
Connected to "dgtest3"
Switchover succeeded, new primary is "dgtest"
UR=A is used in the TNS entries created in DBCS VM DBs that also has data guard enabled.

Related Metalink note
Connections to NOMOUNT/MOUNTED or RESTRICTED Databases Fail [ID 362656.1]

Update 28 January 2022
Entries similar to "[W000 2021-10-25T13:55:24.705+00:00] Failed to attach to dgtest3tns" are due to bug 30870248. Apply the patch if available for the RU.

Monday, April 18, 2016

Valid Node Checking For Registration (VNCR)

Valid Node Checking for Registration (VNCR) allows restriction of instance registration to come only from know servers. This has replaced the use of COST (setting up COST for 11gR1 SE and 11gR2 SE) for protecting against TNS Listener Poison Attack. COST could be still used if it is used for reasons other than listener restriction (for more read 1340831.1).
VNCR is available on 11.2.0.4 and 12c so if the DB is 11.2.0.3 then COST is still the only option available against this attack. Below is an example of setting VNCR on a single instance database. For RAC with SCAN listeners refer the MOS listed at the end of the post.
The listener runs on server with IP 192.168.0.66 (hpc1.domain.net). Currently no DB is registered.
lsnrctl status

LSNRCTL for Linux: Version 11.2.0.4.0 - Production on 22-FEB-2016 13:06:44

Copyright (c) 1991, 2013, Oracle.  All rights reserved.

Connecting to (DESCRIPTION=(ADDRESS=(PROTOCOL=TCP)(HOST=hpc1.domain.net)(PORT=1521)))
STATUS of the LISTENER
------------------------
Alias                     LISTENER
Version                   TNSLSNR for Linux: Version 11.2.0.4.0 - Production
Start Date                22-FEB-2016 13:06:41
Uptime                    0 days 0 hr. 0 min. 3 sec
Trace Level               off
Security                  ON: Local OS Authentication
SNMP                      OFF
Listener Parameter File   /opt/app/oracle/product/11.2.0/std4/network/admin/listener.ora
Listener Log File         /opt/app/oracle/diag/tnslsnr/hpc1/listener/alert/log.xml
Listening Endpoints Summary...
  (DESCRIPTION=(ADDRESS=(PROTOCOL=tcp)(HOST=hpc1.domain.net)(PORT=1521)))
The listener supports no services
The command completed successfully
The DB (something to register with this listener) runs on server with IP 192.168.0.99. To register with the listener on 192.168.0.66 add a remote listener entry on tnsnames.ora and register the database
cat tnsnames.ora
listener_name =
  (DESCRIPTION=
    (ADDRESS=(PROTOCOL=tcp)(HOST=192.168.0.66)(PORT=1521))
  )

SQL> alter system set remote_listener='listener_name' scope=both;
System altered.

SQL> alter system register;
System altered.
Check the listener status
lsnrctl status

LSNRCTL for Linux: Version 11.2.0.4.0 - Production on 22-FEB-2016 13:08:51

Copyright (c) 1991, 2013, Oracle.  All rights reserved.

Connecting to (DESCRIPTION=(ADDRESS=(PROTOCOL=TCP)(HOST=hpc1.domain.net)(PORT=1521)))
STATUS of the LISTENER
------------------------
Alias                     LISTENER
Version                   TNSLSNR for Linux: Version 11.2.0.4.0 - Production
Start Date                22-FEB-2016 13:06:41
Uptime                    0 days 0 hr. 2 min. 10 sec
Trace Level               off
Security                  ON: Local OS Authentication
SNMP                      OFF
Listener Parameter File   /opt/app/oracle/product/11.2.0/std4/network/admin/listener.ora
Listener Log File         /opt/app/oracle/diag/tnslsnr/hpc1/listener/alert/log.xml
Listening Endpoints Summary...
  (DESCRIPTION=(ADDRESS=(PROTOCOL=tcp)(HOST=hpc1.domain.net)(PORT=1521)))
Services Summary...
Service "fgacdb" has 1 instance(s).
  Instance "fgacdb", status READY, has 1 handler(s) for this service...
Service "fgacdbXDB" has 1 instance(s).
  Instance "fgacdb", status READY, has 1 handler(s) for this service...
The command completed successfully
Above shows that remote database has registered with the listener.



In order to prevent this type of remote registrations enable valid node checking on the listener. To enable VNCR on 11.2.0.4 add following entry to listener.ora file In this case the listener name is "LISTENER".
VALID_NODE_CHECKING_REGISTRATION_LISTENER=1
Reload the listener and check the status. The remote listener registration is blocked.
$ lsnrctl reload
$ lsnrctl status

LSNRCTL for Linux: Version 11.2.0.4.0 - Production on 22-FEB-2016 13:24:17

Copyright (c) 1991, 2013, Oracle.  All rights reserved.

Connecting to (DESCRIPTION=(ADDRESS=(PROTOCOL=TCP)(HOST=hpc1.domain.net)(PORT=1521)))
STATUS of the LISTENER
------------------------
Alias                     LISTENER
Version                   TNSLSNR for Linux: Version 11.2.0.4.0 - Production
Start Date                22-FEB-2016 13:06:41
Uptime                    0 days 0 hr. 17 min. 35 sec
Trace Level               off
Security                  ON: Local OS Authentication
SNMP                      OFF
Listener Parameter File   /opt/app/oracle/product/11.2.0/std4/network/admin/listener.ora
Listener Log File         /opt/app/oracle/diag/tnslsnr/hpc1/listener/alert/log.xml
Listening Endpoints Summary...
  (DESCRIPTION=(ADDRESS=(PROTOCOL=tcp)(HOST=hpc1.domain.net)(PORT=1521)))
The listener supports no services
Any attempt to register with the listener is blocked and the listener.log will have an entry similar to following
Listener(VNCR option 1) rejected Registration request from destination 192.168.0.99
22-FEB-2016 13:31:47 * service_register_NSGR * 1182
TNS-01182: Listener rejected registration of service ""
On 12c is VNCR is enabled by default and any attempt for remote registration is rejected and listener log will have an entry as above.

Useful metalink notes
Valid Node Checking For Registration (VNCR) [ID 1600630.1]
How to Enable VNCR on RAC Database to Register only Local Instances [ID 1914282.1]

Related Posts
Implementing Class of Secure Transport (COST) to Restrict Instance Registration in Oracle 11gR2 SE RAC (Solution mentioned in Oracle Security Alert for CVE-2012-1675)
Using Class of Secure Transport (COST) to Restrict Instance Registration in Oracle SE RAC Fails (Solution mentioned in Oracle Security Alert for CVE-2012-1675)

Friday, February 1, 2013

Changing Listener and SCAN Listener Name in 11gR2 RAC

When the 11gR2 clusterware is installed the listener and scan listeners are created with default names. Default listener is named "LISTENER" while the scan listeners have the following naming convention "LISTENER_SCAN#" (# scan listener number).
At times it may be required to setup the cluster with all non-default values thus requiring to rename the default listener names. This post uses the 11gR2 RAC installation with role separation (with COST setup) and already has the listener and scan listener default ports changed.

Changing the scan listener name (read end of the post for known issue)
1. Current scan listener configuration is as follows
cat /opt/app/11.2.0/grid/network/admin/listener.ora
LISTENER=(DESCRIPTION=(ADDRESS_LIST=(ADDRESS=(PROTOCOL=IPC)(KEY=LISTENER))))            # line added by Agent
LISTENER_SCAN1=(DESCRIPTION=(ADDRESS_LIST=(ADDRESS=(PROTOCOL=IPC)(KEY=LISTENER_SCAN1))))                # line added by Agent
ENABLE_GLOBAL_DYNAMIC_ENDPOINT_LISTENER_SCAN1=ON                # line added by Agent
ENABLE_GLOBAL_DYNAMIC_ENDPOINT_LISTENER=ON              # line added by Agent

WALLET_LOCATION =
  (SOURCE =
   (METHOD = FILE)
    (METHOD_DATA =
     (DIRECTORY = /opt/app/11.2.0/grid/network/admin/cost)
    )
  )

SECURE_REGISTER_LISTENER = (IPC,TCP)
SECURE_REGISTER_LISTENER_SCAN1 = (IPC,TCPS) <---- part of COST
2. New scan listener cannot be added while there are scan listeners already configured.
srvctl config scan_listener
SCAN Listener LISTENER_SCAN1 exists. Port: TCP:9120/TCPS:1523

srvctl add scan_listener -l myscanlistener
PRCS-1028 : Single Client Access Name listeners already exist
3. To add a new scan listener remove the default scan listener first. When the new scan listener name is specified it's give only with the prefix of the scan listener name. There's no need to include the "SCAN", this will be added by the clusteware.
srvctl stop scan_listener

srvctl remove scan_listener
Remove scan listener? (y/[n]) y

srvctl add scan_listener -l myscanlistener -p TCP:9120/TCPS:1523

srvctl config scan_listener
SCAN Listener MYSCANLISTENER_SCAN1 exists. Port: TCP:9120/TCPS:1523
Listener.ora file won't be modified at this stage. Entries related to new scan listener name will be added when the scan listener is started. The old scan listener entries will still be there on the listener.ora and must be manually removed. Listener.ora files across all nodes will be updated when the scan listener is started. The COST related scan listener entries must be manually changed
cat /opt/app/11.2.0/grid/network/admin/listener.ora   # before scan listener start
LISTENER=(DESCRIPTION=(ADDRESS_LIST=(ADDRESS=(PROTOCOL=IPC)(KEY=LISTENER))))            # line added by Agent
LISTENER_SCAN1=(DESCRIPTION=(ADDRESS_LIST=(ADDRESS=(PROTOCOL=IPC)(KEY=LISTENER_SCAN1))))                # line added by Agent
ENABLE_GLOBAL_DYNAMIC_ENDPOINT_LISTENER_SCAN1=ON                # line added by Agent
ENABLE_GLOBAL_DYNAMIC_ENDPOINT_LISTENER=ON              # line added by Agent

WALLET_LOCATION =
  (SOURCE =
   (METHOD = FILE)
    (METHOD_DATA =
     (DIRECTORY = /opt/app/11.2.0/grid/network/admin/cost)
    )
  )

SECURE_REGISTER_LISTENER = (IPC,TCP)
SECURE_REGISTER_LISTENER_SCAN1 = (IPC,TCPS)

srvctl start scan_listener

srvctl config scan_listener
SCAN Listener MYSCANLISTENER_SCAN1 exists. Port: TCP:9120/TCPS:1523

srvctl status scan_listener
SCAN Listener MYSCANLISTENER_SCAN1 is enabled
SCAN listener MYSCANLISTENER_SCAN1 is running on node rhel6m1

cat /opt/app/11.2.0/grid/network/admin/listener.ora   # after scan listener start
MYSCANLISTENER_SCAN1=(DESCRIPTION=(ADDRESS_LIST=(ADDRESS=(PROTOCOL=IPC)(KEY=MYSCANLISTENER_SCAN1))))            # line added by Agent
LISTENER=(DESCRIPTION=(ADDRESS_LIST=(ADDRESS=(PROTOCOL=IPC)(KEY=LISTENER))))            # line added by Agent
LISTENER_SCAN1=(DESCRIPTION=(ADDRESS_LIST=(ADDRESS=(PROTOCOL=IPC)(KEY=LISTENER_SCAN1))))                # line added by Agent
ENABLE_GLOBAL_DYNAMIC_ENDPOINT_LISTENER_SCAN1=ON                # line added by Agent
ENABLE_GLOBAL_DYNAMIC_ENDPOINT_LISTENER=ON              # line added by Agent

WALLET_LOCATION =
  (SOURCE =
   (METHOD = FILE)
    (METHOD_DATA =
     (DIRECTORY = /opt/app/11.2.0/grid/network/admin/cost)
    )
  )

SECURE_REGISTER_LISTENER = (IPC,TCP)
SECURE_REGISTER_MYSCANLISTENER_SCAN1 = (IPC,TCPS) <--- must be changed manually

ENABLE_GLOBAL_DYNAMIC_ENDPOINT_MYSCANLISTENER_SCAN1=ON          # line added by Agent
4. Check the scan listener status
lsnrctl status MYSCANLISTENER_SCAN1

LSNRCTL for Linux: Version 11.2.0.3.0 - Production on 24-JAN-2013 10:35:24

Copyright (c) 1991, 2011, Oracle.  All rights reserved.

Connecting to (DESCRIPTION=(ADDRESS=(PROTOCOL=IPC)(KEY=MYSCANLISTENER_SCAN1)))
STATUS of the LISTENER
------------------------
Alias                     MYSCANLISTENER_SCAN1
Version                   TNSLSNR for Linux: Version 11.2.0.3.0 - Production
Start Date                24-JAN-2013 10:34:57
Uptime                    0 days 0 hr. 0 min. 27 sec
Trace Level               off
Security                  ON: Local OS Authentication
SNMP                      OFF
Listener Parameter File   /opt/app/11.2.0/grid/network/admin/listener.ora
Listener Log File         /opt/app/11.2.0/grid/log/diag/tnslsnr/rhel6m1/myscanlistener_scan1/alert/log.xml
Listening Endpoints Summary...
  (DESCRIPTION=(ADDRESS=(PROTOCOL=ipc)(KEY=MYSCANLISTENER_SCAN1)))
  (DESCRIPTION=(ADDRESS=(PROTOCOL=tcps)(HOST=192.168.0.91)(PORT=1523)))
  (DESCRIPTION=(ADDRESS=(PROTOCOL=tcp)(HOST=192.168.0.91)(PORT=9120)))
Services Summary...
Service "std11g2" has 1 instance(s).
  Instance "std11g21", status READY, has 1 handler(s) for this service...
Service "std11g2XDB" has 1 instance(s).
  Instance "std11g21", status READY, has 1 handler(s) for this service...
The command completed successfully



Changing the listener name
1. To add the new listener name on the same port as the current default listener name -s option must be used. This will skip the port check if not an error will be thrown.
srvctl status listener
Listener LISTENER is enabled
Listener LISTENER is running on node(s): rhel6m2,rhel6m1

srvctl add listener -l mylistener -p 9120   # without -s results in an error
PRCN-2061 : Failed to add listener ora.MYLISTENER.lsnr
PRCN-2065 : Port(s) 9120 are not available on the nodes given
PRCN-2067 : Port 9120 is not available across node(s) "rhel6m1-vip,rhel6m2-vip"

srvctl add listener -l mylistener -s -p 9120

srvctl config listener
Name: MYLISTENER
Network: 1, Owner: grid
Home: <CRS home>
End points: TCP:9120
Before removing the default listener modify the asm configuration with the new listener. If not asm configuration will continue to use the old listener name and will report PRCA-1032: ASM Listener LISTENER Does Not Exist when srvctl config asm is run. It's important that this is done before the default listener is dropped if not the default listener need to be added and then new listener name is specified for the asm configuration. More on this is available on 1428284.1
srvctl modify asm -l MYLISTENER
srvctl config asm
ASM home: /opt/app/11.2.0/grid
ASM listener: MYLISTENER
2. Stop the default listener and remove it from the configuration.
srvctl stop listener -l listener
srvctl remove listener -l listener
3. The new listener name related entries are not added to the listener.ora file until the listener is restarted and update will be done across all nodes of the cluster. Similar to scan listener the old entries will still remain the listener.ora file and must be manually removed. At the same time also edit the COST related listener entry.
cat /opt/app/11.2.0/grid/network/admin/listener.ora  # before listener start
MYSCANLISTENER_SCAN1=(DESCRIPTION=(ADDRESS_LIST=(ADDRESS=(PROTOCOL=IPC)(KEY=MYSCANLISTENER_SCAN1))))            # line added by Agent
LISTENER=(DESCRIPTION=(ADDRESS_LIST=(ADDRESS=(PROTOCOL=IPC)(KEY=LISTENER))))            # line added by Agent
ENABLE_GLOBAL_DYNAMIC_ENDPOINT_LISTENER=ON              # line added by Agent

WALLET_LOCATION =
  (SOURCE =
   (METHOD = FILE)
    (METHOD_DATA =
     (DIRECTORY = /opt/app/11.2.0/grid/network/admin/cost)
    )
  )

SECURE_REGISTER_LISTENER = (IPC,TCP)
SECURE_REGISTER_MYSCANLISTENER_SCAN1 = (IPC,TCPS)
ENABLE_GLOBAL_DYNAMIC_ENDPOINT_MYSCANLISTENER_SCAN1=ON          # line added by Agent

srvctl start listener -l mylistener

cat /opt/app/11.2.0/grid/network/admin/listener.ora  # after listener start
MYLISTENER=(DESCRIPTION=(ADDRESS_LIST=(ADDRESS=(PROTOCOL=IPC)(KEY=MYLISTENER))))                # line added by Agent
MYSCANLISTENER_SCAN1=(DESCRIPTION=(ADDRESS_LIST=(ADDRESS=(PROTOCOL=IPC)(KEY=MYSCANLISTENER_SCAN1))))            # line added by Agent
LISTENER=(DESCRIPTION=(ADDRESS_LIST=(ADDRESS=(PROTOCOL=IPC)(KEY=LISTENER))))            # line added by Agent
ENABLE_GLOBAL_DYNAMIC_ENDPOINT_LISTENER=ON              # line added by Agent

WALLET_LOCATION =
  (SOURCE =
   (METHOD = FILE)
    (METHOD_DATA =
     (DIRECTORY = /opt/app/11.2.0/grid/network/admin/cost)
    )
  )

SECURE_REGISTER_MYLISTENER = (IPC,TCP) <-- must be changed manually
SECURE_REGISTER_MYSCANLISTENER_SCAN1 = (IPC,TCPS)
ENABLE_GLOBAL_DYNAMIC_ENDPOINT_MYSCANLISTENER_SCAN1=ON          # line added by Agent
ENABLE_GLOBAL_DYNAMIC_ENDPOINT_MYLISTENER=ON            # line added by Agent
4. Listener status
lsnrctl status mylistener

LSNRCTL for Linux: Version 11.2.0.3.0 - Production on 24-JAN-2013 11:57:18

Copyright (c) 1991, 2011, Oracle.  All rights reserved.

Connecting to (DESCRIPTION=(ADDRESS=(PROTOCOL=IPC)(KEY=MYLISTENER)))
STATUS of the LISTENER
------------------------
Alias                     MYLISTENER
Version                   TNSLSNR for Linux: Version 11.2.0.3.0 - Production
Start Date                24-JAN-2013 11:51:49
Uptime                    0 days 0 hr. 5 min. 29 sec
Trace Level               off
Security                  ON: Local OS Authentication
SNMP                      OFF
Listener Parameter File   /opt/app/11.2.0/grid/network/admin/listener.ora
Listener Log File         /opt/app/11.2.0/grid/log/diag/tnslsnr/rhel6m2/mylistener/alert/log.xml
Listening Endpoints Summary...
  (DESCRIPTION=(ADDRESS=(PROTOCOL=ipc)(KEY=MYLISTENER)))
  (DESCRIPTION=(ADDRESS=(PROTOCOL=tcp)(HOST=192.168.0.86)(PORT=9120)))
  (DESCRIPTION=(ADDRESS=(PROTOCOL=tcp)(HOST=192.168.0.90)(PORT=9120)))
Services Summary...
Service "+ASM" has 1 instance(s).
  Instance "+ASM2", status READY, has 1 handler(s) for this service...
Service "std11g2" has 1 instance(s).
  Instance "std11g22", status READY, has 1 handler(s) for this service...
Service "std11g2XDB" has 1 instance(s).
  Instance "std11g22", status READY, has 1 handler(s) for this service...
The command completed successfully
This concludes the changing of listener and scan listener names.

Known Issues
There were no known application related issues after this listener name change. Application was able to connect (JDBC) using the scan ip. However running cluvfy comp healthcheck resulted in following error
ERROR:
PRVG-0222 : Error establishing connection to database "std11g2". Verification will be skipped for this database.
PRCQ-1000 : An error occurred while establishing connection to database with user name "cvusys" and connect descriptor:
(DESCRIPTION = (LOAD_BALANCE=on)  (ADDRESS = (PROTOCOL = TCP)(HOST = rhel6m-scan)(PORT = null)) (CONNECT_DATA =(SERVER = DEDICATED)(SERVICE_NAME = std11g2)))
IO Error: The Network Adapter could not establish the connection
It could be seen that it's trying to use the scan-ip but the port value is null. When the scan listener name is reverted back to the default value (LISTENER_SCAN1) then the cluvfy comp healthcheck continue without any errors. It cluvfy comp healthcheck depends on the default scan listener name. SR was raised regarding this. However Oracle couldn't confirm or deny that there are components depending on the scan listener name to be of the default format. Final reply for this SR was "I think you are the first person did this change. Since you have followed the steps to change the SCAN Listener using srvctl then it is fine." With a phone call Oracle confirmed the steps are indeed correct and there are no documents available (yet) detailing how to change scan listener name. For the time being the RAC setup work fine and had no adverse effect other than the cluvfy issue.

Related Post
Changing Listener and SCAN Listener Port in 11gR2 RAC

Useful metalink notes
PRCA-1032: ASM Listener LISTENER Does Not Exist On 11.2 SIHA/OHAS Configuration. [ID 1428284.1]

Saturday, January 19, 2013

Changing Listener and SCAN Listener Port in 11gR2 RAC

Unlike the previous edition listener port change in RAC doesn't require any database parameter modification. According to Real Application Clusters Installation Guide During Oracle Database creation, the LOCAL_LISTENER parameter is automatically configured to point to the local listener for the database. The Database Agent sets the LOCAL_LISTENER parameter to a connect descriptor that does not require a TNS alias. You can set a value manually for LOCAL_LISTENER. However, Oracle recommends that you leave the parameter unset so that the Database Agent can maintain it automatically. If you set LOCAL_LISTENER, then the Agent does not automatically update this value. If you do not set LOCAL_LISTENER, then the Database Agent automatically keeps the database associated with the Grid home's node listener updated, even as the ports or IP of that listener are changed.
Steps below will change the port from the default 1521 to 9120. The configuration is a two node 11gR2 Standard Edition RAC with role separation and the solution for Oracle Security Alert for CVE-2012-1675 applied.
1. Current listener and scan listener configurations (run as grid user)
srvctl status listener
Listener LISTENER is enabled
Listener LISTENER is running on node(s): db-02,db-01

srvctl config listener
Name: LISTENER
Network: 1, Owner: grid
Home: 
End points: TCP:1521

$ srvctl status scan_listener
SCAN Listener LISTENER_SCAN1 is enabled
SCAN listener LISTENER_SCAN1 is running on node db-02
SCAN Listener LISTENER_SCAN2 is enabled
SCAN listener LISTENER_SCAN2 is running on node db-01
SCAN Listener LISTENER_SCAN3 is enabled
SCAN listener LISTENER_SCAN3 is running on node db-01

$ srvctl config scan_listener
SCAN Listener LISTENER_SCAN1 exists. Port: TCP:1521/TCPS:2992
SCAN Listener LISTENER_SCAN2 exists. Port: TCP:1521/TCPS:2992
SCAN Listener LISTENER_SCAN3 exists. Port: TCP:1521/TCPS:2992
The TCPS on 2992 is due to the COST setup and not part of the port change mentioned here.
2. As mentioned earlier (on the RAC installation guide) the local_listener is auto set and current configuration of local_listener is using port 1521
SQL> show parameter local

NAME            TYPE    VALUE
--------------- ------- ------------------------------------------------------------------------------------
local_listener  string  (DESCRIPTION=(ADDRESS_LIST=(ADDRESS=(PROTOCOL=TCP)(HOST=192.168.100.171)(PORT=1521))))
The remote listener is restricted to TCPS on the scan vips (again this is due to COST setup.
SQL> show parameter remote

NAME            TYPE    VALUE
--------------- ------- -----------------------------------------------------------------------
remote_listener  string  (ADDRESS_LIST=(ADDRESS=(PROTOCOL=TCPS)(HOST=192.168.100.181)(PORT=2992))
                                      (ADDRESS=(PROTOCOL=TCPS)(HOST=192.168.100.182)(PORT=2992))
                                      (ADDRESS=(PROTOCOL=TCPS)(HOST=192.168.100.183)(PORT=2992)))
3. The listener.ora in GI_HOME does not contain any port information. There's endpoints_listener.ora with port information but this is for backward compatibility for DB versions lower than 11.2 and not applicable in this case as DB is 11.2.
4. Make a note of listener status information which shows where the default port is being used. Run this is as grid user and set ORACLE_HOME variable (ie. . oraenv to +ASM*) if not the command will fail.
lsnrctl status listener # on db1

Connecting to (DESCRIPTION=(ADDRESS=(PROTOCOL=IPC)(KEY=LISTENER)))
STATUS of the LISTENER
------------------------
Alias                     LISTENER
Version                   TNSLSNR for Linux: Version 11.2.0.3.0 - Production
Start Date                18-JAN-2013 12:55:04
Uptime                    0 days 3 hr. 57 min. 32 sec
Trace Level               off
Security                  ON: Local OS Authentication
SNMP                      OFF
Listener Parameter File   /opt/app/11.2.0/grid/network/admin/listener.ora
Listener Log File         /opt/app/oracle/diag/tnslsnr/db-01/listener/alert/log.xml
Listening Endpoints Summary...
  (DESCRIPTION=(ADDRESS=(PROTOCOL=ipc)(KEY=LISTENER)))
  (DESCRIPTION=(ADDRESS=(PROTOCOL=tcp)(HOST=192.168.100.171)(PORT=1521))) <---- vip
  (DESCRIPTION=(ADDRESS=(PROTOCOL=tcp)(HOST=192.168.100.170)(PORT=1521))) <---- ip
Services Summary...
Service "+ASM" has 1 instance(s).
  Instance "+ASM1", status READY, has 1 handler(s) for this service...

lsnrctl status listener  # on db2

Connecting to (DESCRIPTION=(ADDRESS=(PROTOCOL=IPC)(KEY=LISTENER)))
STATUS of the LISTENER
------------------------
Alias                     LISTENER
Version                   TNSLSNR for Linux: Version 11.2.0.3.0 - Production
Start Date                18-JAN-2013 12:54:52
Uptime                    0 days 4 hr. 1 min. 31 sec
Trace Level               off
Security                  ON: Local OS Authentication
SNMP                      OFF
Listener Parameter File   /opt/app/11.2.0/grid/network/admin/listener.ora
Listener Log File         /opt/app/oracle/diag/tnslsnr/db-02/listener/alert/log.xml
Listening Endpoints Summary...
  (DESCRIPTION=(ADDRESS=(PROTOCOL=ipc)(KEY=LISTENER)))
  (DESCRIPTION=(ADDRESS=(PROTOCOL=tcp)(HOST=192.168.100.172)(PORT=1521))) <-- vip
  (DESCRIPTION=(ADDRESS=(PROTOCOL=tcp)(HOST=192.168.100.173)(PORT=1521))) <-- ip
Services Summary...
Service "+ASM" has 1 instance(s).
  Instance "+ASM2", status READY, has 1 handler(s) for this service...
Scan listener status. It's important to check the status of the scan_listener on the node it's currently active. Use the srvctl status to find out on which node the particular scan listener is currently active.
lsnrctl status listener_scan1

Connecting to (DESCRIPTION=(ADDRESS=(PROTOCOL=IPC)(KEY=LISTENER_SCAN1)))
STATUS of the LISTENER
------------------------
Alias                     LISTENER_SCAN1
Version                   TNSLSNR for Linux: Version 11.2.0.3.0 - Production
Start Date                18-JAN-2013 12:55:06
Uptime                    0 days 3 hr. 58 min. 11 sec
Trace Level               off
Security                  ON: Local OS Authentication
SNMP                      OFF
Listener Parameter File   /opt/app/11.2.0/grid/network/admin/listener.ora
Listener Log File         /opt/app/11.2.0/grid/log/diag/tnslsnr/db-02/listener_scan1/alert/log.xml
Listening Endpoints Summary...
  (DESCRIPTION=(ADDRESS=(PROTOCOL=ipc)(KEY=LISTENER_SCAN1)))
  (DESCRIPTION=(ADDRESS=(PROTOCOL=tcps)(HOST=192.168.100.181)(PORT=2992)))<-- scan ip with TCPS due to COST
  (DESCRIPTION=(ADDRESS=(PROTOCOL=tcp)(HOST=192.168.100.181)(PORT=1521))) <-- scan ip with TCP
Services Summary...

lsnrctl status listener_scan2

Connecting to (DESCRIPTION=(ADDRESS=(PROTOCOL=IPC)(KEY=LISTENER_SCAN2)))
STATUS of the LISTENER
------------------------
Alias                     LISTENER_SCAN2
Version                   TNSLSNR for Linux: Version 11.2.0.3.0 - Production
Start Date                18-JAN-2013 12:55:04
Uptime                    0 days 3 hr. 57 min. 57 sec
Trace Level               off
Security                  ON: Local OS Authentication
SNMP                      OFF
Listener Parameter File   /opt/app/11.2.0/grid/network/admin/listener.ora
Listener Log File         /opt/app/11.2.0/grid/log/diag/tnslsnr/db-01/listener_scan2/alert/log.xml
Listening Endpoints Summary...
  (DESCRIPTION=(ADDRESS=(PROTOCOL=ipc)(KEY=LISTENER_SCAN2)))
  (DESCRIPTION=(ADDRESS=(PROTOCOL=tcps)(HOST=192.168.100.182)(PORT=2992))) <-- scan ip with TCPS due to COST
  (DESCRIPTION=(ADDRESS=(PROTOCOL=tcp)(HOST=192.168.100.182)(PORT=1521))) <-- scan ip with TCP
Services Summary...

lsnrctl status listener_scan3

Connecting to (DESCRIPTION=(ADDRESS=(PROTOCOL=IPC)(KEY=LISTENER_SCAN3)))
STATUS of the LISTENER
------------------------
Alias                     LISTENER_SCAN3
Version                   TNSLSNR for Linux: Version 11.2.0.3.0 - Production
Start Date                18-JAN-2013 12:55:04
Uptime                    0 days 3 hr. 57 min. 59 sec
Trace Level               off
Security                  ON: Local OS Authentication
SNMP                      OFF
Listener Parameter File   /opt/app/11.2.0/grid/network/admin/listener.ora
Listener Log File         /opt/app/11.2.0/grid/log/diag/tnslsnr/db-01/listener_scan3/alert/log.xml
Listening Endpoints Summary...
  (DESCRIPTION=(ADDRESS=(PROTOCOL=ipc)(KEY=LISTENER_SCAN3)))
  (DESCRIPTION=(ADDRESS=(PROTOCOL=tcps)(HOST=192.168.100.183)(PORT=2992))) <-- scan ip with TCPS due to COST
  (DESCRIPTION=(ADDRESS=(PROTOCOL=tcp)(HOST=192.168.100.183)(PORT=1521))) <-- scan ip with TCP
Services Summary...



5. To change the port run srvctl as grid user.
srvctl modify listener -l LISTENER -p 9120

srvctl config listener
Name: LISTENER
Network: 1, Owner: grid
Home: 
End points: TCP:9120

srvctl modify scan_listener -p TCP:9120/TCPS:2992

srvctl config scan_listener
SCAN Listener LISTENER_SCAN1 exists. Port: TCP:9120/TCPS:2992
SCAN Listener LISTENER_SCAN2 exists. Port: TCP:9120/TCPS:2992
SCAN Listener LISTENER_SCAN3 exists. Port: TCP:9120/TCPS:2992
Changes are not effective until the listeners are restarted.
srvctl stop listener
srvctl start listener
srvctl stop scan_listener
srvctl start scan_listener
6. Verify the listeners have picked up the new port.
lsnrctl status listener_scan1

Connecting to (DESCRIPTION=(ADDRESS=(PROTOCOL=IPC)(KEY=LISTENER_SCAN1)))
STATUS of the LISTENER
------------------------
Alias                     LISTENER_SCAN1
Version                   TNSLSNR for Linux: Version 11.2.0.3.0 - Production
Start Date                18-JAN-2013 17:57:47
Uptime                    0 days 0 hr. 1 min. 21 sec
Trace Level               off
Security                  ON: Local OS Authentication
SNMP                      OFF
Listener Parameter File   /opt/app/11.2.0/grid/network/admin/listener.ora
Listener Log File         /opt/app/11.2.0/grid/log/diag/tnslsnr/db-02/listener_scan1/alert/log.xml
Listening Endpoints Summary...
  (DESCRIPTION=(ADDRESS=(PROTOCOL=ipc)(KEY=LISTENER_SCAN1)))
  (DESCRIPTION=(ADDRESS=(PROTOCOL=tcps)(HOST=192.168.100.181)(PORT=2992)))
  (DESCRIPTION=(ADDRESS=(PROTOCOL=tcp)(HOST=192.168.100.181)(PORT=9120)))
Services Summary...

lsnrctl status listener_scan2

Connecting to (DESCRIPTION=(ADDRESS=(PROTOCOL=IPC)(KEY=LISTENER_SCAN2)))
STATUS of the LISTENER
------------------------
Alias                     LISTENER_SCAN2
Version                   TNSLSNR for Linux: Version 11.2.0.3.0 - Production
Start Date                18-JAN-2013 17:57:47
Uptime                    0 days 0 hr. 0 min. 45 sec
Trace Level               off
Security                  ON: Local OS Authentication
SNMP                      OFF
Listener Parameter File   /opt/app/11.2.0/grid/network/admin/listener.ora
Listener Log File         /opt/app/11.2.0/grid/log/diag/tnslsnr/db-01/listener_scan2/alert/log.xml
Listening Endpoints Summary...
  (DESCRIPTION=(ADDRESS=(PROTOCOL=ipc)(KEY=LISTENER_SCAN2)))
  (DESCRIPTION=(ADDRESS=(PROTOCOL=tcps)(HOST=192.168.100.182)(PORT=2992)))
  (DESCRIPTION=(ADDRESS=(PROTOCOL=tcp)(HOST=192.168.100.182)(PORT=9120)))
Services Summary...

lsnrctl status listener_scan3

Connecting to (DESCRIPTION=(ADDRESS=(PROTOCOL=IPC)(KEY=LISTENER_SCAN3)))
STATUS of the LISTENER
------------------------
Alias                     LISTENER_SCAN3
Version                   TNSLSNR for Linux: Version 11.2.0.3.0 - Production
Start Date                18-JAN-2013 17:57:49
Uptime                    0 days 0 hr. 0 min. 45 sec
Trace Level               off
Security                  ON: Local OS Authentication
SNMP                      OFF
Listener Parameter File   /opt/app/11.2.0/grid/network/admin/listener.ora
Listener Log File         /opt/app/11.2.0/grid/log/diag/tnslsnr/db-01/listener_scan3/alert/log.xml
Listening Endpoints Summary...
  (DESCRIPTION=(ADDRESS=(PROTOCOL=ipc)(KEY=LISTENER_SCAN3)))
  (DESCRIPTION=(ADDRESS=(PROTOCOL=tcps)(HOST=192.168.100.183)(PORT=2992)))
  (DESCRIPTION=(ADDRESS=(PROTOCOL=tcp)(HOST=192.168.100.183)(PORT=9120)))
Services Summary...

lsnrctl status # on db1 and db2

Connecting to (DESCRIPTION=(ADDRESS=(PROTOCOL=IPC)(KEY=LISTENER)))
STATUS of the LISTENER
------------------------
Alias                     LISTENER
Version                   TNSLSNR for Linux: Version 11.2.0.3.0 - Production
Start Date                18-JAN-2013 17:57:31
Uptime                    0 days 0 hr. 1 min. 54 sec
Trace Level               off
Security                  ON: Local OS Authentication
SNMP                      OFF
Listener Parameter File   /opt/app/11.2.0/grid/network/admin/listener.ora
Listener Log File         /opt/app/oracle/diag/tnslsnr/db-02/listener/alert/log.xml
Listening Endpoints Summary...
  (DESCRIPTION=(ADDRESS=(PROTOCOL=ipc)(KEY=LISTENER)))
  (DESCRIPTION=(ADDRESS=(PROTOCOL=tcp)(HOST=192.168.100.171/2)(PORT=9120)))
  (DESCRIPTION=(ADDRESS=(PROTOCOL=tcp)(HOST=192.168.100.170/3)(PORT=9120)))
6. The DB's local_listener port is still registered on earler port (only one instance shown below).
SQL> show parameter local

NAME            TYPE    VALUE
--------------- ------- ------------------------------------------------------------------------------------
local_listener  string  (DESCRIPTION=(ADDRESS_LIST=(ADDRESS=(PROTOCOL=TCP)(HOST=192.168.100.171)(PORT=1521))))
Restart the database so local_listener is registered with the new port. Run as oracle user
srvctl stop database -d std11g2
srvctl start database -d std11g2

SQL> show parameter local

NAME            TYPE    VALUE
--------------- ------- ------------------------------------------------------------------------------------
local_listener  string  (DESCRIPTION=(ADDRESS_LIST=(ADDRESS=(PROTOCOL=TCP)(HOST=192.168.100.171)(PORT=9120))))
7. If COST is not used (refer 11gR2 Grid Infrastructure Single Client Access Name (SCAN) Explained 887522.1 note and section titled "Is it recommended to use COST feature?") then remote_listener value which is by default set as scan-name:port must also be updated with the new port. Since here COST is used (1340831.1) this step is omitted.
8. Port information on the endpoints_listener.ora would have been automatically edited by the agents and will reflect the new values set.
9. Change the port on any tnsnames.ora files used for connectivity to reflect the new port.
10. EM repository may need recreation with the new port information or manually edit config/emoms.properties and emd/targets.xml with port information.

Useful metalink notes
Changing Default Listener Port Number [ID 359277.1]
Listener port changed after 11.2 upgrade [ID 1269679.1]
Changing Listener Ports On RAC/EXADATA [ID 1473035.1]
11.2 Scan and Node TNS Listener Setup Examples [ID 1070607.1]
How To Configure Scan Listeners With A TCPS Port? [ID 1092753.1]
How to Modify SCAN Setting or SCAN Listener Port after Installation [ID 972500.1]
How to Configure A Second Listener on a Separate Network in 11.2 Grid Infrastructure [ID 1063571.1]
Using the TNS_ADMIN variable and changing the default port number of all Listeners in an 11.2 RAC for an 11.2, 11.1, and 10.2 Database [ID 1306927.1]

Friday, June 15, 2012

Implementing Class of Secure Transport (COST) to Restrict Instance Registration in Oracle 11gR2 SE RAC (Solution mentioned in Oracle Security Alert for CVE-2012-1675)

This post is related to implementing the solution for security vulnerability mentioned in Oracle Security Alert for CVE-2012-1675. There is another post but that is related to version 11gR1 and this one is specific to 11gR2.
The metalink note related to applying the solution in a RAC environment is Using Class of Secure Transport (COST) to Restrict Instance Registration in Oracle RAC [ID 1340831.1]
However this solution fails when tried on a standard edition RAC environment as standard edition Oracle binaries are not linked with TCPS protocol. The standard edition RAC environment created for RHEL 6 is used in this case.
All the steps in section 1 in (1340831.1) up to step 1.4 work same on SE as on EE edition and problem will be encountered trying to implement step 1.5.
As oracle user (since this is a role separated environment) try to set TCPS protocol for the remote listener using scan ips.
show parameter remote_listener

NAME                                 TYPE        VALUE
------------------------------------ ----------- ------------------------------
remote_listener                      string      rhel6m-scan:1521
This environment only used one scan ip (not recommended by Oracle)
srvctl config scan
SCAN name: rhel6m-scan, Network: 1/192.168.0.0/255.255.255.0/eth0
SCAN VIP name: scan1, IP: /rhel6m-scan/192.168.0.91
Trying to set TCPS will give the following error
SQL> alter system set remote_listener='(ADDRESS_LIST=(ADDRESS=(PROTOCOL=TCPS)(HOST=192.168.0.91)(PORT=1523)))' scope=both sid='*';
alter system set remote_listener='(ADDRESS_LIST=(ADDRESS=(PROTOCOL=TCPS)(HOST=192.168.0.91)(PORT=1523)))' scope=both sid='*'
*
ERROR at line 1:
ORA-02097: parameter cannot be modified because specified value is invalid
ORA-00119: invalid specification for system parameter REMOTE_LISTENER
ORA-00130: invalid listener address
'(ADDRESS=(PROTOCOL=TCPS)(HOST=192.168.0.91)(PORT=1523))'
Error messages here doesn't help much as it doesn't specifically say which parameter is invalid unlike in 11gR1 which mentioned invalid protocol. It could be tested indeed it's the protocol that is causing the problem by testing the same command with TCP instead of TCPS, in this case it would work without an error
SQL> alter system set remote_listener='(ADDRESS_LIST=(ADDRESS=(PROTOCOL=TCP)(HOST=192.168.0.91)(PORT=1523)))' scope=both sid='*';

System altered.
Solution is to relink the binaries with tcps same as before. Stop all processes running out of the ORACLE_HOME
srvctl stop home -o $ORACLE_HOME -s status -n `hostname -s`
Change into $ORACLE_HOME/lib and make a backup of the existing libntcps11.a file and copy the libntcps11_ee.a.dbl
cp libntcps11.a libntcps11.a.orig
cp libntcps11_ee.a.dbl libntcps11.a
Execute relink all and monitor the log file
$ relink all
writing relink log to: /opt/app/oracle/product/11.2.0/dbhome_1/install/relink.log
Relinking ends with following text
-L/opt/app/oracle/product/11.2.0/dbhome_1/lib
test ! -f /opt/app/oracle/product/11.2.0/dbhome_1/bin/oracle ||\
           mv -f /opt/app/oracle/product/11.2.0/dbhome_1/bin/oracle /opt/app/oracle/product/11.2.0/dbhome_1/bin/oracleO
mv /opt/app/oracle/product/11.2.0/dbhome_1/rdbms/lib/oracle /opt/app/oracle/product/11.2.0/dbhome_1/bin/oracle
chmod 6751 /opt/app/oracle/product/11.2.0/dbhome_1/bin/oracle
It is important to relink all the oracle homes in the cluster if not the error will still persist
SQL> alter system set remote_listener='(ADDRESS_LIST=(ADDRESS=(PROTOCOL=TCPS)(HOST=192.168.0.91)(PORT=1523)))' scope=both sid='*';
alter system set remote_listener='(ADDRESS_LIST=(ADDRESS=(PROTOCOL=TCPS)(HOST=192.168.0.91)(PORT=1523)))' scope=both sid='*'
*
ERROR at line 1:
ORA-32008: error while processing parameter update at instance std11g21
ORA-02097: parameter cannot be modified because specified value is invalid
ORA-00119: invalid specification for system parameter REMOTE_LISTENER
ORA-00130: invalid listener address
'(ADDRESS=(PROTOCOL=TCPS)(HOST=192.168.0.91)(PORT=1523))'
But to confirm relinking has worked the above command could be executed at instance level
SQL> alter system set remote_listener='(ADDRESS_LIST=(ADDRESS=(PROTOCOL=TCPS)(HOST=192.168.0.91)(PORT=1523)))' scope=both sid='std11g22';

System altered.
Once all oracle homes are relinked execute the original command.
SQL> alter system set remote_listener='(ADDRESS_LIST=(ADDRESS=(PROTOCOL=TCPS)(HOST=192.168.0.91)(PORT=1523)))' scope=both sid='*';

System altered.
Rest of the steps in (1340831.1) could be executed without any additional work.
If the libntcps11_ee.a.dbl file is missing it could obtained by extracting the
database/stage/Components/oracle.network.rsf/11.2.0.3.0/1/DataFiles/filegroup5.jar

Related Post
Using Class of Secure Transport (COST) to Restrict Instance Registration in Oracle SE RAC Fails (Solution mentioned in Oracle Security Alert for CVE-2012-1675)




Update on 11th November 2015
For database version 11.2.0.4 Oracle has introduced "Valid Node Checking for Registration (VNCR)" as an alternative for implementing COST, if the sole reason for implementing cost is preventing remote registration. MOS note 1340831.1 has been updated with this information. VNCR related information could be found on following docs.
How to Enable VNCR on RAC Database to Register only Local Instances [ID 1914282.1]
Valid Node Checking For Registration (VNCR) [ID 1600630.1]

Wednesday, May 9, 2012

Using Class of Secure Transport (COST) to Restrict Instance Registration in Oracle SE RAC Fails (Solution mentioned in Oracle Security Alert for CVE-2012-1675)

Oracle Security Alert CVE-2012-1675 mentions two metalink notes.
Using Class of Secure Transport (COST) to Restrict Instance Registration in Oracle RAC [ID 1340831.1] for RAC systems and Using Class of Secure Transport (COST) to Restrict Instance Registration [ID 1453883.1] for non-RAC systems. Non-RAC systme works fine in Oracle Enterprise and Standard Edition systems. But the solution given in 1340831.1 for "Oracle RAC versions prior to 11.2 (10.2.0.3 - 11.1)" cannot be implemented in Oracle Standard Edition RAC. After the wallet and the patch 12880299 is applied the metalink notes says to add a line to listener.ora for TCPS protocol and restart the listener. (This environment has seperate homes for ASM and DB and listener runs out of ASM home). Oracle database version is 11.1.0.7.11
LISTENER_RAC1 =
  (DESCRIPTION_LIST =
    (DESCRIPTION =
      (ADDRESS = (PROTOCOL = TCP)(HOST = rac1-vip)(PORT = 1521)(IP = FIRST))
      (ADDRESS = (PROTOCOL = TCPS)(HOST = rac1-vip)(PORT = 1523)(IP = FIRST))
      (ADDRESS = (PROTOCOL = TCP)(HOST = 192.168.0.85)(PORT = 1521)(IP = FIRST))
    )
  )

WALLET_LOCATION =
  (SOURCE =
   (METHOD = FILE)
    (METHOD_DATA =
     (DIRECTORY = /opt/app/oracle/product/11.1.0/asm_1/network/admin/cost)
    )
  )

#SECURE_REGISTER_LISTENER_RAC1 = (TCP,TCPS)
It then ask to stop and start the listener on that node. During the start of the listener following error is shown on the command line.
$ASM_HOME/bin/srvctl start listener -n `hostname -s`
rac1:ora.rac1.LISTENER_RAC1.lsnr:
rac1:ora.rac1.LISTENER_RAC1.lsnr:LSNRCTL for Linux: Version 11.1.0.7.0 - Production on 09-MAY-2012 15:01:38
rac1:ora.rac1.LISTENER_RAC1.lsnr:
rac1:ora.rac1.LISTENER_RAC1.lsnr:Copyright (c) 1991, 2008, Oracle.  All rights reserved.
rac1:ora.rac1.LISTENER_RAC1.lsnr:
rac1:ora.rac1.LISTENER_RAC1.lsnr:Starting /opt/app/oracle/product/11.1.0/asm_1/bin/tnslsnr: please wait...
rac1:ora.rac1.LISTENER_RAC1.lsnr:
rac1:ora.rac1.LISTENER_RAC1.lsnr:TNSLSNR for Linux: Version 11.1.0.7.0 - Production
rac1:ora.rac1.LISTENER_RAC1.lsnr:System parameter file is /opt/app/oracle/product/11.1.0/asm_1/network/admin/listener.ora
rac1:ora.rac1.LISTENER_RAC1.lsnr:Log messages written to /opt/app/oracle/diag/tnslsnr/rac1/listener_rac1/alert/log.xml
rac1:ora.rac1.LISTENER_RAC1.lsnr:Listening on: (DESCRIPTION=(ADDRESS=(PROTOCOL=tcp)(HOST=192.168.0.89)(PORT=1521)))
rac1:ora.rac1.LISTENER_RAC1.lsnr:Error listening on: (DESCRIPTION=(ADDRESS=(PROTOCOL=TCPS)(HOST=rac1-vip)(PORT=1523)(IP=FIRST)))
rac1:ora.rac1.LISTENER_RAC1.lsnr:TNS-12557: TNS:protocol adapter not loadable
rac1:ora.rac1.LISTENER_RAC1.lsnr: TNS-12560: TNS:protocol adapter error
rac1:ora.rac1.LISTENER_RAC1.lsnr:  TNS-00527: Protocol Adapter not loadable
rac1:ora.rac1.LISTENER_RAC1.lsnr:
rac1:ora.rac1.LISTENER_RAC1.lsnr:
rac1:ora.rac1.LISTENER_RAC1.lsnr:Listener failed to start. See the error message(s) above...
rac1:ora.rac1.LISTENER_RAC1.lsnr:
rac1:ora.rac1.LISTENER_RAC1.lsnr:
rac1:ora.rac1.LISTENER_RAC1.lsnr:LSNRCTL for Linux: Version 11.1.0.7.0 - Production on 09-MAY-2012 15:01:38
rac1:ora.rac1.LISTENER_RAC1.lsnr:
rac1:ora.rac1.LISTENER_RAC1.lsnr:Copyright (c) 1991, 2008, Oracle.  All rights reserved.
rac1:ora.rac1.LISTENER_RAC1.lsnr:
rac1:ora.rac1.LISTENER_RAC1.lsnr:Connecting to (DESCRIPTION=(ADDRESS=(PROTOCOL=TCP)(HOST=rac1-vip)(PORT=1521)(IP=FIRST)))
rac1:ora.rac1.LISTENER_RAC1.lsnr:TNS-12541: TNS:no listener
rac1:ora.rac1.LISTENER_RAC1.lsnr: TNS-12560: TNS:protocol adapter error
rac1:ora.rac1.LISTENER_RAC1.lsnr:  TNS-00511: No listener
rac1:ora.rac1.LISTENER_RAC1.lsnr:   Linux Error: 111: Connection refused
rac1:ora.rac1.LISTENER_RAC1.lsnr:Connecting to (DESCRIPTION=(ADDRESS=(PROTOCOL=TCPS)(HOST=rac1-vip)(PORT=1523)(IP=FIRST)))
rac1:ora.rac1.LISTENER_RAC1.lsnr:TNS-12557: TNS:protocol adapter not loadable
rac1:ora.rac1.LISTENER_RAC1.lsnr: TNS-12560: TNS:protocol adapter error
rac1:ora.rac1.LISTENER_RAC1.lsnr:  TNS-00527: Protocol Adapter not loadable
rac1:ora.rac1.LISTENER_RAC1.lsnr:
rac1:ora.rac1.LISTENER_RAC1.lsnr:Connecting to (DESCRIPTION=(ADDRESS=(PROTOCOL=TCP)(HOST=192.168.0.85)(PORT=1521)(IP=FIRST)))
rac1:ora.rac1.LISTENER_RAC1.lsnr:TNS-12541: TNS:no listener
rac1:ora.rac1.LISTENER_RAC1.lsnr: TNS-12560: TNS:protocol adapter error
rac1:ora.rac1.LISTENER_RAC1.lsnr:  TNS-00511: No listener
rac1:ora.rac1.LISTENER_RAC1.lsnr:   Linux Error: 111: Connection refused
CRS-0215: Could not start resource 'ora.rac1.LISTENER_RAC1.lsnr'.
On listener.log
System parameter file is /opt/app/oracle/product/11.1.0/asm_1/network/admin/listener.ora
Log messages written to /opt/app/oracle/diag/tnslsnr/rac1/listener_rac1/alert/log.xml
Trace information written to /opt/app/oracle/diag/tnslsnr/rac1/listener_rac1/trace/ora_15792_47190667617024.trc
Trace level is currently 0

Started with pid=15792
Listening on: (DESCRIPTION=(ADDRESS=(PROTOCOL=tcp)(HOST=192.168.0.89)(PORT=1521)))
Error listening on: (DESCRIPTION=(ADDRESS=(PROTOCOL=TCPS)(HOST=rac1-vip)(PORT=1523)(IP=FIRST)))
TNS-12557: TNS:protocol adapter not loadable
 TNS-12560: TNS:protocol adapter error
  TNS-00527: Protocol Adapter not loadable

No longer listening on: (DESCRIPTION=(ADDRESS=(PROTOCOL=tcp)(HOST=192.168.0.89)(PORT=1521)))
On crsd.log
2012-05-09 15:01:37.891: [  CRSRES][1510025536] startRunnable: setting CLI values
2012-05-09 15:01:37.895: [  CRSRES][1510025536] Attempting to start `ora.rac1.LISTENER_RAC1.lsnr` on member `rac1`
2012-05-09 15:01:38.920: [  CRSAPP][1510025536] StartResource error for ora.rac1.LISTENER_RAC1.lsnr error code = 1
2012-05-09 15:01:39.240: [  CRSRES][1510025536] Start of `ora.rac1.LISTENER_RAC1.lsnr` on member `rac1` failed.
2012-05-09 15:01:39.255: [  CRSRES][1510025536] rac2 : CRS-1019: Resource ora.rac1.LISTENER_RAC1.lsnr (application) cannot run on rac2
It could be that in SE the Advance Security Option needed to support TCPS is not there.

In the security alert Oracle mentions "Please note that Oracle has added Oracle Advanced Security SSL/TLS to the Oracle Database Standard Edition license when used with the Real Application Clusters and Oracle has added Oracle Advanced Security SSL/TLS to the Enterprise Edition Real Application Clusters (Oracle RAC) and RAC One Node options so that the directions provided in the Support Notes referenced above can be applied by all Oracle customers without additional cost."

The steps works in a 11.1.0.7.11 Enterprise Edition (EE) RAC without any issue. This could be explained using How To Check if Oracle Advanced Security Option is Installed ? [ID 549989.1] in which it is mentioned "Oracle product development has decided that starting with Oracle 11g Oracle Advanced Security Option will be installed by default when installing Enterprise Edition . There is no way to uninstall or to install it."

Update 31 May 2012
For the steps mentioned in 1340831.1 to work on Oracle SE it requires Oracle binaries to be relinked with the tcps. In this example the listener is running out of ASM home therefore oracle binaries in ASM Home will be relinked with tcps. It is not necessary to relink the Oracle Home if listener is not running out of it
1. cd $ASM_HOME/lib (or $ORACLE_HOME/lib if a separate home for ASM is not used).

2. Make a backup of the current tcps binary
$  cp libntcps11.a libntcps11.a.orig
3. Find libntcps11_ee.a.dbl file in the same lib directory and make a copy of it with the file name libntcps11.a
$ cp libntcps11_ee.a.dbl libntcps11.a
4. To make sure the relink is happening in ASM_HOME set the ORACLE_HOME to ASM_HOME and set relink binary in PATH variable
export ORACLE_HOME=$ASM_HOME
which relink
/opt/app/oracle/product/11.1.0/db_1/bin/relink <-- still using the ORACLE_HOME relink

export PATH=$ORACLE_HOME/bin:$PATH
/opt/app/oracle/product/11.1.0/asm_1/bin/relink
5. Stop all processes running out of ASM_HOME. Since database instance is dependent on asm instance this would mean database instance should be stopped as well. Once all running processes are stopped (except clusterware processes) start the relink with
relink all
and the relink process end with following lines (this is on 11.1.0.7 , on other versions it may be different)
..
..
/app/oracle/product/11.1.0/asm_1/lib
mv -f /opt/app/oracle/product/11.1.0/asm_1/bin/ctxkbtc /opt/app/oracle/product/11.1.0/asm_1/bin/ctxkbtc0
mv ctxkbtc /opt/app/oracle/product/11.1.0/asm_1/bin/ctxkbtc
chmod 751 /opt/app/oracle/product/11.1.0/asm_1/bin/ctxkbtc
Once the relink is complete continue with the steps in the metalink document and the listener will be able to start with TCPS.
If the file libntcps11_ee.a.dbl is missing in $ORACLE_HOME/lib then it could be obtained by extracting the
database/stage/Components/oracle.network.rsf/11.1.0.6.0/1/DataFiles/filegroup7.jar
file in the 11.1.0.6 base installation. Even though the system is upgraded to 11.1.0.7 there's no copy of it in 11.1.0.7 patchset files which mean it is not part of the upgraded processes thus only place to get it is on 11.1.0.6 base instllation.

Wednesday, March 5, 2008

Dispatcher Information on Listener