Wednesday, December 12, 2012

HAIP for Private Interconnect in 11gR2

Multiple NICs would be defined of private interconnect during installation (or after) on 11gR2. When there are multiple NICs defined for private interconnect oracle creates highly available IP (HAIP) for interconnect traffic.
From 11.2.0.2 onwards even if single NICs is used for private interconnect Oracle will use HAIP by default. Because of this v$cluster_interconnects view will show HAIP on 11gR2 (11.2.0.2 or later) compared to earlier versions such as 10g or 11gR1.
For example on 11gR1 IPs used for private interconnect could be identified using v$cluster_interconnects view
SQL> select * from gv$cluster_interconnects order by 1;

   INST_ID NAME            IP_ADDRESS       IS_ SOURCE
---------- --------------- ---------------- --- -------------------------------
         1 eth1            192.168.1.87     NO  Oracle Cluster Repository
         2 eth1            192.168.1.88     NO  Oracle Cluster Repository
Or on the database alert log will have following entries
Cluster communication is configured to use the following interface(s) for this instance
  192.168.1.88
cluster interconnect IPC version:Oracle UDP/IP (generic)
IPC Vendor 1 proto 2
However due to HAIP this is not the same on 11gR2, and what you specified during installation is not what you see on v$cluster_interconnects.
SQL> select * from gv$cluster_interconnects order by 1;

   INST_ID NAME            IP_ADDRESS       IS_ SOURCE
---------- --------------- ---------------- --- -------------------------------
         1 eth1:1          169.254.73.209   NO
         2 eth1:1          169.254.180.78   NO


Grid automatically creates the 169.254.*.* range IPs from the link local addresses that will be used for HAIP. These IPs are visible when ifconfig is executed and will appear on the same NIC assigned to private interconnect IP.
eth1      Link encap:Ethernet  HWaddr 08:00:27:7C:F4:94
          inet addr:192.168.1.88  Bcast:192.168.1.255  Mask:255.255.255.0
          inet6 addr: fe80::a00:27ff:fe7c:f494/64 Scope:Link
          UP BROADCAST RUNNING MULTICAST  MTU:1500  Metric:1
          RX packets:1997903 errors:0 dropped:0 overruns:0 frame:0
          TX packets:1680302 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:1000
          RX bytes:1349158236 (1.2 GiB)  TX bytes:965427458 (920.7 MiB)

eth1:1    Link encap:Ethernet  HWaddr 08:00:27:7C:F4:94
          inet addr:169.254.73.209  Bcast:169.254.255.255  Mask:255.255.0.0
          UP BROADCAST RUNNING MULTICAST  MTU:1500  Metric:1
Instead of getif option iflist could be used to list the HAIP associated interface.
oifcfg getif
eth0  192.168.0.0  global  public
eth1  192.168.1.0  global  cluster_interconnect

oifcfg iflist -p -n
eth0  192.168.0.0  PRIVATE  255.255.255.0
eth1  192.168.1.0  PRIVATE  255.255.255.0
eth1  169.254.0.0  UNKNOWN  255.255.0.0
Both ASM and DB alert log will also list the HAIP instead of the actual IP used during installation. From 11gR2 DB alert log
Cluster communication is configured to use the following interface(s) for this instance
  169.254.73.209
cluster interconnect IPC version:Oracle UDP/IP (generic)
From ASM alert log
Private Interface 'eth1:1' configured from GPnP for use as a private interconnect.
  [name='eth1:1', type=1, ip=169.254.73.209, mac=08-00-27-7c-f4-94, net=169.254.0.0/16, mask=255.255.0.0, use=haip:cluster_interconnect/62]
Public Interface 'eth0' configured from GPnP for use as a public interface.
  [name='eth0', type=1, ip=192.168.0.86, mac=08-00-27-c1-59-a3, net=192.168.0.0/24, mask=255.255.255.0, use=public/1]

Useful metalink notes
11gR2 Grid Infrastructure Redundant Interconnect and ora.cluster_interconnect.haip [ID 1210883.1]