What is Link Local IP ?
- A Link Local IP address is self generated IP address
- It is used for communication with in the network
- Router do not forward packets with the link local IP as its destination IP address .
- Link local IP's are similar to Private IP's .Link Local IP's are generated automatically where as Private IP's are configured manually for each network interface by a network admin
- Link Local Address IP Range is given below
- Starting IP AddressEnding IP Address# of Hosts169.254.1.0169.254.254.25565023
- Link Local IP's are generated by a feature called APIPA (Automatic Private IP Addressing )
- To differentiate link local IP's from Private IP's it use a different IP address range.
How HAIP Works ?
HAIP is introduced from 11.2.0.2 . Realizing the risk of using routable IP addresses for cluster interconnect oracle has introduced HAIP in version 11.2.0.2 . Even if routable IP addresses are set up for private network on the interface, link local IP addresses protect the frames from being forwarded to the Internet.
HAIP allows for redundant cluster interconnect NICs on cluster nodes without requiring any OS level bonding, teaming or aggregating of the NICs
When clusterware on a node is started, a HAIP address is dynamically allocated and linked (link-local) to each NIC configured for use as a cluster interconnect. The HAIP from each interconnect NIC is then used by the the ASM and RDBMS instances and ACFS .
Each NIC defined as a cluster interconnect on a given node will have a static ip address (Private IP) assigned to it and each cluster interconnect NIC on a given node must be on a unique subnet. If any one of the cluster interconnect NICs is down on a node, then the subnet associated with the down NIC is considered not usable by any node of the cluster.
[root@rac2 bin]# ./oifcfg getif
eth1 192.168.56.0 global public
eth2 192.168.10.0 global cluster_interconnect,asm
eth3 192.168.11.0 global cluster_interconnect,asm
From the above example we can see that eth2 and eth3 are on two different subnets .
The number of HAIP addresses is decided by how many private network adapters are active when Grid comes up on the first node in the cluster .
eth1 192.168.56.0 global public
eth2 192.168.10.0 global cluster_interconnect,asm
eth3 192.168.11.0 global cluster_interconnect,asm
From the above example we can see that eth2 and eth3 are on two different subnets .
A unique subnet per private network interface is a requirement from version 11.2 onward. If this configuration requirement is not followed, and if a cable is removed from the first interface listed in routing table, then ARP does not update the ARP cache properly, leading to instance reboot even though the second interface is up and available. To correct this problem scenario, we must use different subnets if multiple interfaces are configured for cluster_interconnect.
The clusterware actually detects that the cable has been unplugged or NIC has been disabled, marks the interface as failed and moves the HAIP 's for this interface to the 2nd NIC, the entry for the failing NIC remains in the routing table and all traffic to that subnet will continue to use the non-functional interface. This will not affect HAIP, but the network communication between the CSS daemons will fail because it does not use HAIP.
When a cluster interconnect NIC fails the HAIP associated with the failed NIC will failover and be linked-local to one of the remaining working physical NICs designated for the cluster interconnect. So the HAIP will now be on a different subnet, one that is working on all nodes.
An ARP broadcast will be made using the new MAC address for the NIC that the HAIP has attached to so that it updates routing table with the new MAC address for the failed over IP .
We can use the "netstat -rn" command to see the routing table to check which ip's active on which interface
The clusterware actually detects that the cable has been unplugged or NIC has been disabled, marks the interface as failed and moves the HAIP 's for this interface to the 2nd NIC, the entry for the failing NIC remains in the routing table and all traffic to that subnet will continue to use the non-functional interface. This will not affect HAIP, but the network communication between the CSS daemons will fail because it does not use HAIP.
When a cluster interconnect NIC fails the HAIP associated with the failed NIC will failover and be linked-local to one of the remaining working physical NICs designated for the cluster interconnect. So the HAIP will now be on a different subnet, one that is working on all nodes.
An ARP broadcast will be made using the new MAC address for the NIC that the HAIP has attached to so that it updates routing table with the new MAC address for the failed over IP .
We can use the "netstat -rn" command to see the routing table to check which ip's active on which interface
The number of HAIP addresses is decided by how many private network adapters are active when Grid comes up on the first node in the cluster .
- If there's only one active private network, Grid will create one;
- If two, Grid will create two;
- If more than two, Grid will create four HAIPs.
Ifconfig -a output list the HAIP's linked to cluster interconnect interfaces . In our example eth2 and eth3 are cluster_interconnect interfaces eth2:1 and eth3:1 are the link local IP's generated automatically by clusterware during node startup and linked to cluster_interconnect NIC.
eth2 Link encap:Ethernet HWaddr 08:00:27:DC:F8:FA
inet addr:192.168.10.102 Bcast:192.168.10.255 Mask:255.255.255.0
inet6 addr: fe80::a00:27ff:fedc:f8fa/64 Scope:Link
UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1
RX packets:45522 errors:0 dropped:0 overruns:0 frame:0
TX packets:125743 errors:0 dropped:0 overruns:0 carrier:0
collisions:0 txqueuelen:1000
RX bytes:28097898 (26.7 MiB) TX bytes:137493817 (131.1 MiB)
eth2:1 Link encap:Ethernet HWaddr 08:00:27:DC:F8:FA
inet addr:169.254.48.139 Bcast:169.254.127.255 Mask:255.255.128.0
UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1
eth3 Link encap:Ethernet HWaddr 08:00:27:6D:AD:3B
inet addr:192.168.11.12 Bcast:192.168.11.255 Mask:255.255.255.0
inet6 addr: fe80::a00:27ff:fe6d:ad3b/64 Scope:Link
UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1
RX packets:65372 errors:0 dropped:0 overruns:0 frame:0
TX packets:47514 errors:0 dropped:0 overruns:0 carrier:0
collisions:0 txqueuelen:1000
RX bytes:51783033 (49.3 MiB) TX bytes:39782962 (37.9 MiB)
eth3:1 Link encap:Ethernet HWaddr 08:00:27:6D:AD:3B
inet addr:169.254.225.100 Bcast:169.254.255.255 Mask:255.255.128.0
UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1
From the below output we can see the routing table entries for each interface and there respective subnets of Private IP's and Link Local IP's (HAIP) .
[root@rac2 bin]# netstat -rn
Kernel IP routing table
Destination Gateway Genmask Flags MSS Window irtt Iface
0.0.0.0 192.168.56.1 0.0.0.0 UG 0 0 0 eth1
169.254.0.0 0.0.0.0 255.255.128.0 U 0 0 0 eth2
169.254.128.0 0.0.0.0 255.255.128.0 U 0 0 0 eth3
192.168.10.0 0.0.0.0 255.255.255.0 U 0 0 0 eth2
192.168.11.0 0.0.0.0 255.255.255.0 U 0 0 0 eth3
192.168.56.0 0.0.0.0 255.255.255.0 U 0 0 0 eth1
[root@rac2 bin]#
Database Alert Log
2019-09-07T12:27:27.337380+05:30
Cluster Communication is configured to use IPs from: GPnP
IP: 169.254.48.139 Subnet: 169.254.0.0
IP: 169.254.225.100 Subnet: 169.254.128.0
KSIPC Loopback IP addresses(OSD):
127.0.0.1
KSIPC Available Transports: UDP:TCP
KSIPC: Client: KCL Transport: UDP
KSIPC: Client: DLM Transport: UDP
KSIPC CAPABILITIES :IPCLW:GRPAM:TOPO:DLL
KSXP: ksxpsg_ipclwtrans: 2 UDP
cluster interconnect IPC version: [IPCLW over UDP(mode 3) ]
IPC Vendor 1 proto 2
ASM Alert Log
System parameters with non-default values:
large_pool_size = 12M
remote_login_passwordfile= "EXCLUSIVE"
asm_diskstring = "/dev/asm*"
asm_diskgroups = "FRA"
asm_power_limit = 1
NOTE: remote asm mode is remote (mode 0x202; from cluster type)
2019-09-07T12:26:35.025874+05:30
Cluster Communication is configured to use IPs from: GPnP
IP: 169.254.48.139 Subnet: 169.254.0.0
IP: 169.254.225.100 Subnet: 169.254.128.0
KSIPC Loopback IP addresses(OSD):
127.0.0.1
KSIPC Available Transports: UDP:TCP
KSIPC: Client: KCL Transport: NONE
KSIPC: Client: DLM Transport: NONE
KSIPC CAPABILITIES :SKGXP:GRPCLSS:TOPO:DLL
KSXP: ksxpsg_ipclwtrans: 0 NONE
cluster interconnect IPC version: Oracle UDP/IP (generic)
IPC Vendor 1 proto 2
To Simulate the failure of an cluster interconnect NIC , i am bringing down one of the interface using the ifdown command .
[root@rac2 bin]# ifdown eth3
Device state: 3 (disconnected)
Now , from the below ifconfig -a output we can see that the HAIP is failed over the other surviving interface and linked to the surviving NIC .At this stage it sends an ARP broadcast message to update the ARP cache with the new MAC address of failed over NIC .
eth2 Link encap:Ethernet HWaddr 08:00:27:DC:F8:FA
inet addr:192.168.10.102 Bcast:192.168.10.255 Mask:255.255.255.0
inet6 addr: fe80::a00:27ff:fedc:f8fa/64 Scope:Link
UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1
RX packets:143024 errors:0 dropped:0 overruns:0 frame:0
TX packets:355394 errors:0 dropped:0 overruns:0 carrier:0
collisions:0 txqueuelen:1000
RX bytes:76154059 (72.6 MiB) TX bytes:284434094 (271.2 MiB)
eth2:1 Link encap:Ethernet HWaddr 08:00:27:DC:F8:FA
inet addr:169.254.48.139 Bcast:169.254.127.255 Mask:255.255.128.0
UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1
eth2:2 Link encap:Ethernet HWaddr 08:00:27:DC:F8:FA
inet addr:169.254.225.100 Bcast:169.254.255.255 Mask:255.255.128.0
UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1
We can see that eth2:1 and eth2:2 are two HAIP's linked to eth2 interface . We can see the HWaddr is same for all the interfaces .We Can also observe that the routing table is updated with subnets of the HAIP's pointing to the same interface .
[root@rac2 bin]# netstat -rn
Kernel IP routing table
Destination Gateway Genmask Flags MSS Window irtt Iface
0.0.0.0 192.168.56.1 0.0.0.0 UG 0 0 0 eth1
169.254.0.0 0.0.0.0 255.255.128.0 U 0 0 0 eth2
169.254.128.0 0.0.0.0 255.255.128.0 U 0 0 0 eth2
192.168.10.0 0.0.0.0 255.255.255.0 U 0 0 0 eth2
192.168.56.0 0.0.0.0 255.255.255.0 U 0 0 0 eth1
Whenever the failed interface comes back the HAIP will be failed back to the interface where it was before failover .
To Simulate i am using the ifup command to bring back the interface which i stopped before . From the below output we can observe that the HAIP is linked back to NIC where it was active before failover . We can also see that routing table got updated .
[root@rac2 bin]# ifup eth3
Active connection state: activated
Active connection path: /org/freedesktop/NetworkManager/ActiveConnection/3
eth2 Link encap:Ethernet HWaddr 08:00:27:DC:F8:FA
inet addr:192.168.10.102 Bcast:192.168.10.255 Mask:255.255.255.0
inet6 addr: fe80::a00:27ff:fedc:f8fa/64 Scope:Link
UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1
RX packets:182354 errors:0 dropped:0 overruns:0 frame:0
TX packets:396528 errors:0 dropped:0 overruns:0 carrier:0
collisions:0 txqueuelen:1000
RX bytes:105303711 (100.4 MiB) TX bytes:316463575 (301.8 MiB)
eth2:1 Link encap:Ethernet HWaddr 08:00:27:DC:F8:FA
inet addr:169.254.48.139 Bcast:169.254.127.255 Mask:255.255.128.0
UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1
eth3 Link encap:Ethernet HWaddr 08:00:27:6D:AD:3B
inet addr:192.168.11.12 Bcast:192.168.11.255 Mask:255.255.255.0
inet6 addr: fe80::a00:27ff:fe6d:ad3b/64 Scope:Link
UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1
RX packets:241335 errors:0 dropped:0 overruns:0 frame:0
TX packets:121335 errors:0 dropped:0 overruns:0 carrier:0
collisions:0 txqueuelen:1000
RX bytes:150786991 (143.8 MiB) TX bytes:93276892 (88.9 MiB)
eth3:1 Link encap:Ethernet HWaddr 08:00:27:6D:AD:3B
inet addr:169.254.225.100 Bcast:169.254.255.255 Mask:255.255.128.0
UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1
[root@rac2 bin]# netstat -rn
Kernel IP routing table
Destination Gateway Genmask Flags MSS Window irtt Iface
0.0.0.0 192.168.56.1 0.0.0.0 UG 0 0 0 eth1
169.254.0.0 0.0.0.0 255.255.128.0 U 0 0 0 eth2
169.254.128.0 0.0.0.0 255.255.128.0 U 0 0 0 eth3
192.168.10.0 0.0.0.0 255.255.255.0 U 0 0 0 eth2
192.168.11.0 0.0.0.0 255.255.255.0 U 0 0 0 eth3
192.168.56.0 0.0.0.0 255.255.255.0 U 0 0 0 eth1
[root@rac2 bin]#
No comments:
Post a Comment