BhanuDBA: How to create network interface bonding in Linux for Public interface

RAC is all about high availability. Every RAC node has a min of two interfaces , one for public and one for Private . We can have separate network for backup (storage network ) . From 12c there is an option to configure separate network for ASM inter communication . If there is a hardware failure in network interface (private network interface card) then the node will not be able to communicate with the other nodes in the cluster and eventually the node is evicted from the cluster and if the public network interface card is down then the users will not be able to connect to the database . Network interfaces are single point of failures .

IP's are coupled with the network interfaces . In the below example eth0 is coupled with IP 192.168.56.101 and eth1 is coupled with IP 172.16.10.11 . If these interfaces are down for any reason then the IP's will not respond .

+--------------------+ Public (192.168.56.101) X

+ NIC-1 (eth0) + =======================:

+--------------------+

+--------------------+ Private (172.16.10.11 )

+ NIC-2 (eth1) + =======================:

+--------------------+

Network interface Bonding in Linux provides high availability of network interface by logically grouping multiple interfaces as a single interface for the outside world .With Bonding we can achieve fault tolerance and load balancing by different configuration modes.

If we have four interfaces available on RAC nodes (eth0 - eth3) we can pair eth0 and eth1 as bond0 . and eth2 and eth3 as bond1 on all the nodes of the cluster ( NOTE : We need to use the same interface names for public and private on all RAC nodes ). Bonding is a logical grouping of two or more interfaces as a single interface and the IP is assigned/linked to this logical group . If one of the network interface in this bond(group) is down then the nodes will still communicate with the other available interface .

Here Bond0 is the master interface and eth0 and eth1 are slaves . Similarly with bond1 also . Bonding works in different modes .

Active-Active : where both interface are active and the load is balanced b/w these interfaces .

Active - Passive : Only one interface is active and the other interface is in standby to support when there is a failure .

To know in detail of the different modes . Please refer the below link .

https://www.thegeekdiary.com/what-are-the-network-bonding-modes-in-centos-rhel/

bond0 - Public 192.168.56.101

----------------------------------------------------

+ +

+ +--------------------+ +

+ + NIC-1 (eth0) + +

+ +--------------------+ +

+ +

+ +--------------------+ +

+ + NIC-2 (eth1) + +

+ +--------------------+ +

+ +

----------------------------------------------------

bond1 - Private 172.16.10.11

----------------------------------------------------

+ +

+ +--------------------+ +

+ + NIC-1 (eth2) + +

+ +--------------------+ +

+ +

+ +--------------------+ +

+ + NIC-2 (eth3) + +

+ +--------------------+ +

+ +

----------------------------------------------------

Steps to migrate Public network interface to bonded interface

STEP : 1 : Take the output of the ifconfig -a on all nodes of the cluster .

STEP : 2 : Stop the Database ,SCAN Listeners ,Local Listeners and VIP on all nodes of the cluster .

srvctl stop database -d <db_uniquename>
srvctl stop scan_listener
srvctl stop scan
srvctl stop vip -force

STEP : 3 : Go to "/etc/sysconfig/network-scripts" location and create ifcfg-bond0 , ifcfg-eth0 and ifcfg-eth1 . Network interface names may vary in your environment . The details shown here specific to my test environment . Please get the correct interface details from OS Team . Ideally bonding will be done by the OS team . As a DBA it is better to know the configuration details .

Create the below three files with the parameters given below on all nodes of the cluster .( Please change the IP/netmask/network according to your environment )

[root@rac1 network-scripts]# pwd

/etc/sysconfig/network-scripts

[root@rac1 network-scripts]#

[root@rac1 network-scripts]# cat ifcfg-bond0

DEVICE=bond0

IPADDR=192.168.56.101

NETWORK=192.168.56.0

NETMASK=255.255.255.0

USERCTL=no

BOOTPROTO=none

ONBOOT=yes

[root@rac1 network-scripts]#

[root@rac1 network-scripts]# cat ifcfg-eth0

HWADDR=08:00:27:00:D9:ED

DEVICE=eth0

BOOTPROTO=none

USERCTL=no

MASTER=bond0

SLAVE=yes

[root@rac1 network-scripts]# cat ifcfg-eth1

HWADDR=08:00:27:04:E7:AC

DEVICE=eth1

BOOTPROTO=none

USERCTL=no

MASTER=bond0

SLAVE=yes

Step : 4 Configure the bonding driver module and restart the network services on all the nodes of the cluster .

[root@rac1 modprobe.d]# pwd

/etc/modprobe.d

[root@rac1 modprobe.d]# vi bonding.conf

alias bond0 bonding

options bond0 mode=balance-alb miimon=100

[root@rac1 modprobe.d]#

Load the bonding Module .

[root@rac1 modprobe.d]# modprobe bonding

Restart the network services

[root@rac1 network-scripts]#

service network restart

Step : 5 To View the current status of the bonding interface and its slave interface check the below file .

[root@rac1 bin]# cat /proc/net/bonding/bond0
Ethernet Channel Bonding Driver: v3.7.1 (April 27, 2011)

Bonding Mode: adaptive load balancing
Primary Slave: None
Currently Active Slave: eth0
MII Status: up
MII Polling Interval (ms): 100
Up Delay (ms): 0
Down Delay (ms): 0

Slave Interface: eth0
MII Status: up
Speed: 1000 Mbps
Duplex: full
Link Failure Count: 0
Permanent HW addr: 08:00:27:00:d9:ed
Slave queue ID: 0

Slave Interface: eth1
MII Status: up
Speed: 1000 Mbps
Duplex: full
Link Failure Count: 0
Permanent HW addr: 08:00:27:04:e7:ac
Slave queue ID: 0

[root@rac1 bin]#

Step : 6 After restarting the network we can see the bond0 interface as MASTER and eth0 and eth1 are SLAVES

[root@rac1 network-scripts]# ifconfig -a

bond0 Link encap:Ethernet HWaddr 08:00:27:00:D9:ED

inet addr:192.168.56.101 Bcast:192.168.56.255 Mask:255.255.255.0

inet6 addr: fe80::a00:27ff:fe00:d9ed/64 Scope:Link

UP BROADCAST RUNNING MASTER MULTICAST MTU:1500 Metric:1

RX packets:16159 errors:0 dropped:1226 overruns:0 frame:0

TX packets:34532 errors:0 dropped:0 overruns:0 carrier:0

collisions:0 txqueuelen:0

RX bytes:2210653 (2.1 MiB) TX bytes:2321079 (2.2 MiB)

eth0 Link encap:Ethernet HWaddr 08:00:27:00:D9:ED

UP BROADCAST RUNNING SLAVE MULTICAST MTU:1500 Metric:1

RX packets:6958 errors:0 dropped:0 overruns:0 frame:0

TX packets:18996 errors:0 dropped:0 overruns:0 carrier:0

collisions:0 txqueuelen:1000

RX bytes:1208245 (1.1 MiB) TX bytes:1308178 (1.2 MiB)

eth1 Link encap:Ethernet HWaddr 08:00:27:04:E7:AC

UP BROADCAST RUNNING SLAVE MULTICAST MTU:1500 Metric:1

RX packets:9201 errors:0 dropped:1226 overruns:0 frame:0

TX packets:15536 errors:0 dropped:0 overruns:0 carrier:0

collisions:0 txqueuelen:1000

RX bytes:1002408 (978.9 KiB) TX bytes:1012901 (989.1 KiB)

Repeat the steps from 3 to 6 on the other nodes of the cluster .

[root@rac2 network-scripts]# pwd

/etc/sysconfig/network-scripts

[root@rac2 network-scripts]# cat ifcfg-bond0

DEVICE=bond0

IPADDR=192.168.56.102

NETWORK=192.168.56.0

NETMASK=255.255.255.0

USERCTL=no

BOOTPROTO=none

ONBOOT=yes

[root@rac2 network-scripts]# cat ifcfg-eth0

HWADDR=08:00:27:34:FD:A9

DEVICE=eth0

BOOTPROTO=none

USERCTL=no

MASTER=bond0

SLAVE=yes

[root@rac2 network-scripts]# cat ifcfg-eth1

HWADDR=08:00:27:DC:F8:FA

DEVICE=eth1

BOOTPROTO=none

USERCTL=no

MASTER=bond0

SLAVE=yes

[root@rac2 modprobe.d]# pwd

/etc/modprobe.d

[root@rac2 modprobe.d]# vi bonding.conf

alias bond0 bonding

options bond0 mode=balance-alb miimon=100

[root@rac2 modprobe.d]#

Load the bonding Module .

[root@rac2 modprobe.d]# modprobe bonding

Restart the network services

[root@rac2 network-scripts]#

service network restart

[root@rac2 network-scripts]# ifconfig -a

bond0 Link encap:Ethernet HWaddr 08:00:27:34:FD:A9

inet addr:192.168.56.102 Bcast:192.168.56.255 Mask:255.255.255.0

inet6 addr: fe80::a00:27ff:fe34:fda9/64 Scope:Link

UP BROADCAST RUNNING MASTER MULTICAST MTU:1500 Metric:1

RX packets:14198 errors:0 dropped:1176 overruns:0 frame:0

TX packets:24503 errors:0 dropped:0 overruns:0 carrier:0

collisions:0 txqueuelen:0

RX bytes:1240495 (1.1 MiB) TX bytes:1514228 (1.4 MiB)

eth0 Link encap:Ethernet HWaddr 08:00:27:34:FD:A9

UP BROADCAST RUNNING SLAVE MULTICAST MTU:1500 Metric:1

RX packets:12644 errors:0 dropped:0 overruns:0 frame:0

TX packets:12549 errors:0 dropped:0 overruns:0 carrier:0

collisions:0 txqueuelen:1000

RX bytes:1082266 (1.0 MiB) TX bytes:788058 (769.5 KiB)

eth1 Link encap:Ethernet HWaddr 08:00:27:DC:F8:FA

UP BROADCAST RUNNING SLAVE MULTICAST MTU:1500 Metric:1

RX packets:1554 errors:0 dropped:1176 overruns:0 frame:0

TX packets:11954 errors:0 dropped:0 overruns:0 carrier:0

collisions:0 txqueuelen:1000

RX bytes:158229 (154.5 KiB) TX bytes:726170 (709.1 KiB)

[root@rac2 network-scripts]#

We can see the interface names and there H/W addresses in the below file "/etc/udev/rules.d/70-persistent-net.rules" . We can change the interface names if required .

[root@rac2 rules.d]# pwd

/etc/udev/rules.d

[root@rac2 rules.d]# cat 70-persistent-net.rules

# This file was automatically generated by the /lib/udev/write_net_rules

# program, run by the persistent-net-generator.rules rules file.

# You can modify it, as long as you keep each rule on a single

# line, and change only the value of the NAME= key.

# PCI device 0x8086:0x100e (e1000)

SUBSYSTEM=="net", ACTION=="add", DRIVERS=="?*", ATTR{address}=="08:00:27:34:fd:a9", ATTR{type}=="1", KERNEL=="eth*", NAME="eth0"

# PCI device 0x8086:0x100e (e1000)

SUBSYSTEM=="net", ACTION=="add", DRIVERS=="?*", ATTR{address}=="08:00:27:44:e1:5e", ATTR{type}=="1", KERNEL=="eth*", NAME="eth2"

# PCI device 0x8086:0x100e (e1000)

SUBSYSTEM=="net", ACTION=="add", DRIVERS=="?*", ATTR{address}=="08:00:27:dc:f8:fa", ATTR{type}=="1", KERNEL=="eth*", NAME="eth1"

# PCI device 0x8086:0x100e (e1000)

SUBSYSTEM=="net", ACTION=="add", DRIVERS=="?*", ATTR{address}=="08:00:27:6d:ad:3b", ATTR{type}=="1", KERNEL=="eth*", NAME="eth3"

[root@rac2 rules.d]#

STEP : 7 By now we have created the bonding interface bond0 on all the nodes of the cluster . its time to change the public interface in RAC cluster to use the newly created bond interface .

PLEASE NOTE : OS Admin might have reassigned the public IP's of the previous interface before bonding to the bonded interface now . In this exercise we are just changing the interface name not the public IP . Public IP remains the same .

Add new public interface by using the below command from any one node .

[root@rac1 bin]# ./oifcfg getif

eth3 172.16.10.0 global cluster_interconnect,asm

eth2 192.168.56.0 global public

[root@rac1 bin]# ./oifcfg setif -global bond0:192.168.56.0:public

[root@rac1 bin]# ./oifcfg getif

eth3 172.16.10.0 global cluster_interconnect,asm

eth2 192.168.56.0 global public

bondo0 192.168.56.0 global public

[root@rac1 bin]#

STEP : 8 Modify the network configuration to use the newly added interface for public network .

[root@rac1 bin]# ./srvctl modify network -subnet 192.168.56.0/255.255.255.0/bond0

[root@rac1 bin]# ./srvctl config network

Network 1 exists

Subnet IPv4: 192.168.56.0/255.255.255.0/bond0, static

Subnet IPv6:

Ping Targets:

Network is enabled

Network is individually enabled on nodes:

Network is individually disabled on nodes:

[root@rac1 bin]#

STEP : 9 We can see the VIP's are started using the newly added interface from the below output.

[root@rac1 bin]# ./srvctl config nodeapps

Network 1 exists

Subnet IPv4: 192.168.56.0/255.255.255.0/bond0, static

Subnet IPv6:

Ping Targets:

Network is enabled

Network is individually enabled on nodes:

Network is individually disabled on nodes:

VIP exists: network number 1, hosting node rac1

VIP Name: rac1-vip.localdomain

VIP IPv4 Address: 192.168.56.91

VIP IPv6 Address:

VIP is enabled.

VIP is individually enabled on nodes:

VIP is individually disabled on nodes:

VIP exists: network number 1, hosting node rac2

VIP Name: rac2-vip.localdomain

VIP IPv4 Address: 192.168.56.92

VIP IPv6 Address:

VIP is enabled.

VIP is individually enabled on nodes:

VIP is individually disabled on nodes:

ONS exists: Local port 6100, remote port 6200, EM port 2016, Uses SSL true

ONS is enabled

ONS is individually enabled on nodes:

ONS is individually disabled on nodes:

STEP : 10 Now its time to change the SCAN to use the new interface name . From the below output we can see the SCAN's are still running using the old interface name

[root@rac1 bin]# ./srvctl config scan

SCAN name: rac-scan, Network: 1

Subnet IPv4: 192.168.56.0/255.255.255.0/eth2, static

Subnet IPv6:

SCAN 1 IPv4 VIP: 10.15.20.30

SCAN VIP is enabled.

SCAN VIP is individually enabled on nodes:

SCAN VIP is individually disabled on nodes:

SCAN 2 IPv4 VIP: 10.15.20.31

SCAN VIP is enabled.

SCAN VIP is individually enabled on nodes:

SCAN VIP is individually disabled on nodes:

SCAN 3 IPv4 VIP: 10.15.20.32

SCAN VIP is enabled.

SCAN VIP is individually enabled on nodes:

SCAN VIP is individually disabled on nodes:

[root@rac1 bin]#

[root@rac1 bin]# ./srvctl stop scan_listener

[root@rac1 bin]# ./srvctl stop scan

STEP : 11 Run the below command to refresh SCAN configuration

[root@rac1 bin]# ./srvctl modify scan -scanname rac-scan

[root@rac1 bin]# ./srvctl start scan

[root@rac1 bin]# ./srvctl start scan_listener

[root@rac1 bin]#

STEP : 12 We can see the SCAN IP's are populated using the new interface bond0 .

[root@rac1 bin]# ./srvctl config scan

SCAN name: rac-scan, Network: 1

Subnet IPv4: 192.168.56.0/255.255.255.0/bond0, static

Subnet IPv6:

SCAN 1 IPv4 VIP: 192.168.56.94

SCAN VIP is enabled.

SCAN VIP is individually enabled on nodes:

SCAN VIP is individually disabled on nodes:

SCAN 2 IPv4 VIP: 192.168.56.95

SCAN VIP is enabled.

SCAN VIP is individually enabled on nodes:

SCAN VIP is individually disabled on nodes:

SCAN 3 IPv4 VIP: 192.168.56.93

SCAN VIP is enabled.

SCAN VIP is individually enabled on nodes:

SCAN VIP is individually disabled on nodes:

[root@rac1 bin]#

STEP : 13 At this point of time we have updated Public interface to bond0 and reconfigured cluster configuration to use the newly added interface .We can delete the old eth2 interface .

[root@rac1 bin]# ./oifcfg getif

eth3 172.16.10.0 global cluster_interconnect,asm

eth2 192.168.56.0 global public

bondo0 192.168.56.0 global public

[root@rac1 bin]# ./oifcfg delif -global eth2

[root@rac1 bin]# ./oifcfg getif

eth3 172.16.10.0 global cluster_interconnect,asm

bondo0 192.168.56.0 global public

[root@rac1 bin]#

STEP : 14 Once we have all the VIP and SCAN IP are up . We can see these IP's are linked to the bond0 interface with the alias names bond0:x

In the below output

bond0 : Public IP

bond0:2 : SCAN-IP

bond0:3 : SCAN-IP

bond0:4 : VIP

we can use the nslookup with the below ip's to know hostname resolutions

[root@rac1 bin]# ifconfig -a

bond0 Link encap:Ethernet HWaddr 08:00:27:00:D9:ED

inet addr:192.168.56.101 Bcast:192.168.56.255 Mask:255.255.255.0

inet6 addr: fe80::a00:27ff:fe00:d9ed/64 Scope:Link

UP BROADCAST RUNNING MASTER MULTICAST MTU:1500 Metric:1

RX packets:15805 errors:0 dropped:1217 overruns:0 frame:0

TX packets:32532 errors:0 dropped:0 overruns:0 carrier:0

collisions:0 txqueuelen:0

RX bytes:2164731 (2.0 MiB) TX bytes:2192363 (2.0 MiB)

bond0:2 Link encap:Ethernet HWaddr 08:00:27:00:D9:ED

inet addr:192.168.56.95 Bcast:192.168.56.255 Mask:255.255.255.0

UP BROADCAST RUNNING MASTER MULTICAST MTU:1500 Metric:1

bond0:3 Link encap:Ethernet HWaddr 08:00:27:00:D9:ED

inet addr:192.168.56.93 Bcast:192.168.56.255 Mask:255.255.255.0

UP BROADCAST RUNNING MASTER MULTICAST MTU:1500 Metric:1

bond0:4 Link encap:Ethernet HWaddr 08:00:27:00:D9:ED

inet addr:192.168.56.91 Bcast:192.168.56.255 Mask:255.255.255.0

UP BROADCAST RUNNING MASTER MULTICAST MTU:1500 Metric:1

eth0 Link encap:Ethernet HWaddr 08:00:27:00:D9:ED

UP BROADCAST RUNNING SLAVE MULTICAST MTU:1500 Metric:1

RX packets:6847 errors:0 dropped:0 overruns:0 frame:0

TX packets:18029 errors:0 dropped:0 overruns:0 carrier:0

collisions:0 txqueuelen:1000

RX bytes:1193268 (1.1 MiB) TX bytes:1245993 (1.1 MiB)

eth1 Link encap:Ethernet HWaddr 08:00:27:04:E7:AC

UP BROADCAST RUNNING SLAVE MULTICAST MTU:1500 Metric:1

RX packets:8958 errors:0 dropped:1217 overruns:0 frame:0

TX packets:14503 errors:0 dropped:0 overruns:0 carrier:0

collisions:0 txqueuelen:1000

RX bytes:971463 (948.6 KiB) TX bytes:946370 (924.1 KiB)

eth2 Link encap:Ethernet HWaddr 08:00:27:44:E1:5E

inet addr:10.15.20.10 Bcast:10.15.20.255 Mask:255.255.255.0

inet6 addr: fe80::a00:27ff:fe44:e15e/64 Scope:Link

UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1

RX packets:6143 errors:0 dropped:0 overruns:0 frame:0

TX packets:8803 errors:0 dropped:0 overruns:0 carrier:0

collisions:0 txqueuelen:1000

RX bytes:2348044 (2.2 MiB) TX bytes:2930811 (2.7 MiB)

eth3 Link encap:Ethernet HWaddr 08:00:27:A0:5A:A4

inet addr:172.16.10.11 Bcast:172.16.10.255 Mask:255.255.255.0

inet6 addr: fe80::a00:27ff:fea0:5aa4/64 Scope:Link

UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1

RX packets:97728 errors:0 dropped:0 overruns:0 frame:0

TX packets:112078 errors:0 dropped:0 overruns:0 carrier:0

collisions:0 txqueuelen:1000

RX bytes:82182520 (78.3 MiB) TX bytes:91906772 (87.6 MiB)

eth3:1 Link encap:Ethernet HWaddr 08:00:27:A0:5A:A4

inet addr:169.254.93.115 Bcast:169.254.255.255 Mask:255.255.0.0

UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1

lo Link encap:Local Loopback

inet addr:127.0.0.1 Mask:255.0.0.0

inet6 addr: ::1/128 Scope:Host

UP LOOPBACK RUNNING MTU:16436 Metric:1

RX packets:56418 errors:0 dropped:0 overruns:0 frame:0

TX packets:56418 errors:0 dropped:0 overruns:0 carrier:0

collisions:0 txqueuelen:0

RX bytes:158344945 (151.0 MiB) TX bytes:158344945 (151.0 MiB)

NODE:2 Here we see only two aliases , Only one SCAN and one VIP is running on this node . The other two SCAN IP's are running on first node .

[oracle@rac2 bin]$ ifconfig -a
bond0 Link encap:Ethernet HWaddr 08:00:27:34:FD:A9
inet addr:192.168.56.102 Bcast:192.168.56.255 Mask:255.255.255.0
inet6 addr: fe80::a00:27ff:fe34:fda9/64 Scope:Link
UP BROADCAST RUNNING MASTER MULTICAST MTU:1500 Metric:1
RX packets:6519 errors:0 dropped:195 overruns:0 frame:0
TX packets:8053 errors:0 dropped:0 overruns:0 carrier:0
collisions:0 txqueuelen:0
RX bytes:876977 (856.4 KiB) TX bytes:986162 (963.0 KiB)

bond0:1 Link encap:Ethernet HWaddr 08:00:27:34:FD:A9
inet addr:192.168.56.92 Bcast:192.168.56.255 Mask:255.255.255.0
UP BROADCAST RUNNING MASTER MULTICAST MTU:1500 Metric:1

bond0:2 Link encap:Ethernet HWaddr 08:00:27:34:FD:A9
inet addr:192.168.56.94 Bcast:192.168.56.255 Mask:255.255.255.0

UP BROADCAST RUNNING MASTER MULTICAST MTU:1500 Metric:1

STEP : 15 Start the database services . At this point we can all the services and resources are up and running .

srvctl start database -d <db_uniquename>

BhanuDBA

How to create network interface bonding in Linux for Public interface

No comments:

Post a Comment

Change Private IP Network Interface /Subnet/Netmask

Total Pageviews