Load balancing with Linux Virtual Server via Tunnelling
A post after almost a year. But finally I have something new to post.
Linux Virtual Server is commonly employed for load balancing between multiple servers. The load balancer and real servers share a common IP address called virtual IP.
There are three methods for using LVS – Direct Routing, NAT and Tunnelling. The first two methods require direct link between your machines and you should be able to control the routers. There’s no such restriction in the tunnelling method – the servers can be even spread over WAN many kilometres apart.
There are just two requirements in case of tunnelling – the real servers must support IPIP Tunnelling and the routers connected to the real servers must allow packets from your real server with SRC IP=Virtual IP even when the router doesn’t route incoming packets to the Virtual IP on that server. This is similar to IP spoofing, so as per standards it is blocked except special cases.
IPIP can be enabled in Linux by enabling the IP Tunneling option when configuring the kernel. If configured as module, it is known as ipip.
Suppose we have 3 servers with IP addresses 192.168.1.1, .2, .3 and the virtual IP which will be shared between all three is 192.168.1.4.
The load balancer must have both the IP addresses, real and virtual –
# ip addr add 192.168.1.1/CIDR brd dev eth0
# ip addr add 192.168.1.4/32 brd 192.168.1.4 dev eth0
# ip link set eth0 up
# ip route ... etc to configure routes
On real servers –
# ip addr add 192.168.1.2/CIDR brd dev eth0
# ip tunnel add mode ipip vip
# ip addr add 192.168.1.4/32 brd dev vip
# ip route add 192.168.1.4/32 dev vip
# ip link set eth0 up
# ip link set vip up
Now we have our real server configured, but there’s one thing that’s left which is enabling loose mode rp_filter for the IPIP interface.
# sysctl -w net.ipv4.conf.vip.rp_filter=2
rp_filter has three modes – 0 which means don’t do any source verification, 1 means strict filtering (default) and 2 means loose verification. If the source address can be found in one of the interfaces, then it will accept (ref: Linux Documentation).
If you happen to use Gentoo with OpenRC init system, you can use netifrc to do it on boot:
config_enp3s0="real ip addresses"
routes_enp3s0="default via "
iptunnel_vip="mode ipip"
config_vip="192.168.1.4/32 brd 192.168.1.4"
routes_vip="192.168.1.4 dev vip"
postup() {
if [ "${IFACE}" == "vip" ]; then
sysctl -w -q net.ipv4.conf.vip.rp_filter=2
fi
}
# cd /etc/init.d; ln -s net.lo net.vip
# rc-update add net.vip default
# cd -
Now, to test that the router connected to real servers allows packet spoofing on the virtual IP, we run traceroute on real server to an external IP address and watch for packets using tcpdump at the load balancer.
On real server –
# traceroute -s 192.168.1.4
On load balancer –
# tcpdump -ln host
You should see UDP port unreachable messages in tcpdump’s output. This indicates that packets successfully went to the real IP.
If the tcpdump/traceroute experiment is successful, you can load balance any service using LVS-Tunnel method. Suppose we want to load balance HTTP and the HTTP server has been setup on real servers. The HTTP server must listen on VIP that is 192.168.1.4 in this example.
On the load balancer –
# ipvsadm -A -t 192.168.1.4:80
# ipvsadm -a -t 192.168.1.4:80 -r 192.168.1.2
# ipvsadm -a -t 192.168.1.4:80 -r 192.168.1.3
As soon as traffic starts hitting our virtual IP, you should see connections in ipvsadm -L .
For more reference / guide, LVS-HOWTO homepage.
You can use stuff like ldirectord/keepalived for automatically removing servers from the IPVS table, because IPVS won’t check if the real server is up before forwarding packets to it. You definitely don’t want packets being routed into a black hole.
NileshGR 
Sorry, I know you posted this a while ago. This is the first example I’ve seen that uses “ip tunnel add”. I’d like to use this method as well, but it’s not clear in your example how you’re dealing with the ARP problem. Are IPs attached to tunnel devices automatically excluded from ARP replies?
LikeLike
As far as I remember, the real servers never reply to ARP who-has request from the router so router will always send packets to your LB. From there it is routed to the real server over a tunnel and then server uses the virtual IP to send it back to the router. Though this works, but you need proper tuning for MTU and stuff because of tunneling / encapsulation due to which I faced a lot of issues such as site loading breaks in between, etc.
LikeLike