Load balancing with Linux Virtual Server via Tunnelling

A post after almost a year. But finally I have something new to post.

Linux Virtual Server is commonly employed for load balancing between multiple servers. The load balancer and real servers share a common IP address called virtual IP.

There are three methods for using LVS – Direct Routing, NAT and Tunnelling. The first two methods require direct link between your machines and you should be able to control the routers. There’s no such restriction in the tunnelling method – the servers can be even spread over WAN many kilometres apart.

There are just two requirements in case of tunnelling – the real servers must support IPIP Tunnelling and the routers connected to the real servers must allow packets from your real server with  SRC IP=Virtual IP  even when the router doesn’t route incoming packets to the Virtual IP on that server. This is similar to IP spoofing, so as per standards it is blocked except special cases.

IPIP can be enabled in Linux by enabling the IP Tunneling option when configuring the kernel. If configured as module, it is known as ipip.

Suppose we have 3 servers with IP addresses, .2, .3 and the virtual IP which will be shared between all three is

The load balancer must have both the IP addresses, real and virtual –

# ip addr add brd  dev eth0
# ip addr add brd dev eth0
# ip link set eth0 up
# ip route ... etc to configure routes

On real servers –

# ip addr add brd  dev eth0
# ip tunnel add mode ipip vip
# ip addr add brd  dev vip
# ip route add dev vip
# ip link set eth0 up
# ip link set vip up

Now we have our real server configured, but there’s one thing that’s left which is enabling loose mode rp_filter  for the IPIP interface.

# sysctl -w net.ipv4.conf.vip.rp_filter=2

rp_filter  has three modes  – 0 which means don’t do any source verification, 1 means strict filtering (default) and 2 means loose verification. If the source address can be found in one of the interfaces, then it will accept (ref: Linux Documentation).

If you happen to use Gentoo with OpenRC init system, you can use netifrc to do it on boot:

config_enp3s0="real ip addresses"
routes_enp3s0="default via "

iptunnel_vip="mode ipip"
config_vip=" brd"
routes_vip=" dev vip"

postup() {
    if [ "${IFACE}" == "vip" ]; then
        sysctl -w -q net.ipv4.conf.vip.rp_filter=2

# cd /etc/init.d; ln -s net.lo net.vip
# rc-update add net.vip default
# cd -

Now, to test that the router connected to real servers allows packet spoofing on the virtual IP, we run traceroute  on real server to an external IP address and watch for packets using tcpdump  at the load balancer.

On real server –

# traceroute -s

On load balancer –

# tcpdump -ln host

You should see UDP port unreachable messages in tcpdump’s output. This indicates that packets successfully went to the real IP.

If the tcpdump/traceroute experiment is successful, you can load balance any service using LVS-Tunnel method. Suppose we want to load balance HTTP and the HTTP server has been setup on real servers. The HTTP server must listen on VIP that is in this example.

On the load balancer –

# ipvsadm -A -t
# ipvsadm -a -t -r
# ipvsadm -a -t -r

As soon as traffic starts hitting our virtual IP, you should see connections in ipvsadm -L .

For more reference / guide, LVS-HOWTO homepage.

You can use stuff like ldirectord/keepalived for automatically removing servers from the IPVS table, because IPVS won’t check if the real server is up before forwarding packets to it. You definitely don’t want packets being routed into a black hole.

2 Comments on “Load balancing with Linux Virtual Server via Tunnelling”

  1. Sorry, I know you posted this a while ago. This is the first example I’ve seen that uses “ip tunnel add”. I’d like to use this method as well, but it’s not clear in your example how you’re dealing with the ARP problem. Are IPs attached to tunnel devices automatically excluded from ARP replies?


    • As far as I remember, the real servers never reply to ARP who-has request from the router so router will always send packets to your LB. From there it is routed to the real server over a tunnel and then server uses the virtual IP to send it back to the router. Though this works, but you need proper tuning for MTU and stuff because of tunneling / encapsulation due to which I faced a lot of issues such as site loading breaks in between, etc.


Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )

Connecting to %s

%d bloggers like this: