Simple Linux Virtual Server Setup for Fedora 15 with LVS-DR forwarding

This post describes how to setup a simple Linux Virtual Server (LVS) using a director with Fedora 15 and direct routing (LVS-DR) as forwarding method. Setting up a LVS helps you to distribute the traffic of your website to various servers. This process is called load balancing. The instructions are based on the LVS-mini-HOWTO where further information can be found.
The purpose of this exercise is to distribute the traffic of a website between two (or more) servers which host a copy website. Furthermore persistent connections are required which means that a client is always redirected to the same server for a defined interval of time. For debugging as telnet (port 23) as http (port 80) are load balanced in this setup since testing for telnet is way more easier.


What do I need?

For realizing and testing this setup you need at least 3 nodes:

(1) A client to address the LVS
(2) A node the redirects the requests (director) and operates a real server at the same time
(3) A node that only works as a real server

Each node needs one network interface card (NIC).

You can add an arbitrary number of additional real servers to the setup to increase performance. The director and real server should have installed at least Fedora 15, for the client the operating system does not matter. The three nodes are located in the same network.

In our example the director has the IP address 192.168.1.12, the real server 192.168.1.10. Both will share the virtual IP address 192.168.1.110 under which the LVS will be reachable.

Preparations for node (2), the director:
Some steps have to be done manually, the rest of the configuration can be done by the configuration script.

Fedora 15 already brings the required kernel ip_vs modules therefore it is not necessary to patch the kernel.

In a first step we install the tool ipvsadm which we use and monitor to configure and the LVS.

[sourcecode language=”bash”]
yum install ipvsadm
[/sourcecode]

Now we have to add the following lines to the file /etc/sysctl.conf:

[sourcecode language=”bash”]
# Controls IP packet forwarding
net.ipv4.ip_forward = 0

# Controls source route verification
net.ipv4.conf.default.rp_filter = 0

# Do not accept source routing
net.ipv4.conf.default.accept_source_route = 0
[/sourcecode]

and afterwards run
[sourcecode language=”bash”]
sysctl -p
[/sourcecode]

to update the kernel parameters.

The configuration script:

Replace “p5p1″ by the name of your NIC (often eth0) before running the configuration script:

[sourcecode language=”bash”]
#!/bin/bash
#—————mini-rc.lvs_dr-director————————
#set ip_forward OFF for lvs-dr director (1 on, 0 off)
#(there is no forwarding in the conventional sense for LVS-DR)

#add ethernet device and routing for VIP 192.168.1.110
/sbin/ifconfig p5p1:110 192.168.1.110 broadcast 192.168.1.110 netmask 255.255.255.255
/sbin/route add -host 192.168.1.110 dev p5p1:110
#listing ifconfig info for VIP 192.168.1.110
/sbin/ifconfig p5p1:110

#check VIP 192.168.1.110 is reachable from self (director)
/bin/ping -c 1 192.168.1.110
#listing routing info for VIP 192.168.1.110
/bin/netstat -rn

#setup_ipvsadm_table
#clear ipvsadm table
/sbin/ipvsadm -C
#installing LVS services with ipvsadm
#add telnet to VIP with round robin scheduling
/sbin/ipvsadm -A -t 192.168.1.110:telnet -s rr
/sbin/ipvsadm -A -t 192.168.1.110:http -s rr
# persistent connection deactivated for it is difficult to debug
#/sbin/ipvsadm -A -t 192.168.1.110:http -s rr -p 600

#forward telnet and http to realserver using direct routing with weight 1
/sbin/ipvsadm -a -t 192.168.1.110:telnet -r 192.168.1.10 -g -w 1
/sbin/ipvsadm -a -t 192.168.1.110:http -r 192.168.1.10 -g -w 1
#check realserver reachable from director
ping -c 1 192.168.1.10

#forward telnet and http to the director itself using direct routing with weight 1
/sbin/ipvsadm -a -t 192.168.1.110:telnet -r 192.168.1.12 -g -w 1
/sbin/ipvsadm -a -t 192.168.1.110:http -r 192.168.1.12 -g -w 1
#check realserver reachable from director
ping -c 1 192.168.1.12

#displaying ipvsadm settings
/sbin/ipvsadm

[/sourcecode]

In case you need persistent connections use the commented line in the script for that.

Preparations for node (3), the real servers:

Before running the configuration script you have also to modify the /etc/sysctl.conf at the real server. It has to contain the following lines. “p2p1″ has to be the name of the NIC (e.g. eth0).

[sourcecode language=”bash”]

net.ipv4.conf.p2p1.arp_ignore = 1
net.ipv4.conf.p2p1.arp_announce = 2
net.ipv4.conf.all.arp_ignore = 1
net.ipv4.conf.all.arp_announce = 2

# Controls IP packet forwarding
net.ipv4.ip_forward = 0

[/sourcecode]

run
[sourcecode language=”bash”]
sysctl -p
[/sourcecode]

The standard gateway for the real servers can be any IP (eg. the client or a DSL router) in your network apart from that of the director. Change “p2p1” to the name the NIC of your real server.

The configuration script:
[sourcecode language=”bash”]

#!/bin/bash
#———-mini-rc.lvs_dr-realserver——————
#installing default gw 192.168.1.1 for vs-dr
/sbin/route add default gw 192.168.1.1
#showing routing table
/bin/netstat -rn
#checking if DEFAULT_GW 192.168.1.1 is reachable
ping -c 1 192.168.1.1

#looking for DIP 192.168.1.12
ping -c 1 192.168.1.12

#looking for VIP (will be on director)
ping -c 1 192.168.1.110

#install_realserver_vip
/sbin/ifconfig lo:110 192.168.1.110 broadcast 192.168.1.110 netmask 0xffffffff up
#ifconfig output
/sbin/ifconfig lo:110
#installing route for VIP 192.168.1.110 on device lo:110
/sbin/route add -host 192.168.1.110 dev lo:110
#listing routing info for VIP 192.168.1.110
/bin/netstat -rn

[/sourcecode]

Thats it! You can try to connect to your LVS now from the client by typing

[sourcecode language=”bash”]
telnet 192.168.1.110
[/sourcecode]

The requests should be processed by the director and by the real server rotatory. You can check this by typing

[sourcecode language=”bash”]
/sbin/ipvsadm
[/sourcecode]

at the director. In the output you should see “Active Connections” for both nodes. If that works you can try to connect via http e.g. by typing 192.168.1.110 in your web browser.

If you want to reach your LVS from the internet you can setup IP forwarding from the relevant ports in your DSL router. The ports have to be forwarded to the virtual IP 192.168.1.110. Deactivate all firewalls for testing!