I’ve recently been building a fresh test environment to mimic a live environment as closely as possible using the virtualisation technology that we have here. My design goals are:
- The test environment should have the same network topology as live
- The test web servers should be load balanced the same as the live servers
- The resulting set of machines should be able to be cloned in order to produce multiple test environments
- This has to work on both Hyper-V and VMWare Infrastructure.
The complexity in the solution stems from the fact that this particular solution has a number of SSL websites which are served up by a pair of web servers, SSL sites cannot share an IP address like non-encrypted sites can so there are a large number of IP addresses and load balancing being done.
I’m going to build a Linux server to act as a router and load balancer between my virtualised test environment and the real world, hopefully by using Linux for this piece I can get this working whilst using as few resources as possible on my virtualisation platform and get it up and running quickly and cheaply.
Here’s the environment I’m aiming for:
This article is mainly an aide-mémoire for me as I’ve built a few of these systems now, and each time it takes me half a day to remember how to configure each of the steps, so here’s how I went about building it.
Firstly download Ubuntu 9.10 Server from Ubuntu’s download site. Create a new virtual machine and add two network adapters to it – attach the first adapter (which will become eth0) to the main network and attach the second adapter to your internal network (which will become eth1):
Start the install and choose the default server install. The kernel that gets installed needs to have support for IPVS, the minimum install option (underneath the F4 option) does not have this support in the kernel so I went for the standard install instead. Work through the setup and don’t add any extra software to the server when it asks what packages to add. Once it’s installed and rebooted, we need to log in and install ipvsadm
$ sudo apt-get install ipvsadm
That’s the only extra piece of software we need, so now we just need to configure it.
Switch To Static IP Address
The server is by default installed with a dynamic DHCP IP address, to save any issues with the server being allocated a different address in the future I’ve had a set of addresses on the network reserved and can now assign a static IP address to this machine.
To change the configuration of the network interface we need to edit /etc/network/interfaces, by default it will look like
auto eth0
iface eth0 inet dhcp
To change it to use a static IP address the networking information can be inserted here
auto eth0
iface eth0 inet static
address 172.21.5.200
netmask 255.255.248.0
To get changes that are made to the interfaces file applied we can restart the network:
$ sudo /etc/init.d/networking restart
We can also configure the network interface for the internal network to have the 192.168.1.1 address:
auto eth0
iface eth0 inet static
address 172.21.5.200
netmask 255.255.248.0
auto eth1
iface eth1 inet static
address 192.168.1.1
netmask 255.255.255.0
At this point we can configure the internal windows servers to use this machine as their default gateway:
Configuring Routing/NAT
We now need to teach it to pass traffic between it’s two network interfaces, one on the internal virtual network and the other on the physical network. For this to work IP Forwarding must be enabled in the Kernel, to enable it you run
# sysctl -w net.ipv4.ip_forward=1
This is not a persistent change though, to make the change permanent make the corresponding change in /etc/sysctl.conf, the following will be presented but commented out:
net.ipv4.ip_forward=1
After making the change to sysctl.conf run the following to pick up the change:
# sysctl –p /etc/sysctl.conf
The virtual network is on eth1 with the physical network on eth0, this will allow all the virtual machine to send traffic to the physical network:
# iptables –A FORWARD –i eth1 –j ACCEPT
# iptables –A FORWARD –o eth0 –j ACCEPT
However, in our environment we require the router to perform network address translation as the addresses on the virtual network are not routable from the physical network. So also:
# iptables –t nat –A POSTROUTING –o eth0 –j MASQUERADE
This should then allow the internal virtual machines to ping machines on the physical network. However these settings are not persistent yet, if the router is rebooted this configuration will be lost. So we can save the current configuration to a file and then put that file somewhere handy:
# iptables-save > /etc/iptables.rules
These rules can be loaded when the network interface is brought online by adding an extra line to /etc/network/interfaces:
auto eth0
iface eth0 inet static
address 172.21.5.200
netmask 255.255.248.0
pre-up iptables-restore < /etc/iptables-rules
The routing rules should now be persistent and survive reboots.
Additional IP Addresses
However we need to have multiple IP addresses on the public side of the router so that each one can have an SSL web site associated with it. To add extra addresses to existing interfaces you can create virtual interfaces again by editing /etc/network/interfaces. The definition in the interfaces file now looks like:
auto eth0
iface eth0 inet static
address 172.21.5.200
netmask 255.255.248.0
pre-up iptables-restore < /etc/iptables-rules
auto eth0:0
iface eth0:0 inet static
address 172.21.5.201
netmask 255.255.248.0
Add Load Balancing onto each of the Virtual IPs
Firstly, define the service that is to be exposed load balanced:
# ipvsadm –A –t 172.21.5.201:443 –s rr
The –s rr uses Round Robin scheduling when assigning new connections to servers. Secondly need to add the servers in that are going to handle requests to the service:
# ipvsadm –a –t 172.21.5.201:443 –r 192.168.1.3 –m –w 100
# ipvsadm –a –t 172.21.5.201:443 –r 192.168.1.4 –m –w 100
Again this configuration isn’t permanent, to make these settings apply automatically at boot time the config needs to be placed into /etc/ipvsadm.rules :
# rules file for ipvsadm
–A –t 172.21.5.201:443 –s rr
-a –t 172.21.5.201:443 –r 192.168.1.3 –m –w 100
–a –t 172.21.5.201:443 –r 192.168.1.4 –m –w 100
This can then be rinsed and repeated for eth0:1, eth0:2 etc depending on how many addresses you need.
So that’s it - we’re good to go, we can now deploy our test environment onto the virtual servers and get on with the testing. Users on the main networks can point a browser at https://172.21.5.200 and https://172.21.5.201 (which will get a nice DNS names assigned to them) and the router and load balancer will forward the requests to the virtual server farm.
Remember :
$ man
is your friend :)