LVS is a software load balancing solution. It’s open-source software, built directly in Linux kernel and it’s free.
The director (load balancer) can be in one DC, and the real servers are in different DCs. The director only needs good bandwidth, Pentium 4 or even P3 is fine since it’s Layer 4 switching (less overhead than Layer 7, eg: HAProxy). The incoming traffic flows from Client -> Director -> Worker. The returning traffic: Worker -> Client. As you can see, the director has a much higher throughput since it only handles incoming requests. The return packets come directly from the workers. We current manage several LVS setups. One example: 3 directors, 12 real servers, in over 5 different DCs spanning across US and Europe. It’s quite easy to set up and manage.
Reference: LVS-Tun is an LVS original. It is based on LVS-DR and has the same high scalability/throughput of LVS-DR. LVS-Tun can be used with realservers that can tunnel (==IPIP encapsulation). The director encapsulates the request packet inside an IPIP packet before sending it to the realserver. The realserver must be able to decapsulate the IPIP packet. Initially only Linux could decapsulate IPIP packets, but recently FreeBSD and W2K can now do it too (hmm 2005, I think Microsoft has dropped support for IPIP). With LVS-DR, the realservers can have almost any OS.
Unlike LVS-DR, with LVS-Tun the realservers can be on a network remote from the director, and can each be on separate networks. Thus the realservers could be in different countries (e.g. a set of ftp mirror sites for a project). If this is the case, the realservers will be generating reply packets with VIP:port->CIP (where port is the LVS’ed service). Not being on the VIP network, the routers for the realservers will have to be programmed to accept outgoing packets with src_addr=VIP:port. Routers normally drop these packets as an anti-spoofing measure. If you aren’t in control of the routers, you’ll just have to inform the people who are, that packets from VIP:port are valid for your business. If they don’t want to help you with your business, then you should find another provider who will. Read more here and here
To detect if the ISPs allow LVS-TUN, follow the tests on this page, more specifically, this test:
realserver# traceroute -s VIRTUAL_IP -n CLIENT_IP 1 * * * 2 * * * 3 * * * 4 * * * 5 * * * 6 * * * 7 * * * 8 * * *
Be patient and wait on the director to see something similar to the following
director# tcpdump -ln host CLIENT_IP tcpdump: listening on eth0 19:20:20.310162 CLIENT_IP > VIRTUAL_IP: icmp: CLIENT_IP udp port 33483 unreachable 19:22:40.639844 CLIENT_IP > VIRTUAL_IP: icmp: CLIENT_IP udp port 33511 unreachable 19:22:45.641061 CLIENT_IP > VIRTUAL_IP: icmp: CLIENT_IP udp port 33512 unreachable 19:23:30.664315 CLIENT_IP > VIRTUAL_IP: icmp: CLIENT_IP udp port 33521 unreachable
If you don’t see anything response on the director, it might be the realserver cannot get any packet out to the client because the ISP’s router dropped these packets.
It is very important that ISPs see the demand/request for LVS-TUN setups to distinguish it from malicious network attacks. Security is good but cannot be too strict or rigid to have flexibility, growth for business. If you have experienced setting up LVS-TUN with other ISPs, webhosting companies, please let me know to add to the list.
List of ISPs support LVS-TUN (allow outgoing spoofed-yet-valid packets for the realservers):
- LayeredTech: at Savvis building in Dallas, their other DataBank DC blocks this. Currently working with LT to unblock. Updated: LT is very accommodating for their clients, they exclude our load balancer’s IP address in the router filter list.
- Hivelocity: blocked but then unblocked, willing to make an exception.
- 1paket at Lambdanet in Germany
- SoftLayer: custom router setup
- WebNX in LA
List of ISPs do NOT support LVS-TUN (drop these packets and are not willing to make exception):
- ThePlanet: denied, not willing to make exception in network filter for this type of packets, against their AUP
Leave a Reply