« Archives in April, 2006

LVS-Tun & ISPs

LVS is a software load balancing solution. It’s open-source software, built directly in Linux kernel and it’s free.

The director (load balancer) can be in one DC, and the real servers are in different DCs. The director only needs good bandwidth, Pentium 4 or even P3 is fine since it’s Layer 4 switching (less overhead than Layer 7, eg: HAProxy). The incoming traffic flows from Client -> Director -> Worker. The returning traffic: Worker -> Client. As you can see, the director has a much higher throughput since it only handles incoming requests. The return packets come directly from the workers. We current manage several LVS setups. One example: 3 directors, 12 real servers, in over 5 different DCs spanning across US and Europe. It’s quite easy to set up and manage.

Reference: LVS-Tun is an LVS original. It is based on LVS-DR and has the same high scalability/throughput of LVS-DR. LVS-Tun can be used with realservers that can tunnel (==IPIP encapsulation). The director encapsulates the request packet inside an IPIP packet before sending it to the realserver. The realserver must be able to decapsulate the IPIP packet. Initially only Linux could decapsulate IPIP packets, but recently FreeBSD and W2K can now do it too (hmm 2005, I think Microsoft has dropped support for IPIP). With LVS-DR, the realservers can have almost any OS.

Unlike LVS-DR, with LVS-Tun the realservers can be on a network remote from the director, and can each be on separate networks. Thus the realservers could be in different countries (e.g. a set of ftp mirror sites for a project). If this is the case, the realservers will be generating reply packets with VIP:port->CIP (where port is the LVS’ed service). Not being on the VIP network, the routers for the realservers will have to be programmed to accept outgoing packets with src_addr=VIP:port. Routers normally drop these packets as an anti-spoofing measure. If you aren’t in control of the routers, you’ll just have to inform the people who are, that packets from VIP:port are valid for your business. If they don’t want to help you with your business, then you should find another provider who will. Read more here and here

To detect if the ISPs allow LVS-TUN, follow the tests on this page, more specifically, this test:

realserver# traceroute -s VIRTUAL_IP -n CLIENT_IP
 1  * * *
 2  * * *
 3  * * *
 4  * * *
 5  * * *
 6  * * *
 7  * * *
 8  * * *

Be patient and wait on the director to see something similar to the following

director# tcpdump -ln host CLIENT_IP
tcpdump: listening on eth0
19:20:20.310162 CLIENT_IP > VIRTUAL_IP: icmp: CLIENT_IP udp port 33483 unreachable
19:22:40.639844 CLIENT_IP > VIRTUAL_IP: icmp: CLIENT_IP udp port 33511 unreachable
19:22:45.641061 CLIENT_IP > VIRTUAL_IP: icmp: CLIENT_IP udp port 33512 unreachable
19:23:30.664315 CLIENT_IP > VIRTUAL_IP: icmp: CLIENT_IP udp port 33521 unreachable

If you don’t see anything response on the director, it might be the realserver cannot get any packet out to the client because the ISP’s router dropped these packets.

It is very important that ISPs see the demand/request for LVS-TUN setups to distinguish it from malicious network attacks. Security is good but cannot be too strict or rigid to have flexibility, growth for business. If you have experienced setting up LVS-TUN with other ISPs, webhosting companies, please let me know to add to the list.

List of ISPs support LVS-TUN (allow outgoing spoofed-yet-valid packets for the realservers):

  1. LayeredTech: at Savvis building in Dallas, their other DataBank DC blocks this. Currently working with LT to unblock. Updated: LT is very accommodating for their clients, they exclude our load balancer’s IP address in the router filter list.
  2. Hivelocity: blocked but then unblocked, willing to make an exception.
  3. 1paket at Lambdanet in Germany
  4. SoftLayer: custom router setup
  5. WebNX in LA

List of ISPs do NOT support LVS-TUN (drop these packets and are not willing to make exception):

  1. ThePlanet: denied, not willing to make exception in network filter for this type of packets, against their AUP

Dual Opteron 248

Just got another ad server, Dual Opteron 248, 2GB RAM, 36 G SCSI. Running the test suite. This seems like a powerful machine. AMD is winning the heart and mind of the web hosting world.

CRC32 hash collision

I was trying to use CRC32() to uniquely identify distinctive domain names because it’s probably the most economical in MySQL datatype (int only takes 4 bytes) comparing to MD5() as a 32-char string. However, it seems hash collisions with CRC32 occur too frequently. Just in a set of about 800k string, got a bunch of duplicate ids. For eg:

fc0591 => 1009521187
123rainerbommert => 1009521187

Also, using the length of the string to make additional distinction does not seem to help. For eg, both has length of 9, same CRC32 but different strings

a1sellers => 3605292603
advertees => 3605292603

Some stats: out of over 7M domains, about 6200 collisions (duplicate CRC32 hashes, mostly twice, but 3 cases 3 strings yield the same hash)

Housing Booms, Busts, and Where Opportunities Occur

Robert Kiyosaki “Is there a real estate bubble?” That’s the question I’m asked repeatedly. When I reply honestly — “I hope so” — my questioners’ fear occasionally turns into anger. There are seven stages of a financial bubble.

read more | digg story