Load balancing howto Copyright 2002 William Stearns Overall idea: C1 C2 C3 modem1----isp1--internet--\ | | | / \ ClientLan-ClientRouter TunnelServer--Internet-> \ / modem2----isp2--internet--/ - C1, C2, and C3 keep permanent IP addresses on TunnelServer's lan, or we masquerade outbound traffic on TunnelServer's IP address. - Client Router dials up N modems to 1..N ISP's. Client Router now has N IP addresses. With each connection, ClientRouter must connect to TunnelServer to inform it of the new IP address to use for return packets. - ClientRouter round robin's outbound packets though any of the modem links, tunneling each packet off to TunnelServer. TunnelServer unwraps the packet and sends it on its way. - Response packets come back and TunnelServer wraps them up in packets heading back to one of ClientRouter's IP addresses. - Before a connection drops, ClientRouter needs to tell TunnelServer to remove the IP address from TunnelServer's target list. Also, to handle unexpected drops, TunnelServer needs to verify that each address is still live. An alternate approach is to have the ClientRouter regularly connect to TunnelServer to update all live IP addresses. GRE = proto 47 IPIP = proto 94 ENCAP = proto 98 sudo iptables -A INPUT -p icmp -i lo -s 127.0.0.1 -m recent --name TUNNEL --update -j LOG [root@sparrow ipt_recent]# cat DEFAULT [root@sparrow ipt_recent]# echo +127.0.0.1 >DEFAULT [root@sparrow ipt_recent]# cat DEFAULT src=127.0.0.1 ttl: 90 last_seen: 62630302 oldest_pkt: 1 last_pkts: 62630302 [root@sparrow ipt_recent]# cat DEFAULT src=127.0.0.1 ttl: 64 last_seen: 62631744 oldest_pkt: 9 last_pkts: 62630302, 62631398, 62631398, 62631398, 62631398, 62631644, 62631644, 62631744, 62631744 Advantages: - Ability to keep connections open even when laptop disconnects/reconnects or moves to new networks. The connections in question should have no traffic flowing over them while the laptop is completely disconnected from the Internet, or the connections may get killed. Keepalives, in particular, should be turned off. (See /proc/sys/net/ipv4/tcp_keepalive_{intvl,probes,time} ; documentation from lartc at the end. ssh keepalives can be turned off in /etc/ssh/{ssh,sshd}_config with the line: KeepAlive no - Bonding 2 or more connections of any form into a shared pipe with linear scalability. - No modification of client apps, works with all IP traffic. Disadvantages: - Increased latency. (cover 3 vs. 4 segment appraoch if ISP doesn't do egress filtering). - Traffic to the tunnel server itself can't be tunneled, hence can't be load balanced or persist across connections (give tunnel server a second IP if this is important). Traffic to the tunnel server is only sent through the first interface. Note: - Traceroute traffic shows the first hop as your tunnel server, and continues the path out from there. Client handed: - short name for tunnel ($TunnelName) Client returns: - transient IP(s) (none means tear down tunnel) Server handed: - short name for tunnel ($TunnelName) - transient IP(s) (none means tear down tunnel) Pulled from identical /etc/tunnel.conf on client and server - client permanent IP ($ClientPermanentIP) - tunnel server IP ($TunnelServerIP) - complete list of candidate interface(s) (none, or none active, means tear down tunnel), ($ClientCandidateIFs) To start: if TransientIPs=`ssh localhost /sbin/client_tunnel start -n wls` ; then ssh MyTunnelServer /sbin/server_tunnel start -n wls $TransientIPs else echo Setting up client failed, not setting up server. fi OR ssh MyTunnelServer /sbin/server_tunnel start -n wls `ssh localhost /sbin/client_tunnel start -n wls` To stop: ssh MyTunnelServer /sbin/server_tunnel stop -n wls ssh localhost /sbin/client_tunnel stop -n wls Assumed that default route is through tunnel, route to tunnel server through all candidate interfaces. Handle interfaces that need a gateway later. Assumed approximately equal bandwidth for each connection; weighting in a future version. Suggest routing each isp's servers through their interface. From http://lartc.org/HOWTO//cvs/2.4routing/lartc.html proc/sys/net/ipv4/tcp_keepalive_time How often TCP sends out keepalive messages when keepalive is enabled. Default: 2hours. /proc/sys/net/ipv4/tcp_keepalive_intvl How frequent probes are retransmitted, when a probe isn't acknowledged. Default: 75 seconds. /proc/sys/net/ipv4/tcp_keepalive_probes How many keepalive probes TCP will send, until it decides that the connection is broken. Default value: 9. Multiplied with tcp_keepalive_intvl, this gives the time a link can be nonresponsive after a keepalive has been sent.