m-vychytavky: LOADBALANCING

03.02.2006 14:28 | blackhole

V tomto seriali, vam ukazeme niektore pekne vychytavky, ktore vam mozu pomoct pri administracii linuxu. V nasledujucom texte je popisane ako vyuzit linux na loadbalancing viacerych liniek do netu.

Prerequisities:
1. iproute2
2. kernel s podporou sietoveho multipathu
3. openvpn alebo iny tunnel software (ak netreba sifrovany tunel staci nam pouzit "ip tunnel ..." prikazy
4. (len v pripade per-packet balancingu) "-m nth" a "-j ROUTE" patch do iptables + jadra

MAPA:

   intenet isp2

      |

    |gw2|

      |

      |tun1

      |eth2                                                             tun1

---------------tun0                                                     tun0------------------

| HOME SERVER |eth1    chello modem          internet isp 1             eth0| HOSTING SERVER | 

|    linux    |-----------OOOOOOO-------+@#$%@#$TQGTDY$W%YQERGQ+------------|      linux     |

---------------                         |                      |            ------------------

       |eth0                            |                      |

       |                            chello router         hosting router

   ---------                            gw1                    gw

      LAN

HOME SERVER

eth2: 172.16.3.2/24
eth1: 172.16.0.2/24
eth0: 192.168.0.110/24
gw1: 172.16.0.1 (chello router)
gw2: 172.16.3.1
tun0: 10.1.0.11
tun1: 10.1.0.10

HOSTING SERVER

eth0: 172.16.2.2/24
gw: 172.16.2.1 (hosting router)
tun0: 10.1.0.2
tun1: 10.1.0.1

LAN: 192.168.0.0/24

Najprv si pripravime routing. Tak aby packety odchadzajuce s adresou eth0 naozaj aj odisli cez eth0 a packety odchadzajuce s adresou eth1 odchadzali cez eth1. Za normalnych okolnosti by isli von interfejsom na ktorom je defaultroue. Cize ked by sme z internetu pingli eth1, tak icmp echo reply nepojde spat cez eth1 interface, ale cez eth0 na ktorom je zhodou okolnosti defaultrouta. Riesenie je policy routing, k tomu hned po priprave tunnelov.

nakonfigurujeme teda dva tunely, napriklad cez openvpn:

HOSTING SERVER openvpn config:

bash-2.05b# cat /etc/openvpn/tun1.conf
dev tun
ifconfig 10.1.0.1 10.1.0.10
up ./tun1.up
secret static.key
proto tcp-server
port 5000
user nobody
group nobody
ping 15
verb 3
comp-lzo
link-mtu 1500
bash-2.05b# cat /etc/openvpn/tun0.conf
dev tun
local 172.16.2.2
remote 72.16.0.2
ifconfig 10.1.0.2 10.1.0.11
up ./tun0.up
secret static.key
proto tcp-client
port 5001
user nobody
group nobody
ping 15
verb 3
link-mtu 1500

HOME SERVER openvpn config:

bash-2.05b# cat /etc/openvpn/tun1.conf
dev tun
local 172.16.3.2
remote 172.16.2.2
ifconfig 10.1.0.10 10.1.0.1
up ./tun1.up
secret static.key
proto tcp-client
port 5000
user nobody
group nobody
ping 15
verb 3
comp-lzo
link-mtu 1500
bash-2.05b# cat /etc/openvpn/tun0.conf
dev tun
ifconfig 10.1.0.11 10.1.0.2
up ./tun0.up
secret static.key
proto tcp-server
port 5001
user nobody
group nobody
ping 15
verb 3
comp-lzo
link-mtu 1500

poznamocka: tun0 mam doma ako "tcp-server", pretoze openvpn ma bug kedy tcp-client ignoruje adresu na ktoru sa ma bingnut, co sposobovalo, ze oba tunely mi komunikovali z adresou jedneho interfejsu a tym padom boli routovane len cez jednu linku.

takze teraz uz spominany policy routing:

do /etc/iproute2/rt_tables dopiseme toto:
10 openvpn
20 isp1
30 isp2

packety zo src adresami jednotlivych sietoviek budeme routovat aj fyzicky cez ne. dalej povieme domacemu kernelu ze pre packety z LANky ma pouzit routovaciu tabulku "openvpn" a zaroven nastavime v tejto routovacej tabulke multipath defaultroutu cez tunely:

bash-2.05b# ip route add default table isp1 via 172.16.0.1 dev eth1
bash-2.05b# ip route add default table isp2 via 172.16.3.1 dev eth2
bash-2.05b# ip rule add from 192.168.0.0/24 table openvpn
bash-2.05b# ip route add default table openvpn nexthop via 10.1.0.1 dev tun1 weight 1 nexthop via 10.1.0.2 dev tun0 weight 1

hosting kernelu tiez musime troska upravit routing:
bash-2.05b# ip route add 192.168.0.0/24 scope global nexthop via 10.1.0.10 dev tun1 weight 1 nexthop via 10.1.0.11 dev tun0 weight 1

v tomto momente mame chodivy per-destination balancing. to v praxi znamena, ze ked komunikujete s jednym servrom, tak cela komunikacia pojde cez jednu linku. pri komunikacii s dalsim servrom uz moze ist komunikacia druhou linkou, alebo opat tou istou (ako sa podari [podla istych pravidiel v kereli, na mojej nove 2.6.12 sa to da uz troska aj tweakovat])

este odporucam mat aspon takyto basic setup na servroch:

echo 1 > /proc/sys/net/ipv4/ip_forward
iptables -A FORWARD -m state --state RELATED, ESTABLISHED -j ACCEPT
iptables -A FORWARD -s 192.168.0.0/24 -j ACCEPT

tym povolime forwarding smerom z LANky, na HOSTING SERVERi este musime tomuto trafficu pridelit nejaku vonkajsiu adresu miesto privatnych 192.168.0.0/24:
iptables -A POSTROUTING -t nat -s 192.168.0.0/24 -j MASQUERADE

taaak, to by bola prva cast, idem si dat nejaky junk food a napisem ako spravit per packet balancing, kde sa kazdy packet trafficu rozklada podla nasich poziadaviek na jednotlive linky.

********************************************************************************************************

takze primitivny per-packet balancing mozme spravit pomocou patchu do jadra a iptables z bezneho iptables patch-o-matic-ng patchu. konkretne nam treba nth patch a ROUTE. po rozbaleni patch-o-matic-ng spustime ./runme extra a povieme co by sme radi. nasledne treba este raz prekompilovat iptables (ano patchomatic potrebuje zdrojaky iptables) a samozrejme aj kernel.

povieme kernelu na HOME SERVERi aby kazdy prvy a druhy packet posielal cez isp2 a kazdy treti cez isp1

bash-2.05b# iptables -A POSTROUTING -s 192.168.0.0/24 -d ! 192.168.0.0/24 -t mangle -m nth --counter 0 --every 3 --packet 0 -j ROUTE --oif tun1 --gw 10.1.0.1
bash-2.05b# iptables -A POSTROUTING -s 192.168.0.0/24 -d ! 192.168.0.0/24 -t mangle -m nth --counter 1 --every 3 --packet 1 -j ROUTE --oif tun1 --gw 10.1.0.1
bash-2.05b# iptables -A POSTROUTING -s 192.168.0.0/24 -d ! 192.168.0.0/24 -t mangle -m nth --counter 2 --every 3 --packet 2 -j ROUTE --oif tun0 --gw 10.1.0.2

a teraz to iste pre HOSTING SERVER:

iptables -A POSTROUTING -d 192.168.0.0/24 -t mangle -m nth --counter 0 --every 3 --packet 0 -j ROUTE --oif tun0 --gw 10.1.0.11
iptables -A POSTROUTING -d 192.168.0.0/24 -t mangle -m nth --counter 1 --every 3 --packet 1 -j ROUTE --oif tun1 --gw 10.1.0.10
iptables -A POSTROUTING -d 192.168.0.0/24 -t mangle -m nth --counter 2 --every 3 --packet 2 -j ROUTE --oif tun1 --gw 10.1.0.10

samozrejme vynechanim posledneho riadku na oboch servroch by sme docielili rozdelovanie trafficu 1:1, ale kedze ja mam jednu linku dvojnasobnej kapacity druhej tak som to rozdelil 1:2

neostava mi nic ine len popriat vela stastia dajte mi vediet ako vam to chodi, ja som bol trocha sklamany z overheadu (trocha pomohlo vyhodit lzo-comp z configu openvpn, ale aj tak..)

ak bude zase trocha casu tak skusim aj patchnutu zebru, vraj by mala tiez slusne balancovat..

takto nejak to vyzera v praxi na mojom HOME SERVERi (miesto eth2 mam eth0 spojenu s LANkou a inym gatewayom do internetu cez isp2 (bolo na to treba este vynat traffic HOMESERVRA do lokalky a na HOSTINGSERVER z balancingu pretoze, aj samotne tunnel packety ktore by mali chodit len cez isp2 by sa rozhadzovali aj na druhu linku:

iptables -A POSTROUTING -t mangle -s 192.168.0.110 -d 192.168.0.0/24 -j ACCEPT ; iptables -A POSTROUTING -t mangle -s 192.168.0.110 -d 172.16.2.2 -j ACCEPT:

Averages for the last 3014 msec

 ----------------------------------------------------------

| IF      |Input                   |Output                 |

 ----------------------------------------------------------

|  eth0   |  1544 kbps|   302 pk/s||  2102 kbps|   328 pk/s|

|  eth1   |   706 kbps|   105 pk/s||   101 kbps|    76 pk/s|

|    lo   |     0 kbps|     0 pk/s||     0 kbps|     0 pk/s|

| tunl0   |     0 kbps|     0 pk/s||     0 kbps|     0 pk/s|

|  gre0   |     0 kbps|     0 pk/s||     0 kbps|     0 pk/s|

|  tun0   |   626 kbps|    84 pk/s||    38 kbps|    61 pk/s|

|  tun1   |  1283 kbps|   173 pk/s||    77 kbps|   122 pk/s|

----------------------------------------------------------

napisane povodne pre: kyberia.sk