Keepalived: Difference between revisions

From Open-Xchange
No edit summary
No edit summary
Line 23: Line 23:


== Configuration example: HTTP ==
== Configuration example: HTTP ==
=== Keepalived configuration file ===


Create a file <pre>/etc/keepalived/keepalived.conf</pre> with following contend (adapt network adresses)
Create a file <pre>/etc/keepalived/keepalived.conf</pre> with following contend (adapt network adresses)
Line 123: Line 125:


== Configuration example: Keepalived for Galera Loadbalancing ==
== Configuration example: Keepalived for Galera Loadbalancing ==
=== Keepalived configuration file ===


Here:
Here:
Line 128: Line 132:
* loadbalancer IP ''10.20.29.174''
* loadbalancer IP ''10.20.29.174''
* Three galera nodes: ''10.20.29.140'', ''10.20.29.142'', ''10.20.29.138''
* Three galera nodes: ''10.20.29.140'', ''10.20.29.142'', ''10.20.29.138''
For the configuration: adjust the router_id, virtual_router_id, and authentication information.


  global_defs {
  global_defs {
Line 142: Line 144:
   # The default state, one should be master, the others should be set to SLAVE.
   # The default state, one should be master, the others should be set to SLAVE.
   state MASTER
   state MASTER
  priority 101
   
   
   # This should be the same on all participating load balancers.
   # This should be the same on all participating load balancers.
   virtual_router_id 19
   virtual_router_id 19
  priority 101
   
   
   # Set the interface whose status to track to trigger a failover.                   
   # Set the interface whose status to track to trigger a failover.                   
Line 161: Line 162:
   # This is the IP address that floats between the loadbalancers.
   # This is the IP address that floats between the loadbalancers.
   virtual_ipaddress {
   virtual_ipaddress {
     10.20.29.174 dev eth0
     10.20.29.174/32 dev eth0
   }
   }
  }
  }
Line 197: Line 198:
  }
  }


=== Networking adjustments ===
=== Adding a second Keepalived node for redundancy ===
 
Set up a second Keepalived node as described above, with the following changes:
 
Change the router_id (to the hostname, for example), the state to BACKUP, and the priority to something lower than the masters priority (e.g. 100).
Make sure the virtual_router_id and authentication information is the same on the backup keepalived node as on the master keepalived node.
 
Now the backup node will notice the master going down and take over. Automatic failback also happens.
 
=== Networking adjustments for the keepalived node ===


The following instructions are valid for debian.
configure in /etc/sysctl.conf: <code>net.ipv4.ip_forward = 1</code>


# For the keepalived node, configure in /etc/sysctl.conf: <code>net.ipv4.ip_forward = 1</code>
=== Networking adjustments for the galera nodes ===
# For the server nodes, use a stanza in the /etc/network/interfaces file. Adjust the IP.


auto lo:0
The galera nodes need the loadbalancer IP configured on some network device in order for galera to be able to bind on this device.
iface lo:0 inet static
 
    address 10.20.30.77
However, creating a fully configured "alias" device is bad, since the galera nodes will pick the loadbalancer IP as primary IP of the node for example for full state transfers (SST). So when trying a SST the galera nodes will try to connect to the loadbalancer on the SST port. This will fail because on the loadbalancer nothing listens on the SST port.
    netmask 255.255.255.255
 
If we instead create a dummy device and only assign an IP to it (without setting all those flags like UP), then galera can bind to the IP, but it won't use the IP as its primary IP. A configuration like this can be created using the following trick. Ad dsome pre-up, post-up, pre-down, post-down lines to the /etc/network/interfaces file as follows:
 
allow-hotplug eth0
iface eth0 inet dhcp
     pre-up echo 1 > /proc/sys/net/ipv4/conf/all/arp_ignore
     pre-up echo 1 > /proc/sys/net/ipv4/conf/all/arp_ignore
     pre-up echo 2 > /proc/sys/net/ipv4/conf/all/arp_announce
     pre-up echo 2 > /proc/sys/net/ipv4/conf/all/arp_announce
     post-up /sbin/route add -host 10.20.30.77 dev lo:0
     post-up ip addr add 10.20.29.174/32 dev dummy0
     pre-down /sbin/route del -host 10.20.30.77 dev lo:0
     pre-down ip addr del 10.20.29.174/32 dev dummy0
    # reset to defaults
     post-down echo 0 > /proc/sys/net/ipv4/conf/all/arp_ignore
     post-down echo 0 > /proc/sys/net/ipv4/conf/all/arp_ignore
     post-down echo 0 > /proc/sys/net/ipv4/conf/all/arp_announce
     post-down echo 0 > /proc/sys/net/ipv4/conf/all/arp_announce
The effect of those settings should be self-explanatory. How to do this on other operating systems needs to be documented here.

Revision as of 08:00, 13 August 2013

Keepalived Loadbalancer

Introduction

this page contains a basic description about how to set up keepalived for Open-Xchange cluster. This example is to work on debian systems. Keepalived mode is Direct Routing.

It is required to have ox servers and loadbalancer connected to the same switch or hub and that there is no filter for network packages between (some virtualization system do filter, too), so that MAC rewriting works.

For more information please see: www.keepalived.org

Software installation

Packages are installed using

test1:~# apt-get install keepalived 

Keepalived requires some kernel modules to be loaded. They are loaded by the ipvsadmm service. So we enable it using dpkg-reconfigure:

dpkg-reconfigure ipvsadm

Answer the questions with "Yes" ("load ... at boot") and then "backup" for "Daemon method".

Configuration example: HTTP

Keepalived configuration file

Create a file

/etc/keepalived/keepalived.conf

with following contend (adapt network adresses)

global_defs {
    router_id OX
}

vrrp_sync_group OX_GROUP {
    group {
        OX_GOUP
    }
}

vrrp_instance OX_VRRP {
    state BACKUP
    interface eth0
    garp_master_delay 10
    virtual_router_id 10
    priority 101
    nopreempt
    advert_int 1
    authentication {
        auth_type AH   # Simple 'PASS' can use
        auth_pass 1234 # example password '1234' 
    }
    virtual_ipaddress {
        10.20.30.77/24 brd 10.20.30.255 dev eth0 # virtual service ip 10.20.30.67
    }
    virtual_ipaddress_excluded {
    }
}

virtual_server_group OX_HTTP {
        10.20.30.77 80         # virtual ip and port 80
}

virtual_server_group OX_OL_PUSH {
        10.20.30.77 44335      # VIP VPORT
}

virtual_server group OX_HTTP {
    delay_loop 3
    lvs_sched  rr
    lvs_method DR
    protocol   TCP
    virtualhost 10.20.30.77

    real_server 10.20.30.123 80 {
        weight 1
        inhibit_on_failure
        HTTP_GET {
            url {
                path /servlet/TestServlet
                status_code 200
            } 
            connect_port 80
            connect_timeout 10
        }
    }

    real_server 10.20.30.321 80 {
        weight 1
        inhibit_on_failure
        HTTP_GET {
            url {
                path /servlet/TestServlet
                status_code 200
            }
            connect_port 80
            connect_timeout 10
        }
    } 
}

virtual_server group OX_OL_PUSH {
    delay_loop 3
    lvs_sched  rr
    lvs_method DR
    protocol   UDP

    real_server 10.20.30.123 44335 {
        weight 1
        inhibit_on_failure
	  TCP_CHECK {
                 connect_port 9999
		  connect_timeout 5
        }
    }

    real_server 10.20.30.321 44335 {
        weight 1
        inhibit_on_failure
        TCP_CHECK {
                 connect_port 9999
		  connect_timeout 5
        }
    }
}

Configuration example: Keepalived for Galera Loadbalancing

Keepalived configuration file

Here:

  • loadbalancer IP 10.20.29.174
  • Three galera nodes: 10.20.29.140, 10.20.29.142, 10.20.29.138
global_defs {
  # This should be unique.
  router_id galera-lb
}

vrrp_instance mysql_pool {
  # The interface we listen on.
  interface eth0

  # The default state, one should be master, the others should be set to SLAVE.
  state MASTER
  priority 101

  # This should be the same on all participating load balancers.
  virtual_router_id 19

  # Set the interface whose status to track to trigger a failover.                   
  track_interface {           
    eth0
  }

  # Password for the loadbalancers to share.
  authentication {
    auth_type PASS
    auth_pass Twagipmiv3
  }

  # This is the IP address that floats between the loadbalancers.
  virtual_ipaddress {
   10.20.29.174/32 dev eth0
  }
}

# Here we add the virtal mysql node
virtual_server 10.20.29.174 3306 {
  delay_loop 6
  # Round robin, but you can use whatever fits your needs.
  lb_algo rr
  lb_kind DR
  protocol TCP

  # For each server add the following. 
  real_server 10.20.29.140 3306 {
    weight 10
    MISC_CHECK {
      misc_path "/etc/keepalived/galera-checker.pl 10.20.29.140"
      misc_timeout 5
    }
  }
  real_server 10.20.29.142 3306 {
    weight 11
    MISC_CHECK {
      misc_path "/etc/keepalived/galera-checker.pl 10.20.29.142"
      misc_timeout 5
    }
  }
  real_server 10.20.29.138 3306 {
    weight 12
    MISC_CHECK {
      misc_path "/etc/keepalived/galera-checker.pl 10.20.29.138"
      misc_timeout 5
    }
  }
}

Adding a second Keepalived node for redundancy

Set up a second Keepalived node as described above, with the following changes:

Change the router_id (to the hostname, for example), the state to BACKUP, and the priority to something lower than the masters priority (e.g. 100). Make sure the virtual_router_id and authentication information is the same on the backup keepalived node as on the master keepalived node.

Now the backup node will notice the master going down and take over. Automatic failback also happens.

Networking adjustments for the keepalived node

configure in /etc/sysctl.conf: net.ipv4.ip_forward = 1

Networking adjustments for the galera nodes

The galera nodes need the loadbalancer IP configured on some network device in order for galera to be able to bind on this device.

However, creating a fully configured "alias" device is bad, since the galera nodes will pick the loadbalancer IP as primary IP of the node for example for full state transfers (SST). So when trying a SST the galera nodes will try to connect to the loadbalancer on the SST port. This will fail because on the loadbalancer nothing listens on the SST port.

If we instead create a dummy device and only assign an IP to it (without setting all those flags like UP), then galera can bind to the IP, but it won't use the IP as its primary IP. A configuration like this can be created using the following trick. Ad dsome pre-up, post-up, pre-down, post-down lines to the /etc/network/interfaces file as follows:

allow-hotplug eth0
iface eth0 inet dhcp
    pre-up echo 1 > /proc/sys/net/ipv4/conf/all/arp_ignore
    pre-up echo 2 > /proc/sys/net/ipv4/conf/all/arp_announce
    post-up ip addr add 10.20.29.174/32 dev dummy0
    pre-down ip addr del 10.20.29.174/32 dev dummy0
    post-down echo 0 > /proc/sys/net/ipv4/conf/all/arp_ignore
    post-down echo 0 > /proc/sys/net/ipv4/conf/all/arp_announce