Keepalived - Revision history

Dominik.epple: /* Keepalived configuration (with health checks) */

2017-09-26T06:44:55Z

Keepalived configuration (with health checks)

← Older revision		Revision as of 06:44, 26 September 2017
Line 389:		Line 389:
	==== Keepalived configuration (with health checks) ====		==== Keepalived configuration (with health checks) ====

	We now configure health checks in <code>keepalived.conf</code>. The recommended way is to configure a custom [[Clustercheck\|clustercheck~~.ox~~]] service on the server nodes and configure it in <code>keepalived.conf</code> on the loadbalancer as follows. Each <code>real_server</code> gets a <code>HTTP_GET</code> child element like		We now configure health checks in <code>keepalived.conf</code>. The recommended way is to configure a custom [[Clustercheck\|clustercheck]] service on the server nodes and configure it in <code>keepalived.conf</code> on the loadbalancer as follows. Each <code>real_server</code> gets a <code>HTTP_GET</code> child element like

	real_server 10.0.0.1 3306 {		real_server 10.0.0.1 3306 {

Dominik.epple at 12:32, 22 September 2017

2017-09-22T12:32:35Z

← Older revision		Revision as of 12:32, 22 September 2017
Line 381:		Line 381:
	The example has been using <code>TUN</code>; for <code>DR</code>, just replace <code>TUN</code> by <code>DR</code> in the <code>virtual_server</code> definitions / <code>lb_kind</code> line.		The example has been using <code>TUN</code>; for <code>DR</code>, just replace <code>TUN</code> by <code>DR</code> in the <code>virtual_server</code> definitions / <code>lb_kind</code> line.

	For <code>NAT</code>, as this method does not use virtual IPs, you can disable/comment out the <code>virtual_ipaddress</code> section in the <code>vrrp_instance</code> section, and you change the virtual servers to run on the native IP on extra ports like 5506 and 5507. And of course you change the <code>lb_kind</code> to <code>NAT</code>,		For <code>NAT</code>, as this method does not use virtual IPs, you can disable/comment out the <code>virtual_ipaddress</code> section in the <code>vrrp_instance</code> section, and you change the virtual servers to run on the native IP on extra ports like 5506 and 5507. And of course you change the <code>lb_kind</code> to <code>NAT</code>.

	After a <code>service keepalived restart</code> you should be able to execute the same client connectivity tests as shown above. (Remember to cleanly unconfigure your manual setup before in order to not measure false success.)		After a <code>service keepalived restart</code> you should be able to execute the same client connectivity tests as shown above. (Remember to cleanly unconfigure your manual setup before in order to not measure false success.)

Dominik.epple: /* Keepalived configuration (with health checks) */

2017-09-22T12:11:38Z

Keepalived configuration (with health checks)

← Older revision		Revision as of 12:11, 22 September 2017
Line 389:		Line 389:
	==== Keepalived configuration (with health checks) ====		==== Keepalived configuration (with health checks) ====

	We now configure health checks in <code>keepalived.conf</code>. The recommended way is to configure a custom [Clustercheck clustercheck.ox] service and configure it as follows. Each <code>real_server</code> gets a <code>HTTP_GET</code> child element like		We now configure health checks in <code>keepalived.conf</code>. The recommended way is to configure a custom [[Clustercheck\|clustercheck.ox]] service on the server nodes and configure it in <code>keepalived.conf</code> on the loadbalancer as follows. Each <code>real_server</code> gets a <code>HTTP_GET</code> child element like

	real_server 10.0.0.1 3306 {		real_server 10.0.0.1 3306 {
Line 406:		Line 406:

	If you only have one (active) ''loadbalancer'' at each given time, you might want to stick with the standard way of letting Keepalived decide (persistently) for a master node. It might have faster failover/recovery times, and avoid (unnecessary) failbacks.		If you only have one (active) ''loadbalancer'' at each given time, you might want to stick with the standard way of letting Keepalived decide (persistently) for a master node. It might have faster failover/recovery times, and avoid (unnecessary) failbacks.

			As usual, thorough testing after config changes is mandatory.

	=== Adding a second Keepalived node for redundancy ===		=== Adding a second Keepalived node for redundancy ===

Dominik.epple at 12:09, 22 September 2017

2017-09-22T12:09:00Z

Show changes

Dominik.epple: /* Loadbalancer */

2017-09-21T14:18:07Z

Loadbalancer

← Older revision		Revision as of 14:18, 21 September 2017
Line 163:		Line 163:
	# Round-Robin / read instance		# Round-Robin / read instance
	/sbin/ipvsadm -A -t 10.0.0.4:5506 -s rr		/sbin/ipvsadm -A -t 10.0.0.4:5506 -s rr
	/sbin/ipvsadm -a -t 10.0.0.4:5506 -r 10.0.0.1 -m -w 10		/sbin/ipvsadm -a -t 10.0.0.4:5506 -r 10.0.0.1:3306 -m -w 10
	/sbin/ipvsadm -a -t 10.0.0.4:5506 -r 10.0.0.2 -m -w 10		/sbin/ipvsadm -a -t 10.0.0.4:5506 -r 10.0.0.2:3306 -m -w 10
	/sbin/ipvsadm -a -t 10.0.0.4:5506 -r 10.0.0.3 -m -w 10		/sbin/ipvsadm -a -t 10.0.0.4:5506 -r 10.0.0.3:3306 -m -w 10
	# Persistent / write instance		# Persistent / write instance
	/sbin/ipvsadm -A -t 10.0.0.4:5507 -s rr -p 86400 -M 0.0.0.0		/sbin/ipvsadm -A -t 10.0.0.4:5507 -s rr -p 86400 -M 0.0.0.0
	/sbin/ipvsadm -a -t 10.0.0.4:5507 -r 10.0.0.1 -m -w 10		/sbin/ipvsadm -a -t 10.0.0.4:5507 -r 10.0.0.1:3306 -m -w 10
	/sbin/ipvsadm -a -t 10.0.0.4:5507 -r 10.0.0.2 -m -w 10		/sbin/ipvsadm -a -t 10.0.0.4:5507 -r 10.0.0.2:3306 -m -w 10
	/sbin/ipvsadm -a -t 10.0.0.4:5507 -r 10.0.0.3 -m -w 10		/sbin/ipvsadm -a -t 10.0.0.4:5507 -r 10.0.0.3:3306 -m -w 10

	Note: For DR and TUN, you need to restart the MySQL service after the networking adjustments; otherwise, the MySQL daemon will not accept packages with the virtual IP as target IP. This has caused a lot of wasted time to quite some people.		Note: For DR and TUN, you need to restart the MySQL service after the networking adjustments; otherwise, the MySQL daemon will not accept packages with the virtual IP as target IP. This has caused a lot of wasted time to quite some people.

Dominik.epple at 06:41, 21 September 2017

2017-09-21T06:41:55Z

← Older revision		Revision as of 06:41, 21 September 2017
Line 52:		Line 52:
	The Tunneling method generally works better in modern virtualized / cloud environments.		The Tunneling method generally works better in modern virtualized / cloud environments.

	==== NAT ~~method~~ ====		==== NAT ====

	Here LVS substitutes the target IP from incoming packages from its own (virtual / loadbalancer) IP to the IP of the designated server, and re-sends the package.		Here LVS substitutes the target IP from incoming packages from its own (virtual / loadbalancer) IP to the IP of the designated server, and re-sends the package.

Dominik.epple at 16:05, 20 September 2017

2017-09-20T16:05:06Z

← Older revision		Revision as of 16:05, 20 September 2017
Line 32:		Line 32:
	=== Routing methods ===		=== Routing methods ===

	LVS provides several modes of routing. We will describe here Direct Routing (<code>DR</code>) ~~and~~ Tunneling (<code>TUN</code>). There are more routing methods available which might come interesting in special cases, but not covered in this document.		LVS provides several modes of routing. We will describe here the Direct Routing (<code>DR</code>) mode, the Tunneling (<code>TUN</code>) mode, and the ''Network Address Translation'' mode (<code>NAT</code>). There are more routing methods available which might come interesting in special cases, but not covered in this document.

	~~When unsure, follow the <code>TUN</code> path. It seems more robust in certain environments than the <code>DR</code> method~~.

	==== Direct Routing ====		==== Direct Routing ====
Line 56:		Line 54:
	==== NAT method ====		==== NAT method ====

	~~We have not worked out~~ / ~~tested~~ some ~~NAT based setup~~ yet, but it ~~sounds promising~~ to ~~get it working~~ in ~~even more restrictive~~ cloud environments, ~~where~~ routers ~~typically reject packages with mismatching IPs/MACs. Feedback welcome~~.		Here LVS substitutes the target IP from incoming packages from its own (virtual / loadbalancer) IP to the IP of the designated server, and re-sends the package.

			If the target server is a different network and the loadbalancer is actually acting as a gateway, there is nothing more to do. However, in the standard setup where the loadbalancer and servers are in the same network, there some additional configuration required in order to make the package roundtrip work: the server must use the loadbalancer as a gateway (at least for the returning MySQL traffic), and the loadbalancer in turn must be configured to act as a gateway and to do IP masquerading for the servers.

			This method is a fairly recent "discovery" and not yet extensively tested, but it promises to be the most robust method and allows for running also in general cloud environments. It needs no additional floating / service IPs, no special requirements on the involved routers, no ARP problems, etc.

	== Software installation on the loadbalancer node ==		== Software installation on the loadbalancer node ==
Line 79:		Line 81:
	10.0.0.4 loadbalancer primary IP		10.0.0.4 loadbalancer primary IP
	10.0.0.5 database client, e.g. OX middleware node		10.0.0.5 database client, e.g. OX middleware node
	10.0.0.10 loadbalancer virtual IP for writing (persistent routing / dedicated write node)		10.0.0.10 only DR/TUN: loadbalancer virtual IP for writing (persistent routing / dedicated write node)
	10.0.0.11 loadbalancer virtual IP for reading (round-robin)		10.0.0.11 only DR/TUN: loadbalancer virtual IP for reading (round-robin)

	Note: with DR and TUN, it is not possible to change the port numbers on routing; thus, for each loadbalancer endpoint, the loadbalancer needs an additional virtual IP. (It is not possible to configure them on different ports on the same (e.g. primary) IP of the loadbalancer.)		Note: with DR and TUN, it is not possible to change the port numbers on routing; thus, for each loadbalancer endpoint, the loadbalancer needs an additional virtual IP. (It is not possible to configure them on different ports on the same (e.g. primary) IP of the loadbalancer.) For NAT, we can use one IP with different ports (5506 for reading, 5507 for writing).

	=== Manual configuration / testing ===		=== Manual configuration / testing ===
Line 88:		Line 90:
	==== Networking adjustments on the server nodes ====		==== Networking adjustments on the server nodes ====

	~~The~~ server nodes need the loadbalancer virtual IP(s) configured on some network device in order for the server processes to be able to bind on this device.		For DR and TUN, the server nodes need the loadbalancer virtual IP(s) configured on some network device in order for the server processes to be able to bind on this device. For DR, it seems natural to configure a dummy device. For TUN, you need a tunl device.

	For DR, it ~~seems natural~~ to ~~configure a dummy device. For TUN, you need a tunl device~~.		For NAT, we need to set the default route for outgoing (or rather, responding) MySQL traffic. It seems it is sufficient to do so with some ip route / ip rule / iptables magic on packages with source port 3306.

	For testing, you can do it manually on the ~~given~~ nodes:		For testing, you can do it manually on the server nodes:

	# for TUN		# for TUN
Line 98:		Line 100:
	ip addr add 10.0.0.10/32 brd 10.0.0.10 dev tunl0		ip addr add 10.0.0.10/32 brd 10.0.0.10 dev tunl0
	ip addr add 10.0.0.11/32 brd 10.0.0.11 dev tunl0		ip addr add 10.0.0.11/32 brd 10.0.0.11 dev tunl0
			echo 1 > /proc/sys/net/ipv4/conf/all/arp_ignore
			echo 2 > /proc/sys/net/ipv4/conf/all/arp_announce

	# for DR		# for DR
	ip addr add 10.0.0.10/32 brd 10.0.0.10 dev dummy0		ip addr add 10.0.0.10/32 brd 10.0.0.10 dev dummy0
	ip addr add 10.0.0.11/32 brd 10.0.0.11 dev dummy0		ip addr add 10.0.0.11/32 brd 10.0.0.11 dev dummy0
			echo 1 > /proc/sys/net/ipv4/conf/all/arp_ignore
			echo 2 > /proc/sys/net/ipv4/conf/all/arp_announce

	~~Then, you solve~~ the ~~''ARP Problem'' by~~		# for NAT
			# default route for MySQL response traffic to the primary IP of the loadbalancer
			ip route add 10.0.0.4/32 dev eth0 table 1
			ip route add default via 10.0.0.4 dev eth0 table 1
			ip rule add fwmark 1 table 1
			iptables -t mangle -A OUTPUT -p tcp --sport 3306 -j MARK --set-mark 1

	~~echo 1~~ > /~~proc~~/~~sys/net/ipv4/conf~~/all/~~arp_ignore~~		Note about NAT: If the depicted approach proves insufficient, a different idea would be to use networking namespaces to have a separate default gateway for the mysql process tree -- but however, it seems it is actually beneficial to affect as few traffic by the "default route" as possible, which is why we propose this specific approach rather than a generic "default gateway" approach. The latter would look like <code>ip route delete default; ip route delete 10.0.0.0/24; ip route add 10.0.0.4/32 dev eth0; ip route add default via 10.0.0.4 dev eth0</code>. This also works in tests, however it is inconvenient for other ops to have all outgoing traffic routed through the loadbalancer, and it seems unclear / to require testing whether this would work for all usecases (like, full SST, etc).
	~~echo 2 > /proc/sys/net/ipv4/conf/~~all~~/arp_announce~~

	==== Loadbalancer ====		==== Loadbalancer ====

	The loadbalancer also needs the virtual IPs configured as secondary IPs:		For <code>DR</code>/<code>TUN</code>, The loadbalancer also needs the virtual IPs configured as secondary IPs (not required for <code>NAT</code>):

			# TUN and DR
	ip addr add 10.0.0.10/32 dev eth0		ip addr add 10.0.0.10/32 dev eth0
	ip addr add 10.0.0.11/32 dev eth0		ip addr add 10.0.0.11/32 dev eth0

			For <code>NAT</code>, we need to allow IP forwarding and masquerading:

			# For NAT
			echo 1 > /proc/sys/net/ipv4/ip_forward
			iptables -t nat -A POSTROUTING -s 10.0.0.1 -j MASQUERADE
			iptables -t nat -A POSTROUTING -s 10.0.0.2 -j MASQUERADE
			iptables -t nat -A POSTROUTING -s 10.0.0.3 -j MASQUERADE

	Then the loadbalancer endpoints themselves can be configured with </code>ipvsadm</code>:		Then the loadbalancer endpoints themselves can be configured with </code>ipvsadm</code>:
Line 141:		Line 160:
	/sbin/ipvsadm -a -t 10.0.0.11:3306 -r 10.0.0.3 -g -w 10		/sbin/ipvsadm -a -t 10.0.0.11:3306 -r 10.0.0.3 -g -w 10

	Note: you need to restart the MySQL service after the networking adjustments; otherwise, the MySQL daemon will not accept packages with the virtual IP as target IP. This has caused a lot of wasted time to quite some people.		# For NAT
			# Round-Robin / read instance
			/sbin/ipvsadm -A -t 10.0.0.4:5506 -s rr
			/sbin/ipvsadm -a -t 10.0.0.4:5506 -r 10.0.0.1 -m -w 10
			/sbin/ipvsadm -a -t 10.0.0.4:5506 -r 10.0.0.2 -m -w 10
			/sbin/ipvsadm -a -t 10.0.0.4:5506 -r 10.0.0.3 -m -w 10
			# Persistent / write instance
			/sbin/ipvsadm -A -t 10.0.0.4:5507 -s rr -p 86400 -M 0.0.0.0
			/sbin/ipvsadm -a -t 10.0.0.4:5507 -r 10.0.0.1 -m -w 10
			/sbin/ipvsadm -a -t 10.0.0.4:5507 -r 10.0.0.2 -m -w 10
			/sbin/ipvsadm -a -t 10.0.0.4:5507 -r 10.0.0.3 -m -w 10

			Note: For DR and TUN, you need to restart the MySQL service after the networking adjustments; otherwise, the MySQL daemon will not accept packages with the virtual IP as target IP. This has caused a lot of wasted time to quite some people.

	Note: if lazy, you can test with one server node, and extend the configuration later to all three nodes.		Note: if lazy, you can test with one server node, and extend the configuration later to all three nodes.

Dominik.epple at 11:49, 20 September 2017

2017-09-20T11:49:51Z

Show changes

Dominik.epple: /* High Level Design */

2017-09-20T11:35:59Z

High Level Design

← Older revision		Revision as of 11:35, 20 September 2017
Line 20:		Line 20:
	=== High Level Design ===		=== High Level Design ===

	The solution consists of several ~~modules~~.		The solution consists of several components.

	Main component is some kernel modules which implements the real loadbalancing / forwarnding functionality. (ip_vs, ip_vs_rr, probably more.)		Main component is some kernel modules which implements the real loadbalancing / forwarnding functionality. (ip_vs, ip_vs_rr, probably more.)

Dominik.epple at 10:31, 20 September 2017

2017-09-20T10:31:20Z

Show changes