29th
HAProxy Performance Tuning
We recently deployed some HAProxy servers to load balance some of our internal server traffic. HAProxy is a high performance HTTP load balancer with a very nice live stats interface.
I’ve been loving HAProxy so far except for one odd latency issue we saw with a few of our most heavily loaded server clusters. HAProxy seemed to be introducing a large amount of latency, about 1s, that we didn’t see if we accessed each of the servers in the pool directly.
The hardest thing in trying to configure HAProxy is that it has too many great options and perhaps too much documentation (at least for me). After re-reading the HAProxy docs for the Nth time I stumbled upon 1 variable change that led to 1 second plus drop in latency.
Here’s how the dropped looked graphically under a constant load (actually increased load after the change).
The change was to the minconn parameter defined for each backend server. Simply increasing the minconn parameter solved our issue.
Here’s the section of the HAProxy docs that helped me understand what the minconn parameter actually did. Pay special attention to “will receive only 10 simultaneous sessions when the proxy will be under 1000 sessions”.
Example :
---------
# be nice with P3 which only has 256 MB of RAM.
listen web_appl 0.0.0.0:80
maxconn 10000
mode http
cookie SERVERID insert nocache indirect
balance roundrobin
server pentium3-800 192.168.1.1:80 cookie s1 weight 8 minconn 10 maxconn 100 check
server opteron-2.0G 192.168.1.2:80 cookie s2 weight 20 minconn 30 maxconn 300 check
server opteron-2.4G 192.168.1.3:80 cookie s3 weight 24 minconn 30 maxconn 300 check
server web-backup1 192.168.2.1:80 cookie s4 check maxconn 200 backup
server web-excuse 192.168.3.1:80 check backup
In the example above, the server 'pentium3-800' will receive at most 100
simultaneous sessions when the proxy instance will reach 10000 sessions, and
will receive only 10 simultaneous sessions when the proxy will be under 1000
sessions.
- David
