Haproxy in front of elasticsearch rest api on multinode cluster?

Elastic Stack Elasticsearch
1

Hi I've setup a 3 node elasticsearch cluster on different VMs locally on my laptop.

I use haproxy for loadbalancing.
Connecting curl or kibana against the loadbalancer works fine, but logstash's monitoring pipeline is giving errors. I did not test with the real pipeline yet.
Looks for me as if logstash wants to keep the connection open but with haproxy in the chain the connection gets closed and logstash seems not to recognize the closure correctly (or it is not forwarded to logstash correctly). So it hits an error and after a while it reconnects again.

I hope anyone of you is using haproxy in front of elasticsearch's rest api and you can provide your configuration. I am new to haproxy.

Mine looks like this:

global

    log         127.0.0.1 local2 debug
    log         127.0.0.1 local0 debug

    chroot      /var/lib/haproxy
    pidfile     /var/run/haproxy.pid
    maxconn     4000

    user        haproxy
    group       haproxy
    daemon
    stats socket /var/lib/haproxy/stats
    tune.ssl.default-dh-param   2048

defaults
    mode                    http
    log                     global
    option                  httplog
    option                  dontlognull
    option http-server-close
    option forwardfor       except 127.0.0.0/8
    option                  redispatch
    retries                 3
    timeout http-request    10s
    timeout queue           1m
    timeout connect         10s
    timeout client          1m
    timeout server          1m
    timeout http-keep-alive 10s
    timeout check           10s
    maxconn                 3000

    
frontend ft_elasticsearch_http
    mode tcp
    option tcpka
    option tcplog
    bind 0.0.0.0:9201 name elasticsearch_http
    default_backend bk_elasticsearch_http


backend bk_elasticsearch_http
    mode tcp
    option httpchk HEAD / HTTP/1.1\r\nAuthorization:\ Basic\ aGVhbHRoY2hlY2s6aGVhbHRoMQ==
    http-check expect status 200
    option tcpka
    option tcplog

    default-server fall 2

    balance source
    option log-health-checks
    default-server inter 1s fall 2
    #server elastic03.internal.de_9200 elastic03.internal.de:9200  check check-ssl verify required ca-file /etc/haproxy/certs/ca/elasticsearch/elastic.pem inter 2s
    #server elastic02.internal.de_9200 elastic02.internal.de:9200  check check-ssl verify required ca-file /etc/haproxy/certs/ca/elasticsearch/elastic.pem inter 2s
    server elastic01.internal.de_9200 elastic01.internal.de:9200  check check-ssl verify required ca-file /etc/haproxy/certs/ca/elasticsearch/elastic.pem inter 2s

When I wireshark the connection without loadbalancer in the chain, I can see tcp keepalives, but with haproxy in between, I don't see any.

Thanks a lot, Andreas

2

Hi @asp

I think this is a known issue with Logstash, see https://github.com/logstash-plugins/logstash-output-elasticsearch/issues/814

There are some suggested workarounds for this in https://github.com/logstash-plugins/logstash-output-elasticsearch/issues/793 but I can't speak to their validity to be honest.

Setting

      sniffing => false

for the Logstash ES output plugin may help though if the problem isn't so much with HAProxy dropping connections but rather Logstash trying to sniff for more endpoints? I'd check if you may have that setting on true (defaults to false also though)?

3

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.