Hi I've setup a 3 node elasticsearch cluster on different VMs locally on my laptop.
I use haproxy for loadbalancing.
Connecting curl or kibana against the loadbalancer works fine, but logstash's monitoring pipeline is giving errors. I did not test with the real pipeline yet.
Looks for me as if logstash wants to keep the connection open but with haproxy in the chain the connection gets closed and logstash seems not to recognize the closure correctly (or it is not forwarded to logstash correctly). So it hits an error and after a while it reconnects again.
I hope anyone of you is using haproxy in front of elasticsearch's rest api and you can provide your configuration. I am new to haproxy.
Mine looks like this:
global log 127.0.0.1 local2 debug log 127.0.0.1 local0 debug chroot /var/lib/haproxy pidfile /var/run/haproxy.pid maxconn 4000 user haproxy group haproxy daemon stats socket /var/lib/haproxy/stats tune.ssl.default-dh-param 2048 defaults mode http log global option httplog option dontlognull option http-server-close option forwardfor except 127.0.0.0/8 option redispatch retries 3 timeout http-request 10s timeout queue 1m timeout connect 10s timeout client 1m timeout server 1m timeout http-keep-alive 10s timeout check 10s maxconn 3000 frontend ft_elasticsearch_http mode tcp option tcpka option tcplog bind 0.0.0.0:9201 name elasticsearch_http default_backend bk_elasticsearch_http backend bk_elasticsearch_http mode tcp option httpchk HEAD / HTTP/1.1\r\nAuthorization:\ Basic\ aGVhbHRoY2hlY2s6aGVhbHRoMQ== http-check expect status 200 option tcpka option tcplog default-server fall 2 balance source option log-health-checks default-server inter 1s fall 2 #server elastic03.internal.de_9200 elastic03.internal.de:9200 check check-ssl verify required ca-file /etc/haproxy/certs/ca/elasticsearch/elastic.pem inter 2s #server elastic02.internal.de_9200 elastic02.internal.de:9200 check check-ssl verify required ca-file /etc/haproxy/certs/ca/elasticsearch/elastic.pem inter 2s server elastic01.internal.de_9200 elastic01.internal.de:9200 check check-ssl verify required ca-file /etc/haproxy/certs/ca/elasticsearch/elastic.pem inter 2s
When I wireshark the connection without loadbalancer in the chain, I can see tcp keepalives, but with haproxy in between, I don't see any.
Thanks a lot, Andreas