[Solved] Does anyone have a working LSF->HAProxy->Logstash 1.5.2 indexers stack?

Elastic Stack Logstash
1

On my production cluster, I have Logstash-Forwarder connecting to my two Logstash 1.4.2 indexers via a HAProxy load balancer.

After testing an upgrade to Logstash 1.5.2 on my test cluster, I upgraded production. And everything broke.

Since my test cluster does not have HAProxy involved, I'm certain that's where the worst of my problems are located.

When I told one production LSF instance to bypass the HAProxy, the indexer started receiving events. (Though it ran into lumberjack pipeline errors...).

LSF logged messages like:

Loading client ssl certificate: /path/to/pemfile.pem and /path/to/keyfile.key
Setting trusted CA from file: /path/to/CAFile.crt
Connecting to [ha.pro.xy.ip]:5043 (haproxy.example.tld) 
Read error looking for ack: EOF

Anyway, long story short, something in how Logstash handles LSF connections changed between 1.4.2 and 1.5.2. That change made HAProxy stop working.

Any suggestions?

Note, as soon as I restored Logstash back to 1.4.2, it started working again.

Oh, and I am using the elasticsearch_http plugin with 1.4.2, and the elasticsearch plugin with http protocol on 1.5.2.

2

What is your haproxy config?

You'll need to put it into TCP mode so its passing through the connection to the SSL LSF endpoint.

I've had some luck with this config, but am still troubleshooting a member that isn't balancing properly, so it may be a wonky config:

listen logstashforwarder
bind x.x.x.x:6783 
mode  tcp
balance  roundrobin
option  tcplog
server logstash-prod-node01 x.x.x.x:6783 check
server logstash-prod-node02 x.x.x.x:6783 check
3

This is haproxy configuration is working for me.
I am also using the latest logstash (1.5.2)

global
chroot /var/lib/haproxy
daemon
group haproxy
maxconn 4000
pidfile /var/run/haproxy.pid
stats socket /var/lib/haproxy/stats
user haproxy

defaults
log 127.0.0.1 local0
log 127.0.0.1 local1 notice
timeout queue 10m
timeout connect 10s
timeout client 10m
timeout server 10m
timeout check 10s

listen logstash
bind 10.0.69.5:6969
mode tcp
balance leastconn
option tcplog
server logstash01 10.0.69.7:6969 check
server logstash02 10.0.69.8:6969 check

listen proxystats
bind 10.0.69.5:80
mode http
stats enable
stats hide-version
stats realm Haproxy\ Statistics
stats uri /

1 Like
4

So, after upgrading Logstash to 1.5.4, my old configuration works just fine. I assume the ssl related bugs they fixed in the new version were the source of my problems.