Lumberjack protocol error

[a.gavric (Aleksandar Gavric)]

Hello everyone i have been working on a scaled up elastic stack infrastructure with two logstash instances behind a reverseproxy running on HAproxy server with loadbalancing. the problem is with beats that are reporting as Established and then disconnects with lumberjack protocol error.

Config files are below:

Filebeat:

• type: log

  Change to true to enable this input configuration.

  enabled: true

  Paths that should be crawled and fetched. Glob based paths.

  paths:

  • /var/log/*.log
    Blockquote
    setup.kibana:
    host: "10.226.100.30:5601"
    Blockquote
    output.logstash:

The Logstash hosts

hosts: ["10.226.100.10:6969"]

Logstash:

input {

beats {

port => 6969

}

}

output {

elasticsearch {

hosts => ["http://logmgmt01cluster.log.lab.aginion.net:9200"]
index => "%{[@metadata][beat]}-%{[@metadata][version]}"
ilm_enabled => true

}
}

Haproxy:

frontend logstash-data
bind 10.226.100.10:6969
mode tcp
option tcplog
log /dev/log local0 notice
default_backend logstash-loadbalance

backend logstash-loadbalance
server logstash01.log.lab.aginion.net logstash01.log.lab.aginion.net:6969 maxconn 128 check
server logstash02.log.lab.aginion.net logstash02.log.lab.aginion.net:6969 maxconn 128 check
log /dev/log local0 notice
balance leastconn

Once filebeat is started with stdout the following is shown:

Connecting to backoff(async(tcp://10.226.100.10:6969))
Connection to backoff(async(tcp://10.226.100.10:6969)) established
Failed to publish events caused by: lumberjack protocol error
Failed to publish events caused by: client is not connected

i assumed that this is because filebeat cannot see logstash through HAProxy as if i connect filebeat directly to logstash with avoiding Haproxy it connects with no issues.
Any comment regarding this would be greatly appreciated.

[a.gavric (Aleksandar Gavric)]

Hello Everyone, you can mark this as closed as i have managed to resolve the issue in the meantime. In case anyone requires this in the future the configuration for Haproxy as loadbalancing for logstash is below:

listen logstash
bind 10.226.100.10:6969
mode tcp
option tcplog
log /dev/log local0 notice
server server1 server1 maxconn 128 check
server server2 server2 maxconn 128 check
balance leastconn