Filebeats logs -> aws elb of logstash

[blackOcean (Sagar Rout)]

Hi,

Using AWS elb for logstash server (3 nodes of EC2 instances), but I am getting client is unable to connect.

filebeat.log
2017-12-12T14:55:06Z ERR  Failed to publish events caused by: client is not connected
2017-12-12T14:55:07Z ERR  Failed to publish events: client is not connected
2017-12-12T14:55:07Z DBG  connect 
2017-12-12T14:55:07Z DBG  3 events out of 3 events sent to logstash. Continue sending
2017-12-12T14:55:07Z ERR  Failed to publish events caused by: lumberjack protocol error
2017-12-12T14:55:07Z DBG  closing
2017-12-12T14:55:07Z DBG  3 events out of 3 events sent to logstash. Continue sending
2017-12-12T14:55:07Z DBG  close connection
2017-12-12T14:55:07Z DBG  close connection

Filebeat is able to send logs when I write all logstash instance manually use logstash loadbalancing rather than AWS elastic loadbalancing.

   hosts: ["ec2-logstash0:5043","ec2-logstash1:5043","ec2-logstash3:5043"]  -- this is working with multiple instance
   loadbalance : true
   worker: 2

I tested with telnet logstash-elb-amazonaws.com 5043, Able to conect.

 And this with aws-elb is not working. 
 hosts: ["logstash-elb-amazonaws.com:5043"]
 loadbalance: true --this property does not affect anything
 worker: 2 -- this property also does not affect anything whether we use it or not.

[andrewkroh]

What protocol did you configure the load balancer for? It should be TCP as the protocol is not HTTP based.

[blackOcean (Sagar Rout)]

Thanks.

I used Network load balancer with TCP. And Boom Boom !!!

It worked but I believe filebeat has some sticky session with logstash. It always sends logs to one logstash instance always. If that dies then only it choose another one and works.

I believe AWS ELB Should distribute the load on multiple logstash rather than on single instance.

Any Idea on this ?

[andrewkroh]

Filebeat will keep its connection to Logstash alive. But you can set a TTL on the connection to allow for rebalancing. https://www.elastic.co/guide/en/beats/filebeat/current/logstash-output.html#_literal_ttl_literal

[blackOcean (Sagar Rout)]

Thanks