# Filebeat only goes to one of the Logstash Servers that is behind an ELB
**URL:** https://discuss.elastic.co/t/filebeat-only-goes-to-one-of-the-logstash-servers-that-is-behind-an-elb/48875
**Category:** Beats
**Tags:** filebeat
**Created:** [April 30, 2016, 10:54pm UTC](https://discuss.elastic.co/t/filebeat-only-goes-to-one-of-the-logstash-servers-that-is-behind-an-elb/48875 "2016-04-30T22:54:14Z")
**Posts on this page:** 10
**Page:** 1
### Author:  [@gcherneski](https://discuss.elastic.co/u/gcherneski)
#### Post date: [April 30, 2016, 10:54pm UTC](https://discuss.elastic.co/t/filebeat-only-goes-to-one-of-the-logstash-servers-that-is-behind-an-elb/48875/1 "2016-04-30T22:54:14Z")
I have the following test scenario:
Filebeat --\> AWS ELB --\> (2) Logstash servers.
Both logstash servers have the exact same configuration.
The log output from a single Filebeat server only hit one of the Logstash servers behind the ELB.
I would expect it to round-robin between the two Logstash servers that are behind the ELB.
I am using TCP for my port on the ELB so there is no session affinity.
Is there a trick to getting the round-robin to happen.
Please advise.
Regards,
Gary
---
### Author:  [@warkolm](https://discuss.elastic.co/u/warkolm)
#### Post date: [May 1, 2016, 1:32am UTC](https://discuss.elastic.co/t/filebeat-only-goes-to-one-of-the-logstash-servers-that-is-behind-an-elb/48875/2 "2016-05-01T01:32:59Z")
Isn't this an ELB problem, not an LS/Beats one?
---
### Author:  [@Christian\_Dahlqvist](https://discuss.elastic.co/u/Christian_Dahlqvist)
#### Post date: [May 1, 2016, 11:42am UTC](https://discuss.elastic.co/t/filebeat-only-goes-to-one-of-the-logstash-servers-that-is-behind-an-elb/48875/3 "2016-05-01T11:42:57Z")
If you use a load balancer exposing a single IP, I believe Filebeat will assume you are trying to connect to a single Logstash instance. Filebeat uses persistent connections so will only reconnect once the connection is broken. If you instead provide the hosts directly in the Filebeat config, you can configure it to [load balance between the specified instances](https://www.elastic.co/guide/en/beats/filebeat/current/logstash-output.html#loadbalance).
---
### Author:  [@gcherneski](https://discuss.elastic.co/u/gcherneski)
#### Post date: [May 1, 2016, 3:14pm UTC](https://discuss.elastic.co/t/filebeat-only-goes-to-one-of-the-logstash-servers-that-is-behind-an-elb/48875/4 "2016-05-01T15:14:28Z")
Thanks for the replies. I appreciate the your time in responding to my questions. I know this looks and smells like an ELB issue but I have a number of other applications sending traffic to ELBs and none of those have this issue. They all round-robin between the _n_ number of backend servers that are behind the ELB. I want to use an ELB to avoid having to hard-code the IPs of the target Logstash servers and to use AWS's auto-scaling capabilities. I put the question to the forum to see if there was something special that needed to be done to use Filebeat with an ELB.
---
### Author:  [@whyapenny](https://discuss.elastic.co/u/whyapenny)
#### Post date: [May 2, 2016, 4:30pm UTC](https://discuss.elastic.co/t/filebeat-only-goes-to-one-of-the-logstash-servers-that-is-behind-an-elb/48875/5 "2016-05-02T16:30:00Z")
might be a stupid question, but did you configure both logstash servers i the hosts and set loadbalance: true, such as:
hosts: ["logstash01:5044", "logstash02.corp.pvt:5044"]
loadbalance: true
loadbalance defaults to off, and will only send to a single logstash instance unless failed over. Take logstash01 down to see if it goes to 02. if so, probably this. otherwise i likely misunderstood your question.
---
### Author:  [@steffens](https://discuss.elastic.co/u/steffens)
#### Post date: [May 2, 2016, 4:30pm UTC](https://discuss.elastic.co/t/filebeat-only-goes-to-one-of-the-logstash-servers-that-is-behind-an-elb/48875/6 "2016-05-02T16:30:45Z")
ELB works by assigning another server per connection made (I think based on DNS). That is, it only works well with non-persistent connections. That is, for ELB to properly load-balance you require your application to reconnect every so often. In most extreme cases one connection per log-line. This mode of operation increases latency, reduces throughput and adds additional packets/bytes to be transmitted in network, which makes this a quite expensive but flexible solution.
Beats-\>logstash is using a persistent TCP connection. In order to work 'well' with ELB, you will need multiple beats-instances + some method to break connections every so often (force filebeat to reconnect). Related ticket: [https://github.com/elastic/beats/issues/661](https://github.com/elastic/beats/issues/661)
Only advantage of beats dropping connections after TTL is, they will reconnect and 'rebalance' based current ELB state. But once connected, beats will send to assigned logstash only. load-balancing will be pretty sub-optimal. One can try to improve by configuring `worker=X` to spawn multiple workers each connection to logstash via ELB.
Personally I'd love to see some cluster-state-whatever-support (e.g. like elasticsearch sniffing support by some clients) for beats to auto-discover logstash nodes being added/removed and dynamically update the load-balancer. But this is something for far future.
---
### Author:  [@gcherneski](https://discuss.elastic.co/u/gcherneski)
#### Post date: [May 4, 2016, 2:38pm UTC](https://discuss.elastic.co/t/filebeat-only-goes-to-one-of-the-logstash-servers-that-is-behind-an-elb/48875/7 "2016-05-04T14:38:45Z")
Rather than beats sniffing for auto-discovery a simplified model would be to have an option in filebeat to either use a persistent connection or to re-establish a connection for each parsed log line. While there is overhead associated with the second model it more simulates a web traffic scenario in that each request, or in this case each parsed log line (single or multi), would be a separate request from filebeat. This would greatly improve the use of data sent from filebeat for a load-balanced model. Filebeat would not have to keep track of the downstream logstash systems, it would send and forget. The load balancer, in the case of AWS, would be able to auto-scale based on load adding as many, or reducing as many, backend logstash servers as needed as load demands change.
---
### Author:  [@steffens](https://discuss.elastic.co/u/steffens)
#### Post date: [May 4, 2016, 4:09pm UTC](https://discuss.elastic.co/t/filebeat-only-goes-to-one-of-the-logstash-servers-that-is-behind-an-elb/48875/8 "2016-05-04T16:09:42Z")
This is related to issue #661. See link in my former post. Only problem with this is, filebeat waits for ACK from logstash before sending another batch events. Worse, in case we implement pipelining it get's a lot more difficult to decide when to drop a connection.
---
### Author:  [@gcherneski](https://discuss.elastic.co/u/gcherneski)
#### Post date: [May 9, 2016, 7:15pm UTC](https://discuss.elastic.co/t/filebeat-only-goes-to-one-of-the-logstash-servers-that-is-behind-an-elb/48875/9 "2016-05-09T19:15:32Z")
Resolved. A new filebeat version came out that supports Redis. I created an AWS ElastiCache Redis cluster to accept the filebeat input and to face logstash. I moved multiline parsing to the filebeat tier and out of the logstash tier because multiline parsing causes logstash to go single threaded. Plus it is better to do multiline parsing as close to the input data as possible. Thanks for the recommendations @steffens. I used a 1.5M+ test record set containing 10 different timestamp formats and get 100% match results. I now have the following configuration: filebeat (many instances) --\> ElastiCacheRedis \<-- logstash (3) --\> elasticsearch cluster (3) \<-- Kibana. The new filebeat functionality works great!
---
### Author:  [@system](https://discuss.elastic.co/u/system)
#### Post date: [July 5, 2017, 9:52pm UTC](https://discuss.elastic.co/t/filebeat-only-goes-to-one-of-the-logstash-servers-that-is-behind-an-elb/48875/10 "2017-07-05T21:52:11Z")