# Loadbalance duplicating the events using logstash **URL:** \ **Category:** Beats\ **Tags:** filebeat\ **Created:** [September 23, 2016, 6:50am UTC](https://discuss.elastic.co/t/loadbalance-duplicating-the-events-using-logstash/61342 "2016-09-23T06:50:01Z")\ **Posts on this page:** 9\ **Page:** 1
**Author:** ![avinash](https://avatars.discourse-cdn.com/v4/letter/a/ed655f/32.png) [@avinash](https://discuss.elastic.co/u/avinash)\ **Post date:** [September 23, 2016, 6:50am UTC](https://discuss.elastic.co/t/loadbalance-duplicating-the-events-using-logstash/61342/1 "2016-09-23T06:50:01Z")
I have configuration as filebeat with loadbalance true & two logstash hosts i.e. logstash1 & logstash2 which are input to elasticsearch. The filebeat configurtion for logstash is as below prospectors: - paths: - /home/testLogs/temp.log spool\_size: 1 publish\_async: true ### Logstash as output logstash: hosts: ["10.186.187.44:5044", "10.186.187.6:5044"] worker: 1 loadbalance: true The logstash input on both logstash I set as below input { beats { port =\> 5044 congestion\_threshold =\> 100000 } } output { stdout {codec =\> rubydebug} elasticsearch { hosts =\> "ipaddress:9200" index =\> "testidx" } } However when I echo some lines into file temp.log, I see that in the elasticsearch processed lines are duplicated docs, it looks to me that both logstash are processing all the lines. and on stdout of both logstash sometime I see forex line1 & line2 processed by both logstash, or sometimes line1 processed by logstash1 & line1, line2, line2 processed by logstash2 However even making loadbalance=flase I still see the duplicated docs. & in stdout I sed line1 & line2 processed twice either by logstash1 or logstash2. So just wanted to know is my configuration wrong that the lines are not distributed on logstashes or is it like its behavior or bug ---
**Author:** ![steffens](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/steffens/32/79630_2.png) [@steffens](https://discuss.elastic.co/u/steffens)\ **Post date:** [September 29, 2016, 9:42am UTC](https://discuss.elastic.co/t/loadbalance-duplicating-the-events-using-logstash/61342/2 "2016-09-29T09:42:36Z")
can you share filebeat configuration + filebeat debug logs + exact shell commands you use to update your file. The outputs in filebeat do normally not duplicate content. But with send-at-least-once semantics in filebeat, if one output did not ACK the events being published, events might be send again to same or another output. To me it sounds more like filebeat is resending the complete file (maybe inode changes). But without filebeat and logstash logs I can not say anything. ---
**Author:** ![avinash](https://avatars.discourse-cdn.com/v4/letter/a/ed655f/32.png) [@avinash](https://discuss.elastic.co/u/avinash)\ **Post date:** [September 30, 2016, 9:55am UTC](https://discuss.elastic.co/t/loadbalance-duplicating-the-events-using-logstash/61342/3 "2016-09-30T09:55:49Z")
Hi Seffens, It looks, it stopped getting duplicate docs, after my system got crashed & I had to restart again. So now the problem seems to be fixed but I still wonder why I was getting the duplicate logs after providing two hosts & loadbalance=true or even false. I dont have older filebeat debug logs which was causing the duplicate events. Even previously i restarted filebeat multiple times which was still causing duplication. I hope it will not start duplicating again. ---
**Author:** ![shog](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/shog/32/12138_2.png) [@shog](https://discuss.elastic.co/u/shog)\ **Post date:** [September 30, 2016, 7:46pm UTC](https://discuss.elastic.co/t/loadbalance-duplicating-the-events-using-logstash/61342/4 "2016-09-30T19:46:25Z")
Hi @avinash, Out of curiosity, do you have increased congestion\_threshold on your beats input? ---
**Author:** ![avinash](https://avatars.discourse-cdn.com/v4/letter/a/ed655f/32.png) [@avinash](https://discuss.elastic.co/u/avinash)\ **Post date:** [October 3, 2016, 4:35am UTC](https://discuss.elastic.co/t/loadbalance-duplicating-the-events-using-logstash/61342/5 "2016-10-03T04:35:14Z")
HI @shog, From the start I used congestion\_threshold very high value i.e. congestion\_threshold =\> 100000, and I didnt change it before or after fixing this issue. ---
**Author:** ![steffens](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/steffens/32/79630_2.png) [@steffens](https://discuss.elastic.co/u/steffens)\ **Post date:** [October 4, 2016, 12:40pm UTC](https://discuss.elastic.co/t/loadbalance-duplicating-the-events-using-logstash/61342/6 "2016-10-04T12:40:45Z")
beats do not duplicate events, but beats have do resend events if the output is not ACKing events (e.g. network failures or timeouts). If logstash/elasticsearch/redis/kafka did process events without the reply ever being received by beats, beats have to resend. Without reply you can not tell if data have been processed or not. This can even happen if just one logstash host is configured and/or load balancing is disabled. It's due to send-at-least-once semantics. Deduplication must either be handled on protocol level or potentially in ES (adds indexing overhead though) by adding some kind of sequence number/id/key. With load-balancer potentially re-balancing failed send attempts it can not be solved on protocol itself unless rebalancing is disabled (potentially blocking the beat if one sink becomes unavailable). Related issue about generating some UUID per event: [https://github.com/elastic/beats/issues/1492](https://github.com/elastic/beats/issues/1492) per event UUIDs should be used with on-disk persistent queues in order remove the chance of duplicate events being send due to new UUID being generated between beat restarts: [https://github.com/elastic/beats/issues/575](https://github.com/elastic/beats/issues/575) But this is some future talk not yet available in beats. ---
**Author:** ![shog](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/shog/32/12138_2.png) [@shog](https://discuss.elastic.co/u/shog)\ **Post date:** [October 4, 2016, 1:31pm UTC](https://discuss.elastic.co/t/loadbalance-duplicating-the-events-using-logstash/61342/7 "2016-10-04T13:31:16Z")
Hi @steffens, So, could lowering congestion\_threshold on beats input help here? My understanding is that logstash output in filebeat has a timeout of 30 secs by default, but @avinash is also using a beats input in LS with congestion\_threshold set to 100k seconds. ---
**Author:** ![steffens](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/steffens/32/79630_2.png) [@steffens](https://discuss.elastic.co/u/steffens)\ **Post date:** [October 4, 2016, 1:51pm UTC](https://discuss.elastic.co/t/loadbalance-duplicating-the-events-using-logstash/61342/8 "2016-10-04T13:51:30Z")
A lower congestion\_threshold can worsen the problem, as logstash might close connection with subset of events published still being actively processed. There is still a chance the timeout in beats did trigger (30s without response is very well possible if logstash is blocked by outputs) or we're indeed seeing some network issues. Increasing timeout to something like 5 minutes or 10 minutes might improve situation if logstash regularly has to wait (e.g. for it's output not responding due to GC-phase in remote server). ---
**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\ **Post date:** [October 14, 2016, 6:50am UTC](https://discuss.elastic.co/t/loadbalance-duplicating-the-events-using-logstash/61342/9 "2016-10-14T06:50:18Z")
This topic was automatically closed after 21 days. New replies are no longer allowed.