# Multiple logstash servers need to be ache
**URL:** https://discuss.elastic.co/t/multiple-logstash-servers-need-to-be-ache/64154
**Category:** Beats
**Tags:** filebeat
**Created:** [October 27, 2016, 2:04pm UTC](https://discuss.elastic.co/t/multiple-logstash-servers-need-to-be-ache/64154 "2016-10-27T14:04:30Z")
**Posts on this page:** 7
**Page:** 1
### Author:  [@nagendrababug](https://discuss.elastic.co/u/nagendrababug)
#### Post date: [October 27, 2016, 2:04pm UTC](https://discuss.elastic.co/t/multiple-logstash-servers-need-to-be-ache/64154/1 "2016-10-27T14:04:30Z")
Hi,
can i use multiple logstash servers in [filebeat.My](http://filebeat.My) requirement is, if one logstash server gets down another server needs to be take responsibilty with out loss and duplicate of logs data.
will it be done with filebeat?
Thanks in advance
---
### Author:  [@steffens](https://discuss.elastic.co/u/steffens)
#### Post date: [October 27, 2016, 3:07pm UTC](https://discuss.elastic.co/t/multiple-logstash-servers-need-to-be-ache/64154/2 "2016-10-27T15:07:17Z")
filebeat supports loadbalancing and/or failover. But it can not guarantee no duplicates, as filebeat has send-at-least-once semantics. Guaranteeing no duplication would require some non-trivial coordination on protocol level and between logstash instances. Or document IDs used to index-or-update documents in ES.
---
### Author:  [@nagendrababug](https://discuss.elastic.co/u/nagendrababug)
#### Post date: [October 28, 2016, 4:18am UTC](https://discuss.elastic.co/t/multiple-logstash-servers-need-to-be-ache/64154/3 "2016-10-28T04:18:22Z")
Thanks Steffens for your reply,
can you please provide a sample filebeat configuration code with multiple logstash servers.
---
### Author:  [@steffens](https://discuss.elastic.co/u/steffens)
#### Post date: [October 28, 2016, 1:37pm UTC](https://discuss.elastic.co/t/multiple-logstash-servers-need-to-be-ache/64154/4 "2016-10-28T13:37:10Z")
See docs: [https://www.elastic.co/guide/en/beats/filebeat/current/load-balancing.html](https://www.elastic.co/guide/en/beats/filebeat/current/load-balancing.html)
---
### Author:  [@nagendrababug](https://discuss.elastic.co/u/nagendrababug)
#### Post date: [November 1, 2016, 1:30pm UTC](https://discuss.elastic.co/t/multiple-logstash-servers-need-to-be-ache/64154/5 "2016-11-01T13:30:05Z")
Hi steffens,
I have been working on Logstash with kafka.I am seeing different behaviour of Logstash while pushing messages into kafka.
Requirement:
1. I will get different files continously in a location assume /tmp/input-logs/ . all are with extension .log.I have to push the messages of that files to a topic(stagin-topic) in kafka broker lets say 10.0.24.33:9092.Later kafka consmer will consumes the messages from topic and display.
2. Some times i will restart logstash, then i want to get latest data without duplicate and loss.
Code I have written for logstash
input
{
file {
path =\> "/file0/file1/logstash-input-logs/\*.log"
start\_position =\> "end"
sincedb\_path =\> "/file0/file1/logstash-conf/input.sincedb"
}
}
output {
kafka {
codec =\> plain {
format =\> "%{message}"
}
bootstrap\_servers =\> "10.0.24.23:9092"
topic\_id =\> "staging-topic"
}
}
Problems I got
3. If i put start\_postion =\> "beginning".It is working fine untill we stop logstash while it is writing data to kafka.When we start logstash again then it
reads data from all files.
4. If i put start\_postion =\> "end". Logstash not writes complete data to kafka ex: I have 100000 records in 20 files, but while consuming from zookeeper topic i can
able to get only 40000 or 25000 and sometimes it is 60000.
5. When i increased pipeline workers count from default 8 to 30.I am able to get all messages but if we restart logstash it is ignoring the messages which are not sent
kafka.
---
### Author:  [@steffens](https://discuss.elastic.co/u/steffens)
#### Post date: [November 2, 2016, 10:43am UTC](https://discuss.elastic.co/t/multiple-logstash-servers-need-to-be-ache/64154/6 "2016-11-02T10:43:41Z")
regarding logstash+kafka issues, please ask in logstash forum.
- kafka is offset based. It operates more like a big distributed append-only file. Using kafka one has to use and keep track of the offset in the consumer. Normally this is done using consumer groups. Using `beginning` (always use offset 0 on start) or `end` (always start at end of file - like `tail -f`) as start position, ignores the last offset processed.
- to not loose data one has to properly manage offsets in consumer
- kafka and kafka protocol is very minimal. It's up to the application deciding wether to retry inserting data or drop data. filebeat for example will retry forever. Not sure about logstash output plugin. With always-retry one only gets send-at-least-once semantics =\> In case of network failures there is always a chance of duplicates. One can try to get some sort of deduplication by defining unique keys per event.
---
### Author:  [@system](https://discuss.elastic.co/u/system)
#### Post date: [November 17, 2016, 2:04pm UTC](https://discuss.elastic.co/t/multiple-logstash-servers-need-to-be-ache/64154/7 "2016-11-17T14:04:37Z")
This topic was automatically closed after 21 days. New replies are no longer allowed.