# At least once delivery setup with logstash kafka **URL:** \ **Category:** Logstash\ **Created:** [February 12, 2019, 9:22am UTC](https://discuss.elastic.co/t/at-least-once-delivery-setup-with-logstash-kafka/167998 "2019-02-12T09:22:21Z")\ **Posts on this page:** 6\ **Page:** 1
**Author:** ![JohanRask](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/johanrask/32/22713_2.png) [@JohanRask](https://discuss.elastic.co/u/JohanRask)\ **Post date:** [February 12, 2019, 9:22am UTC](https://discuss.elastic.co/t/at-least-once-delivery-setup-with-logstash-kafka/167998/1 "2019-02-12T09:22:21Z")
Hi, Recently, we had an issue where all our servers, including kafka + logstash where restarted at the same time. I was hoping that after restart we would get all messages but unfortunately some messages where lost. After some digging and replying of messages it seems like kafka-consumer is committing offsets of messages that have not yet been delivered to elasticsearch (or whatever output you use). This actually makes sense since there is an autocommit everyh 5 seconds and I guess that messages can be in "transit" during that period. Is there something we can do to prevent this from happening? Increase autocommit period, shorter internal queues etc? with enable\_autocommit=false, does this mean that offset is never committed since it does not seem to provide a manual commit. Thanks /Johan Rask ---
**Author:** ![Christian\_Dahlqvist](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/christian_dahlqvist/32/4617_2.png) [@Christian\_Dahlqvist](https://discuss.elastic.co/u/Christian_Dahlqvist)\ **Post date:** [February 12, 2019, 9:27am UTC](https://discuss.elastic.co/t/at-least-once-delivery-setup-with-logstash-kafka/167998/2 "2019-02-12T09:27:14Z")
Have you got a [persistent queue](https://www.elastic.co/guide/en/logstash/6.6/persistent-queues.html) configured for Logstash? ---
**Author:** ![JohanRask](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/johanrask/32/22713_2.png) [@JohanRask](https://discuss.elastic.co/u/JohanRask)\ **Post date:** [February 12, 2019, 9:41am UTC](https://discuss.elastic.co/t/at-least-once-delivery-setup-with-logstash-kafka/167998/3 "2019-02-12T09:41:41Z")
Nope, since we use kafka I consider that my buffer and everything is designed for at-least-once. Is that my only option? ---
**Author:** ![Christian\_Dahlqvist](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/christian_dahlqvist/32/4617_2.png) [@Christian\_Dahlqvist](https://discuss.elastic.co/u/Christian_Dahlqvist)\ **Post date:** [February 12, 2019, 9:43am UTC](https://discuss.elastic.co/t/at-least-once-delivery-setup-with-logstash-kafka/167998/4 "2019-02-12T09:43:06Z")
Logstash has an internal queue, and if you do not use a persistent queue this is in memory, which can lead to events being lost in the event of a crash. You can keep it small and use it together with Kafka. ---
**Author:** ![JohanRask](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/johanrask/32/22713_2.png) [@JohanRask](https://discuss.elastic.co/u/JohanRask)\ **Post date:** [February 12, 2019, 9:54am UTC](https://discuss.elastic.co/t/at-least-once-delivery-setup-with-logstash-kafka/167998/5 "2019-02-12T09:54:18Z")
We where just hoping that we could get away without it.. I will dig into it, and I assume there is some kind of backpressure thingy that prevents an input from filling the queue. Thanks @Christian_Dahlqvist! ---
**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\ **Post date:** [March 12, 2019, 9:54am UTC](https://discuss.elastic.co/t/at-least-once-delivery-setup-with-logstash-kafka/167998/6 "2019-03-12T09:54:21Z")
This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.