Recently, we had an issue where all our servers, including kafka + logstash where restarted at the same time. I was hoping that after restart we would get all messages but unfortunately some messages where lost. After some digging and replying of messages it seems like kafka-consumer is committing offsets of messages that have not yet been delivered to elasticsearch (or whatever output you use).
This actually makes sense since there is an autocommit everyh 5 seconds and I guess that messages can be in "transit" during that period.
Is there something we can do to prevent this from happening? Increase autocommit period, shorter internal queues etc? with enable_autocommit=false, does this mean that offset is never committed since it does not seem to provide a manual commit.
Logstash has an internal queue, and if you do not use a persistent queue this is in memory, which can lead to events being lost in the event of a crash. You can keep it small and use it together with Kafka.
Apache, Apache Lucene, Apache Hadoop, Hadoop, HDFS and the yellow elephant
logo are trademarks of the
Apache Software Foundation
in the United States and/or other countries.