Recently, we had an issue where all our servers, including kafka + logstash where restarted at the same time. I was hoping that after restart we would get all messages but unfortunately some messages where lost. After some digging and replying of messages it seems like kafka-consumer is committing offsets of messages that have not yet been delivered to elasticsearch (or whatever output you use).
This actually makes sense since there is an autocommit everyh 5 seconds and I guess that messages can be in "transit" during that period.
Is there something we can do to prevent this from happening? Increase autocommit period, shorter internal queues etc? with enable_autocommit=false, does this mean that offset is never committed since it does not seem to provide a manual commit.
Thanks /Johan Rask