We are running the latest logstash 2.2.1. I've tried it with acks = 0, 1, and "all" and we've run into the same issue. It now appears that the issue is that it can't handle the initial load caused by the backup of messages. We have 2 logstash instances sitting behind a load balancer which seems to queue up the messages that fail to be processed by logstash once it "chokes". So when we restart logstash and communication opens back up, we get a huge influx of messages that the kafka output can't handle (but the File and Elasticsearch plugins can) so it chokes eventually and we're back at square one.
If I get rid of the backup and only send the real-time messages, the kafka output seems to handle it fine. It's just the initial catch-up that it can't handle. But in production I can't assure that we'll never have to restart logstash or get a build-up of messages like that.
Is there an easy solution to handle this?