# Kafka output plugin loses all data when Kafka is down

**URL:** <https://discuss.elastic.co/t/kafka-output-plugin-loses-all-data-when-kafka-is-down/102468>\
**Category:** Logstash\
**Created:** [October 2, 2017, 6:40pm UTC](https://discuss.elastic.co/t/kafka-output-plugin-loses-all-data-when-kafka-is-down/102468 "2017-10-02T18:40:32Z")\
**Posts on this page:** 3\
**Page:** 1

<div class="post-metadata">

**Author:** ![quadd5856](https://avatars.discourse-cdn.com/v4/letter/q/c6cbf5/32.png) [@quadd5856](https://discuss.elastic.co/u/quadd5856)\
**Post date:** [October 2, 2017, 6:40pm UTC](https://discuss.elastic.co/t/kafka-output-plugin-loses-all-data-when-kafka-is-down/102468/1 "2017-10-02T18:40:32Z")

</div>

I am investigating why we lost data from certain servers during a network outage. We are able to reproduce the conditions and can confirm that when the Kafka server is unavailable, anything sent to logstash is lost.

The setup is **Filebeat 5.5.2 on APP-HOST:**

> output.logstash:  
> hosts: ["LOGSTASH-HOST:5043"]

On **LOGSTASH-HOST - logstash 5.6.2 - kafka plugin 0.10.0.1**

> ```
> kafka.conf
> input {
> beats {
> port => "5043"
> }
> }
> output {
> kafka {
> bootstrap_servers => "KAFKA-HOST:9092"
> topic_id => "issue_logs"
> codec => "json"
> }
> }
> 
> ```

The only addition to the logstash.yml is: (but we have tried without)  
queue.type: persisted

**KAFKA-HOST running Kafka 2.11-0-11.0.1**  
I can confirm receiving data into that Kafka topic using this basic setup.  
To simulate the network outage, I am using iptables to drop all packets hitting port 9092 on the kafka server.

> ```
> iptables -A INPUT -p tcp --dport 9092 -j DROP
> 
> ```

When I enable the above iptables setting, effectively mimicking a network outage, any logs sent through filebeat are seen to arrive at LOGSTASH-HOST - but there is no logging or error messaging to be seen as to the connection status to Kafka or any errors at all in Logstash debug logs.

As soon as I disable the iptables setting (service iptables stop):  
New data will arrive in that Kafka topic - anything sent before this though, is gone!

I have tried these settings to the Logstash kafka.conf file in hopes one of them may help but none of them seem to have any impact on this issue.

> ```
> acks => "1"
> retries => 99
> request_timeout_ms => "5000"
> reconnect_backoff_ms => 50
> retry_backoff_ms => 5000
> block_on_buffer_full => true
> metadata_max_age_ms => 10000
> 
> ```

Am I missing an obvious setting here?  
I can also confirm if I go directly from Filebeat to Kafka, things are fine. Filebeat itself does not lose anything and sends everything to Kafka once it is back up. But having Logstash in the middle is our desired setup at this time.

---

<div class="post-metadata">

**Author:** ![quadd5856](https://avatars.discourse-cdn.com/v4/letter/q/c6cbf5/32.png) [@quadd5856](https://discuss.elastic.co/u/quadd5856)\
**Post date:** [October 2, 2017, 8:22pm UTC](https://discuss.elastic.co/t/kafka-output-plugin-loses-all-data-when-kafka-is-down/102468/2 "2017-10-02T20:22:48Z")

</div>

A big cooincidence we just ran across this below just updated, it appears that this issue is being addressed - I will test this change out and see if it helps.

> <https://github.com/logstash-plugins/logstash-output-kafka/pull/151>

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\
**Post date:** [October 30, 2017, 8:22pm UTC](https://discuss.elastic.co/t/kafka-output-plugin-loses-all-data-when-kafka-is-down/102468/3 "2017-10-30T20:22:55Z")

</div>

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.