# Reliable Kafka Output Plugin config

**URL:** <https://discuss.elastic.co/t/reliable-kafka-output-plugin-config/86998>\
**Category:** Logstash\
**Created:** [May 24, 2017, 3:16pm UTC](https://discuss.elastic.co/t/reliable-kafka-output-plugin-config/86998 "2017-05-24T15:16:17Z")\
**Posts on this page:** 3\
**Page:** 1

<div class="post-metadata">

**Author:** ![yuri1969](https://avatars.discourse-cdn.com/v4/letter/y/ea5d25/32.png) [@yuri1969](https://discuss.elastic.co/u/yuri1969)\
**Post date:** [May 24, 2017, 3:16pm UTC](https://discuss.elastic.co/t/reliable-kafka-output-plugin-config/86998/1 "2017-05-24T15:16:17Z")

</div>

Hello,  
I'm wondering about a way how to make output to Kafka reliable. I am new to Kafka, so I'm sorry about any logical errors.

I use Logstash 5.4.0 and Kafka 0.10.2.1 with a matching Zookeeper. The Kafka setup is rather simple with a single instance and single broker.

My basic idea is to parse events from pseudo-log files on file system and send them to Kafka in a reliable manner.

I thought about network issues, Kafka being down, etc. So I added retries etc. to my Kafka output plugin config. The basic test scenario involving stdin/stdout was:

- fire up both Kafka and Logstash
- Logstash emits events based on stdin input
- read events from a Kafka topic with a simple stdout consumer
- stop Kafka
- keep emitting events in Logstash - they should be retrying, right?
- start Kafka again
- read events from start of the Kafka topic - retried should be present, right?

After I start the Kafka **no new events are shown**. Thus it seems to me like the retry policy doesn't take effect.

Pipeline setup:

```
 input { stdin { } }
 output {
    kafka {
       codec => plain {
          format => "%{message}"
       }
       topic_id => "test"
       client_id => "test_console_client"
       bootstrap_servers => "kafka:9092"
       acks => "all"
       retries => 999 
       retry_backoff_ms => 10000
       block_on_buffer_full => true
   } 
}

```

I've tried to examine the pipeline's internals through `_node/stats/pipeline`. I got equal no. of `"in"` and `"out"` events for the Kafka plugin even when the Kafka was down.

Is my setup wrong? Is the testing idea flawed?

Thanks in advance!

---

<div class="post-metadata">

**Author:** ![yuri1969](https://avatars.discourse-cdn.com/v4/letter/y/ea5d25/32.png) [@yuri1969](https://discuss.elastic.co/u/yuri1969)\
**Post date:** [May 25, 2017, 1:16pm UTC](https://discuss.elastic.co/t/reliable-kafka-output-plugin-config/86998/2 "2017-05-25T13:16:22Z")

</div>

The cause is a setting called **[max.block.ms](http://max.block.ms)**. The default is set to 60 000ms. After this timeout the message is dropped.

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\
**Post date:** [June 22, 2017, 1:19pm UTC](https://discuss.elastic.co/t/reliable-kafka-output-plugin-config/86998/3 "2017-06-22T13:19:35Z")

</div>

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.