# How to restart or kill logstash process when kafka output plugin failed

**URL:** <https://discuss.elastic.co/t/how-to-restart-or-kill-logstash-process-when-kafka-output-plugin-failed/116395>\
**Category:** Logstash\
**Created:** [January 21, 2018, 5:37pm UTC](https://discuss.elastic.co/t/how-to-restart-or-kill-logstash-process-when-kafka-output-plugin-failed/116395 "2018-01-21T17:37:02Z")\
**Posts on this page:** 2\
**Page:** 1

<div class="post-metadata">

**Author:** ![berlin](https://avatars.discourse-cdn.com/v4/letter/b/41988e/32.png) [@berlin](https://discuss.elastic.co/u/berlin)\
**Post date:** [January 21, 2018, 5:37pm UTC](https://discuss.elastic.co/t/how-to-restart-or-kill-logstash-process-when-kafka-output-plugin-failed/116395/1 "2018-01-21T17:37:02Z")

</div>

I am using `docker.elastic.co/logstash/logstash-oss:6.0.0` with `kafka output plugin`,  
Logstash output kafka plugin is not pushing data into kafka when one of the kafka nodes go down or get different broker id.

```
1/19/2018 10:36:47 PM[2018-01-19T20:36:47,283][WARN][org.apache.kafka.clients.NetworkClient] Connection to node 1 could not be established. Broker may not be available.
1/19/2018 11:16:44 PM[2018-01-19T21:16:44,320][INFO][logstash.outputs.kafka] Sending batch to Kafka failed. Will retry after a delay. {:batch_size=>1, :failures=>1, :sleep=>0.01}
1/19/2018 11:46:44 PM[2018-01-19T21:46:44,876][INFO][logstash.outputs.kafka] Sending batch to Kafka failed. Will retry after a delay. {:batch_size=>1, :failures=>1, :sleep=>0.01}
1/21/2018 5:08:48 PM[2018-01-21T15:08:48,645][INFO][logstash.outputs.kafka] Sending batch to Kafka failed. Will retry after a delay. {:batch_size=>1, :failures=>1, :sleep=>0.01}
1/21/2018 5:08:49 PM[2018-01-21T15:08:49,241][INFO][logstash.outputs.kafka] Sending batch to Kafka failed. Will retry after a delay. {:batch_size=>1, :failures=>1, :sleep=>0.01}

```

`retries` parameter might not help in the case because the broker id can be different from the one when logstash container start with (for example kafka brokers can change from [1,2,3] to [1,2,4].

```
  # If you choose to set `retries`, a value greater than zero will cause the
  # client to only retry a fixed number of times. This will result in data loss
  # if a transient error outlasts your retry count.
  #

```

[https://www.elastic.co/guide/en/logstash/5.6/plugins-outputs-kafka.html](https://www.elastic.co/guide/en/logstash/5.6/plugins-outputs-kafka.html)

Is there a way to force logstash to exit / kill the process in this case, this way a new logstash container will be launched with the new brokers ids and the service will start properly.

```
output {
  kafka {
    bootstrap_servers=> "kafka:9092"
    topic_id=> "topic"
    codec=> "json"
    message_key=> "key"
  }
  #stdout { codec => "rubydebug" }
}

```

Thanks

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\
**Post date:** [February 18, 2018, 5:37pm UTC](https://discuss.elastic.co/t/how-to-restart-or-kill-logstash-process-when-kafka-output-plugin-failed/116395/2 "2018-02-18T17:37:07Z")

</div>

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.