# Kafka publish failed with: circuit breaker is open

**URL:** <https://discuss.elastic.co/t/kafka-publish-failed-with-circuit-breaker-is-open/123339>\
**Category:** Beats\
**Tags:** filebeat\
**Created:** [March 9, 2018, 8:32pm UTC](https://discuss.elastic.co/t/kafka-publish-failed-with-circuit-breaker-is-open/123339 "2018-03-09T20:32:06Z")\
**Posts on this page:** 3\
**Page:** 1

<div class="post-metadata">

**Author:** ![andlaz](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/andlaz/32/28643_2.png) [@andlaz](https://discuss.elastic.co/u/andlaz)\
**Post date:** [March 9, 2018, 8:32pm UTC](https://discuss.elastic.co/t/kafka-publish-failed-with-circuit-breaker-is-open/123339/1 "2018-03-09T20:32:06Z")

</div>

hi all, i'm having a problem with the kafka output in that it sends a couple MB worth of log messages right after start, then not a single one. It does not log much about a connection issue, the kafka cluster is healthy, the topics are healthy..

here are some relevant logs from the top

```
2018-03-09T20:18:10.114Z INFO kafka/log.go:36 kafka message: [Initializing new client]
2018-03-09T20:18:10.114Z INFO kafka/log.go:36 client/metadata fetching metadata for all topics from broker [[kafka-1.kafka:9092]] 
2018-03-09T20:18:10.115Z INFO kafka/log.go:36 Connected to broker at [[kafka-1.kafka:9092]] (unregistered)
2018-03-09T20:18:10.117Z INFO kafka/log.go:36 client/brokers registered new broker #[[1001 %!d(string=kafka-1.kafka.logging.svc.cluster.local:9092)]] at %!s(MISSING)
2018-03-09T20:18:10.117Z INFO kafka/log.go:36 client/brokers registered new broker #[[1003 %!d(string=kafka-2.kafka.logging.svc.cluster.local:9092)]] at %!s(MISSING)
2018-03-09T20:18:10.117Z INFO kafka/log.go:36 client/brokers registered new broker #[[1002 %!d(string=kafka-0.kafka.logging.svc.cluster.local:9092)]] at %!s(MISSING)
2018-03-09T20:18:10.117Z INFO kafka/log.go:36 kafka message: [Successfully initialized new client]
2018-03-09T20:18:10.119Z INFO kafka/log.go:36 producer/broker/[[1001]] starting up

2018-03-09T20:18:10.121Z INFO kafka/log.go:36 producer/broker/[[1001 %!d(string=shared) 2]] state change to [open] on %!s(MISSING)/%!d(MISSING)
2018-03-09T20:18:10.121Z INFO kafka/log.go:36 producer/broker/[[1001 %!d(string=kube-system) 1]] state change to [open] on %!s(MISSING)/%!d(MISSING)

2018-03-09T20:18:10.121Z INFO kafka/log.go:36 producer/broker/[[1001 %!d(string=boldt5) 2]] state change to [open] on %!s(MISSING)/%!d(MISSING)

2018-03-09T20:18:10.119Z INFO kafka/log.go:36 producer/broker/[[1002]] starting up

2018-03-09T20:18:10.121Z INFO kafka/log.go:36 producer/broker/[[1001 %!d(string=timeseries) 1]] state change to [open] on %!s(MISSING)/%!d(MISSING)

2018-03-09T20:18:10.119Z INFO kafka/log.go:36 producer/broker/[[1003]] starting up

2018-03-09T20:18:10.121Z INFO kafka/log.go:36 producer/broker/[[1001 %!d(string=logging) 1]] state change to [open] on %!s(MISSING)/%!d(MISSING)

2018-03-09T20:18:10.121Z INFO kafka/log.go:36 producer/broker/[[1003 %!d(string=kube-system) 0]] state change to [open] on %!s(MISSING)/%!d(MISSING)

```

then, after a first small batch is sent

```
2018-03-09T20:21:41.933Z DEBUG [kafka] kafka/client.go:234 Kafka publish failed with: circuit breaker is open
2018-03-09T20:21:41.938Z DEBUG [kafka] kafka/client.go:220 finished kafka batch

```

versions:

filebeat container image: [docker.elastic.co/beats/filebeat:6.2.2](http://docker.elastic.co/beats/filebeat:6.2.2)  
kafka container image: solsson/kafka:0.11.0.0

config:

filebeat.yml -

```
    filebeat.config:
      prospectors:
        # Mounted `filebeat-prospectors` configmap:
        path: ${path.config}/prospectors.d/*.yml
        # Reload prospectors configs as they change:
        reload.enabled: false
      modules:
        path: ${path.config}/modules.d/*.yml
        # Reload module configs as they change:
        reload.enabled: false

    processors:
      - add_cloud_metadata:

    output.kafka:
      # client_id: '${POD_NAMESPACE}-${NODE_NAME}'
      hosts: ['kafka-0.kafka:9092', 'kafka-1.kafka:9092', 'kafka-2.kafka:9092']
      topic: '%{[kubernetes.namespace]}'

      partition.round_robin:
        reachable_only: true

      required_acks: 0
      compression: gzip
      max_message_bytes: 1000000

      # version: 0.11.0.0

    logging.level: debug
    # logging.selectors: [kafka,output]

```

kubernetes.yml -

```
- type: log
  paths:
    - /var/lib/docker/containers/*/*.log
  json.message_key: log
  json.keys_under_root: true
  processors:
    - add_kubernetes_metadata:
        in_cluster: true
        namespace: ${POD_NAMESPACE}
```

---

<div class="post-metadata">

**Author:** ![andlaz](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/andlaz/32/28643_2.png) [@andlaz](https://discuss.elastic.co/u/andlaz)\
**Post date:** [March 12, 2018, 3:36pm UTC](https://discuss.elastic.co/t/kafka-publish-failed-with-circuit-breaker-is-open/123339/2 "2018-03-12T15:36:08Z")

</div>

I guess my question is, how do i even begin to debug this..? Is there a way to see actual connection issues apart from setting `logging.level` ?

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\
**Post date:** [April 9, 2018, 3:36pm UTC](https://discuss.elastic.co/t/kafka-publish-failed-with-circuit-breaker-is-open/123339/3 "2018-04-09T15:36:18Z")

</div>

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.