# Filebeat to kafka failure

**URL:** <https://discuss.elastic.co/t/filebeat-to-kafka-failure/270163>\
**Category:** Beats\
**Tags:** filebeat\
**Created:** [April 15, 2021, 12:56am UTC](https://discuss.elastic.co/t/filebeat-to-kafka-failure/270163 "2021-04-15T00:56:24Z")\
**Posts on this page:** 6\
**Page:** 1

<div class="post-metadata">

**Author:** ![madhan\_dhanikachalam](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/madhan_dhanikachalam/32/86074_2.png) [@madhan\_dhanikachalam](https://discuss.elastic.co/u/madhan_dhanikachalam)\
**Post date:** [April 15, 2021, 12:56am UTC](https://discuss.elastic.co/t/filebeat-to-kafka-failure/270163/1 "2021-04-15T00:56:24Z")

</div>

filebeat on server 1 seems to send log data to kafka perfectly, no issues..

set up filebeat on server2 to send log data to kafka and get the below error

```
2021-04-14T20:46:28.475-0400 INFO [publisher] pipeline/retry.go:221   
retryer: send unwait signal to consumer
2021-04-14T20:46:28.475

```

below is the filebeat.yml

```
filebeat.inputs:

# Each - is an input. Most options can be set at the input level, so
# you can use different inputs for various configurations.
# Below are the input specific configurations.

- type: log

  # Change to true to enable this input configuration.
  enabled: true

  # Paths that should be crawled and fetched. Glob based paths.
  paths:
    - /tmp/filebeat.txt
    #- c:\programdata\elasticsearch\logs\*

  close_inactive: 10m
  ignore_older: 2h
  scan_frequency: 10s
  clean_inactive: 3h
   # Exclude lines. A list of regular expressions to match. It drops the lines that 
are
  
# ================================== Outputs 
===================================

# Configure what output to use when sending the data collected by the beat.

#---- kafka output

output.kafka:
  hosts: ["host1:9092","host2:9092","host3.9092"]
  topic: 'FIT1'
  partition.round_robin:
    reachable_only: true
  required_acks: 1
  compression: gzip
  max_message_bytes: 1000000

```

added the below after reading some other posts

```
 close_inactive: 10m
 ignore_older: 2h
 scan_frequency: 10s
 clean_inactive: 3h

```

still same error.

filebeat version filebeat-7.8.0

---

<div class="post-metadata">

**Author:** ![mtojek](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/mtojek/32/63863_2.png) [@mtojek](https://discuss.elastic.co/u/mtojek)\
**Post date:** [April 15, 2021, 8:37am UTC](https://discuss.elastic.co/t/filebeat-to-kafka-failure/270163/2 "2021-04-15T08:37:21Z")

</div>

Did you try to check if Kafka is reachable on the server2? Did you try it with other client/telnet?

---

<div class="post-metadata">

**Author:** ![madhan\_dhanikachalam](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/madhan_dhanikachalam/32/86074_2.png) [@madhan\_dhanikachalam](https://discuss.elastic.co/u/madhan_dhanikachalam)\
**Post date:** [April 15, 2021, 10:47am UTC](https://discuss.elastic.co/t/filebeat-to-kafka-failure/270163/3 "2021-04-15T10:47:14Z")

</div>

telnet works, there is network connectivity.

---

<div class="post-metadata">

**Author:** ![madhan\_dhanikachalam](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/madhan_dhanikachalam/32/86074_2.png) [@madhan\_dhanikachalam](https://discuss.elastic.co/u/madhan_dhanikachalam)\
**Post date:** [April 15, 2021, 10:54am UTC](https://discuss.elastic.co/t/filebeat-to-kafka-failure/270163/4 "2021-04-15T10:54:48Z")

</div>

tried in debug mode and see this error

Kafka publish failed with: circuit breaker is open

---

<div class="post-metadata">

**Author:** ![madhan\_dhanikachalam](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/madhan_dhanikachalam/32/86074_2.png) [@madhan\_dhanikachalam](https://discuss.elastic.co/u/madhan_dhanikachalam)\
**Post date:** [April 15, 2021, 1:57pm UTC](https://discuss.elastic.co/t/filebeat-to-kafka-failure/270163/5 "2021-04-15T13:57:56Z")

</div>

this worked after upgrading to filebeat7.12

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\
**Post date:** [May 13, 2021, 3:58pm UTC](https://discuss.elastic.co/t/filebeat-to-kafka-failure/270163/6 "2021-05-13T15:58:51Z")

</div>

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.