# Unable send logs from filebeat 6 to kafka

**URL:** <https://discuss.elastic.co/t/unable-send-logs-from-filebeat-6-to-kafka/113513>\
**Category:** Beats\
**Created:** [December 28, 2017, 11:41pm UTC](https://discuss.elastic.co/t/unable-send-logs-from-filebeat-6-to-kafka/113513 "2017-12-28T23:41:35Z")\
**Posts on this page:** 3\
**Page:** 1

<div class="post-metadata">

**Author:** ![ktella](https://avatars.discourse-cdn.com/v4/letter/k/71e660/32.png) [@ktella](https://discuss.elastic.co/u/ktella)\
**Post date:** [December 28, 2017, 11:41pm UTC](https://discuss.elastic.co/t/unable-send-logs-from-filebeat-6-to-kafka/113513/1 "2017-12-28T23:41:35Z")

</div>

Below is my filebeat6 config.

-sh-4.1$ grep -v # filebeat.yml  
filebeat.prospectors:

- type: log  
enabled: true  
paths:
  - /var/log/nginx/ss-xxxxx.access.log  
exclude\_files: ['.gz$']  
tags: ["xxxxx"]

filebeat.config.modules:  
path: ${path.config}/modules.d/\*.yml  
reload.enabled: false  
setup.template.settings:  
index.number\_of\_shards: 3  
name: [xxxxx.corp.xxxxx.com](http://xxxxx.corp.xxxxx.com)  
tags: ["somehosts"]  
env: staging  
setup.kibana:  
host: "[xxx-centos7.corp.xxxx.com:5601](http://xxx-centos7.corp.xxxx.com:5601)"  
output.kafka:  
hosts: ["[xxxx-centos7.corp.xxxx.com:9092](http://xxxx-centos7.corp.xxxx.com:9092)"]  
topic: '%{[fields.log\_topic]}'  
version: 0.11.0.0  
partition.round\_robin:  
reachable\_only: true  
required\_acks: 1  
compression: gzip  
max\_message\_bytes: 1000000  
worker: 8

A similar config of filebeat 5 is able to publish logs to the same kafka instance successfully, but not 6. I'm working on upgrading all my ELK versions 6.x latest ones.

Below is the DEBUG logs in filebeat6.

2017-12-28T17:53:24-05:00 INFO Non-zero metrics in the last 30s: beat.info.uptime.ms=30000 beat.memstats.gc\_next=13153488 beat.memstats.memory\_alloc=9390664 beat.memstats.memory\_total=149112439096 filebeat.harvester.open\_files=1 filebeat.harvester.running=1 libbeat.config.module.running=0 libbeat.output.events.batches=1005 libbeat.output.events.failed=2058240 libbeat.output.events.total=2058240 libbeat.pipeline.clients=1 libbeat.pipeline.events.active=4117 libbeat.pipeline.events.retry=2058240 registrar.states.current=1  
2017-12-28T17:53:54-05:00 INFO Non-zero metrics in the last 30s: beat.info.uptime.ms=30000 beat.memstats.gc\_next=13355504 beat.memstats.memory\_alloc=7904424 beat.memstats.memory\_total=152900545008 filebeat.harvester.open\_files=1 filebeat.harvester.running=1 libbeat.config.module.running=0 libbeat.output.events.batches=968 libbeat.output.events.failed=1982464 libbeat.output.events.total=1982464 libbeat.pipeline.clients=1 libbeat.pipeline.events.active=4117 libbeat.pipeline.events.retry=1982464 registrar.states.current=1

2017-12-28T18:03:14-05:00 DBG [kafka] Kafka publish failed with: circuit breaker is open  
2017-12-28T18:03:14-05:00 DBG [kafka] finished kafka batch  
2017-12-28T18:03:14-05:00 DBG [kafka] Kafka publish failed with: circuit breaker is open  
2017-12-28T18:03:14-05:00 DBG [kafka] finished kafka batch  
2017-12-28T18:03:14-05:00 DBG [kafka] Kafka publish failed with: circuit breaker is open  
2017-12-28T18:03:14-05:00 DBG [kafka] finished kafka batch  
2017-12-28T18:03:14-05:00 DBG [kafka] Kafka publish failed with: circuit breaker is open  
2017-12-28T18:03:14-05:00 DBG [kafka] finished kafka batch  
2017-12-28T18:03:14-05:00 DBG [kafka] Kafka publish failed with: circuit breaker is open

---

<div class="post-metadata">

**Author:** ![steffens](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/steffens/32/79630_2.png) [@steffens](https://discuss.elastic.co/u/steffens)\
**Post date:** [January 4, 2018, 1:01pm UTC](https://discuss.elastic.co/t/unable-send-logs-from-filebeat-6-to-kafka/113513/2 "2018-01-04T13:01:49Z")

</div>

Basically all events have been failed by the output:

```auto
libbeat.output.events.failed=2058240 libbeat.output.events.total=2058240 

```

As the circuit breaker in the kafka output did kick in, I wonder if there are some more/older messages in the logs stating connection/IO errors.

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\
**Post date:** [January 18, 2018, 11:41pm UTC](https://discuss.elastic.co/t/unable-send-logs-from-filebeat-6-to-kafka/113513/3 "2018-01-18T23:41:44Z")

</div>

This topic was automatically closed after 21 days. New replies are no longer allowed.