Unable send logs from filebeat 6 to kafka

[ktella (Kiran Tella)]

Below is my filebeat6 config.

-sh-4.1$ grep -v # filebeat.yml
filebeat.prospectors:

• type: log
  enabled: true
  paths:
  • /var/log/nginx/ss-xxxxx.access.log
    exclude_files: ['.gz$']
    tags: ["xxxxx"]

filebeat.config.modules:
path: ${path.config}/modules.d/*.yml
reload.enabled: false
setup.template.settings:
index.number_of_shards: 3
name: xxxxx.corp.xxxxx.com
tags: ["somehosts"]
env: staging
setup.kibana:
host: "xxx-centos7.corp.xxxx.com:5601"
output.kafka:
hosts: ["xxxx-centos7.corp.xxxx.com:9092"]
topic: '%{[fields.log_topic]}'
version: 0.11.0.0
partition.round_robin:
reachable_only: true
required_acks: 1
compression: gzip
max_message_bytes: 1000000
worker: 8

A similar config of filebeat 5 is able to publish logs to the same kafka instance successfully, but not 6. I'm working on upgrading all my ELK versions 6.x latest ones.

Below is the DEBUG logs in filebeat6.

2017-12-28T17:53:24-05:00 INFO Non-zero metrics in the last 30s: beat.info.uptime.ms=30000 beat.memstats.gc_next=13153488 beat.memstats.memory_alloc=9390664 beat.memstats.memory_total=149112439096 filebeat.harvester.open_files=1 filebeat.harvester.running=1 libbeat.config.module.running=0 libbeat.output.events.batches=1005 libbeat.output.events.failed=2058240 libbeat.output.events.total=2058240 libbeat.pipeline.clients=1 libbeat.pipeline.events.active=4117 libbeat.pipeline.events.retry=2058240 registrar.states.current=1
2017-12-28T17:53:54-05:00 INFO Non-zero metrics in the last 30s: beat.info.uptime.ms=30000 beat.memstats.gc_next=13355504 beat.memstats.memory_alloc=7904424 beat.memstats.memory_total=152900545008 filebeat.harvester.open_files=1 filebeat.harvester.running=1 libbeat.config.module.running=0 libbeat.output.events.batches=968 libbeat.output.events.failed=1982464 libbeat.output.events.total=1982464 libbeat.pipeline.clients=1 libbeat.pipeline.events.active=4117 libbeat.pipeline.events.retry=1982464 registrar.states.current=1

2017-12-28T18:03:14-05:00 DBG [kafka] Kafka publish failed with: circuit breaker is open
2017-12-28T18:03:14-05:00 DBG [kafka] finished kafka batch
2017-12-28T18:03:14-05:00 DBG [kafka] Kafka publish failed with: circuit breaker is open
2017-12-28T18:03:14-05:00 DBG [kafka] finished kafka batch
2017-12-28T18:03:14-05:00 DBG [kafka] Kafka publish failed with: circuit breaker is open
2017-12-28T18:03:14-05:00 DBG [kafka] finished kafka batch
2017-12-28T18:03:14-05:00 DBG [kafka] Kafka publish failed with: circuit breaker is open
2017-12-28T18:03:14-05:00 DBG [kafka] finished kafka batch
2017-12-28T18:03:14-05:00 DBG [kafka] Kafka publish failed with: circuit breaker is open

[steffens (Steffen Siering)]

Basically all events have been failed by the output:

libbeat.output.events.failed=2058240 libbeat.output.events.total=2058240 

As the circuit breaker in the kafka output did kick in, I wonder if there are some more/older messages in the logs stating connection/IO errors.