When kafka broker timeout ,my filebeat always retrying

Elastic Stack Beats
1

my filebeat version 6.4.0 (amd64), libbeat 6.4.0
when kafka broker timeout ,my filebeat always retrying, don't put log,util restart filebeat,Please help me take a look。

filebeat log is this.....2024-01-23T20:38:21.439+0800 INFO kafka/log.go:53 producer/broker/2681 state change to [closing] because read tcp 10.2.36.
76:44752->10.60.26.81:9110: i/o timeout

2024-01-23T20:38:21.439+0800 INFO kafka/log.go:53 Closed connection to broker 10.60.26.81:9110

2024-01-23T20:38:21.439+0800 INFO kafka/log.go:53 producer/leader/my-topic/126 state change to [retrying-
1]

2024-01-23T20:38:21.439+0800 INFO kafka/log.go:53 producer/leader/my-topic/126 abandoning broker 2681

2024-01-23T20:38:21.439+0800 INFO kafka/log.go:53 producer/broker/2681 shut down

2024-01-23T20:38:21.539+0800 INFO kafka/log.go:53 client/metadata fetching metadata for [my-topic] from b
roker 10.61.23.25:9110

2024-01-23T20:38:21.544+0800 INFO kafka/log.go:53 producer/broker/2681 starting up

2024-01-23T20:38:21.544+0800 INFO kafka/log.go:53 producer/broker/2681 state change to [open] on my-topic
/126

2024-01-23T20:38:21.544+0800 INFO kafka/log.go:53 producer/leader/my-topic/126 selected broker 2681

2024-01-23T20:38:21.544+0800 INFO kafka/log.go:53 producer/leader/my-topic/126 state change to [flushing-
1]

2024-01-23T20:38:21.544+0800 INFO kafka/log.go:53 producer/leader/my-topic/126 state change to [normal]

....

2024-01-23T20:57:04.635+0800 INFO kafka/log.go:53 client/metadata fetching metadata for all topics from broker 10.60.26.82
:9110

2024-01-23T20:57:04.635+0800 INFO kafka/log.go:53 kafka message: client/metadata got error from broker while fetching meta
data:%!(EXTRA *errors.errorString=EOF)
2024-01-23T20:57:04.635+0800 INFO kafka/log.go:53 Closed connection to broker 10.60.26.82:9110

2024-01-23T20:57:04.635+0800 INFO kafka/log.go:53 client/brokers deregistered broker #2682 at 10.60.26.82:9110
2024-01-23T20:57:04.635+0800 INFO kafka/log.go:53 client/metadata fetching metadata for all topics from broker 10.60.63.55
:9110

2024-01-23T20:57:04.635+0800 INFO kafka/log.go:53 kafka message: client/metadata got error from broker while fetching meta
data:%!(EXTRA *errors.errorString=EOF)
2024-01-23T20:57:04.635+0800 INFO kafka/log.go:53 Closed connection to broker 10.60.63.55:9110

2024-01-23T20:57:04.635+0800 INFO kafka/log.go:53 client/brokers deregistered broker #70149 at 10.60.63.55:9110
2024-01-23T20:57:04.635+0800 INFO kafka/log.go:53 client/metadata fetching metadata for all topics from broker 10.60.20.18
3:9110

2024-01-23T20:57:04.635+0800 INFO kafka/log.go:53 kafka message: client/metadata got error from broker while fetching meta
data:%!(EXTRA *errors.errorString=EOF)
2024-01-23T20:57:04.635+0800 INFO kafka/log.go:53 Closed connection to broker 10.60.20.183:9110

2024-01-23T20:57:04.635+0800 INFO kafka/log.go:53 client/brokers deregistered broker #20183 at 10.60.20.183:9110
2024-01-23T20:57:04.635+0800 INFO kafka/log.go:53 client/metadata fetching metadata for all topics from broker 10.60.18.13
1:9110
....

2

Filebeat 6.4.0 is about 4.5 years old at this point so my first recommendation would be to upgrade your filebeat to a more recent version.

It will be difficult to know whether your issue is with your config, Kafka, or due to an already fixed bug in Filebeat given the age of the version you've got deployed.

3

thank you! my config is this.

#=========================== Filebeat inputs =============================
filebeat.inputs:
- type: log
  enabled: true
  paths:
    - "/data0/test.log"
  harvester_buffer_size: 5242880
  harvester_limit: 50
  max_bytes: 5242880
  close_inactive: "1h"
  scan_frequency: "10s"

#========================= Filebeat global options ============================
filename: "my-topic"
filebeat:
  registry_file: "${path.data}/.${filename}"
  registry_file_permissions: 0600
  shutdown_timeout: "5s"

#================================ General =====================================
name: "${filename}-10.10.10.10"
max_procs: 1
queue.mem:
    events: 10000
    flush.min_events: 1000
    flush.timeout: 10s

#================================ Processors ===================================
processors:
- drop_fields:
    fields: [ "beat", "prospector", "source", "offset", "host", "input" ]

#================================ Outputs =====================================
#------------------------------- Kafka output ----------------------------------
output.kafka:
    hosts: ['broker1:9110', 'broker2:9110', 'broker3:9110', 'broker4:9110', 'broker5:9110']
    topic: "my-topic"
    version: 0.10.2.1
    username: "test"
    password: "testpass"
    partition.round_robin:
      reachable_only: false
    codec.format:
      string: "%{[message]}"
    client_id: "{{grains['id']}}"
    compression: "snappy"
    retry.backoff: 1000
    bulk_max_size: 10000
    max_message_bytes: 1000000
    channel_buffer_size: 5000
    required_acks: 1

#================================= Paths ======================================
path:
  home: /data0/soft_app/monitor/filebeat
  config: ${path.home}
  data: ${path.home}/data
  logs: ${path.home}/logs


#================================ Logging =====================================
logging.level: debug
logging.selectors: [memqueue, metrics, beat]
logging.metrics.period: 60s
logging.to_files: true
logging.to_syslog: false
logging.files:
  path: ${path.logs}
  name: filebeat-${filename}.log
  rotateeverybytes: 52428800
  keepfiles: 1
  json: true
4

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.