Why does filebeat register a new broker when inserting data into topic of kafka?

I tried to insert data into the cta_test topic of kafka using filebeat.

However, the topic of cta_test had no data.

So I checked the filebeat log and found a new broker registration part there.

I intended to access kafka from the filebeat.yml file to domain "kafka-001-cluster.priv.com ~ kafka-003-cluster.priv.com".

but the connection was reset to "kafka-001.publ.com ~ kafka-003.publ.com."

I wonder why it was reset to kafka-00*.publ.com instead of kafka-00*-cluster.priv.com that I intended.

  • filebeat version : 6.3.0

  • kafka version : 2.0.0

  • filebeat log

## first try log ##
2020-12-30T09:55:07.734+0900    DEBUG   [kafka] kafka/client.go:69      connect: [kafka-001-cluster.priv.com:9092,kafka-002-cluster.priv.com:9092 kafka-003-cluster.priv.com:9092]
2020-12-30T09:55:07.734+0900    INFO    kafka/log.go:36 kafka message: Initializing new client
2020-12-30T09:55:07.734+0900    INFO    kafka/log.go:36 client/metadata fetching metadata for all topics from broker kafka-003-cluster.priv.com:9092
2020-12-30T09:55:07.734+0900    DEBUG   [harvester]     log/log.go:85   End of file reached: /home/logs/access_log; Backoff now.
2020-12-30T09:55:07.794+0900    INFO    kafka/log.go:36 Connected to broker at kafka-003-cluster.priv.com:9092 (unregistered)
2020-12-30T09:55:07.795+0900    INFO    kafka/log.go:36 client/brokers registered new broker #2 at kafka-002.publ.com:9092
2020-12-30T09:55:07.795+0900    INFO    kafka/log.go:36 client/brokers registered new broker #1 at kafka-001.publ.com:9092
2020-12-30T09:55:07.795+0900    INFO    kafka/log.go:36 client/brokers registered new broker #3 at kafka-003.publ.com:9092
2020-12-30T09:55:07.795+0900    INFO    kafka/log.go:36 kafka message: Successfully initialized new client
2020-12-30T09:55:07.795+0900    INFO    kafka/log.go:36 producer/broker/3 starting up
2020-12-30T09:55:07.795+0900    INFO    kafka/log.go:36 producer/broker/3 state ctange to [open] on cta_test/9

2020-12-30T09:55:37.795+0900    INFO    kafka/log.go:36 Failed to connect to broker kafka-003.publ.com:9092: dial tcp 221.223.45.43:9092: i/o timeout
2020-12-30T09:55:37.795+0900    INFO    kafka/log.go:36 producer/broker/3 state ctange to [closing] because dial tcp 221.223.45.43:9092: i/o timeout
2020-12-30T09:55:37.795+0900    INFO    kafka/log.go:36 producer/leader/cta_test/9 state ctange to [retrying-1]
2020-12-30T09:55:37.795+0900    INFO    kafka/log.go:36 producer/leader/cta_test/9 abandoning broker 3
2020-12-30T09:55:37.795+0900    INFO    kafka/log.go:36 producer/broker/3 shut down


2020-12-30T09:55:37.895+0900    INFO    kafka/log.go:36 client/metadata fetching metadata for [cta_test] from broker kafka-003.cluster.priv.com:9092
2020-12-30T09:55:37.896+0900    INFO    kafka/log.go:36 producer/broker/3 starting up
2020-12-30T09:55:37.896+0900    INFO    kafka/log.go:36 producer/broker/3 state ctange to [open] on cta_test/9
2020-12-30T09:55:37.896+0900    INFO    kafka/log.go:36 producer/leader/cta_test/9 selected broker 3
2020-12-30T09:55:37.896+0900    INFO    kafka/log.go:36 producer/leader/cta_test/9 state ctange to [flushing-1]
2020-12-30T09:55:37.896+0900    INFO    kafka/log.go:36 producer/leader/cta_test/9 state ctange to [normal]


## retry log##
kafka/log.go:36 Failed to connect to broker kafka-003.publ.com:9092: dial tcp 221.223.45.43:9092: i/o timeout
2020-12-30T15:13:41.840+0900    INFO    kafka/log.go:36 producer/broker/3 state change to [closing] because dial tcp 221.223.45.43:9092: i/o timeout
2020-12-30T15:13:41.840+0900    INFO    kafka/log.go:36 producer/leader/cta_test/3 state change to [retrying-1]
2020-12-30T15:13:41.840+0900    INFO    kafka/log.go:36 producer/leader/cta_test/3 abandoning broker 3
2020-12-30T15:13:41.840+0900    INFO    kafka/log.go:36 producer/leader/cta_test/9 state change to [retrying-2]
2020-12-30T15:13:41.840+0900    INFO    kafka/log.go:36 producer/leader/cta_test/9 abandoning broker 3
2020-12-30T15:13:41.840+0900    INFO    kafka/log.go:36 producer/broker/3 shut down
2020-12-30T15:13:41.940+0900    INFO    kafka/log.go:36 client/metadata fetching metadata for [cta_test] from broker kafka-003.publ.com:9092
2020-12-30T15:13:41.940+0900    INFO    kafka/log.go:36 client/metadata fetching metadata for [cta_test] from broker kafka-003.publ.com:9092
2020-12-30T15:13:41.941+0900    INFO    kafka/log.go:36 producer/broker/3 starting up
2020-12-30T15:13:41.941+0900    INFO    kafka/log.go:36 producer/broker/3 state change to [open] on cta_test/9
2020-12-30T15:13:41.941+0900    INFO    kafka/log.go:36 producer/leader/cta_test/9 selected broker 3
2020-12-30T15:13:41.941+0900    INFO    kafka/log.go:36 producer/leader/cta_test/9 state change to [flushing-2]
2020-12-30T15:13:41.941+0900    INFO    kafka/log.go:36 producer/leader/cta_test/9 state change to [normal]
2020-12-30T15:13:41.941+0900    INFO    kafka/log.go:36 producer/leader/cta_test/3 selected broker 3
2020-12-30T15:13:41.941+0900    INFO    kafka/log.go:36 producer/broker/3 state change to [open] on cta_test/3
2020-12-30T15:13:41.941+0900    INFO    kafka/log.go:36 producer/leader/cta_test/3 state change to [flushing-1]
2020-12-30T15:13:41.941+0900    INFO    kafka/log.go:36 producer/leader/cta_test/3 state change to [normal]
  • filebeat.yml
#------------------------------- Kafka output ----------------------------------
output.kafka:
  enabled: true

  hosts: ["kafka-001-cluster.priv.com:9092,kafka-002-cluster.priv.com:9092","kafka-003-cluster.priv.com:9092"]

  topic: "cta_test"

  codec.format:
    string: '"offset":%{[offset]}, "message":%{[message]}'