# Why does filebeat register a new broker when inserting data into topic of kafka? **URL:** \ **Category:** Beats\ **Tags:** filebeat\ **Created:** [December 30, 2020, 5:43am UTC](https://discuss.elastic.co/t/why-does-filebeat-register-a-new-broker-when-inserting-data-into-topic-of-kafka/259850 "2020-12-30T05:43:53Z")\ **Posts on this page:** 2\ **Page:** 1
**Author:** ![bae\_park](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/bae_park/32/81552_2.png) [@bae\_park](https://discuss.elastic.co/u/bae_park)\ **Post date:** [December 30, 2020, 5:43am UTC](https://discuss.elastic.co/t/why-does-filebeat-register-a-new-broker-when-inserting-data-into-topic-of-kafka/259850/1 "2020-12-30T05:43:53Z")
I tried to insert data into the cta\_test topic of kafka using filebeat. However, the topic of cta\_test had no data. So I checked the filebeat log and found a new broker registration part there. I intended to access kafka from the filebeat.yml file to domain "[kafka-001-cluster.priv.com](http://kafka-001-cluster.priv.com) ~ [kafka-003-cluster.priv.com](http://kafka-003-cluster.priv.com)". but the connection was reset to "[kafka-001.publ.com](http://kafka-001.publ.com) ~ [kafka-003.publ.com](http://kafka-003.publ.com)." I wonder why it was reset to kafka-00\*.publ.com instead of kafka-00\*-cluster.priv.com that I intended. - filebeat version : 6.3.0 - kafka version : 2.0.0 - filebeat log ```auto ## first try log ## 2020-12-30T09:55:07.734+0900 DEBUG [kafka] kafka/client.go:69 connect: [kafka-001-cluster.priv.com:9092,kafka-002-cluster.priv.com:9092 kafka-003-cluster.priv.com:9092] 2020-12-30T09:55:07.734+0900 INFO kafka/log.go:36 kafka message: Initializing new client 2020-12-30T09:55:07.734+0900 INFO kafka/log.go:36 client/metadata fetching metadata for all topics from broker kafka-003-cluster.priv.com:9092 2020-12-30T09:55:07.734+0900 DEBUG [harvester] log/log.go:85 End of file reached: /home/logs/access_log; Backoff now. 2020-12-30T09:55:07.794+0900 INFO kafka/log.go:36 Connected to broker at kafka-003-cluster.priv.com:9092 (unregistered) 2020-12-30T09:55:07.795+0900 INFO kafka/log.go:36 client/brokers registered new broker #2 at kafka-002.publ.com:9092 2020-12-30T09:55:07.795+0900 INFO kafka/log.go:36 client/brokers registered new broker #1 at kafka-001.publ.com:9092 2020-12-30T09:55:07.795+0900 INFO kafka/log.go:36 client/brokers registered new broker #3 at kafka-003.publ.com:9092 2020-12-30T09:55:07.795+0900 INFO kafka/log.go:36 kafka message: Successfully initialized new client 2020-12-30T09:55:07.795+0900 INFO kafka/log.go:36 producer/broker/3 starting up 2020-12-30T09:55:07.795+0900 INFO kafka/log.go:36 producer/broker/3 state ctange to [open] on cta_test/9 2020-12-30T09:55:37.795+0900 INFO kafka/log.go:36 Failed to connect to broker kafka-003.publ.com:9092: dial tcp 221.223.45.43:9092: i/o timeout 2020-12-30T09:55:37.795+0900 INFO kafka/log.go:36 producer/broker/3 state ctange to [closing] because dial tcp 221.223.45.43:9092: i/o timeout 2020-12-30T09:55:37.795+0900 INFO kafka/log.go:36 producer/leader/cta_test/9 state ctange to [retrying-1] 2020-12-30T09:55:37.795+0900 INFO kafka/log.go:36 producer/leader/cta_test/9 abandoning broker 3 2020-12-30T09:55:37.795+0900 INFO kafka/log.go:36 producer/broker/3 shut down 2020-12-30T09:55:37.895+0900 INFO kafka/log.go:36 client/metadata fetching metadata for [cta_test] from broker kafka-003.cluster.priv.com:9092 2020-12-30T09:55:37.896+0900 INFO kafka/log.go:36 producer/broker/3 starting up 2020-12-30T09:55:37.896+0900 INFO kafka/log.go:36 producer/broker/3 state ctange to [open] on cta_test/9 2020-12-30T09:55:37.896+0900 INFO kafka/log.go:36 producer/leader/cta_test/9 selected broker 3 2020-12-30T09:55:37.896+0900 INFO kafka/log.go:36 producer/leader/cta_test/9 state ctange to [flushing-1] 2020-12-30T09:55:37.896+0900 INFO kafka/log.go:36 producer/leader/cta_test/9 state ctange to [normal] ## retry log## kafka/log.go:36 Failed to connect to broker kafka-003.publ.com:9092: dial tcp 221.223.45.43:9092: i/o timeout 2020-12-30T15:13:41.840+0900 INFO kafka/log.go:36 producer/broker/3 state change to [closing] because dial tcp 221.223.45.43:9092: i/o timeout 2020-12-30T15:13:41.840+0900 INFO kafka/log.go:36 producer/leader/cta_test/3 state change to [retrying-1] 2020-12-30T15:13:41.840+0900 INFO kafka/log.go:36 producer/leader/cta_test/3 abandoning broker 3 2020-12-30T15:13:41.840+0900 INFO kafka/log.go:36 producer/leader/cta_test/9 state change to [retrying-2] 2020-12-30T15:13:41.840+0900 INFO kafka/log.go:36 producer/leader/cta_test/9 abandoning broker 3 2020-12-30T15:13:41.840+0900 INFO kafka/log.go:36 producer/broker/3 shut down 2020-12-30T15:13:41.940+0900 INFO kafka/log.go:36 client/metadata fetching metadata for [cta_test] from broker kafka-003.publ.com:9092 2020-12-30T15:13:41.940+0900 INFO kafka/log.go:36 client/metadata fetching metadata for [cta_test] from broker kafka-003.publ.com:9092 2020-12-30T15:13:41.941+0900 INFO kafka/log.go:36 producer/broker/3 starting up 2020-12-30T15:13:41.941+0900 INFO kafka/log.go:36 producer/broker/3 state change to [open] on cta_test/9 2020-12-30T15:13:41.941+0900 INFO kafka/log.go:36 producer/leader/cta_test/9 selected broker 3 2020-12-30T15:13:41.941+0900 INFO kafka/log.go:36 producer/leader/cta_test/9 state change to [flushing-2] 2020-12-30T15:13:41.941+0900 INFO kafka/log.go:36 producer/leader/cta_test/9 state change to [normal] 2020-12-30T15:13:41.941+0900 INFO kafka/log.go:36 producer/leader/cta_test/3 selected broker 3 2020-12-30T15:13:41.941+0900 INFO kafka/log.go:36 producer/broker/3 state change to [open] on cta_test/3 2020-12-30T15:13:41.941+0900 INFO kafka/log.go:36 producer/leader/cta_test/3 state change to [flushing-1] 2020-12-30T15:13:41.941+0900 INFO kafka/log.go:36 producer/leader/cta_test/3 state change to [normal] ``` - filebeat.yml ```auto #------------------------------- Kafka output ---------------------------------- output.kafka: enabled: true hosts: ["kafka-001-cluster.priv.com:9092,kafka-002-cluster.priv.com:9092","kafka-003-cluster.priv.com:9092"] topic: "cta_test" codec.format: string: '"offset":%{[offset]}, "message":%{[message]}' ``` ---
**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\ **Post date:** [January 27, 2021, 7:43am UTC](https://discuss.elastic.co/t/why-does-filebeat-register-a-new-broker-when-inserting-data-into-topic-of-kafka/259850/2 "2021-01-27T07:43:58Z")
This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.