I tried to insert data into the cta_test topic of kafka using filebeat.
However, the topic of cta_test had no data.
So I checked the filebeat log and found a new broker registration part there.
I intended to access kafka from the filebeat.yml file to domain "kafka-001-cluster.priv.com ~ kafka-003-cluster.priv.com".
but the connection was reset to "kafka-001.publ.com ~ kafka-003.publ.com."
I wonder why it was reset to kafka-00*.publ.com instead of kafka-00*-cluster.priv.com that I intended.
-
filebeat version : 6.3.0
-
kafka version : 2.0.0
-
filebeat log
## first try log ##
2020-12-30T09:55:07.734+0900 DEBUG [kafka] kafka/client.go:69 connect: [kafka-001-cluster.priv.com:9092,kafka-002-cluster.priv.com:9092 kafka-003-cluster.priv.com:9092]
2020-12-30T09:55:07.734+0900 INFO kafka/log.go:36 kafka message: Initializing new client
2020-12-30T09:55:07.734+0900 INFO kafka/log.go:36 client/metadata fetching metadata for all topics from broker kafka-003-cluster.priv.com:9092
2020-12-30T09:55:07.734+0900 DEBUG [harvester] log/log.go:85 End of file reached: /home/logs/access_log; Backoff now.
2020-12-30T09:55:07.794+0900 INFO kafka/log.go:36 Connected to broker at kafka-003-cluster.priv.com:9092 (unregistered)
2020-12-30T09:55:07.795+0900 INFO kafka/log.go:36 client/brokers registered new broker #2 at kafka-002.publ.com:9092
2020-12-30T09:55:07.795+0900 INFO kafka/log.go:36 client/brokers registered new broker #1 at kafka-001.publ.com:9092
2020-12-30T09:55:07.795+0900 INFO kafka/log.go:36 client/brokers registered new broker #3 at kafka-003.publ.com:9092
2020-12-30T09:55:07.795+0900 INFO kafka/log.go:36 kafka message: Successfully initialized new client
2020-12-30T09:55:07.795+0900 INFO kafka/log.go:36 producer/broker/3 starting up
2020-12-30T09:55:07.795+0900 INFO kafka/log.go:36 producer/broker/3 state ctange to [open] on cta_test/9
2020-12-30T09:55:37.795+0900 INFO kafka/log.go:36 Failed to connect to broker kafka-003.publ.com:9092: dial tcp 221.223.45.43:9092: i/o timeout
2020-12-30T09:55:37.795+0900 INFO kafka/log.go:36 producer/broker/3 state ctange to [closing] because dial tcp 221.223.45.43:9092: i/o timeout
2020-12-30T09:55:37.795+0900 INFO kafka/log.go:36 producer/leader/cta_test/9 state ctange to [retrying-1]
2020-12-30T09:55:37.795+0900 INFO kafka/log.go:36 producer/leader/cta_test/9 abandoning broker 3
2020-12-30T09:55:37.795+0900 INFO kafka/log.go:36 producer/broker/3 shut down
2020-12-30T09:55:37.895+0900 INFO kafka/log.go:36 client/metadata fetching metadata for [cta_test] from broker kafka-003.cluster.priv.com:9092
2020-12-30T09:55:37.896+0900 INFO kafka/log.go:36 producer/broker/3 starting up
2020-12-30T09:55:37.896+0900 INFO kafka/log.go:36 producer/broker/3 state ctange to [open] on cta_test/9
2020-12-30T09:55:37.896+0900 INFO kafka/log.go:36 producer/leader/cta_test/9 selected broker 3
2020-12-30T09:55:37.896+0900 INFO kafka/log.go:36 producer/leader/cta_test/9 state ctange to [flushing-1]
2020-12-30T09:55:37.896+0900 INFO kafka/log.go:36 producer/leader/cta_test/9 state ctange to [normal]
## retry log##
kafka/log.go:36 Failed to connect to broker kafka-003.publ.com:9092: dial tcp 221.223.45.43:9092: i/o timeout
2020-12-30T15:13:41.840+0900 INFO kafka/log.go:36 producer/broker/3 state change to [closing] because dial tcp 221.223.45.43:9092: i/o timeout
2020-12-30T15:13:41.840+0900 INFO kafka/log.go:36 producer/leader/cta_test/3 state change to [retrying-1]
2020-12-30T15:13:41.840+0900 INFO kafka/log.go:36 producer/leader/cta_test/3 abandoning broker 3
2020-12-30T15:13:41.840+0900 INFO kafka/log.go:36 producer/leader/cta_test/9 state change to [retrying-2]
2020-12-30T15:13:41.840+0900 INFO kafka/log.go:36 producer/leader/cta_test/9 abandoning broker 3
2020-12-30T15:13:41.840+0900 INFO kafka/log.go:36 producer/broker/3 shut down
2020-12-30T15:13:41.940+0900 INFO kafka/log.go:36 client/metadata fetching metadata for [cta_test] from broker kafka-003.publ.com:9092
2020-12-30T15:13:41.940+0900 INFO kafka/log.go:36 client/metadata fetching metadata for [cta_test] from broker kafka-003.publ.com:9092
2020-12-30T15:13:41.941+0900 INFO kafka/log.go:36 producer/broker/3 starting up
2020-12-30T15:13:41.941+0900 INFO kafka/log.go:36 producer/broker/3 state change to [open] on cta_test/9
2020-12-30T15:13:41.941+0900 INFO kafka/log.go:36 producer/leader/cta_test/9 selected broker 3
2020-12-30T15:13:41.941+0900 INFO kafka/log.go:36 producer/leader/cta_test/9 state change to [flushing-2]
2020-12-30T15:13:41.941+0900 INFO kafka/log.go:36 producer/leader/cta_test/9 state change to [normal]
2020-12-30T15:13:41.941+0900 INFO kafka/log.go:36 producer/leader/cta_test/3 selected broker 3
2020-12-30T15:13:41.941+0900 INFO kafka/log.go:36 producer/broker/3 state change to [open] on cta_test/3
2020-12-30T15:13:41.941+0900 INFO kafka/log.go:36 producer/leader/cta_test/3 state change to [flushing-1]
2020-12-30T15:13:41.941+0900 INFO kafka/log.go:36 producer/leader/cta_test/3 state change to [normal]
- filebeat.yml
#------------------------------- Kafka output ----------------------------------
output.kafka:
enabled: true
hosts: ["kafka-001-cluster.priv.com:9092,kafka-002-cluster.priv.com:9092","kafka-003-cluster.priv.com:9092"]
topic: "cta_test"
codec.format:
string: '"offset":%{[offset]}, "message":%{[message]}'