Metricbeat cannot get metrics from kafka running in k8

Elastic Stack Beats
1

My configuration contains:

                        - condition:
                            equals:
                              kubernetes.labels:
                                strimzi_io/name: vikafka-kafka
                          config:
                            - module: kafka
                              metricsets: ["partition","consumergroup"]
                              hosts: ["${data.host}:9092"]
                              enabled: true

Which does find the pods. But I get the sequence of repeating logs:

default/metricbeat-h4qbr[metricbeat]: 2019-08-20T23:24:57.453Z	INFO	kafka/log.go:53	Connected to broker at 10.12.196.222:9092 (unregistered)
default/metricbeat-h4qbr[metricbeat]:
default/metricbeat-h4qbr[metricbeat]: 2019-08-20T23:24:57.454Z	INFO	kafka/log.go:53	Connected to broker at 10.12.196.222:9092 (unregistered)
default/metricbeat-h4qbr[metricbeat]:
default/metricbeat-h4qbr[metricbeat]: 2019-08-20T23:24:57.455Z	INFO	kafka/log.go:53	Connected to broker at 10.12.196.222:9092 (unregistered)
default/metricbeat-h4qbr[metricbeat]:
default/metricbeat-h4qbr[metricbeat]: 2019-08-20T23:24:57.459Z	INFO	kafka/log.go:53	Connected to broker at 10.12.196.222:9092 (unregistered)
default/metricbeat-h4qbr[metricbeat]:
default/metricbeat-h4qbr[metricbeat]: 2019-08-20T23:24:57.471Z	INFO	kafka/log.go:53	Closed connection to broker 10.12.196.222:9092
default/metricbeat-h4qbr[metricbeat]:
default/metricbeat-h4qbr[metricbeat]: 2019-08-20T23:24:57.471Z	INFO	module/wrapper.go:247	Error fetching data for metricset kafka.consumergroup: error in connect: No advertised broker with address 10.12.196.222:9092 found
default/metricbeat-h4qbr[metricbeat]: 2019-08-20T23:24:57.471Z	INFO	kafka/log.go:53	Closed connection to broker 10.12.196.222:9092
default/metricbeat-h4qbr[metricbeat]:
default/metricbeat-h4qbr[metricbeat]: 2019-08-20T23:24:57.471Z	INFO	module/wrapper.go:247	Error fetching data for metricset kafka.partition: error in connect: No advertised broker with address 10.12.196.222:9092 found
default/metricbeat-h4qbr[metricbeat]: 2019-08-20T23:24:57.472Z	INFO	kafka/log.go:53	Closed connection to broker 10.12.196.222:9092
default/metricbeat-h4qbr[metricbeat]:
default/metricbeat-h4qbr[metricbeat]: 2019-08-20T23:24:57.472Z	INFO	module/wrapper.go:247	Error fetching data for metricset kafka.consumergroup: error in connect: No advertised broker with address 10.12.196.222:9092 found
default/metricbeat-h4qbr[metricbeat]: 2019-08-20T23:24:57.476Z	INFO	kafka/log.go:53	Closed connection to broker 10.12.196.222:9092
default/metricbeat-h4qbr[metricbeat]:
default/metricbeat-h4qbr[metricbeat]: 2019-08-20T23:24:57.477Z	INFO	module/wrapper.go:247	Error fetching data for metricset kafka.partition: error in connect: No advertised broker with address 10.12.196.222:9092 found

Not sure what the complaint is. kafkcat shows

$ kafkacat -L -b 10.12.196.222                                                                                                            ops/git/master !+
Metadata for all topics (from broker -1: 10.12.196.222:9092/bootstrap):
 3 brokers:
  broker 0 at vikafka-kafka-0.vikafka-kafka-brokers.default.svc.cluster.local:9092 (controller)
  broker 2 at vikafka-kafka-2.vikafka-kafka-brokers.default.svc.cluster.local:9092
  broker 1 at vikafka-kafka-1.vikafka-kafka-brokers.default.svc.cluster.local:9092
 0 topics:

And from within the metricbeat container I can ping the IP 10.12.196.222 and the DNS vikafka-kafka-0.vikafka-kafka-brokers.default.svc.cluster.local, so at least connectivity exists.

2

We were able to get metricbeat to connect only by hard coding the k8 dns names -- not using autodiscover

          deployment:
            overrideModules:
              kubernetes:
                enabled: true
                config:
                  - module: kafka
                    hosts:
                      - "vikafka-kafka-0.vikafka-kafka-brokers.default.svc.cluster.local:9092"
                      - "vikafka-kafka-1.vikafka-kafka-brokers.default.svc.cluster.local:9092"
                      - "vikafka-kafka-2.vikafka-kafka-brokers.default.svc.cluster.local:9092"
                    metricsets: [consumergroup, partition]

So still at a loss how to make this work with autodiscover

3

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.