Error fetching data for metricset kafka.partition

Elastic Stack Beats
1

Hi,
I am running kafka and elk on kubernetes. I am trying to send kafka metrics to elk to view Metricbeat Kafka dashboard. Metricbeat is running as daemonset.

Here is my metricbeat config:

  • module: kafka
    enabled: true
    metricsets:
    - partition
    - consumergroup
    period: 10s
    hosts: ["my-connect-cp-kafka.default.svc.cluster.local:9092"]

Errors I get:
2020-06-21T05:58:49.267Z INFO module/wrapper.go:259 Error fetching data for metricset kafka.partition: error in connect: No advertised broker with address my-connect-cp-kafka.default.svc.cluster.local:9092 found

There are 3 kafka pods. I have tried with localhost:9092, PLAINTEXT://headless service name:9092

Please advice.

2

My first question is to verify twice if the address is accessible from inside the pod.

3

Thanks for responding. I am inside metricsbeat pod able to verify connection to kafka .

[root@metricbeat-metricbeat-9cfrl metricbeat]# telnet my-connect-cp-kafka.default.svc.cluster.local 9092
Trying 10.100.249.247...
Connected to my-connect-cp-kafka.default.svc.cluster.local.
Escape character is '^]'.
[root@metricbeat-metricbeat-9cfrl metricbeat]# telnet kube-state-metrics.kube-system.svc.cluster.local 8080
Trying 10.100.31.164...
Connected to kube-state-metrics.kube-system.svc.cluster.local.
Escape character is '^]'.

4

I took all steps mentioned in https://www.elastic.co/guide/en/beats/metricbeat/current/metricbeat-module-kafka.html . Looks like there was a problem with kafka connection. I fixed that this morning but I don’t see any metrics on Metricbeat Kafka dashboard or any errors from the metric beat pod logs. I am also wondering how to debug these issues.

I have tried with Kafka 2.2.2 and also latest kafka version. I have applied kafka-acls that was mentioned in the document.

5

When I turned on debug(logging.level: debug):
"message": "error in connect: No advertised broker with address my-connect-cp-kafka.default.svc.cluster.local:9092 found"

I am able to create topics etc on the kafka cluster. So, kafka pods are healthy.