Error fetching data for metricset kafka.partition

I am running kafka and elk on kubernetes. I am trying to send kafka metrics to elk to view Metricbeat Kafka dashboard. Metricbeat is running as daemonset.

Here is my metricbeat config:

  • module: kafka
    enabled: true
    - partition
    - consumergroup
    period: 10s
    hosts: ["my-connect-cp-kafka.default.svc.cluster.local:9092"]

Errors I get:
2020-06-21T05:58:49.267Z INFO module/wrapper.go:259 Error fetching data for metricset kafka.partition: error in connect: No advertised broker with address my-connect-cp-kafka.default.svc.cluster.local:9092 found

There are 3 kafka pods. I have tried with localhost:9092, PLAINTEXT://headless service name:9092

Please advice.

My first question is to verify twice if the address is accessible from inside the pod.

Thanks for responding. I am inside metricsbeat pod able to verify connection to kafka .

[root@metricbeat-metricbeat-9cfrl metricbeat]# telnet my-connect-cp-kafka.default.svc.cluster.local 9092
Connected to my-connect-cp-kafka.default.svc.cluster.local.
Escape character is '^]'.
[root@metricbeat-metricbeat-9cfrl metricbeat]# telnet kube-state-metrics.kube-system.svc.cluster.local 8080
Connected to kube-state-metrics.kube-system.svc.cluster.local.
Escape character is '^]'.

I took all steps mentioned in . Looks like there was a problem with kafka connection. I fixed that this morning but I don’t see any metrics on Metricbeat Kafka dashboard or any errors from the metric beat pod logs. I am also wondering how to debug these issues.

I have tried with Kafka 2.2.2 and also latest kafka version. I have applied kafka-acls that was mentioned in the document.

When I turned on debug(logging.level: debug):
"message": "error in connect: No advertised broker with address my-connect-cp-kafka.default.svc.cluster.local:9092 found"

I am able to create topics etc on the kafka cluster. So, kafka pods are healthy.

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.