Metricbeat not able to collect metrics from Kerberised Kafka

Issue: Metric beat is not to collect metrics from Kerberized Kafka.
It works well with non-kerberized Kafka

  • MetricBeat Version: 6.6.2
  • Operating System: CentOs 7.4.x

logs snapshot:


Logs when metric beat is running on kerberised setup:

2019-03-25T05:51:14.506Z INFO kafka/log.go:53 Closed connection to broker testautomation-slv-01.gvs.ggn:6667

2019-03-25T05:51:14.507Z INFO kafka/log.go:53 Connected to broker at testautomation-slv-01.gvs.ggn:6667 (unregistered)

2019-03-25T05:51:14.757Z INFO kafka/log.go:53 Closed connection to broker testautomation-slv-01.gvs.ggn:6667

2019-03-25T05:51:14.758Z DEBUG [publish] pipeline/processor.go:308 Publish event: {
"@timestamp": "2019-03-25T05:51:14.001Z",
"@metadata": {
"beat": "metricbeat",
"type": "doc",
"version": "6.6.2"
"error": {
"message": "EOF"
"metricset": {
"name": "consumergroup",
"module": "kafka",
"host": "testautomation-slv-01.gvs.ggn:6667",
"rtt": 756169
"host": {
"name": "testautomation-slv-01.gvs.ggn"
"beat": {
"name": "testautomation-slv-01.gvs.ggn",
"hostname": "testautomation-slv-01.gvs.ggn",
"version": "6.6.2"
"event": {
"dataset": "kafka.consumergroup",
"duration": 756169315
2019-03-25T05:51:15.310Z INFO pipeline/output.go:95 Connecting to backoff(elasticsearch(
2019-03-25T05:51:15.311Z DEBUG [elasticsearch] elasticsearch/client.go:697 ES Ping(url=
2019-03-25T05:51:15.314Z DEBUG [elasticsearch] elasticsearch/client.go:720 Ping status code: 200
2019-03-25T05:51:15.314Z INFO elasticsearch/client.go:721 Connected to Elasticsearch version 6.2.4
2019-03-25T05:51:15.315Z DEBUG [elasticsearch] elasticsearch/client.go:739 HEAD
2019-03-25T05:51:15.323Z INFO template/load.go:130 Template already exists and will not be overwritten.
2019-03-25T05:51:15.323Z INFO instance/beat.go:894 Template successfully loaded.
2019-03-25T05:51:15.323Z INFO pipeline/output.go:105 Connection to backoff(elasticsearch( established
2019-03-25T05:51:15.330Z DEBUG [elasticsearch] elasticsearch/client.go:323 PublishEvents: 2 events have been published to elasticsearch in 7.39981ms.
2019-03-25T05:51:15.330Z DEBUG [publisher] memqueue/ackloop.go:160 ackloop: receive ack [0: 0, 2]
2019-03-25T05:51:15.330Z DEBUG [publisher] memqueue/eventloop.go:535 broker ACK events: count=2, start-seq=1, end-seq=2

2019-03-25T05:51:15.330Z DEBUG [publisher] memqueue/ackloop.go:128 ackloop: return ack to broker loop:2

Hi @mmahaja,

can you share the metricbeat configuration?
afaik we don't have kerberos specific support.

@pmercado is there a plan to enable metricbeat with kerberos support?

@pmercado yes kerberos specific support in metricbeat.

so that metricbeat can interact with kerberised components like KAFKA from system and publish component specific metrics (say kafka.partition.offset.newest for KAFKA) to ElasticSearch.

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.