Metricbeat not able to collect metrics from Kerberised Kafka

Elastic Stack Beats
1

Issue: Metric beat is not to collect metrics from Kerberized Kafka.
It works well with non-kerberized Kafka

  • MetricBeat Version: 6.6.2
  • Operating System: CentOs 7.4.x

logs snapshot:

Blockquote

Logs when metric beat is running on kerberised setup:

}
2019-03-25T05:51:14.506Z INFO kafka/log.go:53 Closed connection to broker testautomation-slv-01.gvs.ggn:6667

2019-03-25T05:51:14.507Z INFO kafka/log.go:53 Connected to broker at testautomation-slv-01.gvs.ggn:6667 (unregistered)

2019-03-25T05:51:14.757Z INFO kafka/log.go:53 Closed connection to broker testautomation-slv-01.gvs.ggn:6667

2019-03-25T05:51:14.758Z DEBUG [publish] pipeline/processor.go:308 Publish event: {
"@timestamp": "2019-03-25T05:51:14.001Z",
"@metadata": {
"beat": "metricbeat",
"type": "doc",
"version": "6.6.2"
},
"error": {
"message": "EOF"
},
"metricset": {
"name": "consumergroup",
"module": "kafka",
"host": "testautomation-slv-01.gvs.ggn:6667",
"rtt": 756169
},
"host": {
"name": "testautomation-slv-01.gvs.ggn"
},
"beat": {
"name": "testautomation-slv-01.gvs.ggn",
"hostname": "testautomation-slv-01.gvs.ggn",
"version": "6.6.2"
},
"event": {
"dataset": "kafka.consumergroup",
"duration": 756169315
}
}
2019-03-25T05:51:15.310Z INFO pipeline/output.go:95 Connecting to backoff(elasticsearch(http://192.168.194.143:9200))
2019-03-25T05:51:15.311Z DEBUG [elasticsearch] elasticsearch/client.go:697 ES Ping(url=http://192.168.194.143:9200)
2019-03-25T05:51:15.314Z DEBUG [elasticsearch] elasticsearch/client.go:720 Ping status code: 200
2019-03-25T05:51:15.314Z INFO elasticsearch/client.go:721 Connected to Elasticsearch version 6.2.4
2019-03-25T05:51:15.315Z DEBUG [elasticsearch] elasticsearch/client.go:739 HEAD http://192.168.194.143:9200/_template/jyoti-miqproddc-new_kerb1
2019-03-25T05:51:15.323Z INFO template/load.go:130 Template already exists and will not be overwritten.
2019-03-25T05:51:15.323Z INFO instance/beat.go:894 Template successfully loaded.
2019-03-25T05:51:15.323Z INFO pipeline/output.go:105 Connection to backoff(elasticsearch(http://192.168.194.143:9200)) established
2019-03-25T05:51:15.330Z DEBUG [elasticsearch] elasticsearch/client.go:323 PublishEvents: 2 events have been published to elasticsearch in 7.39981ms.
2019-03-25T05:51:15.330Z DEBUG [publisher] memqueue/ackloop.go:160 ackloop: receive ack [0: 0, 2]
2019-03-25T05:51:15.330Z DEBUG [publisher] memqueue/eventloop.go:535 broker ACK events: count=2, start-seq=1, end-seq=2

2019-03-25T05:51:15.330Z DEBUG [publisher] memqueue/ackloop.go:128 ackloop: return ack to broker loop:2

2

Hi @mmahaja,

can you share the metricbeat configuration?
afaik we don't have kerberos specific support.

3

@pmercado is there a plan to enable metricbeat with kerberos support?

4

@pmercado yes kerberos specific support in metricbeat.

so that metricbeat can interact with kerberised components like KAFKA from system and publish component specific metrics (say kafka.partition.offset.newest for KAFKA) to ElasticSearch.

5

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.