K8s autodiscovery doesn't accept metrics_path

I am running metricbeat 7.4.0 as DaemonSet and another Spring boot 2 application deployment annotated as following;

spec:
  replicas: 1
  selector:
    matchLabels:
      app: petclinic
  template:
    metadata:
      labels:
        app: petclinic
      annotations:
        co.elastic.metrics/module: prometheus
        co.elastic.metrics/hosts: '${data.host}:${data.port}'
        co.elastic.metrics/metrics_path: '/metrics/prometheus'
        co.elastic.metrics/period: 1m

Metricbeat discovery seems to be working fine with other components, but not with prometheus since it always tries to invoke the default /metrics endpoint instead of the one I specified in the metrics_path. I can see metricbeat discovery debug messages retrieving the map without the metrics_path, so it is safe to assume this annotation is not being picked up by autodiscovery.

Here is metricbeat log messages:

2019-10-06T13:13:52.573Z	DEBUG	[autodiscover]	autodiscover/autodiscover.go:166	Got a start event: map[config:[0xc0016a7ce0] host:172.17.0.3 id:504a7244-340b-4f28-9d38-e411947d5a72.petclinic kubernetes:{"annotations":{"co":{"elastic":{"logs/multiline":{"match":"after","negate":"true","pattern":"^[0-9]{4}-[0-9]{2}-[0-9]{2}"},"metrics/hosts":"${data.host}:${data.port}","metrics/metrics_path":"/metrics/prometheus","metrics/module":"prometheus","metrics/period":"1m","metrics/query":{"format":"prometheus"}}}},"container":{"id":"b74a9b8f6bb814bc1e3c2e0ac2514a94ae1bc5ad397bee7948bd01b2e2e6aad1","image":"petclinic:latest","name":"petclinic","runtime":"docker"},"labels":{"app":"petclinic","pod-template-hash":"7d6cc78d55"},"namespace":"default","node":{"name":"minikube"},"pod":{"name":"petclinic-deployment-7d6cc78d55-jkzln","uid":"504a7244-340b-4f28-9d38-e411947d5a72"},"replicaset":{"name":"petclinic-deployment-7d6cc78d55"}} meta:{"kubernetes":{"container":{"image":"petclinic:latest","name":"petclinic"},"labels":{"app":"petclinic","pod-template-hash":"7d6cc78d55"},"namespace":"default","node":{"name":"minikube"},"pod":{"name":"petclinic-deployment-7d6cc78d55-jkzln","uid":"504a7244-340b-4f28-9d38-e411947d5a72"},"replicaset":{"name":"petclinic-deployment-7d6cc78d55"}}} port:8080 provider:7c8ec85b-dbbc-4f00-9ee4-e5944f9d2b0c start:true]
2019-10-06T13:13:52.573Z	DEBUG	[autodiscover]	autodiscover/autodiscover.go:191	Generated config: map[enabled:true hosts:[172.17.0.3:8080] metricsets:[collector] module:prometheus period:1m timeout:3s]
2019-10-06T13:13:52.573Z	DEBUG	[autodiscover]	autodiscover/autodiscover.go:253	Got a meta field in the event
2019-10-06T13:13:52.573Z	DEBUG	[autodiscover]	cfgfile/list.go:62	Starting reload procedure, current runners: 1
2019-10-06T13:13:52.573Z	DEBUG	[autodiscover]	cfgfile/list.go:80	Start list: 1, Stop list: 0
2019-10-06T13:13:52.573Z	DEBUG	[autodiscover]	cfgfile/list.go:101	Starting runner: prometheus [metricsets=1]
E1006 13:23:25.607833       1 reflector.go:342] github.com/elastic/beats/libbeat/common/kubernetes/watcher.go:235: expected type *v1beta1.ReplicaSet, but watch event object had type *v1.ReplicaSet

The last error may not be related; I believe it is K8s version incompatibility with the version of metricbeat. I am using K8s version 1.16.0

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.