I am running metricbeat 7.4.0 as DaemonSet and another Spring boot 2 application deployment annotated as following;
spec:
replicas: 1
selector:
matchLabels:
app: petclinic
template:
metadata:
labels:
app: petclinic
annotations:
co.elastic.metrics/module: prometheus
co.elastic.metrics/hosts: '${data.host}:${data.port}'
co.elastic.metrics/metrics_path: '/metrics/prometheus'
co.elastic.metrics/period: 1m
Metricbeat discovery seems to be working fine with other components, but not with prometheus since it always tries to invoke the default /metrics endpoint instead of the one I specified in the metrics_path. I can see metricbeat discovery debug messages retrieving the map without the metrics_path, so it is safe to assume this annotation is not being picked up by autodiscovery.
Here is metricbeat log messages:
2019-10-06T13:13:52.573Z DEBUG [autodiscover] autodiscover/autodiscover.go:166 Got a start event: map[config:[0xc0016a7ce0] host:172.17.0.3 id:504a7244-340b-4f28-9d38-e411947d5a72.petclinic kubernetes:{"annotations":{"co":{"elastic":{"logs/multiline":{"match":"after","negate":"true","pattern":"^[0-9]{4}-[0-9]{2}-[0-9]{2}"},"metrics/hosts":"${data.host}:${data.port}","metrics/metrics_path":"/metrics/prometheus","metrics/module":"prometheus","metrics/period":"1m","metrics/query":{"format":"prometheus"}}}},"container":{"id":"b74a9b8f6bb814bc1e3c2e0ac2514a94ae1bc5ad397bee7948bd01b2e2e6aad1","image":"petclinic:latest","name":"petclinic","runtime":"docker"},"labels":{"app":"petclinic","pod-template-hash":"7d6cc78d55"},"namespace":"default","node":{"name":"minikube"},"pod":{"name":"petclinic-deployment-7d6cc78d55-jkzln","uid":"504a7244-340b-4f28-9d38-e411947d5a72"},"replicaset":{"name":"petclinic-deployment-7d6cc78d55"}} meta:{"kubernetes":{"container":{"image":"petclinic:latest","name":"petclinic"},"labels":{"app":"petclinic","pod-template-hash":"7d6cc78d55"},"namespace":"default","node":{"name":"minikube"},"pod":{"name":"petclinic-deployment-7d6cc78d55-jkzln","uid":"504a7244-340b-4f28-9d38-e411947d5a72"},"replicaset":{"name":"petclinic-deployment-7d6cc78d55"}}} port:8080 provider:7c8ec85b-dbbc-4f00-9ee4-e5944f9d2b0c start:true]
2019-10-06T13:13:52.573Z DEBUG [autodiscover] autodiscover/autodiscover.go:191 Generated config: map[enabled:true hosts:[172.17.0.3:8080] metricsets:[collector] module:prometheus period:1m timeout:3s]
2019-10-06T13:13:52.573Z DEBUG [autodiscover] autodiscover/autodiscover.go:253 Got a meta field in the event
2019-10-06T13:13:52.573Z DEBUG [autodiscover] cfgfile/list.go:62 Starting reload procedure, current runners: 1
2019-10-06T13:13:52.573Z DEBUG [autodiscover] cfgfile/list.go:80 Start list: 1, Stop list: 0
2019-10-06T13:13:52.573Z DEBUG [autodiscover] cfgfile/list.go:101 Starting runner: prometheus [metricsets=1]
E1006 13:23:25.607833 1 reflector.go:342] github.com/elastic/beats/libbeat/common/kubernetes/watcher.go:235: expected type *v1beta1.ReplicaSet, but watch event object had type *v1.ReplicaSet
The last error may not be related; I believe it is K8s version incompatibility with the version of metricbeat. I am using K8s version 1.16.0