# Autodiscover does not collect prometheus metrics **URL:** https://discuss.elastic.co/t/autodiscover-does-not-collect-prometheus-metrics/218450 **Category:** Beats **Tags:** metricbeat **Created:** [February 8, 2020, 9:39pm UTC](https://discuss.elastic.co/t/autodiscover-does-not-collect-prometheus-metrics/218450 "2020-02-08T21:39:46Z") **Posts on this page:** 7 **Page:** 1
### Author: ![Babadofar](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/babadofar/32/44803_2.png) [@Babadofar](https://discuss.elastic.co/u/Babadofar) #### Post date: [February 8, 2020, 9:39pm UTC](https://discuss.elastic.co/t/autodiscover-does-not-collect-prometheus-metrics/218450/1 "2020-02-08T21:39:46Z")
I have exposed prometheus formatted metrics on several pods, using the prometheus module and listing each service separately works well, but I can not get autodiscover working. Using debug logging for all modules, I can not see any indication of something wrong. Running EKS, kubernetes version 1.14. Thankful for any help. This is the config which does not work: ``` metricbeat.autodiscover: providers: - type: kubernetes include_annotations: ["*"] hints.enabled: false namespace: jx-staging templates: - condition: contains: kubernetes.annotations.prometheus.io.scrape: "true" config: - module: prometheus metricsets: ["collector"] hosts: ["${data.host}:8080"] metrics_path: '/management/prometheus' processors: - add_kubernetes_metadata: in_cluster: true ``` And some logs ` ``` 2020-02-08T21:21:40.374Z DEBUG [autodiscover] kubernetes/kubernetes.go:136 Watcher Pod add: &Pod{ObjectMeta:k8s_io_apimachinery_pkg_apis_meta_v1.ObjectMeta{Name:jx-usermanagment-7fc96974c9-5mzvl,GenerateName:jx-usermanagment-7fc96974c9-,Namespace:jx-staging,SelfLink:/api/v1/namespaces/jx-staging/pods/jx-usermanagment-7fc96974c9-5mzvl,UID:8a0baeb8-4ab6-11ea-991f-0a8d81bb25f4,ResourceVersion:21092442,Generation:0,CreationTimestamp:2020-02-08 21:04:04 +0000 UTC,DeletionTimestamp:,DeletionGracePeriodSeconds:nil,Labels:map[string]string{app: jx-usermanagment,draft: draft-app,pod-template-hash: 7fc96974c9,},Annotations:map[string]string{co.elastic.metrics/hosts: usermanagment,co.elastic.metrics/metricsets: collector,co.elastic.metrics/module: prometheus,co.elastic.metrics/path: /management/prometheus,co.elastic.metrics/period: 1m,kubernetes.io/psp: eks.privileged,prometheus.io/scrape: true,},OwnerReferences:[{apps/v1 ReplicaSet jx-usermanagment-7fc96974c9 8a0895f4-4ab6-11ea-991f-0a8d81bb25f4 0xc0009e8017 0xc0009e8018}],Finalizers:[],ClusterName:,ManagedFields:[],},Spec:PodSpec{Volumes:[{default-token-2tbq2 {nil nil nil nil nil SecretVolumeSource{SecretName:default-token-2tbq2,Items:[],DefaultMode:*420,Optional:nil,} nil nil nil nil nil nil nil nil nil nil nil nil nil nil nil nil nil nil nil nil nil nil}}],Containers:[{usermanagment 523883733847.dkr.ecr.eu-west-1.amazonaws.com/mentor-medier/usermanagment:1.0.48 [] [] [{ 0 8080 TCP }] [{ nil SecretEnvSource{LocalObjectReference:LocalObjectReference{Name:user-db-secret,},Optional:nil,}}] [{dburl jdbc:mysql://mentordigital-campaign-staging.czaidexyizsq.eu-west-1.rds.amazonaws.com/users nil} {dbuser userman nil} {defaultReturnUrl https://user-login-poc-jx-staging.mentordigital.io nil}] {map[cpu:{{500 -3} {} 500m DecimalSI} memory:{{536870912 0} {} BinarySI}] map[cpu:{{400 -3} {} 400m DecimalSI} memory:{{536870912 0} {} BinarySI}]} [{default-token-2tbq2 true /var/run/secrets/kubernetes.io/serviceaccount }] [] &Probe{Handler:Handler{Exec:nil,HTTPGet:&HTTPGetAction{Path:/management/health,Port:8080,Host:,Scheme:HTTP,HTTPHeaders:[],},TCPSocket:nil,},InitialDelaySeconds:60,TimeoutSeconds:1,PeriodSeconds:10,SuccessThreshold:1,FailureThreshold:3,} &Probe{Handler:Handler{Exec:nil,HTTPGet:&HTTPGetAction{Path:/management/health,Port:8080,Host:,Scheme:HTTP,HTTPHeaders:[],},TCPSocket:nil,},InitialDelaySeconds:0,TimeoutSeconds:1,PeriodSeconds:10,SuccessThreshold:1,FailureThreshold:3,} nil /dev/termination-log File IfNotPresent nil false false false}],RestartPolicy:Always,TerminationGracePeriodSeconds:*30,ActiveDeadlineSeconds:nil,DNSPolicy:ClusterFirst,NodeSelector:map[string]string{},ServiceAccountName:default,DeprecatedServiceAccount:default,NodeName:ip-192-168-40-176.eu-west-1.compute.internal,HostNetwork:false,HostPID:false,HostIPC:false,SecurityContext:&PodSecurityContext{SELinuxOptions:nil,RunAsUser:nil,RunAsNonRoot:nil,SupplementalGroups:[],FSGroup:nil,RunAsGroup:nil,Sysctls:[],WindowsOptions:nil,},ImagePullSecrets:[],Hostname:,Subdomain:,Affinity:nil,SchedulerName:default-scheduler,InitContainers:[],AutomountServiceAccountToken:nil,Tolerations:[{node.kubernetes.io/not-ready Exists NoExecute 0xc0009e81e0} {node.kubernetes.io/unreachable Exists NoExecute 0xc0009e8200}],HostAliases:[],PriorityClassName:,Priority:*0,DNSConfig:nil,ShareProcessNamespace:nil,ReadinessGates:[],RuntimeClassName:nil,EnableServiceLinks:*true,PreemptionPolicy:nil,Overhead:ResourceList{},TopologySpreadConstraints:[],EphemeralContainers:[],},Status:PodStatus{Phase:Running,Conditions:[{Initialized True 0001-01-01 00:00:00 +0000 UTC 2020-02-08 21:04:04 +0000 UTC } {Ready True 0001-01-01 00:00:00 +0000 UTC 2020-02-08 21:04:52 +0000 UTC } {ContainersReady True 0001-01-01 00:00:00 +0000 UTC 2020-02-08 21:04:52 +0000 UTC } {PodScheduled True 0001-01-01 00:00:00 +0000 UTC 2020-02-08 21:04:04 +0000 UTC }],Message:,Reason:,HostIP:192.168.40.176,PodIP:192.168.37.87,StartTime:2020-02-08 21:04:04 +0000 UTC,ContainerStatuses:[{usermanagment {nil ContainerStateRunning{StartedAt:2020-02-08 21:04:05 +0000 UTC,} nil} {nil nil nil} true 0 523883733847.dkr.ecr.eu-west-1.amazonaws.com/mentor-medier/usermanagment:1.0.48 docker-pullable://523883733847.dkr.ecr.eu-west-1.amazonaws.com/mentor-medier/usermanagment@sha256:622c29790e31702180387fa96977db813596bcca1328ad2d34909abb7da2b4ca docker://120100ebac89210040156acf84bed5f23d1b11d3ea68e72f198296374d98083e}],QOSClass:Burstable,InitContainerStatuses:[],NominatedNodeName:,PodIPs:[],EphemeralContainerStatuses:[],},} 2020-02-08T21:21:40.374Z DEBUG [autodiscover] kubernetes/kubernetes.go:324 Generated hints {"metrics":{"hosts":"usermanagment","metricsets":"collector","module":"prometheus","path":"/management/prometheus","period":"1m"}} 2020-02-08T21:21:40.374Z DEBUG [autodiscover] kubernetes/kubernetes.go:329 Generated builder event map[container:{"id":"120100ebac89210040156acf84bed5f23d1b11d3ea68e72f198296374d98083e","image":"523883733847.dkr.ecr.eu-west-1.amazonaws.com/mentor-medier/usermanagment:1.0.48","name":"usermanagment","runtime":"docker"} hints:{"metrics":{"hosts":"usermanagment","metricsets":"collector","module":"prometheus","path":"/management/prometheus","period":"1m"}} host:192.168.37.87 kubernetes:{"annotations":{"co":{"elastic":{"metrics/hosts":"usermanagment","metrics/metricsets":"collector","metrics/module":"prometheus","metrics/path":"/management/prometheus","metrics/period":"1m"}},"kubernetes":{"io/psp":"eks.privileged"},"prometheus":{"io/scrape":"true"}},"container":{"id":"120100ebac89210040156acf84bed5f23d1b11d3ea68e72f198296374d98083e","image":"523883733847.dkr.ecr.eu-west-1.amazonaws.com/mentor-medier/usermanagment:1.0.48","name":"usermanagment","runtime":"docker"},"labels":{"app":"jx-usermanagment","draft":"draft-app","pod-template-hash":"7fc96974c9"},"namespace":"jx-staging","node":{"name":"ip-192-168-40-176.eu-west-1.compute.internal"},"pod":{"name":"jx-usermanagment-7fc96974c9-5mzvl","uid":"8a0baeb8-4ab6-11ea-991f-0a8d81bb25f4"},"replicaset":{"name":"jx-usermanagment-7fc96974c9"}} port:8080] 2020-02-08T21:21:40.374Z DEBUG [autodiscover] autodiscover/autodiscover.go:165 Got a start event: map[config:[] host:192.168.37.87 id:8a0baeb8-4ab6-11ea-991f-0a8d81bb25f4.usermanagment kubernetes:{"annotations":{"co":{"elastic":{"metrics/hosts":"usermanagment","metrics/metricsets":"collector","metrics/module":"prometheus","metrics/path":"/management/prometheus","metrics/period":"1m"}},"kubernetes":{"io/psp":"eks.privileged"},"prometheus":{"io/scrape":"true"}},"container":{"id":"120100ebac89210040156acf84bed5f23d1b11d3ea68e72f198296374d98083e","image":"523883733847.dkr.ecr.eu-west-1.amazonaws.com/mentor-medier/usermanagment:1.0.48","name":"usermanagment","runtime":"docker"},"labels":{"app":"jx-usermanagment","draft":"draft-app","pod-template-hash":"7fc96974c9"},"namespace":"jx-staging","node":{"name":"ip-192-168-40-176.eu-west-1.compute.internal"},"pod":{"name":"jx-usermanagment-7fc96974c9-5mzvl","uid":"8a0baeb8-4ab6-11ea-991f-0a8d81bb25f4"},"replicaset":{"name":"jx-usermanagment-7fc96974c9"}} meta:{"kubernetes":{"container":{"image":"523883733847.dkr.ecr.eu-west-1.amazonaws.com/mentor-medier/usermanagment:1.0.48","name":"usermanagment"},"labels":{"app":"jx-usermanagment","draft":"draft-app","pod-template-hash":"7fc96974c9"},"namespace":"jx-staging","node":{"name":"ip-192-168-40-176.eu-west-1.compute.internal"},"pod":{"name":"jx-usermanagment-7fc96974c9-5mzvl","uid":"8a0baeb8-4ab6-11ea-991f-0a8d81bb25f4"},"replicaset":{"name":"jx-usermanagment-7fc96974c9"}}} port:8080 provider:a1766f07-0726-4c00-a271-668ce0338bac start:true] 2020-02-08T21:21:40.374Z DEBUG [autodiscover] autodiscover/autodiscover.go:252 Got a meta field in the event 2020-0` ``` ---
### Author: ![Babadofar](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/babadofar/32/44803_2.png) [@Babadofar](https://discuss.elastic.co/u/Babadofar) #### Post date: [February 21, 2020, 9:36am UTC](https://discuss.elastic.co/t/autodiscover-does-not-collect-prometheus-metrics/218450/2 "2020-02-21T09:36:56Z")
The documentation is perhaps a little misleading on the kubernetes autodiscover annotations. The customary annotation to use is `kubernetes.annotations.prometheus.io/scrape: "true"`with a slash between io and scrape, while the docs has a dot: [https://www.elastic.co/guide/en/beats/metricbeat/current/configuration-autodiscover.html#\_kubernetes](https://www.elastic.co/guide/en/beats/metricbeat/current/configuration-autodiscover.html#_kubernetes) The docs are like this: ``` metricbeat.autodiscover: providers: - type: kubernetes include_annotations: ["prometheus.io.scrape"] templates: - condition: contains: kubernetes.annotations.prometheus.io.scrape: "true" config: - module: prometheus metricsets: ["collector"] hosts: "${data.host}:${data.port}" ``` but they should be ``` metricbeat.autodiscover: providers: - type: kubernetes include_annotations: ["*"] templates: - condition: contains: kubernetes.annotations.prometheus.io/scrape: "true" config: - module: prometheus metricsets: ["collector"] hosts: "${data.host}:${data.port}" ``` ---
### Author: ![Mario\_Castro](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/mario_castro/32/35107_2.png) [@Mario\_Castro](https://discuss.elastic.co/u/Mario_Castro) #### Post date: [February 25, 2020, 10:26am UTC](https://discuss.elastic.co/t/autodiscover-does-not-collect-prometheus-metrics/218450/3 "2020-02-25T10:26:45Z")
Thanks for the heads up! ---
### Author: ![ChrsMark](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/chrsmark/32/55858_2.png) [@ChrsMark](https://discuss.elastic.co/u/ChrsMark) #### Post date: [February 25, 2020, 10:45am UTC](https://discuss.elastic.co/t/autodiscover-does-not-collect-prometheus-metrics/218450/4 "2020-02-25T10:45:34Z")
Hi @Babadofar! With a quick glance I think that the documentation is not wrong since under the sample configuration you posted it mentions `This configuration launches a prometheus module for all containers of pods annotated prometheus.io.scrape=true.` So the condition matches the annotation that is mentioned below. Does this make things clear or I miss something 🙂 ? Regards, C. ---
### Author: ![ChrsMark](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/chrsmark/32/55858_2.png) [@ChrsMark](https://discuss.elastic.co/u/ChrsMark) #### Post date: [February 25, 2020, 10:56am UTC](https://discuss.elastic.co/t/autodiscover-does-not-collect-prometheus-metrics/218450/5 "2020-02-25T10:56:43Z")
However, as @exekias pointed `prometheus.io/scrape` annotation is commonly used so we may need to update the docs to avoid such confusing situations! @Babadofar thanks for mentioning this, feel free to open a GH issue for this. Otherwise I can do it! 🙂 ---
### Author: ![ChrsMark](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/chrsmark/32/55858_2.png) [@ChrsMark](https://discuss.elastic.co/u/ChrsMark) #### Post date: [March 4, 2020, 7:19am UTC](https://discuss.elastic.co/t/autodiscover-does-not-collect-prometheus-metrics/218450/6 "2020-03-04T07:19:16Z")
Here is the merged PR: [https://github.com/elastic/beats/pull/16597](https://github.com/elastic/beats/pull/16597). ---
### Author: ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system) #### Post date: [April 1, 2020, 7:19am UTC](https://discuss.elastic.co/t/autodiscover-does-not-collect-prometheus-metrics/218450/7 "2020-04-01T07:19:27Z")
This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.