Autodiscover does not collect prometheus metrics

Babadofar · February 8, 2020, 9:39pm

I have exposed prometheus formatted metrics on several pods, using the prometheus module and listing each service separately works well, but I can not get autodiscover working. Using debug logging for all modules, I can not see any indication of something wrong.
Running EKS, kubernetes version 1.14.
Thankful for any help.

This is the config which does not work:

metricbeat.autodiscover:
  providers:
  - type: kubernetes
    include_annotations: ["*"]
    hints.enabled: false
    namespace: jx-staging
    templates:
      - condition:
          contains:
            kubernetes.annotations.prometheus.io.scrape: "true"
        config:
          - module: prometheus
            metricsets: ["collector"]
            hosts: ["${data.host}:8080"]
            metrics_path: '/management/prometheus'
            processors:
              - add_kubernetes_metadata:
                  in_cluster: true

And some logs `

	2020-02-08T21:21:40.374Z	DEBUG	[autodiscover]	kubernetes/kubernetes.go:136	Watcher Pod add: &Pod{ObjectMeta:k8s_io_apimachinery_pkg_apis_meta_v1.ObjectMeta{Name:jx-usermanagment-7fc96974c9-5mzvl,GenerateName:jx-usermanagment-7fc96974c9-,Namespace:jx-staging,SelfLink:/api/v1/namespaces/jx-staging/pods/jx-usermanagment-7fc96974c9-5mzvl,UID:8a0baeb8-4ab6-11ea-991f-0a8d81bb25f4,ResourceVersion:21092442,Generation:0,CreationTimestamp:2020-02-08 21:04:04 +0000 UTC,DeletionTimestamp:<nil>,DeletionGracePeriodSeconds:nil,Labels:map[string]string{app: jx-usermanagment,draft: draft-app,pod-template-hash: 7fc96974c9,},Annotations:map[string]string{co.elastic.metrics/hosts: usermanagment,co.elastic.metrics/metricsets: collector,co.elastic.metrics/module: prometheus,co.elastic.metrics/path: /management/prometheus,co.elastic.metrics/period: 1m,kubernetes.io/psp: eks.privileged,prometheus.io/scrape: true,},OwnerReferences:[{apps/v1 ReplicaSet jx-usermanagment-7fc96974c9 8a0895f4-4ab6-11ea-991f-0a8d81bb25f4 0xc0009e8017 0xc0009e8018}],Finalizers:[],ClusterName:,ManagedFields:[],},Spec:PodSpec{Volumes:[{default-token-2tbq2 {nil nil nil nil nil SecretVolumeSource{SecretName:default-token-2tbq2,Items:[],DefaultMode:*420,Optional:nil,} nil nil nil nil nil nil nil nil nil nil nil nil nil nil nil nil nil nil nil nil nil nil}}],Containers:[{usermanagment 523883733847.dkr.ecr.eu-west-1.amazonaws.com/mentor-medier/usermanagment:1.0.48 [] []  [{ 0 8080 TCP }] [{ nil SecretEnvSource{LocalObjectReference:LocalObjectReference{Name:user-db-secret,},Optional:nil,}}] [{dburl jdbc:mysql://mentordigital-campaign-staging.czaidexyizsq.eu-west-1.rds.amazonaws.com/users nil} {dbuser userman nil} {defaultReturnUrl https://user-login-poc-jx-staging.mentordigital.io nil}] {map[cpu:{{500 -3} {<nil>} 500m DecimalSI} memory:{{536870912 0} {<nil>}  BinarySI}] map[cpu:{{400 -3} {<nil>} 400m DecimalSI} memory:{{536870912 0} {<nil>}  BinarySI}]} [{default-token-2tbq2 true /var/run/secrets/kubernetes.io/serviceaccount  <nil> }] [] &Probe{Handler:Handler{Exec:nil,HTTPGet:&HTTPGetAction{Path:/management/health,Port:8080,Host:,Scheme:HTTP,HTTPHeaders:[],},TCPSocket:nil,},InitialDelaySeconds:60,TimeoutSeconds:1,PeriodSeconds:10,SuccessThreshold:1,FailureThreshold:3,} &Probe{Handler:Handler{Exec:nil,HTTPGet:&HTTPGetAction{Path:/management/health,Port:8080,Host:,Scheme:HTTP,HTTPHeaders:[],},TCPSocket:nil,},InitialDelaySeconds:0,TimeoutSeconds:1,PeriodSeconds:10,SuccessThreshold:1,FailureThreshold:3,} nil /dev/termination-log File IfNotPresent nil false false false}],RestartPolicy:Always,TerminationGracePeriodSeconds:*30,ActiveDeadlineSeconds:nil,DNSPolicy:ClusterFirst,NodeSelector:map[string]string{},ServiceAccountName:default,DeprecatedServiceAccount:default,NodeName:ip-192-168-40-176.eu-west-1.compute.internal,HostNetwork:false,HostPID:false,HostIPC:false,SecurityContext:&PodSecurityContext{SELinuxOptions:nil,RunAsUser:nil,RunAsNonRoot:nil,SupplementalGroups:[],FSGroup:nil,RunAsGroup:nil,Sysctls:[],WindowsOptions:nil,},ImagePullSecrets:[],Hostname:,Subdomain:,Affinity:nil,SchedulerName:default-scheduler,InitContainers:[],AutomountServiceAccountToken:nil,Tolerations:[{node.kubernetes.io/not-ready Exists  NoExecute 0xc0009e81e0} {node.kubernetes.io/unreachable Exists  NoExecute 0xc0009e8200}],HostAliases:[],PriorityClassName:,Priority:*0,DNSConfig:nil,ShareProcessNamespace:nil,ReadinessGates:[],RuntimeClassName:nil,EnableServiceLinks:*true,PreemptionPolicy:nil,Overhead:ResourceList{},TopologySpreadConstraints:[],EphemeralContainers:[],},Status:PodStatus{Phase:Running,Conditions:[{Initialized True 0001-01-01 00:00:00 +0000 UTC 2020-02-08 21:04:04 +0000 UTC  } {Ready True 0001-01-01 00:00:00 +0000 UTC 2020-02-08 21:04:52 +0000 UTC  } {ContainersReady True 0001-01-01 00:00:00 +0000 UTC 2020-02-08 21:04:52 +0000 UTC  } {PodScheduled True 0001-01-01 00:00:00 +0000 UTC 2020-02-08 21:04:04 +0000 UTC  }],Message:,Reason:,HostIP:192.168.40.176,PodIP:192.168.37.87,StartTime:2020-02-08 21:04:04 +0000 UTC,ContainerStatuses:[{usermanagment {nil ContainerStateRunning{StartedAt:2020-02-08 21:04:05 +0000 UTC,} nil} {nil nil nil} true 0 523883733847.dkr.ecr.eu-west-1.amazonaws.com/mentor-medier/usermanagment:1.0.48 docker-pullable://523883733847.dkr.ecr.eu-west-1.amazonaws.com/mentor-medier/usermanagment@sha256:622c29790e31702180387fa96977db813596bcca1328ad2d34909abb7da2b4ca docker://120100ebac89210040156acf84bed5f23d1b11d3ea68e72f198296374d98083e}],QOSClass:Burstable,InitContainerStatuses:[],NominatedNodeName:,PodIPs:[],EphemeralContainerStatuses:[],},}
	2020-02-08T21:21:40.374Z	DEBUG	[autodiscover]	kubernetes/kubernetes.go:324	Generated hints {"metrics":{"hosts":"usermanagment","metricsets":"collector","module":"prometheus","path":"/management/prometheus","period":"1m"}}
	2020-02-08T21:21:40.374Z	DEBUG	[autodiscover]	kubernetes/kubernetes.go:329	Generated builder event map[container:{"id":"120100ebac89210040156acf84bed5f23d1b11d3ea68e72f198296374d98083e","image":"523883733847.dkr.ecr.eu-west-1.amazonaws.com/mentor-medier/usermanagment:1.0.48","name":"usermanagment","runtime":"docker"} hints:{"metrics":{"hosts":"usermanagment","metricsets":"collector","module":"prometheus","path":"/management/prometheus","period":"1m"}} host:192.168.37.87 kubernetes:{"annotations":{"co":{"elastic":{"metrics/hosts":"usermanagment","metrics/metricsets":"collector","metrics/module":"prometheus","metrics/path":"/management/prometheus","metrics/period":"1m"}},"kubernetes":{"io/psp":"eks.privileged"},"prometheus":{"io/scrape":"true"}},"container":{"id":"120100ebac89210040156acf84bed5f23d1b11d3ea68e72f198296374d98083e","image":"523883733847.dkr.ecr.eu-west-1.amazonaws.com/mentor-medier/usermanagment:1.0.48","name":"usermanagment","runtime":"docker"},"labels":{"app":"jx-usermanagment","draft":"draft-app","pod-template-hash":"7fc96974c9"},"namespace":"jx-staging","node":{"name":"ip-192-168-40-176.eu-west-1.compute.internal"},"pod":{"name":"jx-usermanagment-7fc96974c9-5mzvl","uid":"8a0baeb8-4ab6-11ea-991f-0a8d81bb25f4"},"replicaset":{"name":"jx-usermanagment-7fc96974c9"}} port:8080]
	2020-02-08T21:21:40.374Z	DEBUG	[autodiscover]	autodiscover/autodiscover.go:165	Got a start event: map[config:[] host:192.168.37.87 id:8a0baeb8-4ab6-11ea-991f-0a8d81bb25f4.usermanagment kubernetes:{"annotations":{"co":{"elastic":{"metrics/hosts":"usermanagment","metrics/metricsets":"collector","metrics/module":"prometheus","metrics/path":"/management/prometheus","metrics/period":"1m"}},"kubernetes":{"io/psp":"eks.privileged"},"prometheus":{"io/scrape":"true"}},"container":{"id":"120100ebac89210040156acf84bed5f23d1b11d3ea68e72f198296374d98083e","image":"523883733847.dkr.ecr.eu-west-1.amazonaws.com/mentor-medier/usermanagment:1.0.48","name":"usermanagment","runtime":"docker"},"labels":{"app":"jx-usermanagment","draft":"draft-app","pod-template-hash":"7fc96974c9"},"namespace":"jx-staging","node":{"name":"ip-192-168-40-176.eu-west-1.compute.internal"},"pod":{"name":"jx-usermanagment-7fc96974c9-5mzvl","uid":"8a0baeb8-4ab6-11ea-991f-0a8d81bb25f4"},"replicaset":{"name":"jx-usermanagment-7fc96974c9"}} meta:{"kubernetes":{"container":{"image":"523883733847.dkr.ecr.eu-west-1.amazonaws.com/mentor-medier/usermanagment:1.0.48","name":"usermanagment"},"labels":{"app":"jx-usermanagment","draft":"draft-app","pod-template-hash":"7fc96974c9"},"namespace":"jx-staging","node":{"name":"ip-192-168-40-176.eu-west-1.compute.internal"},"pod":{"name":"jx-usermanagment-7fc96974c9-5mzvl","uid":"8a0baeb8-4ab6-11ea-991f-0a8d81bb25f4"},"replicaset":{"name":"jx-usermanagment-7fc96974c9"}}} port:8080 provider:a1766f07-0726-4c00-a271-668ce0338bac start:true]
	2020-02-08T21:21:40.374Z	DEBUG	[autodiscover]	autodiscover/autodiscover.go:252	Got a meta field in the event
	2020-0`

Babadofar · February 21, 2020, 9:36am

The documentation is perhaps a little misleading on the kubernetes autodiscover annotations. The customary annotation to use is kubernetes.annotations.prometheus.io/scrape: "true"with a slash between io and scrape, while the docs has a dot: https://www.elastic.co/guide/en/beats/metricbeat/current/configuration-autodiscover.html#_kubernetes
The docs are like this:

metricbeat.autodiscover:
  providers:
    - type: kubernetes
      include_annotations: ["prometheus.io.scrape"]
      templates:
        - condition:
            contains:
              kubernetes.annotations.prometheus.io.scrape: "true"
          config:
            - module: prometheus
              metricsets: ["collector"]
              hosts: "${data.host}:${data.port}"

but they should be

  metricbeat.autodiscover:
  providers:
   - type: kubernetes
     include_annotations: ["*"]
     templates:
       - condition:
           contains:
             kubernetes.annotations.prometheus.io/scrape: "true"
         config:
           - module: prometheus
             metricsets: ["collector"]
             hosts: "${data.host}:${data.port}"

Mario_Castro · February 25, 2020, 10:26am

Thanks for the heads up!

ChrsMark · February 25, 2020, 10:45am

Hi @Babadofar!

With a quick glance I think that the documentation is not wrong since under the sample configuration you posted it mentions This configuration launches a prometheus module for all containers of pods annotated prometheus.io.scrape=true. So the condition matches the annotation that is mentioned below. Does this make things clear or I miss something ?

Regards,
C.

ChrsMark · February 25, 2020, 10:56am

However, as @exekias pointed prometheus.io/scrape annotation is commonly used so we may need to update the docs to avoid such confusing situations!

@Babadofar thanks for mentioning this, feel free to open a GH issue for this. Otherwise I can do it!

ChrsMark · March 4, 2020, 7:19am

Here is the merged PR: https://github.com/elastic/beats/pull/16597.

system · April 1, 2020, 7:19am

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.