I have exposed prometheus formatted metrics on several pods, using the prometheus module and listing each service separately works well, but I can not get autodiscover working. Using debug logging for all modules, I can not see any indication of something wrong.
Running EKS, kubernetes version 1.14.
Thankful for any help.
This is the config which does not work:
metricbeat.autodiscover:
providers:
- type: kubernetes
include_annotations: ["*"]
hints.enabled: false
namespace: jx-staging
templates:
- condition:
contains:
kubernetes.annotations.prometheus.io.scrape: "true"
config:
- module: prometheus
metricsets: ["collector"]
hosts: ["${data.host}:8080"]
metrics_path: '/management/prometheus'
processors:
- add_kubernetes_metadata:
in_cluster: true
And some logs `
2020-02-08T21:21:40.374Z DEBUG [autodiscover] kubernetes/kubernetes.go:136 Watcher Pod add: &Pod{ObjectMeta:k8s_io_apimachinery_pkg_apis_meta_v1.ObjectMeta{Name:jx-usermanagment-7fc96974c9-5mzvl,GenerateName:jx-usermanagment-7fc96974c9-,Namespace:jx-staging,SelfLink:/api/v1/namespaces/jx-staging/pods/jx-usermanagment-7fc96974c9-5mzvl,UID:8a0baeb8-4ab6-11ea-991f-0a8d81bb25f4,ResourceVersion:21092442,Generation:0,CreationTimestamp:2020-02-08 21:04:04 +0000 UTC,DeletionTimestamp:<nil>,DeletionGracePeriodSeconds:nil,Labels:map[string]string{app: jx-usermanagment,draft: draft-app,pod-template-hash: 7fc96974c9,},Annotations:map[string]string{co.elastic.metrics/hosts: usermanagment,co.elastic.metrics/metricsets: collector,co.elastic.metrics/module: prometheus,co.elastic.metrics/path: /management/prometheus,co.elastic.metrics/period: 1m,kubernetes.io/psp: eks.privileged,prometheus.io/scrape: true,},OwnerReferences:[{apps/v1 ReplicaSet jx-usermanagment-7fc96974c9 8a0895f4-4ab6-11ea-991f-0a8d81bb25f4 0xc0009e8017 0xc0009e8018}],Finalizers:[],ClusterName:,ManagedFields:[],},Spec:PodSpec{Volumes:[{default-token-2tbq2 {nil nil nil nil nil SecretVolumeSource{SecretName:default-token-2tbq2,Items:[],DefaultMode:*420,Optional:nil,} nil nil nil nil nil nil nil nil nil nil nil nil nil nil nil nil nil nil nil nil nil nil}}],Containers:[{usermanagment 523883733847.dkr.ecr.eu-west-1.amazonaws.com/mentor-medier/usermanagment:1.0.48 [] [] [{ 0 8080 TCP }] [{ nil SecretEnvSource{LocalObjectReference:LocalObjectReference{Name:user-db-secret,},Optional:nil,}}] [{dburl jdbc:mysql://mentordigital-campaign-staging.czaidexyizsq.eu-west-1.rds.amazonaws.com/users nil} {dbuser userman nil} {defaultReturnUrl https://user-login-poc-jx-staging.mentordigital.io nil}] {map[cpu:{{500 -3} {<nil>} 500m DecimalSI} memory:{{536870912 0} {<nil>} BinarySI}] map[cpu:{{400 -3} {<nil>} 400m DecimalSI} memory:{{536870912 0} {<nil>} BinarySI}]} [{default-token-2tbq2 true /var/run/secrets/kubernetes.io/serviceaccount <nil> }] [] &Probe{Handler:Handler{Exec:nil,HTTPGet:&HTTPGetAction{Path:/management/health,Port:8080,Host:,Scheme:HTTP,HTTPHeaders:[],},TCPSocket:nil,},InitialDelaySeconds:60,TimeoutSeconds:1,PeriodSeconds:10,SuccessThreshold:1,FailureThreshold:3,} &Probe{Handler:Handler{Exec:nil,HTTPGet:&HTTPGetAction{Path:/management/health,Port:8080,Host:,Scheme:HTTP,HTTPHeaders:[],},TCPSocket:nil,},InitialDelaySeconds:0,TimeoutSeconds:1,PeriodSeconds:10,SuccessThreshold:1,FailureThreshold:3,} nil /dev/termination-log File IfNotPresent nil false false false}],RestartPolicy:Always,TerminationGracePeriodSeconds:*30,ActiveDeadlineSeconds:nil,DNSPolicy:ClusterFirst,NodeSelector:map[string]string{},ServiceAccountName:default,DeprecatedServiceAccount:default,NodeName:ip-192-168-40-176.eu-west-1.compute.internal,HostNetwork:false,HostPID:false,HostIPC:false,SecurityContext:&PodSecurityContext{SELinuxOptions:nil,RunAsUser:nil,RunAsNonRoot:nil,SupplementalGroups:[],FSGroup:nil,RunAsGroup:nil,Sysctls:[],WindowsOptions:nil,},ImagePullSecrets:[],Hostname:,Subdomain:,Affinity:nil,SchedulerName:default-scheduler,InitContainers:[],AutomountServiceAccountToken:nil,Tolerations:[{node.kubernetes.io/not-ready Exists NoExecute 0xc0009e81e0} {node.kubernetes.io/unreachable Exists NoExecute 0xc0009e8200}],HostAliases:[],PriorityClassName:,Priority:*0,DNSConfig:nil,ShareProcessNamespace:nil,ReadinessGates:[],RuntimeClassName:nil,EnableServiceLinks:*true,PreemptionPolicy:nil,Overhead:ResourceList{},TopologySpreadConstraints:[],EphemeralContainers:[],},Status:PodStatus{Phase:Running,Conditions:[{Initialized True 0001-01-01 00:00:00 +0000 UTC 2020-02-08 21:04:04 +0000 UTC } {Ready True 0001-01-01 00:00:00 +0000 UTC 2020-02-08 21:04:52 +0000 UTC } {ContainersReady True 0001-01-01 00:00:00 +0000 UTC 2020-02-08 21:04:52 +0000 UTC } {PodScheduled True 0001-01-01 00:00:00 +0000 UTC 2020-02-08 21:04:04 +0000 UTC }],Message:,Reason:,HostIP:192.168.40.176,PodIP:192.168.37.87,StartTime:2020-02-08 21:04:04 +0000 UTC,ContainerStatuses:[{usermanagment {nil ContainerStateRunning{StartedAt:2020-02-08 21:04:05 +0000 UTC,} nil} {nil nil nil} true 0 523883733847.dkr.ecr.eu-west-1.amazonaws.com/mentor-medier/usermanagment:1.0.48 docker-pullable://523883733847.dkr.ecr.eu-west-1.amazonaws.com/mentor-medier/usermanagment@sha256:622c29790e31702180387fa96977db813596bcca1328ad2d34909abb7da2b4ca docker://120100ebac89210040156acf84bed5f23d1b11d3ea68e72f198296374d98083e}],QOSClass:Burstable,InitContainerStatuses:[],NominatedNodeName:,PodIPs:[],EphemeralContainerStatuses:[],},}
2020-02-08T21:21:40.374Z DEBUG [autodiscover] kubernetes/kubernetes.go:324 Generated hints {"metrics":{"hosts":"usermanagment","metricsets":"collector","module":"prometheus","path":"/management/prometheus","period":"1m"}}
2020-02-08T21:21:40.374Z DEBUG [autodiscover] kubernetes/kubernetes.go:329 Generated builder event map[container:{"id":"120100ebac89210040156acf84bed5f23d1b11d3ea68e72f198296374d98083e","image":"523883733847.dkr.ecr.eu-west-1.amazonaws.com/mentor-medier/usermanagment:1.0.48","name":"usermanagment","runtime":"docker"} hints:{"metrics":{"hosts":"usermanagment","metricsets":"collector","module":"prometheus","path":"/management/prometheus","period":"1m"}} host:192.168.37.87 kubernetes:{"annotations":{"co":{"elastic":{"metrics/hosts":"usermanagment","metrics/metricsets":"collector","metrics/module":"prometheus","metrics/path":"/management/prometheus","metrics/period":"1m"}},"kubernetes":{"io/psp":"eks.privileged"},"prometheus":{"io/scrape":"true"}},"container":{"id":"120100ebac89210040156acf84bed5f23d1b11d3ea68e72f198296374d98083e","image":"523883733847.dkr.ecr.eu-west-1.amazonaws.com/mentor-medier/usermanagment:1.0.48","name":"usermanagment","runtime":"docker"},"labels":{"app":"jx-usermanagment","draft":"draft-app","pod-template-hash":"7fc96974c9"},"namespace":"jx-staging","node":{"name":"ip-192-168-40-176.eu-west-1.compute.internal"},"pod":{"name":"jx-usermanagment-7fc96974c9-5mzvl","uid":"8a0baeb8-4ab6-11ea-991f-0a8d81bb25f4"},"replicaset":{"name":"jx-usermanagment-7fc96974c9"}} port:8080]
2020-02-08T21:21:40.374Z DEBUG [autodiscover] autodiscover/autodiscover.go:165 Got a start event: map[config:[] host:192.168.37.87 id:8a0baeb8-4ab6-11ea-991f-0a8d81bb25f4.usermanagment kubernetes:{"annotations":{"co":{"elastic":{"metrics/hosts":"usermanagment","metrics/metricsets":"collector","metrics/module":"prometheus","metrics/path":"/management/prometheus","metrics/period":"1m"}},"kubernetes":{"io/psp":"eks.privileged"},"prometheus":{"io/scrape":"true"}},"container":{"id":"120100ebac89210040156acf84bed5f23d1b11d3ea68e72f198296374d98083e","image":"523883733847.dkr.ecr.eu-west-1.amazonaws.com/mentor-medier/usermanagment:1.0.48","name":"usermanagment","runtime":"docker"},"labels":{"app":"jx-usermanagment","draft":"draft-app","pod-template-hash":"7fc96974c9"},"namespace":"jx-staging","node":{"name":"ip-192-168-40-176.eu-west-1.compute.internal"},"pod":{"name":"jx-usermanagment-7fc96974c9-5mzvl","uid":"8a0baeb8-4ab6-11ea-991f-0a8d81bb25f4"},"replicaset":{"name":"jx-usermanagment-7fc96974c9"}} meta:{"kubernetes":{"container":{"image":"523883733847.dkr.ecr.eu-west-1.amazonaws.com/mentor-medier/usermanagment:1.0.48","name":"usermanagment"},"labels":{"app":"jx-usermanagment","draft":"draft-app","pod-template-hash":"7fc96974c9"},"namespace":"jx-staging","node":{"name":"ip-192-168-40-176.eu-west-1.compute.internal"},"pod":{"name":"jx-usermanagment-7fc96974c9-5mzvl","uid":"8a0baeb8-4ab6-11ea-991f-0a8d81bb25f4"},"replicaset":{"name":"jx-usermanagment-7fc96974c9"}}} port:8080 provider:a1766f07-0726-4c00-a271-668ce0338bac start:true]
2020-02-08T21:21:40.374Z DEBUG [autodiscover] autodiscover/autodiscover.go:252 Got a meta field in the event
2020-0`