AutoDiscovery & Kubernetes & Promteheus within single namespace

Elastic Stack Beats
1

Hi

I'm trying to setup autodiscovery within Kubernetes to collect metrics from Prometheus endpoints. My configuration looks like:

metricbeat.config:
  modules:
    path: ${path.config}/modules.d/*.yml
    reload.enabled: true
    reload.period: 10s
#============================== Dashboards =====================================
# These settings control loading the sample dashboards to the Kibana index.
# Loading the dashboards is disabled by default and can be enabled either by
# setting the options here, or by using the `-setup` CLI flag or the `setup` cmd.
setup.dashboards.enabled: true

#============================== Kibana =========================================
# Starting with Beats version 6.0.0, the dashboards are loaded via the Kibana API.
# This requires a Kibana endpoint configuration.
setup.kibana.host: "{{ .Release.Name }}-kibana-internal:5601"
setup.kibana.protocol: "http"
setup.kibana.path: /monitor

#================================= Outputs =====================================
# Configure what output to use when sending the data collected by the beat.

#========================== Elasticsearch output ===============================
output.elasticsearch:
  hosts: ["{{ .Release.Name }}-elasticsearch-client:9200"]
  protocol: "http"
#  protocol: "https"
#  username: "elastic"
#  password: "changeme"
# podAnnotations:
#  sb4b-component: monitor
output.file:
  enabled: false

# don't show metrics on console
logging.metrics.enabled: true

logging.level: debug

#============================= Autodiscover ==================+=================
# allows tracking Kubernetes APIs to respond to container start and stop events
metricbeat.autodiscover:
  providers:
    - type: kubernetes
      include_annotations: ["prometheus.io/scrape"]
      namespace: {{ .Release.Namespace }}
      templates:
        - condition:
            contains:
              kubernetes.annotations.prometheus.io/scrape: "true"
          config:
            - module: prometheus
              enabled: true
              metricsets: ["collector"]
              metrics_path: /actuator/prometheus
              hosts: "${data.host}:${data.port}"

But I'm getting this output:

2019-05-15T15:52:11.536Z        DEBUG   [autodiscover]  autodiscover/autodiscover.go:195        Check failed for config &{{<nil> } <nil> 0xc42002a720}: 1 error: missing required field accessing 'namespace', won't start runner
2019-05-15T15:52:11.536Z        DEBUG   [autodiscover]  autodiscover/autodiscover.go:165        Got a start event: map[id:35587e41-76f5-11e9-b68e-42010aac0070.demo-corda-node start:true host:10.16.3.124 port:8083 kubernetes:{"annota
2019-05-15T15:52:11.536Z        DEBUG   [autodiscover]  autodiscover/autodiscover.go:183        Generated configs: [0xc420649080]
2019-05-15T15:52:11.536Z        DEBUG   [autodiscover]  autodiscover/autodiscover.go:243        Got a meta field in the event
2019-05-15T15:52:11.536Z        WARN    [cfgwarn]       collector/collector.go:57       BETA: The prometheus collector metricset is beta
2019-05-15T15:52:11.536Z        DEBUG   [autodiscover]  autodiscover/autodiscover.go:195        Check failed for config &{{<nil> } <nil> 0xc42002b3a0}: 1 error: missing required field accessing 'namespace', won't start runner
2019-05-15T15:52:11.536Z        DEBUG   [autodiscover]  autodiscover/autodiscover.go:165        Got a start event: map[start:true host:10.16.3.124 kubernetes:{"annotations":{"co":{"elastic":{"metrics/period":"1m"}},"prometheus":{"io/path":"/actuator/prometheus","i...
2019-05-15T15:52:11.536Z        DEBUG   [autodiscover]  autodiscover/autodiscover.go:183        Generated configs: [0xc4205e0150]
2019-05-15T15:52:11.536Z        DEBUG   [autodiscover]  autodiscover/autodiscover.go:243        Got a meta field in the event
2019-05-15T15:52:11.536Z        DEBUG   [autodiscover]  autodiscover/autodiscover.go:195        Check failed for config &{{<nil> } <nil> 0xc4203c48e0}: missing required field accessing 'module', won't start runner
2019-05-15T15:52:11.537Z        DEBUG   [autodiscover]  autodiscover/autodiscover.go:165        Got a start event: map[config:[0xc42065b050] provider:bb146bab-1561-4427-8f88-3e156140640b id:35587e41-76f5-11e9-b68e-42010aac0070.init-app-demo start:true host:10.16.3.124kubern...
2019-05-15T15:52:11.537Z        DEBUG   [autodiscover]  autodiscover/autodiscover.go:183        Generated configs: [0xc42065b050]
2019-05-15T15:52:11.537Z        DEBUG   [autodiscover]  autodiscover/autodiscover.go:243        Got a meta field in the event
2019-05-15T15:52:11.537Z        DEBUG   [autodiscover]  autodiscover/autodiscover.go:195        Check failed for config &{{<nil> } <nil> 0xc4203c54e0}: missing required field accessing 'module', won't start runner
2019-05-15T15:52:11.537Z        DEBUG   [autodiscover]  autodiscover/autodiscover.go:165        Got a start event: map[meta:{"kubernetes":{"container":{"name":"apm-server"},"labels":{"app":"demo","deployment":"apm-server","pod-template-hash":"2247305717","release":"demo"},"namespace":"demo-marvel-test","node":{"name":"gke-demo-test-....
2019-05-15T15:52:11.537Z        DEBUG   [autodiscover]  autodiscover/autodiscover.go:183        Generated configs: [] 
2019-05-15T15:52:11.537Z        DEBUG   [autodiscover]  autodiscover/autodiscover.go:243        Got a meta field in the event

Any ideas of what could be wrong? The exception messages could be a bit more helpful in the area. Event with debug the output is limited. Currently I do no understand why there are issues with the "module" and "namespace" field:

error: missing required field accessing 'namespace'

It seems to discover the various pods quite fine. But it fails to template and enable the matching configuration. The namespace property itself is also working, because it does not discover fields outside the specified namespace.

Thank you, Regards Remo

2

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.