Kubernetes Autodiscover Setup & Logging

Elastic Stack Beats
1

Hi,

I'm attempting to get the Kubernetes auto-discovery working, but failed so far. My configuration looks like:

  metricbeat.config:
    modules:
      path: ${path.config}/modules.d/*.yml
      reload.enabled: false

  setup.dashboards.enabled: false
  setup.kibana.host: "kibana-internal:5601"
  setup.kibana.protocol: "http"
  setup.kibana.path: /monitor

  output.elasticsearch:
    hosts: ["elasticsearch:9200"]
    protocol: "http"

  output.file:
    enabled: false

  logging.metrics.enabled: true
  logging.level: debug

  metricbeat.autodiscover:
    providers:
      - type: kubernetes
        include_annotations: ["prometheus.io/scrape"]
        namespace: ${POD_NAMESPACE}
        templates:
          - condition.equals:
              kubernetes.annotations.prometheus.io/scrape: "true"
            config:
              - module: prometheus
                metrics_path: /actuator/prometheus
                metricsets: ["collector"]
                hosts: "${data.host}:${data.kubernetes.annotations.prometheus.io/port}"

POD_NAMESPACE is an environment variable passed to the metricbeat pod. ServiceAccount, Role and RoleBinding in place to have permissions to list pods in the current namespace (no cluster roles).

the strange thing is:

2019-06-26T13:34:13.899Z        INFO    [autodiscover]  autodiscover/autodiscover.go:105        Starting autodiscover manager
2019-06-26T13:34:13.899Z        INFO    kubernetes/watcher.go:182       kubernetes: Performing a resource sync for *v1.PodList
2019-06-26T13:34:13.902Z        DEBUG   [kubernetes]    kubernetes/watcher.go:189       Got 3 items from the resource sync

Problem is that there are not 3 resources (pods I assume) in the local namespace, but around 15. It only seems to find two, three pods. None of those passing the condition.equals.

I attempted a few things:

  • remove condition.equals => starts collecting from those three pods that do not have prometheus
  • hard-code hosts and namespace
  • use cluster roles
  • logger: debug => only see Watcher Pod Add events for those three resources.

Is there something I missed? Something else that is filters the pod list? Other services also query the list of pods without issues. Potentially it would be worthwhile to create a ticket to add more logging statements in the area.

Thank you, Regards Remo

2

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.