Metricbeats autodiscovery failing in kubernetes

[Alex_Armstrong]

Autodiscover not working for metricbeat 6.4.0 in kubernetes 1.9.6.
Nginx module in this use case, uwsgi also tried.
Declaring the module and giving an nginx ip outside of autodiscover works. below is the configmap being used.
Any ideas on some additional ways to set this up or problems that would stop the autodiscover from working.

apiVersion: v1
kind: ConfigMap
metadata:
  name: metricbeat-deployment-config
  namespace: kube-system
  labels:
    k8s-app: metricbeat
data:
  metricbeat.yml: |-
    metricbeat.config.modules:
      # Mounted `metricbeat-daemonset-modules` configmap:
      path: ${path.config}/modules.d/*.yml
      # Reload module configs as they change:
      reload.enabled: false

    processors:
      - add_cloud_metadata:

    output.elasticsearch:
      hosts: ['${ELASTICSEARCH_HOST:elasticsearch}:${ELASTICSEARCH_PORT:9200}']
---
apiVersion: v1
kind: ConfigMap
metadata:
  name: metricbeat-deployment-modules
  namespace: kube-system
  labels:
    k8s-app: metricbeat
data:
  autodiscover.yml: |-
    metricbeat.autodiscover:
      providers:
        - type: kubernetes
          host: ${HOSTNAME}
          #hints.enabled: true
          templates:
            - condition:
                contains:
                  kubernetes.container.name: nginx
              config:
                - module: nginx
                  metricsets: ["stubstatus"]
                  enable: true
                  period: 10s
                  hosts: ["${data.host}:80"]
                  server_status_path: "nginx_status"

  kubernetes.yml: |-
    - module: kubernetes
      metricsets:
        - state_node
        - state_deployment
        - state_replicaset
        - state_pod
        - state_container
      period: 10s
      host: ${NODE_NAME}
      hosts: ["kube-state-metrics.monitoring.svc:8080"]

[exekias (Carlos Pérez Aradros)]

Hi @Alex_Armstrong,

Do you see any error in the logs? can you please attach them?

[Alex_Armstrong]

No errors or warnings relating to this in the logs.
I have tried on both the daemonset and deployment of metricbeat to no success.
Let me know if you need any additional info for troubleshooting this.

[exekias (Carlos Pérez Aradros)]

I'm wondering, could you share the pod definition for the nginx container? I want to check that the names are right to match your condition.

Best regards

[Alex_Armstrong]

I can - It also has the annotation in from where we tried getting "hints.enabled: true" to work with it - though that is currently disabled in the config above (hopefully no conflict there).

apiVersion: extensions/v1beta1
kind: Deployment
metadata:
  namespace: env-stg
  name: app-proxy
  labels:
    k8s-app: app
    role: proxy
    env: env-stg
spec:
  replicas: 1
  strategy:
    rollingUpdate:
      maxSurge: 25%
      maxUnavailable: 25%
    type: RollingUpdate
  template:
    metadata:
      labels:
        k8s-app: app
        role: proxy
        env: env-stg
      annotations:
        co.elastic.logs/module: nginx
        co.elastic.logs/fileset.stdout: access
        co.elastic.logs/fileset.stderr: error
        co.elastic.metrics/module: nginx
        co.elastic.metrics/metricsets: stubstatus
        co.elastic.metrics/hosts: '${data.host}:80'
        co.elastic.metrics/period: 10s
        co.elastic.metrics/server_status_path: "nginx_status"
        prometheus.io.scrape: "true"
    spec:
      containers:
      - name: nginx
        image: nginx
        ports:
        - containerPort: 80
          name: transport
        volumeMounts:
        - name: app-config
          mountPath: /etc/nginx/sites-enabled/app.conf
          subPath: app.conf
        - name: nginx-config
          mountPath: /etc/nginx/nginx.conf
          subPath: nginx.conf
      terminationGracePeriodSeconds: 600
      volumes:
      - name: app-config
        configMap:
      name: app-proxy-configmap
          items:
          - key: nginx.conf
            path: nginx.conf
          - key: app.conf
            path: app.conf
      - name: nginx-config
        configMap:
          name: app-proxy-configmap

[exekias (Carlos Pérez Aradros)]

Oh,

I see the issue now: server_status_path is not a valid hint, you would need to use a template for this case. We are considering allowing any type of param to the module, but it has some security implications

[Alex_Armstrong]

server_status_path is valid in the nginx module posted at the top though (which also has hints disabled). Even if we remove the commented hints line from the configuration at the top and remove the annotations we still encounter the error. I can repost the config if it helps?

[exekias (Carlos Pérez Aradros)]

I think that would help, can you post just the module config?

[Alex_Armstrong]

Running the below with the internal IP of one of the nginx pods will bring metrics in for that pod. As this doesn't use autodiscover it is limited to us updating the list of ip addresses for the various nginx pods as they change which is difficult to maintain.

- module: nginx
  metricsets: ["stubstatus"]
  enable: true
  period: 10s
  hosts: ["<ip addr>:80"]
  server_status_path: "nginx_status"