# Metricbeats autodiscovery failing in kubernetes

**URL:** https://discuss.elastic.co/t/metricbeats-autodiscovery-failing-in-kubernetes/147071
**Category:** Beats
**Tags:** metricbeat
**Created:** [September 3, 2018, 12:06pm UTC](https://discuss.elastic.co/t/metricbeats-autodiscovery-failing-in-kubernetes/147071 "2018-09-03T12:06:05Z")
**Posts on this page:** 10
**Page:** 1

<div class="post-metadata">

### Author: ![Alex\_Armstrong](https://avatars.discourse-cdn.com/v4/letter/a/da6949/32.png) [@Alex\_Armstrong](https://discuss.elastic.co/u/Alex_Armstrong)
#### Post date: [September 3, 2018, 12:06pm UTC](https://discuss.elastic.co/t/metricbeats-autodiscovery-failing-in-kubernetes/147071/1 "2018-09-03T12:06:05Z")

</div>

Autodiscover not working for metricbeat 6.4.0 in kubernetes 1.9.6.  
Nginx module in this use case, uwsgi also tried.  
Declaring the module and giving an nginx ip outside of autodiscover works. below is the configmap being used.  
Any ideas on some additional ways to set this up or problems that would stop the autodiscover from working.

```
apiVersion: v1
kind: ConfigMap
metadata:
  name: metricbeat-deployment-config
  namespace: kube-system
  labels:
    k8s-app: metricbeat
data:
  metricbeat.yml: |-
    metricbeat.config.modules:
      # Mounted `metricbeat-daemonset-modules` configmap:
      path: ${path.config}/modules.d/*.yml
      # Reload module configs as they change:
      reload.enabled: false

    processors:
      - add_cloud_metadata:

    output.elasticsearch:
      hosts: ['${ELASTICSEARCH_HOST:elasticsearch}:${ELASTICSEARCH_PORT:9200}']
---
apiVersion: v1
kind: ConfigMap
metadata:
  name: metricbeat-deployment-modules
  namespace: kube-system
  labels:
    k8s-app: metricbeat
data:
  autodiscover.yml: |-
    metricbeat.autodiscover:
      providers:
        - type: kubernetes
          host: ${HOSTNAME}
          #hints.enabled: true
          templates:
            - condition:
                contains:
                  kubernetes.container.name: nginx
              config:
                - module: nginx
                  metricsets: ["stubstatus"]
                  enable: true
                  period: 10s
                  hosts: ["${data.host}:80"]
                  server_status_path: "nginx_status"

  kubernetes.yml: |-
    - module: kubernetes
      metricsets:
        - state_node
        - state_deployment
        - state_replicaset
        - state_pod
        - state_container
      period: 10s
      host: ${NODE_NAME}
      hosts: ["kube-state-metrics.monitoring.svc:8080"]
```

---

<div class="post-metadata">

### Author: ![exekias](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/exekias/32/28718_2.png) [@exekias](https://discuss.elastic.co/u/exekias)
#### Post date: [September 3, 2018, 1:59pm UTC](https://discuss.elastic.co/t/metricbeats-autodiscovery-failing-in-kubernetes/147071/2 "2018-09-03T13:59:27Z")

</div>

Hi @Alex_Armstrong,

Do you see any error in the logs? can you please attach them?

---

<div class="post-metadata">

### Author: ![Alex\_Armstrong](https://avatars.discourse-cdn.com/v4/letter/a/da6949/32.png) [@Alex\_Armstrong](https://discuss.elastic.co/u/Alex_Armstrong)
#### Post date: [September 3, 2018, 2:22pm UTC](https://discuss.elastic.co/t/metricbeats-autodiscovery-failing-in-kubernetes/147071/3 "2018-09-03T14:22:37Z")

</div>

No errors or warnings relating to this in the logs.  
I have tried on both the daemonset and deployment of metricbeat to no success.  
Let me know if you need any additional info for troubleshooting this.

---

<div class="post-metadata">

### Author: ![exekias](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/exekias/32/28718_2.png) [@exekias](https://discuss.elastic.co/u/exekias)
#### Post date: [September 4, 2018, 1:45pm UTC](https://discuss.elastic.co/t/metricbeats-autodiscovery-failing-in-kubernetes/147071/4 "2018-09-04T13:45:01Z")

</div>

I'm wondering, could you share the pod definition for the nginx container? I want to check that the names are right to match your condition.

Best regards

---

<div class="post-metadata">

### Author: ![Alex\_Armstrong](https://avatars.discourse-cdn.com/v4/letter/a/da6949/32.png) [@Alex\_Armstrong](https://discuss.elastic.co/u/Alex_Armstrong)
#### Post date: [September 4, 2018, 2:56pm UTC](https://discuss.elastic.co/t/metricbeats-autodiscovery-failing-in-kubernetes/147071/5 "2018-09-04T14:56:58Z")

</div>

I can - It also has the annotation in from where we tried getting "hints.enabled: true" to work with it - though that is currently disabled in the config above (hopefully no conflict there).

```
apiVersion: extensions/v1beta1
kind: Deployment
metadata:
  namespace: env-stg
  name: app-proxy
  labels:
    k8s-app: app
    role: proxy
    env: env-stg
spec:
  replicas: 1
  strategy:
    rollingUpdate:
      maxSurge: 25%
      maxUnavailable: 25%
    type: RollingUpdate
  template:
    metadata:
      labels:
        k8s-app: app
        role: proxy
        env: env-stg
      annotations:
        co.elastic.logs/module: nginx
        co.elastic.logs/fileset.stdout: access
        co.elastic.logs/fileset.stderr: error
        co.elastic.metrics/module: nginx
        co.elastic.metrics/metricsets: stubstatus
        co.elastic.metrics/hosts: '${data.host}:80'
        co.elastic.metrics/period: 10s
        co.elastic.metrics/server_status_path: "nginx_status"
        prometheus.io.scrape: "true"
    spec:
      containers:
      - name: nginx
        image: nginx
        ports:
        - containerPort: 80
          name: transport
        volumeMounts:
        - name: app-config
          mountPath: /etc/nginx/sites-enabled/app.conf
          subPath: app.conf
        - name: nginx-config
          mountPath: /etc/nginx/nginx.conf
          subPath: nginx.conf
      terminationGracePeriodSeconds: 600
      volumes:
      - name: app-config
        configMap:
      name: app-proxy-configmap
          items:
          - key: nginx.conf
            path: nginx.conf
          - key: app.conf
            path: app.conf
      - name: nginx-config
        configMap:
          name: app-proxy-configmap
```

---

<div class="post-metadata">

### Author: ![exekias](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/exekias/32/28718_2.png) [@exekias](https://discuss.elastic.co/u/exekias)
#### Post date: [September 5, 2018, 11:02pm UTC](https://discuss.elastic.co/t/metricbeats-autodiscovery-failing-in-kubernetes/147071/6 "2018-09-05T23:02:22Z")

</div>

Oh,

I see the issue now: `server_status_path` is not a valid hint, you would need to use a template for this case. We are considering allowing any type of param to the module, but it has some security implications

---

<div class="post-metadata">

### Author: ![Alex\_Armstrong](https://avatars.discourse-cdn.com/v4/letter/a/da6949/32.png) [@Alex\_Armstrong](https://discuss.elastic.co/u/Alex_Armstrong)
#### Post date: [September 6, 2018, 2:45pm UTC](https://discuss.elastic.co/t/metricbeats-autodiscovery-failing-in-kubernetes/147071/7 "2018-09-06T14:45:16Z")

</div>

`server_status_path` is valid in the nginx module posted at the top though (which also has hints disabled). Even if we remove the commented hints line from the configuration at the top and remove the annotations we still encounter the error. I can repost the config if it helps?

---

<div class="post-metadata">

### Author: ![exekias](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/exekias/32/28718_2.png) [@exekias](https://discuss.elastic.co/u/exekias)
#### Post date: [September 10, 2018, 10:25am UTC](https://discuss.elastic.co/t/metricbeats-autodiscovery-failing-in-kubernetes/147071/8 "2018-09-10T10:25:14Z")

</div>

I think that would help, can you post just the module config?

---

<div class="post-metadata">

### Author: ![Alex\_Armstrong](https://avatars.discourse-cdn.com/v4/letter/a/da6949/32.png) [@Alex\_Armstrong](https://discuss.elastic.co/u/Alex_Armstrong)
#### Post date: [September 12, 2018, 2:13pm UTC](https://discuss.elastic.co/t/metricbeats-autodiscovery-failing-in-kubernetes/147071/9 "2018-09-12T14:13:02Z")

</div>

Running the below with the internal IP of one of the nginx pods will bring metrics in for that pod. As this doesn't use autodiscover it is limited to us updating the list of ip addresses for the various nginx pods as they change which is difficult to maintain.

```
- module: nginx
  metricsets: ["stubstatus"]
  enable: true
  period: 10s
  hosts: ["<ip addr>:80"]
  server_status_path: "nginx_status"
```

---

<div class="post-metadata">

### Author: ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)
#### Post date: [October 10, 2018, 2:13pm UTC](https://discuss.elastic.co/t/metricbeats-autodiscovery-failing-in-kubernetes/147071/10 "2018-10-10T14:13:04Z")

</div>

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.
