# Default Dashboard "\[Metricbeat Kubernetes\] Overview ECS" doesn't work

**URL:** <https://discuss.elastic.co/t/default-dashboard-metricbeat-kubernetes-overview-ecs-doesnt-work/203101>\
**Category:** Beats\
**Tags:** metricbeat\
**Created:** [October 10, 2019, 6:57pm UTC](https://discuss.elastic.co/t/default-dashboard-metricbeat-kubernetes-overview-ecs-doesnt-work/203101 "2019-10-10T18:57:36Z")\
**Posts on this page:** 11\
**Page:** 1

<div class="post-metadata">

**Author:** ![xwiz](https://avatars.discourse-cdn.com/v4/letter/x/ad7895/32.png) [@xwiz](https://discuss.elastic.co/u/xwiz)\
**Post date:** [October 10, 2019, 6:57pm UTC](https://discuss.elastic.co/t/default-dashboard-metricbeat-kubernetes-overview-ecs-doesnt-work/203101/1 "2019-10-10T18:57:36Z")

</div>

Hello!  
I am trying to configure monitoring of my Kubernetes cluster. For this, I did:

1. Elasticsearch server and kibana version 7.1.1
2. Installed filebeat and metricbeat 7.1.1 on my Kubernetes cluster via yaml files.
3. Installed the standard metricbeat kubernetes dashboards.

I met some problems:

1. Some dashboards "[Metricbeat Kubernetes] Overview ECS" work, some do not have data (screenshot "metricbeat"). Although there should be enough data
2. The dashboards on the "[Metricbeat Kubernetes] API server ECS" page are completely empty.

How can I configure the correct transfer and display of data in my Kibana?

The settings for transmitting metrics in yaml I used the following:

```
---
apiVersion: v1
kind: ConfigMap
metadata:
  name: metricbeat-daemonset-config
  namespace: kube-system
  labels:
    k8s-app: metricbeat
data:
  metricbeat.yml: |-
    metricbeat.config.modules:
      # Mounted `metricbeat-daemonset-modules` configmap:
      path: ${path.config}/modules.d/*.yml
      # Reload module configs as they change:
      reload.enabled: false

    # To enable hints based autodiscover uncomment this:
    metricbeat.autodiscover:
      providers:
        - type: kubernetes
          host: ${NODE_NAME}
          hints.enabled: true

    processors:
      - add_cloud_metadata:

    cloud.id: ${ELASTIC_CLOUD_ID}
    cloud.auth: ${ELASTIC_CLOUD_AUTH}

    output.elasticsearch:
      hosts: ['${ELASTICSEARCH_HOST:elasticsearch}:${ELASTICSEARCH_PORT:9200}']
      username: ${ELASTICSEARCH_USERNAME}
      password: ${ELASTICSEARCH_PASSWORD}
    setup.dashboards.enabled: true
    setup.kibana.host: "172.16.0.184:5601"
    setup.kibana.protocol: "http"
    setup.kibana.username: "user"
    setup.kibana.password: "password"
    setup.template.settings:
        index.number_of_shards: 5
        index.number_of_replicas: 1
        index.number_of_routing_shards: 30
---
apiVersion: v1
kind: ConfigMap
metadata:
  name: metricbeat-daemonset-modules
  namespace: kube-system
  labels:
    k8s-app: metricbeat
data:
  system.yml: |-
    - module: system
      period: 10s
      metricsets:
        - cpu
        - load
        - memory
        - network
        - process
        - process_summary
        #- core
        #- diskio
        #- socket
      processes: ['.*']
      process.include_top_n:
        by_cpu: 5 # include top 5 processes by CPU
        by_memory: 5 # include top 5 processes by memory

    - module: system
      period: 1m
      metricsets:
        - filesystem
        - fsstat
      processors:
      - drop_event.when.regexp:
          system.filesystem.mount_point: '^/(sys|cgroup|proc|dev|etc|host|lib)($|/)'
  kubernetes.yml: |-
    - module: kubernetes
      metricsets:
        - node
        - system
        - pod
        - container
        - volume
      period: 10s
      host: ${NODE_NAME}
      hosts: ["${HOST_IP}:10255"]

```

 ![metricbeat](https://us1.discourse-cdn.com/elastic/original/3X/c/3/c3265c73ec33d74969e4eca2d013222088427cff.jpeg) ![metricbeat2](https://us1.discourse-cdn.com/elastic/original/3X/b/4/b47925631667426998d1395cb22650b4a5579ef7.jpeg) ![photo_2019-10-10_18-47-17](https://us1.discourse-cdn.com/elastic/original/3X/5/e/5e8e906b7d849a312e9f3730fb489fca20efdd27.jpeg)

---

<div class="post-metadata">

**Author:** ![Alex\_Kristiansen](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/alex_kristiansen/32/46086_2.png) [@Alex\_Kristiansen](https://discuss.elastic.co/u/Alex_Kristiansen)\
**Post date:** [October 14, 2019, 1:43pm UTC](https://discuss.elastic.co/t/default-dashboard-metricbeat-kubernetes-overview-ecs-doesnt-work/203101/2 "2019-10-14T13:43:58Z")

</div>

xwiz,

Are there any errors in your metricbeat logs, if so can you paste us a snippet?

---

<div class="post-metadata">

**Author:** ![Alex\_Kristiansen](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/alex_kristiansen/32/46086_2.png) [@Alex\_Kristiansen](https://discuss.elastic.co/u/Alex_Kristiansen)\
**Post date:** [October 14, 2019, 2:30pm UTC](https://discuss.elastic.co/t/default-dashboard-metricbeat-kubernetes-overview-ecs-doesnt-work/203101/3 "2019-10-14T14:30:37Z")

</div>

Another thing: It looks like the dashboards you're trying to use require both the `state_deployment` and `state_node` metricsets, which I don't see enabled.

---

<div class="post-metadata">

**Author:** ![xwiz](https://avatars.discourse-cdn.com/v4/letter/x/ad7895/32.png) [@xwiz](https://discuss.elastic.co/u/xwiz)\
**Post date:** [October 15, 2019, 10:27am UTC](https://discuss.elastic.co/t/default-dashboard-metricbeat-kubernetes-overview-ecs-doesnt-work/203101/4 "2019-10-15T10:27:26Z")

</div>

I have only one type of errors. There is an example on the screenshot:  
 ![%D0%B8%D0%B7%D0%BE%D0%B1%D1%80%D0%B0%D0%B6%D0%B5%D0%BD%D0%B8%D0%B5](https://us1.discourse-cdn.com/elastic/original/3X/a/3/a308d4fc6b2f9da68a63739697a501f2a5c4b053.png)

This log I received from filebeat and this error appear only for metricbeat's containers. It's strange because I use oss filebeat 7.1.1, oss metricbeat 7.1.1 and oss elasticsearch 7.1.1

Maybe can I give a conclusion from some command to make it clearer?

---

<div class="post-metadata">

**Author:** ![xwiz](https://avatars.discourse-cdn.com/v4/letter/x/ad7895/32.png) [@xwiz](https://discuss.elastic.co/u/xwiz)\
**Post date:** [October 15, 2019, 10:30am UTC](https://discuss.elastic.co/t/default-dashboard-metricbeat-kubernetes-overview-ecs-doesnt-work/203101/5 "2019-10-15T10:30:44Z")

</div>

But I have this lines in my yaml:

```
data:
  # This module requires `kube-state-metrics` up and running under `kube-system` namespace
  kubernetes.yml: |-
    - module: kubernetes
      metricsets:
        - state_node
        - state_deployment
        - state_replicaset
        - state_pod
        - state_container

```

This is not enough?

---

<div class="post-metadata">

**Author:** ![xwiz](https://avatars.discourse-cdn.com/v4/letter/x/ad7895/32.png) [@xwiz](https://discuss.elastic.co/u/xwiz)\
**Post date:** [October 15, 2019, 12:03pm UTC](https://discuss.elastic.co/t/default-dashboard-metricbeat-kubernetes-overview-ecs-doesnt-work/203101/6 "2019-10-15T12:03:22Z")

</div>

I tried to find all metricsets in my elasticsearch:

```
GET metricbeat-*/_search
{
  "size": 0, 
  "query": {
      "exists": {
          "field": "metricset.name"
      }
  },
  "aggs" : {
      "products" : {
          "terms" : {
              "field" : "metricset.name",
              "size" : 500
          }
      }
  }
}

```

I get the result:

```
{
  "took" : 33,
  "timed_out" : false,
  "_shards" : {
    "total" : 7,
    "successful" : 7,
    "skipped" : 0,
    "failed" : 0
  },
  "hits" : {
    "total" : {
      "value" : 10000,
      "relation" : "gte"
    },
    "max_score" : null,
    "hits" : []
  },
  "aggregations" : {
    "products" : {
      "doc_count_error_upper_bound" : 0,
      "sum_other_doc_count" : 0,
      "buckets" : [
        {
          "key" : "volume",
          "doc_count" : 1305543
        },
        {
          "key" : "network",
          "doc_count" : 1262298
        },
        {
          "key" : "process",
          "doc_count" : 725722
        },
        {
          "key" : "container",
          "doc_count" : 714964
        },
        {
          "key" : "pod",
          "doc_count" : 629736
        },
        {
          "key" : "system",
          "doc_count" : 272216
        },
        {
          "key" : "cpu",
          "doc_count" : 101099
        },
        {
          "key" : "load",
          "doc_count" : 101099
        },
        {
          "key" : "memory",
          "doc_count" : 101097
        },
        {
          "key" : "process_summary",
          "doc_count" : 101097
        },
        {
          "key" : "node",
          "doc_count" : 101096
        },
        {
          "key" : "filesystem",
          "doc_count" : 16863
        },
        {
          "key" : "fsstat",
          "doc_count" : 16863
        }
      ]
    }
  }
}

```

There is not **state\_node**. So I changed filter of dashbord "Kubernetes - Nodes ECS" from

```
event.module:kubernetes AND metricset.name:state_node

```

to

```
event.module:kubernetes AND metricset.name:node

```

and now it works:

 ![%D0%B8%D0%B7%D0%BE%D0%B1%D1%80%D0%B0%D0%B6%D0%B5%D0%BD%D0%B8%D0%B5](https://us1.discourse-cdn.com/elastic/original/3X/6/4/6401d2d6def941c194603a0d385c54920030f1ac.png)

But I don't receive any **deployment states** , other dashboard don't work. How do I get these states?

---

<div class="post-metadata">

**Author:** ![pmercado](https://avatars.discourse-cdn.com/v4/letter/p/59ef9b/32.png) [@pmercado](https://discuss.elastic.co/u/pmercado)\
**Post date:** [October 15, 2019, 7:38pm UTC](https://discuss.elastic.co/t/default-dashboard-metricbeat-kubernetes-overview-ecs-doesnt-work/203101/7 "2019-10-15T19:38:14Z")

</div>

Hi @xwiz,

from the data you are sending I'd say you are not fetching any state metrics data.  
Can you make sure that:

- [kube-state-metrics](https://github.com/kubernetes/kube-state-metrics) is deployed
- that the kube-state-metrics service is exposed and reachable to metricbeat?

If the service is called kube-state-metrics and it is configured on 8080 (default settings if you use their upstream manifests), and if metricbeat is deployed in cluster, this should work:

```
- module: kubernetes
  enabled: true
  metricsets:
    - state_deployment
    - state_node
    - state_pod
    - state_container
  period: 10s
  hosts: ["kube-state-metrics.kube-system:8080"]
  in_cluster: true
```

---

<div class="post-metadata">

**Author:** ![xwiz](https://avatars.discourse-cdn.com/v4/letter/x/ad7895/32.png) [@xwiz](https://discuss.elastic.co/u/xwiz)\
**Post date:** [October 16, 2019, 6:41am UTC](https://discuss.elastic.co/t/default-dashboard-metricbeat-kubernetes-overview-ecs-doesnt-work/203101/8 "2019-10-16T06:41:48Z")

</div>

Hi Pablo,  
I deployed kube-state-metrics before and service was enabled:

```
[root@kube2 metric_conf]# kubectl get svc --all-namespaces
NAMESPACE NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE
kube-system kube-state-metrics ClusterIP 10.105.70.199 <none> 8080/TCP,8081/TCP 6d15h

```

I use next yaml kube-states-metrics:

```
---
apiVersion: rbac.authorization.k8s.io/v1
# kubernetes versions before 1.8.0 should use rbac.authorization.k8s.io/v1beta1
kind: ClusterRoleBinding
metadata:
  name: kube-state-metrics
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: ClusterRole
  name: kube-state-metrics
subjects:
- kind: ServiceAccount
  name: kube-state-metrics
  namespace: kube-system
---
apiVersion: rbac.authorization.k8s.io/v1
# kubernetes versions before 1.8.0 should use rbac.authorization.k8s.io/v1beta1
kind: ClusterRole
metadata:
  name: kube-state-metrics
rules:
- apiGroups: [""]
  resources:
  - configmaps
  - secrets
  - nodes
  - pods
  - services
  - resourcequotas
  - replicationcontrollers
  - limitranges
  - persistentvolumeclaims
  - persistentvolumes
  - namespaces
  - endpoints
  verbs: ["list", "watch"]
- apiGroups: ["extensions"]
  resources:
  - daemonsets
  - deployments
  - replicasets
  - ingresses
  verbs: ["list", "watch"]
- apiGroups: ["apps"]
  resources:
  - daemonsets
  - deployments
  - replicasets
  - statefulsets
  verbs: ["list", "watch"]
- apiGroups: ["batch"]
  resources:
  - cronjobs
  - jobs
  verbs: ["list", "watch"]
- apiGroups: ["autoscaling"]
  resources:
  - horizontalpodautoscalers
  verbs: ["list", "watch"]
- apiGroups: ["policy"]
  resources:
  - poddisruptionbudgets
  verbs: ["list", "watch"]
- apiGroups: ["certificates.k8s.io"]
  resources:
  - certificatesigningrequests
  verbs: ["list", "watch"]
- apiGroups: ["storage.k8s.io"]
  resources:
  - storageclasses
  verbs: ["list", "watch"]
- apiGroups: ["autoscaling.k8s.io"]
  resources:
  - verticalpodautoscalers
  verbs: ["list", "watch"]
---
apiVersion: apps/v1
kind: Deployment
metadata:
  labels:
    k8s-app: kube-state-metrics
  name: kube-state-metrics
  namespace: kube-system
spec:
  selector:
    matchLabels:
      k8s-app: kube-state-metrics
  replicas: 1
  template:
    metadata:
      labels:
        k8s-app: kube-state-metrics
    spec:
      serviceAccountName: kube-state-metrics
      containers:
      - name: kube-state-metrics
        image: quay.io/coreos/kube-state-metrics:v1.7.1
        ports:
        - name: http-metrics
          containerPort: 8080
        - name: telemetry
          containerPort: 8081
        readinessProbe:
          httpGet:
            path: /healthz
            port: 8080
          initialDelaySeconds: 5
          timeoutSeconds: 5
---
apiVersion: v1
kind: ServiceAccount
metadata:
  name: kube-state-metrics
  namespace: kube-system
---
apiVersion: v1
kind: Service
metadata:
  name: kube-state-metrics
  namespace: kube-system
  labels:
    k8s-app: kube-state-metrics
  annotations:
    prometheus.io/scrape: 'true'
spec:
  ports:
  - name: http-metrics
    port: 8080
    targetPort: http-metrics
    protocol: TCP
  - name: telemetry
    port: 8081
    targetPort: telemetry
    protocol: TCP
  selector:
    k8s-app: kube-state-metrics

```

Yesterday I tried to use yaml from [this artcle](https://kauri.io/article/935f4e17a10243139b41546780f43c42/v4/(35)-collect-metrics-with-elastic-metricbeat-for-monitoring-kubernetes). But metricsets "state\_node", "state\_deployment", "state\_replicaset", "state\_pod", "state\_container" are still unreacheable.

It looks like my metricbeat cannot access kube-states-metrics. How can I check and fix it?

---

<div class="post-metadata">

**Author:** ![xwiz](https://avatars.discourse-cdn.com/v4/letter/x/ad7895/32.png) [@xwiz](https://discuss.elastic.co/u/xwiz)\
**Post date:** [October 16, 2019, 7:07am UTC](https://discuss.elastic.co/t/default-dashboard-metricbeat-kubernetes-overview-ecs-doesnt-work/203101/9 "2019-10-16T07:07:06Z")

</div>

I found this error in my elastic:

 ![%D0%B8%D0%B7%D0%BE%D0%B1%D1%80%D0%B0%D0%B6%D0%B5%D0%BD%D0%B8%D0%B5](https://us1.discourse-cdn.com/elastic/original/3X/9/1/91aaef1a6b6e029151e870ea1110b9621fee8857.png)

Why does metricbeat cannot connect to kube-state-metrics if service run correctly on default ports?

---

<div class="post-metadata">

**Author:** ![xwiz](https://avatars.discourse-cdn.com/v4/letter/x/ad7895/32.png) [@xwiz](https://discuss.elastic.co/u/xwiz)\
**Post date:** [October 16, 2019, 8:08am UTC](https://discuss.elastic.co/t/default-dashboard-metricbeat-kubernetes-overview-ecs-doesnt-work/203101/10 "2019-10-16T08:08:46Z")

</div>

I have resolved this problem. For this I changed line "hosts" to next view:

```
hosts: ["kube-state-metrics.kube-system.svc.cluster.local:8080"]

```

Now all of my dashboards work:

 ![%D0%B8%D0%B7%D0%BE%D0%B1%D1%80%D0%B0%D0%B6%D0%B5%D0%BD%D0%B8%D0%B5](https://us1.discourse-cdn.com/elastic/original/3X/8/6/86029d46b1d02507c0d9260f78d6826ccbc9d69b.png)

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\
**Post date:** [November 13, 2019, 8:11am UTC](https://discuss.elastic.co/t/default-dashboard-metricbeat-kubernetes-overview-ecs-doesnt-work/203101/11 "2019-11-13T08:11:48Z")

</div>

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.