# Error running Metricbeat 7.6 on AKS K8S cluster

**URL:** <https://discuss.elastic.co/t/error-running-metricbeat-7-6-on-aks-k8s-cluster/246608>\
**Category:** Beats\
**Tags:** metricbeat\
**Created:** [August 27, 2020, 11:32am UTC](https://discuss.elastic.co/t/error-running-metricbeat-7-6-on-aks-k8s-cluster/246608 "2020-08-27T11:32:30Z")\
**Posts on this page:** 7\
**Page:** 1

<div class="post-metadata">

**Author:** ![mirii1994](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/mirii1994/32/77245_2.png) [@mirii1994](https://discuss.elastic.co/u/mirii1994)\
**Post date:** [August 27, 2020, 11:32am UTC](https://discuss.elastic.co/t/error-running-metricbeat-7-6-on-aks-k8s-cluster/246608/1 "2020-08-27T11:32:30Z")

</div>

I'm trying to run metricbeat 7.6.2 on my AKS cluster (k8s version 1.16),  
And getting the following errors:

```auto
2020-08-27T11:00:03.417Z	INFO	module/wrapper.go:252	Error fetching data for metricset kubernetes.volume: error doing HTTP request to fetch 'volume' Metricset data: error making http request: Get https://aks-agentpool-XXX:10250/stats/summary: x509: certificate signed by unknown authority
2020-08-27T11:00:03.494Z	INFO	module/wrapper.go:252	Error fetching data for metricset kubernetes.node: error doing HTTP request to fetch 'node' Metricset data: error making http request: Get https://aks-agentpool-XXX:10250/stats/summary: x509: certificate signed by unknown authority
2020-08-27T11:00:03.548Z	INFO	module/wrapper.go:252	Error fetching data for metricset kubernetes.system: error doing HTTP request to fetch 'system' Metricset data: error making http request: Get https://aks-agentpool-XXX:10250/stats/summary: x509: certificate signed by unknown authority
2020-08-27T11:00:03.572Z	INFO	module/wrapper.go:252	Error fetching data for metricset kubernetes.pod: error doing HTTP request to fetch 'pod' Metricset data: error making http request: Get https://aks-agentpool-XXX:10250/stats/summary: x509: certificate signed by unknown authority
2020-08-27T11:00:03.644Z	INFO	module/wrapper.go:252	Error fetching data for metricset kubernetes.container: error doing HTTP request to fetch 'container' Metricset data: error making http request: Get https://aks-agentpool-XXX:10250/stats/summary: x509: certificate signed by unknown authority

```

My `metricbeat-daemonset-config` is:

```auto
kubernetes.yml: |-
    - module: kubernetes
      metricsets:
        - node
        - system
        - pod
        - container
        - volume
      period: 10s
      host: ${NODE_NAME}
      hosts: ["https://${HOSTNAME}:10250"]
      bearer_token_file: /var/run/secrets/kubernetes.io/serviceaccount/token
      ssl.certificate_authorities:
        - '/var/run/secrets/kubernetes.io/serviceaccount/ca.crt'

```

**Some notes:**

- Latest Metricbeat version that's working for me is 7.4.
- Metricbeat 7.6 is ok on EKS with this config.

Any suggestions?

---

<div class="post-metadata">

**Author:** ![ChrsMark](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/chrsmark/32/55858_2.png) [@ChrsMark](https://discuss.elastic.co/u/ChrsMark)\
**Post date:** [August 31, 2020, 12:29pm UTC](https://discuss.elastic.co/t/error-running-metricbeat-7-6-on-aks-k8s-cluster/246608/2 "2020-08-31T12:29:21Z")

</div>

Hi!

The error indicates that your certificates are not capable to be used in order to reach Kubelet's API.  
Do you actually want to enable `ssl`?

In general you can bypass this using `ssl.verification_mode: "none"`, otherwise you need to make sure that you are using the proper certificate to access the API.

See [https://github.com/elastic/beats/blob/f2956098ef62b0fec1ae02e7cb9659dd6b9e6fe9/deploy/kubernetes/metricbeat-kubernetes.yaml#L116](https://github.com/elastic/beats/blob/f2956098ef62b0fec1ae02e7cb9659dd6b9e6fe9/deploy/kubernetes/metricbeat-kubernetes.yaml#L116)

C.

---

<div class="post-metadata">

**Author:** ![mirii1994](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/mirii1994/32/77245_2.png) [@mirii1994](https://discuss.elastic.co/u/mirii1994)\
**Post date:** [September 3, 2020, 11:41am UTC](https://discuss.elastic.co/t/error-running-metricbeat-7-6-on-aks-k8s-cluster/246608/3 "2020-09-03T11:41:21Z")

</div>

@ChrsMark  
Yeah I did notice that, but I saw in the documentation that it is not recommended to set this field to "none".  
Is the `ssl.verification_mode` field refers to the pods certificate (the one that's in `/var/run/secrets/kubernetes.io/serviceaccount/ca.crt`, or to the certificate that I'm using in the `output.ssl.certificate_authorities` one?

---

<div class="post-metadata">

**Author:** ![ChrsMark](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/chrsmark/32/55858_2.png) [@ChrsMark](https://discuss.elastic.co/u/ChrsMark)\
**Post date:** [September 3, 2020, 12:42pm UTC](https://discuss.elastic.co/t/error-running-metricbeat-7-6-on-aks-k8s-cluster/246608/4 "2020-09-03T12:42:33Z")

</div>

It refers to `ssl.certificate_authorities` that is used to access Kubelet's API (`/var/run/secrets/kubernetes.io/serviceaccount/ca.crt`). It is also in the same config section: [https://github.com/elastic/beats/blob/f2956098ef62b0fec1ae02e7cb9659dd6b9e6fe9/deploy/kubernetes/metricbeat-kubernetes.yaml#L119](https://github.com/elastic/beats/blob/f2956098ef62b0fec1ae02e7cb9659dd6b9e6fe9/deploy/kubernetes/metricbeat-kubernetes.yaml#L119)

---

<div class="post-metadata">

**Author:** ![mirii1994](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/mirii1994/32/77245_2.png) [@mirii1994](https://discuss.elastic.co/u/mirii1994)\
**Post date:** [September 3, 2020, 2:30pm UTC](https://discuss.elastic.co/t/error-running-metricbeat-7-6-on-aks-k8s-cluster/246608/5 "2020-09-03T14:30:32Z")

</div>

@ChrsMark Thanks for the quick response. Just to make me understand it better- do you know what is the change from 7.4 to 7.5 that makes those errors when `ssl.verification_mode` is not `none`?

---

<div class="post-metadata">

**Author:** ![ChrsMark](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/chrsmark/32/55858_2.png) [@ChrsMark](https://discuss.elastic.co/u/ChrsMark)\
**Post date:** [September 3, 2020, 2:38pm UTC](https://discuss.elastic.co/t/error-running-metricbeat-7-6-on-aks-k8s-cluster/246608/6 "2020-09-03T14:38:03Z")

</div>

The initial error you see has to do with an invalid certificate. In this, I'm not sure what could be the reason. It could be the certificate itself, or Kubelet's config that changes some config option.

On our side, from `7.6.2` we switched to accessing secure port of Kubelet API. PR: [https://github.com/elastic/beats/pull/16063](https://github.com/elastic/beats/pull/16063)

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\
**Post date:** [October 1, 2020, 4:38pm UTC](https://discuss.elastic.co/t/error-running-metricbeat-7-6-on-aks-k8s-cluster/246608/7 "2020-10-01T16:38:09Z")

</div>

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.
