# Metricbeat DNS error in kubernetes Cluster

**URL:** <https://discuss.elastic.co/t/metricbeat-dns-error-in-kubernetes-cluster/263303>\
**Category:** Beats\
**Tags:** metricbeat\
**Created:** [February 4, 2021, 9:42pm UTC](https://discuss.elastic.co/t/metricbeat-dns-error-in-kubernetes-cluster/263303 "2021-02-04T21:42:48Z")\
**Posts on this page:** 10\
**Page:** 1

<div class="post-metadata">

**Author:** ![umesh2020](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/umesh2020/32/126038_2.png) [@umesh2020](https://discuss.elastic.co/u/umesh2020)\
**Post date:** [February 4, 2021, 9:42pm UTC](https://discuss.elastic.co/t/metricbeat-dns-error-in-kubernetes-cluster/263303/1 "2021-02-04T21:42:48Z")

</div>

I have installed metric beat using helm chart and I am getting following DNS error. 2021-02-04T21:27:38.074Z WARN [transport] transport/tcp.go:52 DNS lookup failure "k8s-rke-cluster1-node2": lookup k8s-rke-cluster1-node2 on 10.43.0.10:53: server misbehaving

16:27:38.075

2021-02-04T21:27:38.074Z INFO module/wrapper.go:259 Error fetching data for metricset kubernetes.system: error doing HTTP request to fetch 'system' Metricset data: error making http request: Get "[https://k8s-rke-cluster1-node2:10250/stats/summary](https://k8s-rke-cluster1-node2:10250/stats/summary)": lookup k8s-rke-cluster1-node2 on 10.43.0.10:53: server misbehaving

I have the following entry in the metric-beats.yml, but still it doesn't help:  
spec:  
template:  
spec:  
hostNetwork: true  
dnsPolicy: ClusterFirstWithHostNet

Can you please help.

Thanks  
Umesh

---

<div class="post-metadata">

**Author:** ![umesh2020](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/umesh2020/32/126038_2.png) [@umesh2020](https://discuss.elastic.co/u/umesh2020)\
**Post date:** [February 5, 2021, 6:12pm UTC](https://discuss.elastic.co/t/metricbeat-dns-error-in-kubernetes-cluster/263303/2 "2021-02-05T18:12:00Z")

</div>

When I looked at DNS, logs, I am getting NXDOMAIN error. Is there some configuration I need to update to  
[INFO] 10.42.1.161:44476 - 51541 "A IN k8s-rke-cluster1-node1.logging.cluster.local. udp 62 false 512" NXDOMAIN qr,aa,rd 155 0.000165481s  
[INFO] 10.42.1.161:59219 - 6274 "A IN k8s-rke-cluster1-node1.logging. udp 48 false 512" NXDOMAIN qr,rd,ra 123 0.019766515s  
[INFO] 10.42.1.161:59138 - 1373 "A IN wire-read.svc.cluster.local. udp 45 false 512" NXDOMAIN qr,aa,rd 138 0.000230953s

---

<div class="post-metadata">

**Author:** ![ChrsMark](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/chrsmark/32/55858_2.png) [@ChrsMark](https://discuss.elastic.co/u/ChrsMark)\
**Post date:** [February 8, 2021, 10:37am UTC](https://discuss.elastic.co/t/metricbeat-dns-error-in-kubernetes-cluster/263303/3 "2021-02-08T10:37:10Z")

</div>

Hi!

Can you share your [configuration part](https://github.com/elastic/beats/blob/master/deploy/kubernetes/metricbeat-kubernetes.yaml#L111-L113) that sets up the `kubelet` api endpoint?

Also could you `exec` inside metricbeat Pod and try to reach the `kubelet` api manually with curl?

```auto
token=$(cat /var/run/secrets/kubernetes.io/serviceaccount/token)
curl -H "Authorization: Bearer $token" https://${HOSTNAME}:10250/stats/summary --insecure

```

Note: The above `curl` is the equivalent that Metricbeat will try to perform so as to get the metrcis from `kubelet`.

C.

---

<div class="post-metadata">

**Author:** ![umesh2020](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/umesh2020/32/126038_2.png) [@umesh2020](https://discuss.elastic.co/u/umesh2020)\
**Post date:** [February 8, 2021, 4:51pm UTC](https://discuss.elastic.co/t/metricbeat-dns-error-in-kubernetes-cluster/263303/4 "2021-02-08T16:51:58Z")

</div>

Hi Chris,  
Please find the details you requested for:

```auto
      metricbeatConfig:
        metricbeat.yml: |
          metricbeat.modules:
          - module: kubernetes
            metricsets:
              - container
              - node
              - pod
              - system
              - volume
            period: 10s
            host: "${NODE_NAME}"
            hosts: ["https://${NODE_NAME}:10250"]
            bearer_token_file: /var/run/secrets/kubernetes.io/serviceaccount/token
            ssl.verification_mode: "none"
            # If using Red Hat OpenShift remove ssl.verification_mode entry and
            # uncomment these settings:
            #ssl.certificate_authorities:
              #- /var/run/secrets/kubernetes.io/serviceaccount/service-ca.crt
            processors:
            - add_kubernetes_metadata: ~

```

Also, I was able to execute the curl command and reach the kubelet api without any issues.

```auto
> umesh@umeshs-MacBook-Pro Local-ELK % kubectl exec -it my-metricbeat-metricbeat-4tht9 -- /bin/bash
> [root@k8s-rke-cluster1-node2 metricbeat]#
> [root@k8s-rke-cluster1-node2 metricbeat]#
> [root@k8s-rke-cluster1-node2 metricbeat]#
> [root@k8s-rke-cluster1-node2 metricbeat]# token=$(cat /var/run/secrets/kubernetes.io/serviceaccount/token)
> [root@k8s-rke-cluster1-node2 metricbeat]# curl -H "Authorization: Bearer $token" https://${HOSTNAME}:10250/stats/summary --insecure
> {
> "node": {
> "nodeName": "k8s-rke-cluster1-node2",
> "systemContainers": [
> {
> "name": "kubelet",
> "startTime": "2021-02-08T03:14:19Z",
> "cpu": {
> "time": "2021-02-08T16:48:42Z",
> "usageNanoCores": 32691702,
> "usageCoreNanoSeconds": 1536862822056
> },
> "memory": {
> "time": "2021-02-08T16:48:42Z",
> "usageBytes": 179445760,
> "workingSetBytes": 57331712,
> "rssBytes": 49405952,
> "pageFaults": 32014689,
> "majorPageFaults": 81
> }
> },
> {
> "name": "runtime",
> "startTime": "2021-02-08T03:14:16Z",
> "cpu": {
> "time": "2021-02-08T16:48:38Z",
> "usageNanoCores": 30031690,
> "usageCoreNanoSeconds": 1571225630130
> },
> "memory": {
> "time": "2021-02-08T16:48:38Z",
> "usageBytes": 6586425344,
> "workingSetBytes": 3134005248,
> "rssBytes": 87359488,
> "pageFaults": 70791372,
> "majorPageFaults": 333
> }
> },
> {
> "name": "pods",
> "startTime": "2021-02-08T03:14:18Z",
> "cpu": {
> "time": "2021-02-08T16:48:35Z",
> "usageNanoCores": 142213300,
> "usageCoreNanoSeconds": 9085480733540
> },
> "memory": {
> "time": "2021-02-08T16:48:35Z",
> "availableBytes": 13652520960,
> "usageBytes": 3504705536,
> "workingSetBytes": 3173511168,
> "rssBytes": 3094118400,
> "pageFaults": 0,
> "majorPageFaults": 0
> }
> }
> ],
> "startTime": "2021-02-08T03:14:11Z",
> "cpu": {
> "time": "2021-02-08T16:48:35Z",
> "usageNanoCores": 241303905,
> "usageCoreNanoSeconds": 13295512173772
> },
> "memory": {
> "time": "2021-02-08T16:48:35Z",
> "availableBytes": 10005635072,
> "usageBytes": 11078508544,
> "workingSetBytes": 6820397056,
> "rssBytes": 3609812992,
> "pageFaults": 444927,
> "majorPageFaults": 84
> },

```

---

<div class="post-metadata">

**Author:** ![ChrsMark](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/chrsmark/32/55858_2.png) [@ChrsMark](https://discuss.elastic.co/u/ChrsMark)\
**Post date:** [February 9, 2021, 1:52pm UTC](https://discuss.elastic.co/t/metricbeat-dns-error-in-kubernetes-cluster/263303/5 "2021-02-09T13:52:03Z")

</div>

> [@ChrsMark](#):
>
> ```auto
> curl -H "Authorization: Bearer $token" https://${HOSTNAME}:10250/stats/summary --insecure
> 
> ```

Could you try with `curl -H "Authorization: Bearer $token" https://${NODE_NAME}:10250/stats/summary --insecure `?  
`NODE_NAME` instead of `HOSTNAME`.  
Also could you please share the value of these env vars?

---

<div class="post-metadata">

**Author:** ![umesh2020](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/umesh2020/32/126038_2.png) [@umesh2020](https://discuss.elastic.co/u/umesh2020)\
**Post date:** [February 9, 2021, 9:20pm UTC](https://discuss.elastic.co/t/metricbeat-dns-error-in-kubernetes-cluster/263303/6 "2021-02-09T21:20:25Z")

</div>

Hi Chris,  
Please find the details you requested for. Looks like if the HOSTNAME has domain name whereas NODE\_NAME doesn't. My resolv.conf does have the domain-name though.  
Thanks  
Umesh

```auto
[root@k8s-rke-cluster1-node2 metricbeat]# curl -H "Authorization: Bearer $token" https://${NODE_NAME}:10250/stats/summary --insecure
curl: (6) Could not resolve host: k8s-rke-cluster1-node2; Unknown error
[root@k8s-rke-cluster1-node2 metricbeat]# echo ${HOSTNAME}
k8s-rke-cluster1-node2.pfg.dom
[root@k8s-rke-cluster1-node2 metricbeat]# echo ${NODE_NAME}
k8s-rke-cluster1-node2
[root@k8s-rke-cluster1-node2 metricbeat]# cat /etc/resolv.conf
nameserver 10.43.0.10
search logging.svc.cluster.local svc.cluster.local cluster.local pfg.dom
options ndots:5

```

---

<div class="post-metadata">

**Author:** ![ChrsMark](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/chrsmark/32/55858_2.png) [@ChrsMark](https://discuss.elastic.co/u/ChrsMark)\
**Post date:** [February 10, 2021, 11:22am UTC](https://discuss.elastic.co/t/metricbeat-dns-error-in-kubernetes-cluster/263303/7 "2021-02-10T11:22:14Z")

</div>

Hey again.

So `NODE_NAME` should be the name of the node as it is assigned on Pod's spec by k8s. If you run `kubectl get nodes` you should see `k8s-rke-cluster1-node2`. However, I'm not sure why it is different from the `HOSTNAME` and how `HOSTNAME` is set in your case, do you have any special DNS configuration that may result in this?

In any case, in order to tackle this just change `hosts: ["https://${NODE_NAME}:10250"]` in your configuration to `hosts: ["https://${HOSTNAME}:10250"]`. This should do the trick.

---

<div class="post-metadata">

**Author:** ![umesh2020](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/umesh2020/32/126038_2.png) [@umesh2020](https://discuss.elastic.co/u/umesh2020)\
**Post date:** [February 12, 2021, 5:43pm UTC](https://discuss.elastic.co/t/metricbeat-dns-error-in-kubernetes-cluster/263303/8 "2021-02-12T17:43:46Z")

</div>

Hi Chris,  
I don't have any special config DNS config. We use rancher , so I am not sure whether rancher has any special config.

Thanks & Regards,  
Umesh

---

<div class="post-metadata">

**Author:** ![ChrsMark](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/chrsmark/32/55858_2.png) [@ChrsMark](https://discuss.elastic.co/u/ChrsMark)\
**Post date:** [February 22, 2021, 8:47am UTC](https://discuss.elastic.co/t/metricbeat-dns-error-in-kubernetes-cluster/263303/9 "2021-02-22T08:47:37Z")

</div>

Hey!

So did replacing with `HOSTNAME` work?

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\
**Post date:** [March 22, 2021, 10:48am UTC](https://discuss.elastic.co/t/metricbeat-dns-error-in-kubernetes-cluster/263303/10 "2021-03-22T10:48:37Z")

</div>

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.