# Metricbeat kubernetes module can't connect to kubelet

**URL:** <https://discuss.elastic.co/t/metricbeat-kubernetes-module-cant-connect-to-kubelet/161939>\
**Category:** Beats\
**Tags:** metricbeat\
**Created:** [December 23, 2018, 7:03am UTC](https://discuss.elastic.co/t/metricbeat-kubernetes-module-cant-connect-to-kubelet/161939 "2018-12-23T07:03:46Z")\
**Posts on this page:** 4\
**Page:** 1

<div class="post-metadata">

**Author:** ![dnutels](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/dnutels/32/36772_2.png) [@dnutels](https://discuss.elastic.co/u/dnutels)\
**Post date:** [December 23, 2018, 7:03am UTC](https://discuss.elastic.co/t/metricbeat-kubernetes-module-cant-connect-to-kubelet/161939/1 "2018-12-23T07:03:47Z")

</div>

Hi.

We have a setup, where Metricbeat is deployed as a DaemonSet on a Kubernetes cluster (spefcifically -- AWS EKS).

All seems to be functioning properly, but the kubelet connection.

To clarify, the following module:

```
- module: kubernetes
  enabled: true
  metricsets:
    - state_pod
  period: 10s
  hosts: ["kube-state-metrics.system:8080"]

```

works properly (the events flow into logstash/elastic).

This module configuration, however, doesn't work in any variants of `hosts` value (localhost/kubernetes.default/whatever):

```
- module: kubernetes
  period: 10s
  metricsets:
    - pod
  hosts: ["localhost:10255"]
  enabled: true
  add_metadata: true
  in_cluster: true

```

> **NOTE** : using cluster IP instead of `localhost` (so that it goes to control plane) also works (although doesn't retrieve the needed information, of course).
> 
> The configuration above was taken directly from the Metricbeat documentation and immediately struck me as odd -- how does `localhost` get translated (from within Metricbeat docker) to corresponding kubelet?

The error is, as one would expect, in light of the above:

```
error making http request: Get http://localhost:10255/stats/summary: 
dial tcp [::1]:10255: connect: cannot assign requested address

```

which indicates some sort of connectivity issue.

However, when SSH-ing to any node Metricbeat is deployed on, `http://localhost:10255/stats/summary` provides the correct output:

```
{
  "node": {
   "nodeName": "...",
   "systemContainers": [
    {
     "name": "pods",
     "startTime": "2018-12-06T11:22:07Z",
     "cpu": {
      "time": "2018-12-23T06:54:06Z",
      ...
     },
     "memory": {
      "time": "2018-12-23T06:54:06Z",
      "availableBytes": 17882275840,
      ....

```

I must be missing something very obvious. Any suggestion would do.

NOTE: cross-posted to [SO](https://stackoverflow.com/questions/53910022/metricbeat-kubernetes-module-can-t-connect-to-kubelet)

* * *

**Update 29.12.2018**

The solution was to inject `hostIP` via downward API:

```auto
apiVersion: v1
kind: DaemonSet
metadata:
  ...
spec:
  hostNetwork: false
  containers:
    - name: metrictbeat
      ...
      env:
        - name: HOST_IP
          valueFrom:
            fieldRef:
              fieldPath: status.hostIP  

```

and then use the environment variable in the metricbeat configuration:

```
- module: kubernetes
  period: 10s
  metricsets:
    - pod
  hosts: ['${HOST_IP}:10255']
  enabled: true
  add_metadata: true
  in_cluster: true

```

**Note** the single quotes in `hosts` above... I vaguely recall environment variables interpolation not working otherwise (single quotes or nothing).

**Also note** that `hostNetwork` setting should be left at default, which is `false`or explicitly set to `false`

---

<div class="post-metadata">

**Author:** ![Kaiyan\_Sheng](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/kaiyan_sheng/32/38247_2.png) [@Kaiyan\_Sheng](https://discuss.elastic.co/u/Kaiyan_Sheng)\
**Post date:** [December 28, 2018, 6:49pm UTC](https://discuss.elastic.co/t/metricbeat-kubernetes-module-cant-connect-to-kubelet/161939/2 "2018-12-28T18:49:03Z")

</div>

Hi @dnutels, have you tried replacing localhost with the actual nodeName in the config file?

---

<div class="post-metadata">

**Author:** ![dnutels](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/dnutels/32/36772_2.png) [@dnutels](https://discuss.elastic.co/u/dnutels)\
**Post date:** [December 29, 2018, 3:49am UTC](https://discuss.elastic.co/t/metricbeat-kubernetes-module-cant-connect-to-kubelet/161939/3 "2018-12-29T03:49:17Z")

</div>

@Kaiyan_Sheng I ended up injecting `hostIP` via downward API, with `hostNetwork` at `false`. Why `nodeName` instead?

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\
**Post date:** [January 26, 2019, 4:01am UTC](https://discuss.elastic.co/t/metricbeat-kubernetes-module-cant-connect-to-kubelet/161939/4 "2019-01-26T04:01:09Z")

</div>

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.