Metricbeat kubernetes module can't connect to kubelet

Elastic Stack Beats
1

Hi.

We have a setup, where Metricbeat is deployed as a DaemonSet on a Kubernetes cluster (spefcifically -- AWS EKS).

All seems to be functioning properly, but the kubelet connection.

To clarify, the following module:

- module: kubernetes
  enabled: true
  metricsets:
    - state_pod
  period: 10s
  hosts: ["kube-state-metrics.system:8080"]

works properly (the events flow into logstash/elastic).

This module configuration, however, doesn't work in any variants of hosts value (localhost/kubernetes.default/whatever):

- module: kubernetes
  period: 10s
  metricsets:
    - pod
  hosts: ["localhost:10255"]
  enabled: true
  add_metadata: true
  in_cluster: true

NOTE: using cluster IP instead of localhost (so that it goes to control plane) also works (although doesn't retrieve the needed information, of course).

The configuration above was taken directly from the Metricbeat documentation and immediately struck me as odd -- how does localhost get translated (from within Metricbeat docker) to corresponding kubelet?

The error is, as one would expect, in light of the above:

error making http request: Get http://localhost:10255/stats/summary: 
dial tcp [::1]:10255: connect: cannot assign requested address

which indicates some sort of connectivity issue.

However, when SSH-ing to any node Metricbeat is deployed on, http://localhost:10255/stats/summary provides the correct output:

{
  "node": {
   "nodeName": "...",
   "systemContainers": [
    {
     "name": "pods",
     "startTime": "2018-12-06T11:22:07Z",
     "cpu": {
      "time": "2018-12-23T06:54:06Z",
      ...
     },
     "memory": {
      "time": "2018-12-23T06:54:06Z",
      "availableBytes": 17882275840,
      ....

I must be missing something very obvious. Any suggestion would do.

NOTE: cross-posted to SO

Update 29.12.2018

The solution was to inject hostIP via downward API:

apiVersion: v1
kind: DaemonSet
metadata:
  ...
spec:
  hostNetwork: false
  containers:
    - name: metrictbeat
      ...
      env:
        - name: HOST_IP
          valueFrom:
            fieldRef:
              fieldPath: status.hostIP

and then use the environment variable in the metricbeat configuration:

- module: kubernetes
  period: 10s
  metricsets:
    - pod
  hosts: ['${HOST_IP}:10255']
  enabled: true
  add_metadata: true
  in_cluster: true

Note the single quotes in hosts above... I vaguely recall environment variables interpolation not working otherwise (single quotes or nothing).

Also note that hostNetwork setting should be left at default, which is falseor explicitly set to false

1 Like
2

Hi @dnutels, have you tried replacing localhost with the actual nodeName in the config file?

3

@Kaiyan_Sheng I ended up injecting hostIP via downward API, with hostNetwork at false. Why nodeName instead?

4

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.