Hi.
We have a setup, where Metricbeat is deployed as a DaemonSet on a Kubernetes cluster (spefcifically -- AWS EKS).
All seems to be functioning properly, but the kubelet connection.
To clarify, the following module:
- module: kubernetes
enabled: true
metricsets:
- state_pod
period: 10s
hosts: ["kube-state-metrics.system:8080"]
works properly (the events flow into logstash/elastic).
This module configuration, however, doesn't work in any variants of hosts
value (localhost/kubernetes.default/whatever):
- module: kubernetes
period: 10s
metricsets:
- pod
hosts: ["localhost:10255"]
enabled: true
add_metadata: true
in_cluster: true
NOTE: using cluster IP instead of
localhost
(so that it goes to control plane) also works (although doesn't retrieve the needed information, of course).The configuration above was taken directly from the Metricbeat documentation and immediately struck me as odd -- how does
localhost
get translated (from within Metricbeat docker) to corresponding kubelet?
The error is, as one would expect, in light of the above:
error making http request: Get http://localhost:10255/stats/summary:
dial tcp [::1]:10255: connect: cannot assign requested address
which indicates some sort of connectivity issue.
However, when SSH-ing to any node Metricbeat is deployed on, http://localhost:10255/stats/summary
provides the correct output:
{
"node": {
"nodeName": "...",
"systemContainers": [
{
"name": "pods",
"startTime": "2018-12-06T11:22:07Z",
"cpu": {
"time": "2018-12-23T06:54:06Z",
...
},
"memory": {
"time": "2018-12-23T06:54:06Z",
"availableBytes": 17882275840,
....
I must be missing something very obvious. Any suggestion would do.
NOTE: cross-posted to SO
Update 29.12.2018
The solution was to inject hostIP
via downward API:
apiVersion: v1
kind: DaemonSet
metadata:
...
spec:
hostNetwork: false
containers:
- name: metrictbeat
...
env:
- name: HOST_IP
valueFrom:
fieldRef:
fieldPath: status.hostIP
and then use the environment variable in the metricbeat configuration:
- module: kubernetes
period: 10s
metricsets:
- pod
hosts: ['${HOST_IP}:10255']
enabled: true
add_metadata: true
in_cluster: true
Note the single quotes in hosts
above... I vaguely recall environment variables interpolation not working otherwise (single quotes or nothing).
Also note that hostNetwork
setting should be left at default, which is false
or explicitly set to false