Hello
I'm using ECK 1.19 and I'm implementing monitoring with beat use the following:
---
apiVersion: beat.k8s.elastic.co/v1beta1
kind: Beat
metadata:
name: metricbeat
spec:
type: metricbeat
version: 7.16.2
elasticsearchRef:
name: ha-elastic
kibanaRef:
name: kibana
config:
http.enabled: true
http.port: 5066
http.host: 0.0.0.0
monitoring.enabled: false
metricbeat:
autodiscover:
providers:
- hints:
default_config: {}
enabled: "true"
host: ${HOSTNAME}
type: kubernetes
modules:
- module: system
period: 10s
metricsets:
- cpu
- load
- memory
- network
- process
- process_summary
process:
include_top_n:
by_cpu: 5
by_memory: 5
processes:
- .*
- module: system
period: 1m
metricsets:
- filesystem
- fsstat
processors:
- drop_event:
when:
regexp:
system:
filesystem:
mount_point: ^/(sys|cgroup|proc|dev|etc|host|lib)($|/)
- module: kubernetes
period: 10s
enabled: true
labels.dedot: true
add_metadata: true
host: ${HOSTNAME}
hosts:
- https://kube:10250
bearer_token_file: /var/run/secrets/kubernetes.io/serviceaccount/token
ssl:
verification_mode: none
metricsets:
- node
- system
- pod
- volume
- event
- container
processors:
- add_cloud_metadata: {}
- add_host_metadata: {}
daemonSet:
podTemplate:
metadata:
labels:
stack-monitoring.elastic.co/type: beat
annotations:
co.elastic.logs/enabled: "true"
spec:
serviceAccountName: metricbeat
automountServiceAccountToken: true # some older Beat versions are depending on this settings presence in k8s context
containers:
- args:
- -e
- -c
- /etc/beat.yml
- -system.hostfs=/hostfs
ports:
- containerPort: 5066
name: monitoring
protocol: TCP
name: metricbeat
volumeMounts:
- mountPath: /hostfs/sys/fs/cgroup
name: cgroup
- mountPath: /var/run/docker.sock
name: dockersock
- mountPath: /hostfs/proc
name: proc
securityContext:
runAsUser: 0
terminationGracePeriodSeconds: 30
volumes:
- hostPath:
path: /sys/fs/cgroup
name: cgroup
- hostPath:
path: /var/run/docker.sock
name: dockersock
- hostPath:
path: /proc
name: proc
---
# permissions needed for metricbeat
# source: https://www.elastic.co/guide/en/beats/metricbeat/current/metricbeat-module-kubernetes.html
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRole
metadata:
name: metricbeat
rules:
- apiGroups:
- ""
resources:
- nodes
- nodes/stats
- namespaces
- events
- pods
verbs:
- get
- list
- watch
- apiGroups:
- "extensions"
resources:
- replicasets
verbs:
- get
- list
- watch
- apiGroups:
- apps
resources:
- statefulsets
- deployments
- replicasets
verbs:
- get
- list
- watch
- apiGroups:
- ""
resources:
- nodes/stats
verbs:
- get
- list
- watch
- nonResourceURLs:
- /metrics
verbs:
- get
---
apiVersion: v1
kind: ServiceAccount
metadata:
name: metricbeat
namespace: default
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:
name: metricbeat
subjects:
- kind: ServiceAccount
name: metricbeat
namespace: default
roleRef:
kind: ClusterRole
name: metricbeat
apiGroup: rbac.authorization.k8s.io
Reviewing the logs of the metricbeat pod I found the following:
2022-01-25T19:07:48.255Z WARN [transport] transport/tcp.go:52 DNS lookup failure "kube": lookup kube on 10.24.23.10:53: no such host
2022-01-25T19:07:48.316Z ERROR module/wrapper.go:259 Error fetching data for metricset kubernetes.system: error doing HTTP request to fetch 'system' Metricset data: error making http request: Get "https://kube:10250/stats/summary": lookup kube on 10.24.23.10:53: no such host
2022-01-25T19:07:48.516Z ERROR [kubernetes.container] container/container.go:93 error making http request: Get "https://kube:10250/stats/summary": lookup kube on 10.24.23.10:53: no such host
2022-01-25T19:07:49.448Z ERROR [kubernetes.pod] pod/pod.go:94 error making http request: Get "https://kube:10250/stats/summary": lookup kube on 10.24.23.10:53: no such host
2022-01-25T19:07:51.368Z ERROR module/wrapper.go:259 Error fetching data for metricset kubernetes.volume: error doing HTTP request to fetch 'volume' Metricset data: error making http request: Get "https://kube:10250/stats/summary": lookup kube on 10.24.23.10:53: no such host
2022-01-25T19:07:51.794Z ERROR [kubernetes.node] node/node.go:95 error making http request: Get "https://kube:10250/stats/summary": lookup kube on 10.24.23.10:53: no such host
2022-01-25T19:07:57.819Z ERROR module/wrapper.go:259 Error fetching data for metricset kubernetes.system: error doing HTTP request to fetch 'system' Metricset data: error making http request: Get "https://kube:10250/stats/summary": lookup kube on 10.24.23.10:53: no such host
2022-01-25T19:07:58.755Z WARN [transport] transport/tcp.go:52 DNS lookup failure "kube": lookup kube on 10.24.23.10:53: no such host
2022-01-25T19:07:58.755Z ERROR [kubernetes.container] container/container.go:93 error making http request: Get "https://kube:10250/stats/summary": lookup kube on 10.24.23.10:53: no such host
2022-01-25T19:07:59.448Z ERROR [kubernetes.pod] pod/pod.go:94 error making http request: Get "https://kube:10250/stats/summary": lookup kube on 10.24.23.10:53: no such host
2022-01-25T19:08:01.368Z ERROR module/wrapper.go:259 Error fetching data for metricset kubernetes.volume: error doing HTTP request to fetch 'volume' Metricset data: error making http request: Get "https://kube:10250/stats/summary": lookup kube on 10.24.23.10:53: no such host
2022-01-25T19:08:01.793Z ERROR [kubernetes.node] node/node.go:95 error making http request: Get "https://kube:10250/stats/summary": lookup kube on 10.24.23.10:53: no such host
Follow the steps in this post.
That could be happening?