Metricbeats can't recognize kubernetes dns

Hello
I'm using ECK 1.19 and I'm implementing monitoring with beat use the following:

---
apiVersion: beat.k8s.elastic.co/v1beta1
kind: Beat
metadata:
  name: metricbeat
spec:
  type: metricbeat
  version: 7.16.2
  elasticsearchRef:
    name: ha-elastic
  kibanaRef:
    name: kibana
  config:
    http.enabled: true
    http.port: 5066
    http.host: 0.0.0.0
    monitoring.enabled: false
    metricbeat:
      autodiscover:
        providers:
          - hints:
              default_config: {}
              enabled: "true"
            host: ${HOSTNAME}
            type: kubernetes
      modules:
        - module: system
          period: 10s
          metricsets:
            - cpu
            - load
            - memory
            - network
            - process
            - process_summary
          process:
            include_top_n:
              by_cpu: 5
              by_memory: 5
          processes:
            - .*
        - module: system
          period: 1m
          metricsets:
            - filesystem
            - fsstat
          processors:
            - drop_event:
                when:
                  regexp:
                    system:
                      filesystem:
                        mount_point: ^/(sys|cgroup|proc|dev|etc|host|lib)($|/)
        - module: kubernetes
          period: 10s
          enabled: true
          labels.dedot: true
          add_metadata: true
          host: ${HOSTNAME}
          hosts:
            - https://kube:10250
          bearer_token_file: /var/run/secrets/kubernetes.io/serviceaccount/token
          ssl:
            verification_mode: none
          metricsets:
            - node
            - system
            - pod
            - volume
            - event
            - container
    processors:
      - add_cloud_metadata: {}
      - add_host_metadata: {}
  daemonSet:
    podTemplate:
      metadata:
        labels:
          stack-monitoring.elastic.co/type: beat
        annotations:
          co.elastic.logs/enabled: "true"
      spec:
        serviceAccountName: metricbeat
        automountServiceAccountToken: true # some older Beat versions are depending on this settings presence in k8s context
        containers:
          - args:
              - -e
              - -c
              - /etc/beat.yml
              - -system.hostfs=/hostfs
            ports:
              - containerPort: 5066
                name: monitoring
                protocol: TCP
            name: metricbeat
            volumeMounts:
              - mountPath: /hostfs/sys/fs/cgroup
                name: cgroup
              - mountPath: /var/run/docker.sock
                name: dockersock
              - mountPath: /hostfs/proc
                name: proc
        securityContext:
          runAsUser: 0
        terminationGracePeriodSeconds: 30
        volumes:
          - hostPath:
              path: /sys/fs/cgroup
            name: cgroup
          - hostPath:
              path: /var/run/docker.sock
            name: dockersock
          - hostPath:
              path: /proc
            name: proc
---
# permissions needed for metricbeat
# source: https://www.elastic.co/guide/en/beats/metricbeat/current/metricbeat-module-kubernetes.html
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRole
metadata:
  name: metricbeat
rules:
- apiGroups:
  - ""
  resources:
  - nodes
  - nodes/stats
  - namespaces
  - events
  - pods
  verbs:
  - get
  - list
  - watch
- apiGroups:
  - "extensions"
  resources:
  - replicasets
  verbs:
  - get
  - list
  - watch
- apiGroups:
  - apps
  resources:
  - statefulsets
  - deployments
  - replicasets
  verbs:
  - get
  - list
  - watch
- apiGroups:
  - ""
  resources:
  - nodes/stats
  verbs:
  - get
  - list
  - watch
- nonResourceURLs:
  - /metrics
  verbs:
  - get
---
apiVersion: v1
kind: ServiceAccount
metadata:
  name: metricbeat
  namespace: default
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:
  name: metricbeat
subjects:
  - kind: ServiceAccount
    name: metricbeat
    namespace: default
roleRef:
  kind: ClusterRole
  name: metricbeat
  apiGroup: rbac.authorization.k8s.io

Reviewing the logs of the metricbeat pod I found the following:

2022-01-25T19:07:48.255Z	WARN	[transport]	transport/tcp.go:52	DNS lookup failure "kube": lookup kube on 10.24.23.10:53: no such host
2022-01-25T19:07:48.316Z	ERROR	module/wrapper.go:259	Error fetching data for metricset kubernetes.system: error doing HTTP request to fetch 'system' Metricset data: error making http request: Get "https://kube:10250/stats/summary": lookup kube on 10.24.23.10:53: no such host
2022-01-25T19:07:48.516Z	ERROR	[kubernetes.container]	container/container.go:93	error making http request: Get "https://kube:10250/stats/summary": lookup kube on 10.24.23.10:53: no such host
2022-01-25T19:07:49.448Z	ERROR	[kubernetes.pod]	pod/pod.go:94	error making http request: Get "https://kube:10250/stats/summary": lookup kube on 10.24.23.10:53: no such host
2022-01-25T19:07:51.368Z	ERROR	module/wrapper.go:259	Error fetching data for metricset kubernetes.volume: error doing HTTP request to fetch 'volume' Metricset data: error making http request: Get "https://kube:10250/stats/summary": lookup kube on 10.24.23.10:53: no such host
2022-01-25T19:07:51.794Z	ERROR	[kubernetes.node]	node/node.go:95	error making http request: Get "https://kube:10250/stats/summary": lookup kube on 10.24.23.10:53: no such host
2022-01-25T19:07:57.819Z	ERROR	module/wrapper.go:259	Error fetching data for metricset kubernetes.system: error doing HTTP request to fetch 'system' Metricset data: error making http request: Get "https://kube:10250/stats/summary": lookup kube on 10.24.23.10:53: no such host
2022-01-25T19:07:58.755Z	WARN	[transport]	transport/tcp.go:52	DNS lookup failure "kube": lookup kube on 10.24.23.10:53: no such host
2022-01-25T19:07:58.755Z	ERROR	[kubernetes.container]	container/container.go:93	error making http request: Get "https://kube:10250/stats/summary": lookup kube on 10.24.23.10:53: no such host
2022-01-25T19:07:59.448Z	ERROR	[kubernetes.pod]	pod/pod.go:94	error making http request: Get "https://kube:10250/stats/summary": lookup kube on 10.24.23.10:53: no such host
2022-01-25T19:08:01.368Z	ERROR	module/wrapper.go:259	Error fetching data for metricset kubernetes.volume: error doing HTTP request to fetch 'volume' Metricset data: error making http request: Get "https://kube:10250/stats/summary": lookup kube on 10.24.23.10:53: no such host
2022-01-25T19:08:01.793Z	ERROR	[kubernetes.node]	node/node.go:95	error making http request: Get "https://kube:10250/stats/summary": lookup kube on 10.24.23.10:53: no such host

Follow the steps in this post.

That could be happening?

Hi,

did you jump inside the pod to check if the domain is unavailable? Maybe you should use a different one?

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.