[Metricbeat] metricbeat-metric pod keeps on crashing after deployment in kubernetes

Hi Community,

I have deployed the Metricbeat (v7.10.2) helm chart on the kubernetes. The deamonset pods and metricbeat-kube-state-metrics pod are running fine, but the metricbeat-metricbeat-metrics pod is keep on crashing. I am not sure which configuration is wrong and from logs there is only one error message I could observe, "Exiting: all modules are disabled".

=============Configuration================

deployment:
  enabled: true
  metricbeatConfig:
    metricbeat.yml: |
      metricbeat.modules:
      - module: kubernetes
        enabled: false
        metricsets:
          - state_node
          - state_deployment
          - state_replicaset
          - state_pod
          - state_container
        period: 5m
        host: ${NODE_NAME}
        hosts: ["kube-state-metrics:8080"]
      output.elasticsearch:
        hosts: ["https://{es_url}:443"]
        username: {es_username}
        password: {es_password}

===============Pod state==========================

metricbeat-kube-state-metrics-75c5fc65d9-cx2fs   1/1     Running            0          2d12h
metricbeat-metricbeat-48wsw                      0/1     Running            0          8s
metricbeat-metricbeat-metrics-6675dbc84d-ptfqf   0/1     CrashLoopBackOff   1          12s
metricbeat-metricbeat-zqk4x                      1/1     Running            0          11h

===============Logs from metricbeat-metricbeat-metrics==============

2022-06-16T03:49:23.353Z INFO instance/beat.go:645 Home path: [/usr/share/metricbeat] Config path: [/usr/share/metricbeat] Data path: [/usr/share/metricbeat/data] Logs path: [/usr/share/metricbeat/logs]

2022-06-16T03:49:23.355Z INFO instance/beat.go:653 Beat ID: 7a3f3048-6ca4-4655-8033-8cb7a77eb611

2022-06-16T03:49:23.356Z INFO [api] api/server.go:62 Starting stats endpoint

2022-06-16T03:49:23.356Z INFO [api] api/server.go:64 Metrics endpoint listening on: 127.0.0.1:5066 (configured: localhost)

2022-06-16T03:49:23.356Z INFO [seccomp] seccomp/seccomp.go:124 Syscall filter successfully installed

2022-06-16T03:49:23.356Z INFO [beat] instance/beat.go:981 Beat info {"system_info": {"beat": {"path": {"config": "/usr/share/metricbeat", "data": "/usr/share/metricbeat/data", "home": "/usr/share/metricbeat", "logs": "/usr/share/metricbeat/logs"}, "type": "metricbeat", "uuid": "7a3f3048-6ca4-4655-8033-8cb7a77eb611"}}}

2022-06-16T03:49:23.356Z INFO [beat] instance/beat.go:990 Build info {"system_info": {"build": {"commit": "aacf9ecd9c494aa0908f61fbca82c906b16562a8", "libbeat": "7.10.2", "time": "2021-01-12T22:24:31.000Z", "version": "7.10.2"}}}

2022-06-16T03:49:23.356Z INFO [beat] instance/beat.go:993 Go runtime info {"system_info": {"go": {"os":"linux","arch":"amd64","max_procs":2,"version":"go1.14.12"}}}

2022-06-16T03:49:23.357Z INFO [beat] instance/beat.go:997 Host info {"system_info": {"host": {"architecture":"x86_64","boot_time":"2022-06-09T18:03:48Z","containerized":false,"name":"metricbeat-metricbeat-metrics-6675dbc84d-ptfqf","ip":["127.0.0.1/8","10.0.1.91/32"],"kernel_version":"5.4.190-107.353.amzn2.x86_64","mac":["9e:35:b9:96:2d:63"],"os":{"family":"redhat","platform":"centos","name":"CentOS Linux","version":"7 (Core)","major":7,"minor":9,"patch":2009,"codename":"Core"},"timezone":"UTC","timezone_offset_sec":0,"id":"3d458769eadb4ca1ac689c5da40af2bf"}}}

2022-06-16T03:49:23.357Z INFO [beat] instance/beat.go:1026 Process info {"system_info": {"process": {"capabilities": {"inheritable":null,"permitted":["chown","dac_override","fowner","fsetid","kill","setgid","setuid","setpcap","net_bind_service","net_raw","sys_chroot","mknod","audit_write","setfcap"],"effective":["chown","dac_override","fowner","fsetid","kill","setgid","setuid","setpcap","net_bind_service","net_raw","sys_chroot","mknod","audit_write","setfcap"],"bounding":["chown","dac_override","fowner","fsetid","kill","setgid","setuid","setpcap","net_bind_service","net_raw","sys_chroot","mknod","audit_write","setfcap"],"ambient":null}, "cwd": "/usr/share/metricbeat", "exe": "/usr/share/metricbeat/metricbeat", "name": "metricbeat", "pid": 1, "ppid": 0, "seccomp": {"mode":"filter","no_new_privs":true}, "start_time": "2022-06-16T03:49:22.990Z"}}}

2022-06-16T03:49:23.357Z INFO instance/beat.go:299 Setup Beat: metricbeat; Version: 7.10.2

2022-06-16T03:49:23.357Z INFO eslegclient/connection.go:99 elasticsearch url: https://{es_url}:443

2022-06-16T03:49:23.358Z INFO [publisher] pipeline/module.go:113 Beat name: metricbeat-metricbeat-metrics-6675dbc84d-ptfqf

2022-06-16T03:49:23.361Z INFO [api] api/server.go:66 Stats endpoint (127.0.0.1:5066) finished: accept tcp 127.0.0.1:5066: use of closed network connection

2022-06-16T03:49:23.361Z INFO instance/beat.go:424 metricbeat stopped.

2022-06-16T03:49:23.361Z ERROR instance/beat.go:956 Exiting: all modules are disabled

Exiting: all modules are disabled

enabled: false

Remove that or set to

enabled: true

Oh I forgot @Charlie_Guo Welcome to the community!

@stephenb That's my bad, I missed the flag. Thank you for pointing out my mistake and I can feel your warm welcome. :grinning:

1 Like