Hi Community,
I have deployed the Metricbeat (v7.10.2) helm chart on the kubernetes. The deamonset pods and metricbeat-kube-state-metrics pod are running fine, but the metricbeat-metricbeat-metrics pod is keep on crashing. I am not sure which configuration is wrong and from logs there is only one error message I could observe, "Exiting: all modules are disabled".
=============Configuration================
deployment:
enabled: true
metricbeatConfig:
metricbeat.yml: |
metricbeat.modules:
- module: kubernetes
enabled: false
metricsets:
- state_node
- state_deployment
- state_replicaset
- state_pod
- state_container
period: 5m
host: ${NODE_NAME}
hosts: ["kube-state-metrics:8080"]
output.elasticsearch:
hosts: ["https://{es_url}:443"]
username: {es_username}
password: {es_password}
===============Pod state==========================
metricbeat-kube-state-metrics-75c5fc65d9-cx2fs 1/1 Running 0 2d12h
metricbeat-metricbeat-48wsw 0/1 Running 0 8s
metricbeat-metricbeat-metrics-6675dbc84d-ptfqf 0/1 CrashLoopBackOff 1 12s
metricbeat-metricbeat-zqk4x 1/1 Running 0 11h
===============Logs from metricbeat-metricbeat-metrics==============
2022-06-16T03:49:23.353Z INFO instance/beat.go:645 Home path: [/usr/share/metricbeat] Config path: [/usr/share/metricbeat] Data path: [/usr/share/metricbeat/data] Logs path: [/usr/share/metricbeat/logs]
2022-06-16T03:49:23.355Z INFO instance/beat.go:653 Beat ID: 7a3f3048-6ca4-4655-8033-8cb7a77eb611
2022-06-16T03:49:23.356Z INFO [api] api/server.go:62 Starting stats endpoint
2022-06-16T03:49:23.356Z INFO [api] api/server.go:64 Metrics endpoint listening on: 127.0.0.1:5066 (configured: localhost)
2022-06-16T03:49:23.356Z INFO [seccomp] seccomp/seccomp.go:124 Syscall filter successfully installed
2022-06-16T03:49:23.356Z INFO [beat] instance/beat.go:981 Beat info {"system_info": {"beat": {"path": {"config": "/usr/share/metricbeat", "data": "/usr/share/metricbeat/data", "home": "/usr/share/metricbeat", "logs": "/usr/share/metricbeat/logs"}, "type": "metricbeat", "uuid": "7a3f3048-6ca4-4655-8033-8cb7a77eb611"}}}
2022-06-16T03:49:23.356Z INFO [beat] instance/beat.go:990 Build info {"system_info": {"build": {"commit": "aacf9ecd9c494aa0908f61fbca82c906b16562a8", "libbeat": "7.10.2", "time": "2021-01-12T22:24:31.000Z", "version": "7.10.2"}}}
2022-06-16T03:49:23.356Z INFO [beat] instance/beat.go:993 Go runtime info {"system_info": {"go": {"os":"linux","arch":"amd64","max_procs":2,"version":"go1.14.12"}}}
2022-06-16T03:49:23.357Z INFO [beat] instance/beat.go:997 Host info {"system_info": {"host": {"architecture":"x86_64","boot_time":"2022-06-09T18:03:48Z","containerized":false,"name":"metricbeat-metricbeat-metrics-6675dbc84d-ptfqf","ip":["127.0.0.1/8","10.0.1.91/32"],"kernel_version":"5.4.190-107.353.amzn2.x86_64","mac":["9e:35:b9:96:2d:63"],"os":{"family":"redhat","platform":"centos","name":"CentOS Linux","version":"7 (Core)","major":7,"minor":9,"patch":2009,"codename":"Core"},"timezone":"UTC","timezone_offset_sec":0,"id":"3d458769eadb4ca1ac689c5da40af2bf"}}}
2022-06-16T03:49:23.357Z INFO [beat] instance/beat.go:1026 Process info {"system_info": {"process": {"capabilities": {"inheritable":null,"permitted":["chown","dac_override","fowner","fsetid","kill","setgid","setuid","setpcap","net_bind_service","net_raw","sys_chroot","mknod","audit_write","setfcap"],"effective":["chown","dac_override","fowner","fsetid","kill","setgid","setuid","setpcap","net_bind_service","net_raw","sys_chroot","mknod","audit_write","setfcap"],"bounding":["chown","dac_override","fowner","fsetid","kill","setgid","setuid","setpcap","net_bind_service","net_raw","sys_chroot","mknod","audit_write","setfcap"],"ambient":null}, "cwd": "/usr/share/metricbeat", "exe": "/usr/share/metricbeat/metricbeat", "name": "metricbeat", "pid": 1, "ppid": 0, "seccomp": {"mode":"filter","no_new_privs":true}, "start_time": "2022-06-16T03:49:22.990Z"}}}
2022-06-16T03:49:23.357Z INFO instance/beat.go:299 Setup Beat: metricbeat; Version: 7.10.2
2022-06-16T03:49:23.357Z INFO eslegclient/connection.go:99 elasticsearch url: https://{es_url}:443
2022-06-16T03:49:23.358Z INFO [publisher] pipeline/module.go:113 Beat name: metricbeat-metricbeat-metrics-6675dbc84d-ptfqf
2022-06-16T03:49:23.361Z INFO [api] api/server.go:66 Stats endpoint (127.0.0.1:5066) finished: accept tcp 127.0.0.1:5066: use of closed network connection
2022-06-16T03:49:23.361Z INFO instance/beat.go:424 metricbeat stopped.
2022-06-16T03:49:23.361Z ERROR instance/beat.go:956 Exiting: all modules are disabled
Exiting: all modules are disabled
