Hello World!
I'm running same setup: metricbeat.yml and modules.d/logstash-xpack.yml, yet one of the instance everything works fine, yet in another (almost identical) instance it is not working at all:
NOT properly working instance log:
# docker-compose logs --follow
metricbeat | 2023-01-09T19:27:27.960Z INFO instance/beat.go:697 Home path: [/usr/share/metricbeat] Config path: [/usr/share/metricbeat] Data path: [/usr/share/metricbeat/data] Logs path: [/usr/share/metricbeat/logs] Hostfs Path: [/hostfs]
metricbeat | 2023-01-09T19:27:27.961Z INFO instance/beat.go:705 Beat ID: 0bcf478d-1c42-46ff-a341-dd77d5c6e87c
metricbeat | 2023-01-09T19:27:27.963Z WARN [add_cloud_metadata] add_cloud_metadata/provider_aws_ec2.go:95 error when check request status for getting IMDSv2 token: http request status 405. No token in the metadata request will be used.
metricbeat | 2023-01-09T19:27:27.973Z INFO [add_cloud_metadata] add_cloud_metadata/add_cloud_metadata.go:105 add_cloud_metadata: hosting provider type detected as gcp, metadata={"cloud":{"account":{"id":"deep-timer-155418"},"availability_zone":"us-east1-d","instance":{"id":"4420301225678806969","name":"kibanaXX"},"machine":{"type":"n1-highmem-2"},"project":{"id":"deep-timer-155418"},"provider":"gcp","service":{"name":"GCE"}}}
metricbeat | 2023-01-09T19:27:27.989Z INFO [seccomp] seccomp/seccomp.go:124 Syscall filter successfully installed
metricbeat | 2023-01-09T19:27:27.989Z INFO [beat] instance/beat.go:1051 Beat info {"system_info": {"beat": {"path": {"config": "/usr/share/metricbeat", "data": "/usr/share/metricbeat/data", "home": "/usr/share/metricbeat", "logs": "/usr/share/metricbeat/logs"}, "type": "metricbeat", "uuid": "0bcf478d-1c42-46ff-a341-dd77d5c6e87c"}}}
metricbeat | 2023-01-09T19:27:27.989Z INFO [beat] instance/beat.go:1060 Build info {"system_info": {"build": {"commit": "692b4aac606e457bd2f5ef092d2d23c2fa950828", "libbeat": "7.17.8", "time": "2022-12-03T00:45:11.000Z", "version": "7.17.8"}}}
metricbeat | 2023-01-09T19:27:27.989Z INFO [beat] instance/beat.go:1063 Go runtime info {"system_info": {"go": {"os":"linux","arch":"amd64","max_procs":2,"version":"go1.18.5"}}}
metricbeat | 2023-01-09T19:27:27.989Z INFO [beat] instance/beat.go:1067 Host info {"system_info": {"host": {"architecture":"x86_64","boot_time":"2023-01-07T12:42:11Z","containerized":true,"name":"dffc61f6ceb6","ip":["127.0.0.1/8","172.22.0.4/16"],"kernel_version":"4.9.0-19-amd64","mac":["02:42:ac:16:00:04"],"os":{"type":"linux","family":"debian","platform":"ubuntu","name":"Ubuntu","version":"20.04.5 LTS (Focal Fossa)","major":20,"minor":4,"patch":5,"codename":"focal"},"timezone":"UTC","timezone_offset_sec":0}}}
metricbeat | 2023-01-09T19:27:27.990Z INFO [beat] instance/beat.go:1096 Process info {"system_info": {"process": {"capabilities": {"inheritable":["chown","dac_override","dac_read_search","fowner","fsetid","kill","setgid","setuid","setpcap","linux_immutable","net_bind_service","net_broadcast","net_admin","net_raw","ipc_lock","ipc_owner","sys_module","sys_rawio","sys_chroot","sys_ptrace","sys_pacct","sys_admin","sys_boot","sys_nice","sys_resource","sys_time","sys_tty_config","mknod","lease","audit_write","audit_control","setfcap","mac_override","mac_admin","syslog","wake_alarm","block_suspend","audit_read"],"permitted":["chown","dac_override","dac_read_search","fowner","fsetid","kill","setgid","setuid","setpcap","linux_immutable","net_bind_service","net_broadcast","net_admin","net_raw","ipc_lock","ipc_owner","sys_module","sys_rawio","sys_chroot","sys_ptrace","sys_pacct","sys_admin","sys_boot","sys_nice","sys_resource","sys_time","sys_tty_config","mknod","lease","audit_write","audit_control","setfcap","mac_override","mac_admin","syslog","wake_alarm","block_suspend","audit_read"],"effective":["chown","dac_override","dac_read_search","fowner","fsetid","kill","setgid","setuid","setpcap","linux_immutable","net_bind_service","net_broadcast","net_admin","net_raw","ipc_lock","ipc_owner","sys_module","sys_rawio","sys_chroot","sys_ptrace","sys_pacct","sys_admin","sys_boot","sys_nice","sys_resource","sys_time","sys_tty_config","mknod","lease","audit_write","audit_control","setfcap","mac_override","mac_admin","syslog","wake_alarm","block_suspend","audit_read"],"bounding":["chown","dac_override","dac_read_search","fowner","fsetid","kill","setgid","setuid","setpcap","linux_immutable","net_bind_service","net_broadcast","net_admin","net_raw","ipc_lock","ipc_owner","sys_module","sys_rawio","sys_chroot","sys_ptrace","sys_pacct","sys_admin","sys_boot","sys_nice","sys_resource","sys_time","sys_tty_config","mknod","lease","audit_write","audit_control","setfcap","mac_override","mac_admin","syslog","wake_alarm","block_suspend","audit_read"],"ambient":null}, "cwd": "/usr/share/metricbeat", "exe": "/usr/share/metricbeat/metricbeat", "name": "metricbeat", "pid": 7, "ppid": 1, "seccomp": {"mode":"filter"}, "start_time": "2023-01-09T19:27:27.790Z"}}}
metricbeat | 2023-01-09T19:27:27.990Z INFO instance/beat.go:291 Setup Beat: metricbeat; Version: 7.17.8
metricbeat | 2023-01-09T19:27:27.990Z INFO [index-management] idxmgmt/std.go:184 Set output.elasticsearch.index to 'metricbeat-7.17.8' as ILM is enabled.
metricbeat | 2023-01-09T19:27:27.991Z INFO [esclientleg] eslegclient/connection.go:105 elasticsearch url: http://FQDN:9200
metricbeat | 2023-01-09T19:27:27.991Z INFO [publisher] pipeline/module.go:113 Beat name: dffc61f6ceb6
metricbeat | 2023-01-09T19:27:28.017Z INFO [monitoring] log/log.go:142 Starting metrics logging every 30s
metricbeat | 2023-01-09T19:27:28.018Z INFO instance/beat.go:456 metricbeat start running.
metricbeat | 2023-01-09T19:27:28.018Z INFO cfgfile/reload.go:164 Config reloader started
metricbeat | 2023-01-09T19:27:28.019Z INFO cfgfile/reload.go:224 Loading of config files completed.
metricbeat | 2023-01-09T19:27:29.025Z INFO [publisher_pipeline_output] pipeline/output.go:143 Connecting to backoff(elasticsearch(http://FQDN:9200))
metricbeat | 2023-01-09T19:27:29.025Z INFO [publisher] pipeline/retry.go:219 retryer: send unwait signal to consumer
metricbeat | 2023-01-09T19:27:29.025Z INFO [publisher] pipeline/retry.go:223 done
metricbeat | 2023-01-09T19:27:29.043Z INFO [esclientleg] eslegclient/connection.go:285 Attempting to connect to Elasticsearch version 7.17.8
metricbeat | 2023-01-09T19:27:29.100Z INFO [esclientleg] eslegclient/connection.go:285 Attempting to connect to Elasticsearch version 7.17.8
metricbeat | 2023-01-09T19:27:29.118Z INFO [index-management] idxmgmt/std.go:261 Auto ILM enable success.
metricbeat | 2023-01-09T19:27:29.144Z INFO [index-management.ilm] ilm/std.go:170 ILM policy metricbeat exists already.
metricbeat | 2023-01-09T19:27:29.144Z INFO [index-management] idxmgmt/std.go:397 Set setup.template.name to '{metricbeat-7.17.8 {now/d}-000001}' as ILM is enabled.
metricbeat | 2023-01-09T19:27:29.144Z INFO [index-management] idxmgmt/std.go:402 Set setup.template.pattern to 'metricbeat-7.17.8-*' as ILM is enabled.
metricbeat | 2023-01-09T19:27:29.144Z INFO [index-management] idxmgmt/std.go:436 Set settings.index.lifecycle.rollover_alias in template to {metricbeat-7.17.8 {now/d}-000001} as ILM is enabled.
metricbeat | 2023-01-09T19:27:29.145Z INFO [index-management] idxmgmt/std.go:440 Set settings.index.lifecycle.name in template to {metricbeat {"policy":{"phases":{"hot":{"actions":{"rollover":{"max_age":"30d","max_size":"50gb"}}}}}}} as ILM is enabled.
metricbeat | 2023-01-09T19:27:29.185Z INFO template/load.go:110 Template "metricbeat-7.17.8" already exists and will not be overwritten.
metricbeat | 2023-01-09T19:27:29.185Z INFO [index-management] idxmgmt/std.go:297 Loaded index template.
metricbeat | 2023-01-09T19:27:29.244Z INFO [index-management.ilm] ilm/std.go:126 Index Alias metricbeat-7.17.8 exists already.
metricbeat | 2023-01-09T19:27:29.261Z INFO [publisher_pipeline_output] pipeline/output.go:151 Connection to backoff(elasticsearch(http://FQDN:9200)) established
metricbeat | 2023-01-09T19:27:58.028Z INFO [monitoring] log/log.go:184 Non-zero metrics in the last 30s {"monitoring": {"metrics": {"beat":{"cgroup":{"cpu":{"cfs":{"period":{"us":100000}},"id":"/"},"cpuacct":{"id":"/","total":{"ns":271974961}},"memory":{"id":"/","mem":{"limit":{"bytes":9223372036854771712},"usage":{"bytes":36872192}}}},"cpu":{"system":{"ticks":50,"time":{"ms":52}},"total":{"ticks":230,"time":{"ms":240},"value":0},"user":{"ticks":180,"time":{"ms":188}}},"handles":{"limit":{"hard":1048576,"soft":1048576},"open":12},"info":{"ephemeral_id":"44e26517-10a8-4e12-ae3e-8e52ec0d6738","uptime":{"ms":30152},"version":"7.17.8"},"memstats":{"gc_next":19406712,"memory_alloc":17462584,"memory_sys":28918792,"memory_total":35708096,"rss":124112896},"runtime":{"goroutines":36}},"libbeat":{"config":{"module":{"running":1,"starts":1},"reloads":1,"scans":1},"output":{"events":{"acked":6,"active":0,"batches":3,"duplicates":6,"total":12},"read":{"bytes":11523},"type":"elasticsearch","write":{"bytes":38844}},"pipeline":{"clients":2,"events":{"active":0,"published":12,"retry":4,"total":12},"queue":{"acked":12,"max_events":4096}}},"metricbeat":{"logstash":{"node":{"events":6,"success":6},"node_stats":{"events":6,"success":6}}},"system":{"cpu":{"cores":2},"load":{"1":0.11,"15":0.09,"5":0.11,"norm":{"1":0.055,"15":0.045,"5":0.055}}}}}}
I've tried following Debug | Metricbeat Reference [7.17] | Elastic to enable debugging, yet event that was logging appears to be OK, but again, same event isn't making into .monitoring-logstash-7-mb-YYYY.MM.DD.
Please advise)
Thank you in advance!