Metricbeat' logstash module is NOT working properly (no errors)

Hello World!

I'm running same setup: metricbeat.yml and modules.d/logstash-xpack.yml, yet one of the instance everything works fine, yet in another (almost identical) instance it is not working at all:

NOT properly working instance log:

# docker-compose logs --follow
metricbeat  | 2023-01-09T19:27:27.960Z	INFO	instance/beat.go:697	Home path: [/usr/share/metricbeat] Config path: [/usr/share/metricbeat] Data path: [/usr/share/metricbeat/data] Logs path: [/usr/share/metricbeat/logs] Hostfs Path: [/hostfs]
metricbeat  | 2023-01-09T19:27:27.961Z	INFO	instance/beat.go:705	Beat ID: 0bcf478d-1c42-46ff-a341-dd77d5c6e87c
metricbeat  | 2023-01-09T19:27:27.963Z	WARN	[add_cloud_metadata]	add_cloud_metadata/provider_aws_ec2.go:95	error when check request status for getting IMDSv2 token: http request status 405. No token in the metadata request will be used.
metricbeat  | 2023-01-09T19:27:27.973Z	INFO	[add_cloud_metadata]	add_cloud_metadata/add_cloud_metadata.go:105	add_cloud_metadata: hosting provider type detected as gcp, metadata={"cloud":{"account":{"id":"deep-timer-155418"},"availability_zone":"us-east1-d","instance":{"id":"4420301225678806969","name":"kibanaXX"},"machine":{"type":"n1-highmem-2"},"project":{"id":"deep-timer-155418"},"provider":"gcp","service":{"name":"GCE"}}}
metricbeat  | 2023-01-09T19:27:27.989Z	INFO	[seccomp]	seccomp/seccomp.go:124	Syscall filter successfully installed
metricbeat  | 2023-01-09T19:27:27.989Z	INFO	[beat]	instance/beat.go:1051	Beat info	{"system_info": {"beat": {"path": {"config": "/usr/share/metricbeat", "data": "/usr/share/metricbeat/data", "home": "/usr/share/metricbeat", "logs": "/usr/share/metricbeat/logs"}, "type": "metricbeat", "uuid": "0bcf478d-1c42-46ff-a341-dd77d5c6e87c"}}}
metricbeat  | 2023-01-09T19:27:27.989Z	INFO	[beat]	instance/beat.go:1060	Build info	{"system_info": {"build": {"commit": "692b4aac606e457bd2f5ef092d2d23c2fa950828", "libbeat": "7.17.8", "time": "2022-12-03T00:45:11.000Z", "version": "7.17.8"}}}
metricbeat  | 2023-01-09T19:27:27.989Z	INFO	[beat]	instance/beat.go:1063	Go runtime info	{"system_info": {"go": {"os":"linux","arch":"amd64","max_procs":2,"version":"go1.18.5"}}}
metricbeat  | 2023-01-09T19:27:27.989Z	INFO	[beat]	instance/beat.go:1067	Host info	{"system_info": {"host": {"architecture":"x86_64","boot_time":"2023-01-07T12:42:11Z","containerized":true,"name":"dffc61f6ceb6","ip":["127.0.0.1/8","172.22.0.4/16"],"kernel_version":"4.9.0-19-amd64","mac":["02:42:ac:16:00:04"],"os":{"type":"linux","family":"debian","platform":"ubuntu","name":"Ubuntu","version":"20.04.5 LTS (Focal Fossa)","major":20,"minor":4,"patch":5,"codename":"focal"},"timezone":"UTC","timezone_offset_sec":0}}}
metricbeat  | 2023-01-09T19:27:27.990Z	INFO	[beat]	instance/beat.go:1096	Process info	{"system_info": {"process": {"capabilities": {"inheritable":["chown","dac_override","dac_read_search","fowner","fsetid","kill","setgid","setuid","setpcap","linux_immutable","net_bind_service","net_broadcast","net_admin","net_raw","ipc_lock","ipc_owner","sys_module","sys_rawio","sys_chroot","sys_ptrace","sys_pacct","sys_admin","sys_boot","sys_nice","sys_resource","sys_time","sys_tty_config","mknod","lease","audit_write","audit_control","setfcap","mac_override","mac_admin","syslog","wake_alarm","block_suspend","audit_read"],"permitted":["chown","dac_override","dac_read_search","fowner","fsetid","kill","setgid","setuid","setpcap","linux_immutable","net_bind_service","net_broadcast","net_admin","net_raw","ipc_lock","ipc_owner","sys_module","sys_rawio","sys_chroot","sys_ptrace","sys_pacct","sys_admin","sys_boot","sys_nice","sys_resource","sys_time","sys_tty_config","mknod","lease","audit_write","audit_control","setfcap","mac_override","mac_admin","syslog","wake_alarm","block_suspend","audit_read"],"effective":["chown","dac_override","dac_read_search","fowner","fsetid","kill","setgid","setuid","setpcap","linux_immutable","net_bind_service","net_broadcast","net_admin","net_raw","ipc_lock","ipc_owner","sys_module","sys_rawio","sys_chroot","sys_ptrace","sys_pacct","sys_admin","sys_boot","sys_nice","sys_resource","sys_time","sys_tty_config","mknod","lease","audit_write","audit_control","setfcap","mac_override","mac_admin","syslog","wake_alarm","block_suspend","audit_read"],"bounding":["chown","dac_override","dac_read_search","fowner","fsetid","kill","setgid","setuid","setpcap","linux_immutable","net_bind_service","net_broadcast","net_admin","net_raw","ipc_lock","ipc_owner","sys_module","sys_rawio","sys_chroot","sys_ptrace","sys_pacct","sys_admin","sys_boot","sys_nice","sys_resource","sys_time","sys_tty_config","mknod","lease","audit_write","audit_control","setfcap","mac_override","mac_admin","syslog","wake_alarm","block_suspend","audit_read"],"ambient":null}, "cwd": "/usr/share/metricbeat", "exe": "/usr/share/metricbeat/metricbeat", "name": "metricbeat", "pid": 7, "ppid": 1, "seccomp": {"mode":"filter"}, "start_time": "2023-01-09T19:27:27.790Z"}}}
metricbeat  | 2023-01-09T19:27:27.990Z	INFO	instance/beat.go:291	Setup Beat: metricbeat; Version: 7.17.8
metricbeat  | 2023-01-09T19:27:27.990Z	INFO	[index-management]	idxmgmt/std.go:184	Set output.elasticsearch.index to 'metricbeat-7.17.8' as ILM is enabled.
metricbeat  | 2023-01-09T19:27:27.991Z	INFO	[esclientleg]	eslegclient/connection.go:105	elasticsearch url: http://FQDN:9200
metricbeat  | 2023-01-09T19:27:27.991Z	INFO	[publisher]	pipeline/module.go:113	Beat name: dffc61f6ceb6
metricbeat  | 2023-01-09T19:27:28.017Z	INFO	[monitoring]	log/log.go:142	Starting metrics logging every 30s
metricbeat  | 2023-01-09T19:27:28.018Z	INFO	instance/beat.go:456	metricbeat start running.
metricbeat  | 2023-01-09T19:27:28.018Z	INFO	cfgfile/reload.go:164	Config reloader started
metricbeat  | 2023-01-09T19:27:28.019Z	INFO	cfgfile/reload.go:224	Loading of config files completed.
metricbeat  | 2023-01-09T19:27:29.025Z	INFO	[publisher_pipeline_output]	pipeline/output.go:143	Connecting to backoff(elasticsearch(http://FQDN:9200))
metricbeat  | 2023-01-09T19:27:29.025Z	INFO	[publisher]	pipeline/retry.go:219	retryer: send unwait signal to consumer
metricbeat  | 2023-01-09T19:27:29.025Z	INFO	[publisher]	pipeline/retry.go:223	  done
metricbeat  | 2023-01-09T19:27:29.043Z	INFO	[esclientleg]	eslegclient/connection.go:285	Attempting to connect to Elasticsearch version 7.17.8
metricbeat  | 2023-01-09T19:27:29.100Z	INFO	[esclientleg]	eslegclient/connection.go:285	Attempting to connect to Elasticsearch version 7.17.8
metricbeat  | 2023-01-09T19:27:29.118Z	INFO	[index-management]	idxmgmt/std.go:261	Auto ILM enable success.
metricbeat  | 2023-01-09T19:27:29.144Z	INFO	[index-management.ilm]	ilm/std.go:170	ILM policy metricbeat exists already.
metricbeat  | 2023-01-09T19:27:29.144Z	INFO	[index-management]	idxmgmt/std.go:397	Set setup.template.name to '{metricbeat-7.17.8 {now/d}-000001}' as ILM is enabled.
metricbeat  | 2023-01-09T19:27:29.144Z	INFO	[index-management]	idxmgmt/std.go:402	Set setup.template.pattern to 'metricbeat-7.17.8-*' as ILM is enabled.
metricbeat  | 2023-01-09T19:27:29.144Z	INFO	[index-management]	idxmgmt/std.go:436	Set settings.index.lifecycle.rollover_alias in template to {metricbeat-7.17.8 {now/d}-000001} as ILM is enabled.
metricbeat  | 2023-01-09T19:27:29.145Z	INFO	[index-management]	idxmgmt/std.go:440	Set settings.index.lifecycle.name in template to {metricbeat {"policy":{"phases":{"hot":{"actions":{"rollover":{"max_age":"30d","max_size":"50gb"}}}}}}} as ILM is enabled.
metricbeat  | 2023-01-09T19:27:29.185Z	INFO	template/load.go:110	Template "metricbeat-7.17.8" already exists and will not be overwritten.
metricbeat  | 2023-01-09T19:27:29.185Z	INFO	[index-management]	idxmgmt/std.go:297	Loaded index template.
metricbeat  | 2023-01-09T19:27:29.244Z	INFO	[index-management.ilm]	ilm/std.go:126	Index Alias metricbeat-7.17.8 exists already.
metricbeat  | 2023-01-09T19:27:29.261Z	INFO	[publisher_pipeline_output]	pipeline/output.go:151	Connection to backoff(elasticsearch(http://FQDN:9200)) established

metricbeat  | 2023-01-09T19:27:58.028Z	INFO	[monitoring]	log/log.go:184	Non-zero metrics in the last 30s	{"monitoring": {"metrics": {"beat":{"cgroup":{"cpu":{"cfs":{"period":{"us":100000}},"id":"/"},"cpuacct":{"id":"/","total":{"ns":271974961}},"memory":{"id":"/","mem":{"limit":{"bytes":9223372036854771712},"usage":{"bytes":36872192}}}},"cpu":{"system":{"ticks":50,"time":{"ms":52}},"total":{"ticks":230,"time":{"ms":240},"value":0},"user":{"ticks":180,"time":{"ms":188}}},"handles":{"limit":{"hard":1048576,"soft":1048576},"open":12},"info":{"ephemeral_id":"44e26517-10a8-4e12-ae3e-8e52ec0d6738","uptime":{"ms":30152},"version":"7.17.8"},"memstats":{"gc_next":19406712,"memory_alloc":17462584,"memory_sys":28918792,"memory_total":35708096,"rss":124112896},"runtime":{"goroutines":36}},"libbeat":{"config":{"module":{"running":1,"starts":1},"reloads":1,"scans":1},"output":{"events":{"acked":6,"active":0,"batches":3,"duplicates":6,"total":12},"read":{"bytes":11523},"type":"elasticsearch","write":{"bytes":38844}},"pipeline":{"clients":2,"events":{"active":0,"published":12,"retry":4,"total":12},"queue":{"acked":12,"max_events":4096}}},"metricbeat":{"logstash":{"node":{"events":6,"success":6},"node_stats":{"events":6,"success":6}}},"system":{"cpu":{"cores":2},"load":{"1":0.11,"15":0.09,"5":0.11,"norm":{"1":0.055,"15":0.045,"5":0.055}}}}}}

I've tried following Debug | Metricbeat Reference [7.17] | Elastic to enable debugging, yet event that was logging appears to be OK, but again, same event isn't making into .monitoring-logstash-7-mb-YYYY.MM.DD.

Please advise)

Thank you in advance!

What is the difference between the instances?

Also, share your metricbeat.yml.

there are few minor difference in the .env file (they should not affect anything), and i use the default metricbeat.yml (see bellow)

# cat .env | cut -d\= -f1
CONTAINER_NAME
ELASTICSEARCH_HOSTS
ELASTICSEARCH_HOSTS_LOGSTASH
ELASTICSEARCH_PASSWORD
ELASTICSEARCH_USERNAME
VERSION
# cat metricbeat.yml
metricbeat.config.modules:
  path: ${path.config}/modules.d/*.yml
  reload.enabled: false

processors:
  - add_cloud_metadata: ~
  - add_docker_metadata: ~

output.elasticsearch:
  hosts: '${ELASTICSEARCH_HOSTS:elasticsearch:9200}'
  username: '${ELASTICSEARCH_USERNAME:elastic}'
  password: '${ELASTICSEARCH_PASSWORD:}'
#