Addressing Filebeat's memory leak and performance issues with high log volume

I changed filebeat.input to filestream id and enabled both modules. Still the same behavior.

Every 2.0s: systemctl status filebeat                                                                                                                                                                     HOSTls: Thu Mar  7 15:35:10 2024

● filebeat.service - Filebeat sends log files to Logstash or directly to Elasticsearch.
   Loaded: loaded (/usr/lib/systemd/system/filebeat.service; enabled; vendor preset: disabled)
   Active: active (running) since Thu 2024-03-07 15:27:31 CET; 7min ago
     Docs: https://www.elastic.co/beats/filebeat
 Main PID: 322416 (filebeat)
    Tasks: 8 (limit: 74760)
   Memory: 45.9M (max: 6.0G)
   CGroup: /system.slice/filebeat.service
           └─322416 /usr/share/filebeat/bin/filebeat --environment systemd -c /etc/filebeat/filebeat.yml --path.home /usr/share/filebeat --path.config /etc/filebeat --path.data /var/lib/filebeat --path.logs /var/log/filebeat

Mar 07 15:27:31 HOSTols systemd[1]: Started Filebeat sends log files to Logstash or directly to Elasticsearch..



─────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────
  512.08kB  2.89% 88.42%   512.08kB  2.89%  regexp.compile
(pprof) top
Showing nodes accounting for 15643.65kB, 88.42% of 17691.72kB total
Showing top 10 nodes out of 60
      flat  flat%   sum%        cum   cum%
 5488.12kB 31.02% 31.02%  5488.12kB 31.02%  github.com/goccy/go-json/internal/decoder.init.0
 3786.34kB 21.40% 52.42%  3786.34kB 21.40%  github.com/elastic/beats/v7/libbeat/asset.GetFields
 2756.97kB 15.58% 68.01%  2756.97kB 15.58%  github.com/goccy/go-json/internal/encoder.init.0
  522.06kB  2.95% 70.96%   522.06kB  2.95%  cloud.google.com/go/pubsub/apiv1/pubsubpb.init
  522.06kB  2.95% 73.91%   522.06kB  2.95%  github.com/googleapis/gnostic/openapiv2.init
  517.33kB  2.92% 76.83%   517.33kB  2.92%  regexp/syntax.(*compiler).inst
  513.31kB  2.90% 79.73%   513.31kB  2.90%  google.golang.org/protobuf/internal/filedesc.(*File).initDecls
  512.88kB  2.90% 82.63%   512.88kB  2.90%  google.golang.org/protobuf/internal/strs.(*Builder).grow
  512.50kB  2.90% 85.53%   512.50kB  2.90%  runtime.allocm
  512.08kB  2.89% 88.42%   512.08kB  2.89%  regexp.compile
(pprof)
─────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────
Entering interactive mode (type "help" for commands, "o" for options)
(pprof) top
Showing nodes accounting for 10ms, 100% of 10ms total
Showing top 10 nodes out of 22
      flat  flat%   sum%        cum   cum%
      10ms   100%   100%       10ms   100%  runtime/internal/syscall.Syscall6
         0     0%   100%       10ms   100%  github.com/elastic/beats/v7/libbeat/api.makeAPIHandler.func1
         0     0%   100%       10ms   100%  github.com/elastic/elastic-agent-libs/monitoring.(*Func).Visit
         0     0%   100%       10ms   100%  github.com/elastic/elastic-agent-libs/monitoring.(*Registry).Visit (inline)
         0     0%   100%       10ms   100%  github.com/elastic/elastic-agent-libs/monitoring.(*Registry).doVisit
         0     0%   100%       10ms   100%  github.com/elastic/elastic-agent-libs/monitoring.CollectStructSnapshot
         0     0%   100%       10ms   100%  github.com/elastic/elastic-agent-system-metrics/metric/system/host.ReportInfo.func1
         0     0%   100%       10ms   100%  github.com/elastic/go-sysinfo.Host
         0     0%   100%       10ms   100%  github.com/elastic/go-sysinfo/providers/linux.(*reader).network
         0     0%   100%       10ms   100%  github.com/elastic/go-sysinfo/providers/linux.linuxSystem.Host

HEAP a second before MEM reaching 6 GB CAP and reseting

very 2.0s: systemctl status filebeat                                                                                                                                                                     HOSTNAME: Thu Mar  7 15:55:55 2024

● filebeat.service - Filebeat sends log files to Logstash or directly to Elasticsearch.
   Loaded: loaded (/usr/lib/systemd/system/filebeat.service; enabled; vendor preset: disabled)
   Active: active (running) since Thu 2024-03-07 15:55:54 CET; 1s ago
     Docs: https://www.elastic.co/beats/filebeat
 Main PID: 331174 (filebeat)
    Tasks: 8 (limit: 74760)
   Memory: 121.4M (max: 6.0G)
   CGroup: /system.slice/filebeat.service
           └─331174 /usr/share/filebeat/bin/filebeat --environment systemd -c /etc/filebeat/filebeat.yml --path.home /usr/share/filebeat --path.config /etc/filebeat --path.data /var/lib/filebeat --path.logs /var/log/filebeat

Mar 07 15:55:54 HOSTNAME systemd[1]: filebeat.service: Service RestartSec=100ms expired, scheduling restart.
Mar 07 15:55:54 HOSTNAME systemd[1]: filebeat.service: Scheduled restart job, restart counter is at 7.
Mar 07 15:55:54 HOSTNAME systemd[1]: Stopped Filebeat sends log files to Logstash or directly to Elasticsearch..
Mar 07 15:55:54 HOSTNAME systemd[1]: Started Filebeat sends log files to Logstash or directly to Elasticsearch..
─────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────
.bash_history                        .cache/                              elks-kibana-hearbteat.pub            pprof/                               .ssh/
.bash_logout                         .config/                             .local/                              .python_history                      teraz-do-pobrania-heartbeat/
.bash_profile                        eclipse-jee-neon-3-win32-x86_64.zip  logstash/                            scripts/
[HOSTNAME]~$ watch sudo tail /var/log/filebeat/filebeat-20240307-236.ndjson
alerts/                              .bashrc                              elks-kibana-hearbteat                metricbeat-8.10.2-x86_64.rpm         sftp/
.bash_history                        .cache/                              elks-kibana-hearbteat.pub            pprof/                               .ssh/
.bash_logout                         .config/                             .local/                              .python_history                      teraz-do-pobrania-heartbeat/
.bash_profile                        eclipse-jee-neon-3-win32-x86_64.zip  logstash/                            scripts/
[HOSTNAME]~$ watch sudo tail /var/log/filebeat/filebeat-20240307-2
filebeat-20240307-234.ndjson  filebeat-20240307-235.ndjson  filebeat-20240307-236.ndjson  filebeat-20240307-237.ndjson  filebeat-20240307-238.ndjson  filebeat-20240307-239.ndjson  filebeat-20240307-240.ndjson  filebeat-20240307-241.ndjson
[HOSTNAME]~$ watch sudo tail /var/log/filebeat/filebeat-20240307-2
filebeat-20240307-234.ndjson  filebeat-20240307-235.ndjson  filebeat-20240307-236.ndjson  filebeat-20240307-237.ndjson  filebeat-20240307-238.ndjson  filebeat-20240307-239.ndjson  filebeat-20240307-240.ndjson  filebeat-20240307-241.ndjson
[HOSTNAME]~$ watch sudo tail /var/log/filebeat/filebeat-20240307-241.ndjson
[HOSTNAME]~$ sudo vi /etc/filebeat/filebeat.yml
[HOSTNAME]~$ sudo systemctl restart filebeat
[HOSTNAME]~$
─────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────
Type: inuse_space
Time: Mar 7, 2024 at 3:55pm (CET)
Entering interactive mode (type "help" for commands, "o" for options)
(pprof) top
Showing nodes accounting for 3.66GB, 97.97% of 3.73GB total
Dropped 323 nodes (cum <= 0.02GB)
      flat  flat%   sum%        cum   cum%
    3.66GB 97.97% 97.97%     3.66GB 97.97%  github.com/elastic/beats/v7/filebeat/inputsource/common/dgram.DatagramReaderFactory.func1.1
         0     0% 97.97%     0.03GB  0.76%  github.com/elastic/beats/v7/filebeat/beater.(*countingClient).Publish
         0     0% 97.97%     3.66GB 97.97%  github.com/elastic/beats/v7/filebeat/inputsource/common/dgram.(*Listener).Start.func1
         0     0% 97.97%     3.66GB 97.97%  github.com/elastic/beats/v7/filebeat/inputsource/common/dgram.(*Listener).connectAndRun
         0     0% 97.97%     0.03GB  0.76%  github.com/elastic/beats/v7/libbeat/publisher/pipeline.(*client).Publish
         0     0% 97.97%     0.03GB  0.76%  github.com/elastic/beats/v7/libbeat/publisher/pipeline.(*client).publish
         0     0% 97.97%     0.03GB  0.76%  github.com/elastic/beats/v7/libbeat/publisher/processing.(*group).Run
         0     0% 97.97%     3.66GB 97.99%  github.com/elastic/go-concert/unison.(*TaskGroup).Go.func1