Filebeat is leaking memory

Elastic Stack Beats
1

There are about 100,000 log files on the VM, the characteristics of the VM OS Windows Server 2019, CPU 4, RAM 4GB. To collect logs and then send them to logstash, filebeat 7.16.0 is used. After starting filebeat, it begins to consume RAM memory to the maximum, as a result, the filebeat service stops, collecting logs ends with an error:

ERROR	[logstash]	logstash/async.go:280	Failed to publish events caused by: write tcp 10.0.10.5:53536->10.0.10.6:5044: wsasend: An existing connection was forcibly closed by the remote host.
panic: reflect.Select: too many cases (max 65536)

My filebeat config:

filebeat.inputs:
- type: filestream
  enabled: true
  paths:
    - C:\logs\GT\xml\*\*.log
  fields_under_root: true
  fields:
    system: xml
    subsystem: GT  
  parsers:
    - multiline:
        type: pattern
        pattern: '^\[[0-9]{4}-[0-9]{2}-[0-9]{2}'
        negate: true
        match: after

- type: filestream
  paths:
    - C:\logs\ERRORS\xml\*\*.log
  fields_under_root: true
  fields:
    system: xml
    subsystem: errors
  parsers:
    - multiline:
        type: pattern
        pattern: '^\[[0-9]{4}-[0-9]{2}-[0-9]{2}'
        negate: true
        match: after

filebeat.config.modules:
  path: ${path.config}/modules.d/*.yml
  reload.enabled: false

setup.template.settings:
  index.number_of_shards: 1
  
tags: [xmlgate]

output.logstash:
  hosts: ["10.0.10.6:5044"]
  template.name: "xmlgate"
  template.path: "filebeat.template.json"
  template.overwrite: false

logging.metrics.enabled: false

I tried using the parameters queue.mem: events: 2048 - 32; flush.min_events: 2048 - 16; flush.timeout: 10-2s - no result.
What parameters can help reduce filebeat's RAM memory consumption when processing such a number of already existing log files on disk (~100,000) + new log files (~20 per minute)?

1 Like
2

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.