Filebeat 6.5.X - massive memory usage

[urvi (joshi)]

Hi team,

I am using filebeat version 6.5.X. I have observed memory used by filebeat keep on increasing. memstat shows memory usage as below :
"memstats":{"gc_next":11992835024,"memory_alloc":6185623680,"memory_total":4964881788840}}
memory_alloc":6185623680 - ~ 6GB

while on task manager it shows memory usage 9196.8 MB (~ 9 GB ).
It is using 28% of RAM is too much.

1. Can you please suggest some tweaked in filebeat config ?
2. Filebeat is light wight component then why it is using this much memory. We have 2-3 incident when it almost reach to 50% of total memory.

Details :
Registery file size : 10515 KB (10 MB )

Total size of D:\data\logfiles : 58 GB
1 day log produce nearly 10 GB data.

Below is config file :

=========================== Filebeat inputs =============================

filebeat.inputs:

# Each - is an input. Most options can be set at the input level, so
# you can use different inputs for various configurations.
# Below are the input specific configurations.
- type: log

  # Change to true to enable this input configuration.
  enabled: true

  # Paths that should be crawled and fetched. Glob based paths.
  paths:
    #- /var/log/*.log
    - D:\DATA\LogFiles\**

  # Exclude lines. A list of regular expressions to match. It drops the lines that are
  # matching any regular expression from the list.
  #exclude_lines: ['^DBG']

  # Include lines. A list of regular expressions to match. It exports the lines that are
  # matching any regular expression from the list.
  #include_lines: ['^ERR', '^WARN']

  # Exclude files. A list of regular expressions to match. Filebeat drops the files that
  # are matching any regular expression from the list. By default, no files are dropped.
  exclude_files: ['^D:\\(?i)(data)\\LogFiles\\GHS\\','^D:\\(?i)(data)\\LogFiles\\(?i)(AutoSys)\\','\b(\w*HouseKeeping\w*)\b','\b(\w*gmt_c_\w*)\b','\b(\w*TPMonitor.Application_\w*)\b','\.zip$','\.cal$','\.gz$','\.xml$','\.txt.*\d*$','\.ser$','\.mdmp$','\.csv$','\.hprof$']
  ignore_older: 72h
  close_removed: true
  clean_inactive: 72h5m

[jsoriano (Jaime Soriano)]

Hi @urvi,

There were some memory leaks fixed in 7.x, most of them were backported to 6.8. Most of these issues were related to dynamic configuration and configuration reload, that it seems you are not using, but it could worth to try in your case to see if you had any improvement. Could you try a more recent version of filebeat?

Your configuration looks fine, except for the path, the backslashes should be escaped, or the path quoted:

paths:
    - 'D:\DATA\LogFiles\**' 

Were you having other issues apart of the big memory usage? Like log files not being collected, or logs being sent multiple times?

For further investigation we would need to know more about your use case.

• Do logs have something special (for example binary content or very long lines)?
• How many files there can be in this D:\DATA\LogFiles\ directory?
• How is the evolution of memory usage? (does it grow fast on startup or it increases slowly but continuously...)

Having heap and goroutine profiles can be also helpful.