I am using filebeat version 6.5.X. I have observed memory used by filebeat keep on increasing. memstat shows memory usage as below :
memory_alloc":6185623680 - ~ 6GB
while on task manager it shows memory usage 9196.8 MB (~ 9 GB ).
It is using 28% of RAM is too much.
- Can you please suggest some tweaked in filebeat config ?
- Filebeat is light wight component then why it is using this much memory. We have 2-3 incident when it almost reach to 50% of total memory.
Registery file size : 10515 KB (10 MB )
Total size of D:\data\logfiles : 58 GB
1 day log produce nearly 10 GB data.
Below is config file :
=========================== Filebeat inputs ============================= filebeat.inputs: # Each - is an input. Most options can be set at the input level, so # you can use different inputs for various configurations. # Below are the input specific configurations. - type: log # Change to true to enable this input configuration. enabled: true # Paths that should be crawled and fetched. Glob based paths. paths: #- /var/log/*.log - D:\DATA\LogFiles\** # Exclude lines. A list of regular expressions to match. It drops the lines that are # matching any regular expression from the list. #exclude_lines: ['^DBG'] # Include lines. A list of regular expressions to match. It exports the lines that are # matching any regular expression from the list. #include_lines: ['^ERR', '^WARN'] # Exclude files. A list of regular expressions to match. Filebeat drops the files that # are matching any regular expression from the list. By default, no files are dropped. exclude_files: ['^D:\\(?i)(data)\\LogFiles\\GHS\\','^D:\\(?i)(data)\\LogFiles\\(?i)(AutoSys)\\','\b(\w*HouseKeeping\w*)\b','\b(\w*gmt_c_\w*)\b','\b(\w*TPMonitor.Application_\w*)\b','\.zip$','\.cal$','\.gz$','\.xml$','\.txt.*\d*$','\.ser$','\.mdmp$','\.csv$','\.hprof$'] ignore_older: 72h close_removed: true clean_inactive: 72h5m