Filebeat beta1: Insane memory footprint

[shog]

Hi,

Version: Filebeat beta1
System: CentOS6

I observe an insane memory footprint of filebeat beta1.

  PID USER          PR  NI  VIRT  RES  SHR S %CPU %MEM    TIME+  COMMAND            
23818 filebeat      15  -5 30.5g  27g 1556 S 13.9 57.6 593:35.22 filebeat           

This is only after ~20hrs of process uptime. On this host I have four prospectors defined and they are scanning around 2000 files, which are in total worth less than 13GB.

Settings that might potentially be interesting:

      harvester_buffer_size: 131072
  publish_async: true

I see similar memory pattern across entire beta1 setup. Please let me know what info would be valuable for you to track issue. I did not observe this with alpha5.

[warkolm (Mark Walkom)]

What does your config look like?

[ruflin]

Can you please share the full config files and the log files if possible? Which other beats are you running? What is your setup? beats -> LS -> ES? Is the memory constantly increasing or do you see it going up and down?

[steffens (Steffen Siering)]

please share full output section in filebeat.yml and other settings, but prospectors. The publish_async option looks telling. But as publish_async changes interaction with output I need to know about outputs in order to figure there might be some issue. You using multiline? Have you checked filebeat logs for output errors (e.g. EOF or connections being dropped)?

In prospectors have you a common strategy using any of the close_... settings?

Have you tried to disable publish_async?

Filebeat supports generation of a memory profile using -memprofile <out file name>. Can you run filebeat with this setting and share the generated profile with us?