Filebeat is taking too much of memory on ubuntu 16

[Narayanan_Sukumar]

Hello,

I am using filebeat version 5.4.2 with ubuntu 16 and it is taking more memory resulting OOM.

==================
root@xxxxxx:/usr/share/filebeat/bin# ./filebeat -configtest -c /etc/filebeat/filebeat.yml -e 
2017/08/07 07:46:29.806878 metrics.go:23: INFO Metrics logging every 30s
2017/08/07 07:46:29.806651 beat.go:285: INFO Home path: [/usr/share/filebeat/bin] Config path: [/usr/share/filebeat/bin] Data path: [/usr/share/filebeat/bin/data] Logs path: [/usr/share/filebeat/bin/logs]
2017/08/07 07:46:29.809448 beat.go:186: INFO Setup Beat: filebeat; Version: 5.4.2
2017/08/07 07:46:29.809640 outputs.go:108: INFO Activated kafka as output plugin.
2017/08/07 07:46:29.810638 publish.go:295: INFO Publisher name: xxxxxxxxxxxx
2017/08/07 07:46:29.810755 async.go:63: INFO Flush Interval set to: 1s
2017/08/07 07:46:29.810770 async.go:64: INFO Max Bulk Size set to: 2048
2017/08/07 07:46:29.810874 modules.go:93: ERR Not loading modules. Module directory not found: /usr/share/filebeat/bin/module
Config OK
==================

filebeat.prospectors:
- input_type: log
  paths:
    - /srv/xxxx/src/logs/*.log 
    - /home/xxxx/.pm2/logs/*.log
    - /var/log/xxxx/*.log
  tags: ["xxxxxxxx"]
logging.files:
  path: /var/log/mybeat
  name: mybeat.log
output.kafka:
  enabled: true
  hosts: ["xxxxxxx"]
  topic: amadeus
  version: 0.9.0.0
  worker: 1
  compression: gzip
  logging.to_files: true
  logging.files: /tmp/filebeat_kafka.log
=====================

I have also tried to limit the memory usage in systemd config file at /lib/systemd/system/filebeat.service with 1GB of memory.

Please let me know if we need any other optimization needed.

[steffens (Steffen Siering)]

Filebeat is trying to process all logs concurrently. How many harvesters have you running? You can try the harvester limit setting, to reduce the number of concurrent files being processed. Plus, how big are these events? filebeat holds up to 2048 events in it's spooler + the harvesters can hold on additional events to be published. If lines become very long and you have a many harvester, this might affect memory usage.
In addition to the spooler holding 2k events, another 2k events are concurrently processed by the outputs + the kafka output keeps a copy of the json encoding.

You can enable profiling via -httpprof localhost:6060 in filebeat. Using curl you can get a memory profile via curl http://localhost:6060/debug/pprof/heap. Having a look at a few heap profiles can help identifying which component is using the most memory.