We use Filebeat (in Docker, version 7.8) to send logs to Logstash. Every night at 0:01 logrotate runs on the machine where Filebeat is running. It is rotating ~20-25 logfiles. Logrotate is configured with following options:
daily missingok rotate 7 compress copytruncate dateext dateformat .%Y-%m-%d-%s.rotlog
From the moment logrotate starts, Filebeat starts allocating a lot of memory.
Filebeat is reporting, that the file got truncated, sure, that is true.
We've already tried out:
- mem limit configured to container - this ends up in a restart loop of that container
close_inactiveto 30min (in the night logs are used less)
client_inactivity_timeoutin Logstash (in the night logs are used less)
- split the log rotation into time frames (cron is starting different jobs between 0:01 and 0:10)
All these attempts have not helped to throttle the memory consumption of Filebeat. After the first rotation, the consumption is very high.
So what's the problem here?