Filebeat process holding deleted files

Elastic Stack Beats
1

There was alert regarding disk space on our filebeat server

[root@filebeat3 ~]# df -h
Filesystem Size Used Avail Use% Mounted on
/dev/vda1 9.9G 3.5G 5.9G 38% /
tmpfs 12G 0 12G 0% /dev/shm
/dev/mapper/filebeat3 168G 137G 23G 87% /data

When I check /data it’s only using 45GB

-bash-4.1$ sudo du -sh
46G

So I checked if there are files still open that are deleted.

[root@filebeat3]# lsof +L1 /data
COMMAND PID USER FD TYPE DEVICE SIZE/OFF NLINK NODE NAME
filebeat 29838 root 12r REG 253,0 5000018459 0 1703949 /data/scribe/publishing-service/publishing-service_00003 (deleted)
filebeat 29838 root 13r REG 253,0 5000004481 0 9306120 /data/scribe/merge-service/merge-service_00000 (deleted)
filebeat 29838 root 14r REG 253,0 5000005120 0 1703951 /data/scribe/idml-service/idml-service_00020 (deleted)
filebeat 29838 root 15r REG 253,0 5000014528 0 1703952 /data/scribe/publishing-service/publishing-service_00004 (deleted)
filebeat 29838 root 16r REG 253,0 5000016288 0 9306123 /data/scribe/basevariant-service/basevariant-service_00001 (deleted)
... many more lines similar...

[root@filebeat3]# ls -la data/scribe/attribute-extraction-kafka-connect-service/attribute-extraction-kafka-connect-service_00000
ls: cannot access data/scribe/attribute-extraction-kafka-connect-service/attribute-extraction-kafka-connect-service_00000: No such file or directory

So filebeat is still holding the deleted files.. Restarted the daemon and the disk space are reclaimed. :wink:

[root@filebeat3]# /etc/init.d/filebeat stop
Stopping filebeat: [ OK ]

[root@filebeat3]# df -h
Filesystem Size Used Avail Use% Mounted on
/dev/vda1 9.9G 3.5G 5.9G 38% /
tmpfs 12G 0 12G 0% /dev/shm
/dev/mapper/filebeat3 168G 47G 113G 30% /data

Has anyone encountered this as well.. is there anyway to avoid having to restart filebeat process to clear the space?

Thanks,
Rick

2

Filebeat does keep files open until ignore_older is reached. In version 1.2 the close_older feature will be added: https://github.com/elastic/beats/pull/718

If you want to close files directly, use force_close_files but read the docs in detail to be aware of the potential side affects: https://www.elastic.co/guide/en/beats/filebeat/1.1/configuration-filebeat-options.html#_force_close_files

3

I am facing the similar issue, I get disk usage alert quite frequently, and when I check and restart the filebeat, I get around 10-20 GB free space. So, what does actually happen when I restart the filebeat, why filebeat uses this much space, it should supposedly send the logs to logstash and not keep it to itself? Since, I cannot add "sudo service filebeat restart" into my script as a potential solution for the disk usage, how can I fix this issue without restarting the filebeat?

4

@tanmayghosh2507 Filebeat keeps the file handler open for a certain time to not have to open the file all the time and directly read new lines. The file is closed based on close_older after a certain time. Which version of filebeat are you using? Could you share your config?