Filebeat does not have the capabilities to handle deleting files from a host's filesystem after they have been processed. The best option is to use a cron job or scheduled task on your OS to delete them after a safe period of time.
You will want to consider configuring options like ignore_older
and the various close
options, to be explicit around when Filebeat will no longer read the files.
Logstash can use mode
on the file
input, to run a script or command to delete the files. An example of this would look like;
input {
file {
path => "/path/to/log/files/*.log"
mode => "read"
file_completed_action => "delete"
}
}
It is important to note that using mode => "read"
means that Logstash considers these files to be complete - ie they are not being written to - and so will delete these files as soon as it reaches the end of the file.