Deleting log files after they have finished processing

Filebeat does not have the capabilities to handle deleting files from a host's filesystem after they have been processed. The best option is to use a cron job or scheduled task on your OS to delete them after a safe period of time.

You will want to consider configuring options like ignore_older and the various close options, to be explicit around when Filebeat will no longer read the files.

Logstash can use mode on the file input, to run a script or command to delete the files. An example of this would look like;

input {

 file {
    path => "/path/to/log/files/*.log"
    mode => "read"
    file_completed_action => "delete"
  }
}

It is important to note that using mode => "read" means that Logstash considers these files to be complete - ie they are not being written to - and so will delete these files as soon as it reaches the end of the file.

2 Likes