Remove File After Read

[OpSecMonkey]

Hey I have a quick question. I was wondering if there is anyway to remove a file after it has been read by logstash. I have a feeling mine keeps hanging on the same file over and over but when I check the query there is only 1 entry so thats a plus. But I dont want it to keep reading the same file over and over and over.

I have roughly 400 .csv files that range from 400 megs to as small as 100 megs.

I have provided a copy of my config file I am using.

input {
    file {
    path => "/home/dtengine/Documents/conversion/*.*"
    start_position => "beginning"
    sincedb_path => "dev/null"
    }
}
    filter {
    csv {
    separator => ","
    }
}
output {
    elasticsearch {    host => ["http://localhost:9200"] }
    index => "some_name"
}
stdout {codec => rubydebug}
}

[yaauie (Ry Biesemeyer)]

@OpSecMonkey

sincedb_path => "dev/null"

The path to your system's null device is /dev/null (that leading slash is important). My guess is that with your current configuration the File Input Plugin is creating a sincedb and placing it at the path dev/null relative to your config directory; if this is the case, it could explain why the input is "remembering" where it left off and refusing to reprocess files after a restart.

---

If you do need to delete files after they have been processed, the File Input Plugin's file_completed_action directive may be what you are looking for.

[OpSecMonkey]

Got it. I will change it and add the / thanks for noticing that. I will go ahead and stop my process and fix it. Its been on one CSV thats about 40 gigs that I had added for about 12 hrs.

[OpSecMonkey]

How would I add file_completed_action & 'file_sort_direction'

Would it be like

input {
    file {
    path => "/home/dtengine/Documents/conversion/*.*"
    start_position => "beginning"
    file_sort_direction => "asc"
    file_completed_action => "delete"
    sincedb_path => "/dev/null"