Rereading a refreshed file

[lovecraft (Sean)]

Hi All,

I am new to Logstash and trying to work out a config.

My scenario is that I have a utility which runs periodically and outputs a report as a multiline JSON object in a file. The filename remains the same across runs but the content is replaced with every run.

I want to take the content of the file and pass it into Elastic search as a single document. So there is a one document per run of the utility.

I have a config which does this BUT it only does it once at the startup of Logstash. When the content of the file is updated a reread of the file is not triggered.

I suspect the file is being 'unwatched' after the first run. Can someone suggest a better config please?

Thanks++

Here is my config:

input { 
    file {
        codec => json
        mode => read
        delimiter => "EOF"
        path => "/journals/journal.json"
    } 
} 
output { 
    stdout{
        codec => rubydebug
    }
    elasticsearch {
        index => "host-journals"
        document_type => "default"
        hosts => ["http://127.0.0.1:9200"]
    } 
}

[Badger]

I don't think you can do that using a file input.

You could use an exec input to periodically cat the file, and then use a fingerprint filter to set the document_id option on the elasticsearch output, so that if it re-reads the file with the same content it overwrites the same document in elasticsearch.

[lovecraft (Sean)]

Thanks for the tip!

Some shell slight of hand seems to work just fine with the exec input...

input {
    exec {
        command => "if [ -f /journals/journal.json ] ;then /bin/cat /journals/journal.json && mv /journals/journal.json /journals/journal.`date +%Y-%m-%d.%H:%M:%S` ; fi"
        interval => 30
    }
}

I used the drop filter to prevent Logstash from pushing anything to elasticsearch if the exec'd command produces no output...

filter {
  if [message] == "" {
    drop { }
  }
}

This all seems to do what I need it to do. Profuse thanks!

-Seàn