Rereading a refreshed file

Hi All,

I am new to Logstash and trying to work out a config.

My scenario is that I have a utility which runs periodically and outputs a report as a multiline JSON object in a file. The filename remains the same across runs but the content is replaced with every run.

I want to take the content of the file and pass it into Elastic search as a single document. So there is a one document per run of the utility.

I have a config which does this BUT it only does it once at the startup of Logstash. When the content of the file is updated a reread of the file is not triggered.

I suspect the file is being 'unwatched' after the first run. Can someone suggest a better config please?

Thanks++

Here is my config:

input { 
    file {
        codec => json
        mode => read
        delimiter => "EOF"
        path => "/journals/journal.json"
    } 
} 
output { 
    stdout{
        codec => rubydebug
    }
    elasticsearch {
        index => "host-journals"
        document_type => "default"
        hosts => ["http://127.0.0.1:9200"]
    } 
}

I don't think you can do that using a file input.

You could use an exec input to periodically cat the file, and then use a fingerprint filter to set the document_id option on the elasticsearch output, so that if it re-reads the file with the same content it overwrites the same document in elasticsearch.

Thanks for the tip!

Some shell slight of hand seems to work just fine with the exec input...

input {
    exec {
        command => "if [ -f /journals/journal.json ] ;then /bin/cat /journals/journal.json && mv /journals/journal.json /journals/journal.`date +%Y-%m-%d.%H:%M:%S` ; fi"
        interval => 30
    }
}

I used the drop filter to prevent Logstash from pushing anything to elasticsearch if the exec'd command produces no output...

filter {
  if [message] == "" {
    drop { }
  }
}

This all seems to do what I need it to do. Profuse thanks!

-Seàn

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.