File updates in read mode

Elastic Stack Logstash
1

I have input file plugin configured in read mode where I read the entire file and use multiline to concat it into a single event.

input {
  file {
    path => "/absolute/path/to/xmls/**/*.xml"
    start_position => "beginning"
    max_open_files => 10000
    mode => "read"
    close_older => "1 minute"
    codec => multiline {
      charset => "ISO-8859-1"
      pattern => "\Z"
      what => "previous"
    }
  }
}

On initial run it worked perfectly. However I am confused as to what happens when an XML file, with the same filename gets added again with the last modified updated?

In theory, the file should be read again? Currently it doesn't seem to do so.

I tried by adding an XML file with a filename that got processed earlier. The XML files last modified has changed to a more recent time but with 10 minutes into waiting, the file has yet to be read by Logstash.

The reason why I want to re-use XML files with the same name is that these have content that occasionally get updated so I want to pull them in again and my elsticsearch output upserts the content if needed (based on document_id and XML internal updated timestamp).

Could someone explain to me how the file updates should work? Could it be that I have to disable .sincedb file from being created (/dev/null)?

2

it is going by inode. hence it is not reading it because logstash thinks it has already read file sitting on same inode.

check mode detail explanation that I put, as I had exact same problem

https://discuss.elastic.co/t/elk-kibana-showing-data-that-was-already-deleted-wont-show-new-data/181631/2

3

Thank-you for the info. Your post helped me steer to the right solution.

I did a simple test by doing a rename of the filename, which doesn't change the inode. However it seems that Logstash parsed the file even if the inode remained the same.

So it is possible that it might be enough to just have a different named file but since if I will be serving timestamped files to make Logstash do updates it wouldn't really matter.

4

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.