How can I get Logstash to open and process files modified more than 24 hours ago?
Logstash is skipping files that were written over 24 hours ago. I found a topic that references the Logstash forwarder accepting a "dead time" setting, which appears to allow you to adjust this. However I do not see it as an available setting for input file.
logged message:
{:timestamp=>"2017-03-22T16:56:25.516000-0400", :message=>"_discover_file: /log_storage/upload/*.dat: new: /log_storage/upload/REF.AA.201703181100.dat (exclude is [])", :level=>:debug, :file=>"filewatch/watch.rb", :line=>"310", :method=>"_discover_file"}
{:timestamp=>"2017-03-22T16:56:25.517000-0400", :message=>"_discover_file: /log_storage/upload/REF.AA.201703181100.dat: skipping because it was last modified more than 86400.0 seconds ago", :level=>:debug, :file=>"filewatch/watch.rb", :line=>"330", :method=>"_discover_file"}
I have tried setting the ignore_older setting to a value older than my files but that doesn't work as expect. Though in the debug logs it shows it opening the files and writing an entry to sincedb it doesn't actually process the contents of the file.
logged message with "ignore_older" set to 864000:
{:timestamp=>"2017-03-22T17:12:21.600000-0400", :message=>"_open_file: /log_storage/upload/REF.AA.201703181100.dat: opening", :level=>:debug, :file=>"filewatch/tail_base.rb", :line=>"86", :method=>"_open_file"}
{:timestamp=>"2017-03-22T17:12:21.601000-0400", :message=>"/log_storage/upload/REF.AA.201703181100.dat: initial create, no sincedb, seeking to end 48865762", :level=>:debug, :file=>"filewatch/tail_base.rb", :line=>"149", :method=>"_add_to_sincedb"}
{:timestamp=>"2017-03-22T17:12:21.608000-0400", :message=>"each: file grew: /log_storage/upload/REF.AA.201703181100.dat: old size 0, new size 48865762", :level=>:debug, :file=>"filewatch/watch.rb", :line=>"254", :method=>"each"}