Skipping older files

How can I get Logstash to open and process files modified more than 24 hours ago?

Logstash is skipping files that were written over 24 hours ago. I found a topic that references the Logstash forwarder accepting a "dead time" setting, which appears to allow you to adjust this. However I do not see it as an available setting for input file.

logged message:
{:timestamp=>"2017-03-22T16:56:25.516000-0400", :message=>"_discover_file: /log_storage/upload/*.dat: new: /log_storage/upload/REF.AA.201703181100.dat (exclude is [])", :level=>:debug, :file=>"filewatch/watch.rb", :line=>"310", :method=>"_discover_file"}
{:timestamp=>"2017-03-22T16:56:25.517000-0400", :message=>"_discover_file: /log_storage/upload/REF.AA.201703181100.dat: skipping because it was last modified more than 86400.0 seconds ago", :level=>:debug, :file=>"filewatch/watch.rb", :line=>"330", :method=>"_discover_file"}

I have tried setting the ignore_older setting to a value older than my files but that doesn't work as expect. Though in the debug logs it shows it opening the files and writing an entry to sincedb it doesn't actually process the contents of the file.

logged message with "ignore_older" set to 864000:
{:timestamp=>"2017-03-22T17:12:21.600000-0400", :message=>"_open_file: /log_storage/upload/REF.AA.201703181100.dat: opening", :level=>:debug, :file=>"filewatch/tail_base.rb", :line=>"86", :method=>"_open_file"}
{:timestamp=>"2017-03-22T17:12:21.601000-0400", :message=>"/log_storage/upload/REF.AA.201703181100.dat: initial create, no sincedb, seeking to end 48865762", :level=>:debug, :file=>"filewatch/tail_base.rb", :line=>"149", :method=>"_add_to_sincedb"}
{:timestamp=>"2017-03-22T17:12:21.608000-0400", :message=>"each: file grew: /log_storage/upload/REF.AA.201703181100.dat: old size 0, new size 48865762", :level=>:debug, :file=>"filewatch/watch.rb", :line=>"254", :method=>"each"}

Are you setting start_position => beginning for your file input?

Thank you for the response. I did not have it set. After setting it, Logstash still did not read older files. However when I set "start_position" to beginning AND I set "ignore_older" to greater than 86,400 seconds Logstash would read older files. So it appears at least in Logstash 2.3.4, "ignore_older" is set to 86,400 by default.

Yes, in Logstash 2.3 the documented default value is 86400:

That was my problem I was looking at the current documentation, not the documentation for 2.3. Thank you for your support.

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.