Logstash single date for logs

Elastic Stack Logstash
1

This might sound counter intuitive, but I am trying to get my logstash config file to ONLY read the previous day's logs. I have the last 90 days of logs saved (except for current day's, it archives at midnight every night) and I only want logstash to read the previous day's log. How would I go about this?

Thank you!

2

How are the files named? Are you saying there's no log file for the current day until midnight when everything from the last 24 hours are flushed to disk, and it's at that point you want Logstash to kick in?

3

Correct, no log for today until midnight. They are named SyslogCatchAll-year-month-day.txt. So at midnigfht tonight, today's file will be saved as SyslogCatchAll-2017-10-23.txt and tomorrow that will be the only file I want logstash to read (as there are 90 days worth of archived logs).

4

A couple of options:

  • Generate a sincedb file that tricks Logstash into thinking it has processed all the old files.
  • Make Logstash process all the old logs but add a filter that drops all events not coming from the 2017-10-23 file (look at the path field).
  • Generate a new Logstash configuration file every midnight.
  • I'm not sure how the file input treats symlinks, but perhaps you can point Logstash to a file that's a symlink to the current logfile. Every midnight you point that symlink to a new file.
5

I think I will go with either option 2 or 3 (if I can automate it). But for now I changed the path to just yesterday's log to try and fix some issues. I am trying to change the logstash @timestamp to the timestamp in each entry and it is not working. Here is what I have in my filter section:

filter {
grok {
named_captures_only => 'false'
match => { "message" => "%{TIMESTAMP_ISO8601:timestamp}%stuff"}
}
mutate {
convert => {"@timestamp" => "string"}
}
date {
match => ["timestamp","ISO8601"]
target => "@timestamp"
}
...
I could have this wrong, but I read that the target option in the mutate filter reads strings, so I converted @timestamp to a string to try to match @timestamp with timestamp from the file. In kibana, the @timestamp is still being autogenerated by logstash.

6

Nevermind, I fixed this date issue. Thank you!!!!

7

Another and possibly easier option might be to configure your logstash instance to look for files in an empty directory. Then simply manually or programmatically copy your log files in there. If each file that's copied has a unique filename then you won't have to worry about sincedb or soft links.

8

Another and possibly easier option might be to configure your logstash instance to look for files in an empty directory. Then simply manually or programmatically copy your log files in there.

Yes, that's a good idea.

9

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.