Drop/cancel events with a timestamp older than a given amount of seconds

LogStash 6.6

My log files have lines like this

01/07/19 18:31:33.452 blah blah blah

where the time is local.

I need to discard all lines older than a given amount of time. Let's say, for example, 24 hours.
I have a grok filter to capture the timestamp into a variable:

grok {
    pattern_definitions => { "START_TIMESTAMP" => "%{DATE_US} %{TIME}" }
    match => { "message" => [
        '^%{START_TIMESTAMP:eventtimestamp}',
        ...  <other patterns here> ...
   }
}

After that, I do a lot of filtering within the Ruby filter.
So I was wondering if I can, somehow, check how old is the content of "eventtimestamp", and perform event.cancel if needed.

Reading similar posts, I believe they suggest to convert it first using filter date, so I tried something like this

date {
    match => ["eventtimestamp", "MM/dd/yy HH:mm:ss.SSS"]
    target => "eventtimestamp"
}

but it is unclear to me what to do with it after that inside by Ruby filter code. Something like this

if ( Time.now - event.get("eventtimestamp") ) > 86400
    event.cancel
end

does not work, as it seems to be comparing Timestamp and Rational objects.
What am I missing here? Most probably something trivial I don't see as I am 100% new to Ruby... :slight_smile:

Or maybe it is easier to do it using pure LogStash comparisons, before entering the filter plugin?

Thanks a lot in advance.
Jose

Once you have parsed the log timestamp into @timestamp using the date filter you can use the age filter to calculate age and base your drop decision on this.

Hi, Crhistian,
my understanding is that Age filter is not available anymore in LogStash 6.x

You can use

if ( Time.now.to_i - event.get("eventtimestamp").to_i ) > 86400
    event.cancel
end

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.