Drop/cancel events with a timestamp older than a given amount of seconds

Elastic Stack Logstash
1

LogStash 6.6

My log files have lines like this

01/07/19 18:31:33.452 blah blah blah

where the time is local.

I need to discard all lines older than a given amount of time. Let's say, for example, 24 hours.
I have a grok filter to capture the timestamp into a variable:

grok {
    pattern_definitions => { "START_TIMESTAMP" => "%{DATE_US} %{TIME}" }
    match => { "message" => [
        '^%{START_TIMESTAMP:eventtimestamp}',
        ...  <other patterns here> ...
   }
}

After that, I do a lot of filtering within the Ruby filter.
So I was wondering if I can, somehow, check how old is the content of "eventtimestamp", and perform event.cancel if needed.

Reading similar posts, I believe they suggest to convert it first using filter date, so I tried something like this

date {
    match => ["eventtimestamp", "MM/dd/yy HH:mm:ss.SSS"]
    target => "eventtimestamp"
}

but it is unclear to me what to do with it after that inside by Ruby filter code. Something like this

if ( Time.now - event.get("eventtimestamp") ) > 86400
    event.cancel
end

does not work, as it seems to be comparing Timestamp and Rational objects.
What am I missing here? Most probably something trivial I don't see as I am 100% new to Ruby... :slight_smile:

Or maybe it is easier to do it using pure LogStash comparisons, before entering the filter plugin?

Thanks a lot in advance.
Jose

2

Once you have parsed the log timestamp into @timestamp using the date filter you can use the age filter to calculate age and base your drop decision on this.

3

Hi, Crhistian,
my understanding is that Age filter is not available anymore in LogStash 6.x

4

You can use

if ( Time.now.to_i - event.get("eventtimestamp").to_i ) > 86400
    event.cancel
end
1 Like
5

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.