Give events without timestamp the timestamp from the last event


(Thorsten Peter) #1

Hi,
i'm currently trying to give events from the initialization dialog (they have no timestamp) the timestamp from the next possible event which has a timestamp. the list is upside down which means the latest entry is at the top. The interesting part (logs with and without timestamp) look somehow like that:

 66 4  0 days 00:00:09  2015/12/28 10:19:59 Ring redundancy role HRP Client.
 67 4  0 days 00:00:08  2015/12/28 10:19:58 Cold start performed
 68 3  0 days 00:00:02   Port Monitoring Barrier disabled.
 69 3  0 days 00:00:02   Port Mirroring disabled.
 70 3  0 days 00:03:17   Link down on port 1
 71 3  0 days 00:00:23   Link up on port 1

I can distinguish the messages by simple doing:

 if  [message] =~ /^[ , 0-9]+\t\d+\t \d+ days \d\d:\d\d:\d\d \t[\d]+/\d\d\/\d\d/{
            grok {...}
            date {...}
}elseif [message] =~ /^[ , 0-9]+\t\d+\t \d+ days \d\d:\d\d:\d\d \t [\w]+/{
            grok {...}
            date {???}
}

The problem with that is obviously that the events without a timestamp get data dateparsefaliure.
That leads the my question, is there a possibility to take the last known timestamp (in this case 2015/12/28 10:19:58) and write it to all the following events down below?
I found something called aggregate filter but im not sure if i can use thatin my case, and if i can, how?

Thanks in advance!


#2

You could do it with aggregate using something like this.

  mutate { add_field => { "static" => "1" } }
  if [message] =~ /2015/ {
    aggregate {
      task_id => "%{static}"
      code => "map['something'] = event.get('message')"
    }
  } else {
    aggregate {
      task_id => "%{static}"
      code => "event.set('something', map['something'])"
    }
  }

Or you could do it using a ruby class variable. Use one of these in the first block and the other in the second.

    ruby { code => '@@t = event.get("timestamp")' }
    ruby { code => 'event.set("timestamp", @@t)' }

(Thorsten Peter) #3

Hey Badger,
i tried your suggestion with the ruby class variable because it was exactly what i was looking for.
It works great that way!
Thank you very much for your time :slight_smile:


(system) #4

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.