Logstash date filter in UNIX_MS from two fields


(Frances Buontempo) #1

I have some data from snort which has two subfields, in an !event¬ fields:
"event-microsecond" => 289367,
"event-second" => 1493741082,

I am trying to compose these into a float and then tell Elasticsearch it's the timestamp field.

filter {
mutate { add_field =>
{ "full_timestamp" => "[event][event-second].[event][event-microsecond]" }
}
mutate { convert => { "full_timestamp" => "float" } }
date {
locale => "en"
match => ["full_timestamp", "UNIX_MS" ]
target => "@timestamp"
}
}
give me 0.0 for the full_timestamp, so I'm doing something wrong when I try to make the new field.

I then tried
mutate { add_field => { "full_timestamp" => "%{[event][event-second]}.%{[event][event-microsecond]}" } }
and got
"full_timestamp" => 1493741082.289367,
"@timestamp" => 1970-01-18T06:55:41.082Z,

The new field looks about right, but the date is wrong by miles; it should be 2nd May 2017.


(Magnus Bäck) #2

UNIX_MS expects the input to be the time in milliseconds since the epoch, so to get the expected results you'll want full_timestamp to contain 1493741082289. You can use a ruby filter to divide event-microsecond by 1000 to turn it into milliseconds, then use a mutate filter to simply concatenate event-second and event-microsecond (with no intervening decimal point). Well, the concatenation could of course also be done with the same ruby filter.


(Frances Buontempo) #3

I don't know much ruby but have tried this which seems to work:
(advice on something neater more than welcome)

filter {
    ruby {
      code => "s = sprintf('%03d', (event.get('[event][event-microsecond]').to_i/1000)); s = s + '000'; event.set('milliseconds', s[0..2])"
      add_tag => ["ran the code"]
    }

    mutate { add_field =>
      { "full_timestamp" => "%{[event][event-second]}%{milliseconds}" }
    }
    mutate { convert => { "full_timestamp" => "integer" } }
    date  {
        locale => "en"
        match => ["full_timestamp", "UNIX_MS" ]
        target => "@timestamp"
    }
}

giving:
"milliseconds" => "289",
"full_timestamp" => 1493741082289,
"@timestamp" => 2017-05-02T16:04:42.289Z,


(Magnus Bäck) #4

Shorter:

event.set('full_timestamp', event.get('[event][event-second]').to_s + (event.get('[event][event-microsecond]') / 1000).to_s)

(Frances Buontempo) #5

Still leaving the mutate to make it an integer?


(Magnus Bäck) #6

No, you can drop that if you want.


(Frances Buontempo) #7

Perfect.
Is the date plugin treating it as an integer, so the ruby?


(Magnus Bäck) #8

The date filter parses strings. I'm sure it's fine with parsing an integer field but it'll start with converting the integer into a string.


(Frances Buontempo) #9

One final observation,

I suspect I need
'[event][event-second]').to_s + (event.get('[event][event-microsecond]') / 1000000).to_s)
to change microseconds (I had /1000 before, which is milliseconds)


(Magnus Bäck) #10

The desired result is milliseconds so it's correct to divide by 1000. The input is 1493741082 and 289367 and you want 1493741082289.


(system) #11

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.