Logstash date filter in UNIX_MS from two fields

Elastic Stack Logstash
1

I have some data from snort which has two subfields, in an !event¬ fields:
"event-microsecond" => 289367,
"event-second" => 1493741082,

I am trying to compose these into a float and then tell Elasticsearch it's the timestamp field.

filter {
mutate { add_field =>
{ "full_timestamp" => "[event][event-second].[event][event-microsecond]" }
}
mutate { convert => { "full_timestamp" => "float" } }
date {
locale => "en"
match => ["full_timestamp", "UNIX_MS" ]
target => "@timestamp"
}
}
give me 0.0 for the full_timestamp, so I'm doing something wrong when I try to make the new field.

I then tried
mutate { add_field => { "full_timestamp" => "%{[event][event-second]}.%{[event][event-microsecond]}" } }
and got
"full_timestamp" => 1493741082.289367,
"@timestamp" => 1970-01-18T06:55:41.082Z,

The new field looks about right, but the date is wrong by miles; it should be 2nd May 2017.

2

UNIX_MS expects the input to be the time in milliseconds since the epoch, so to get the expected results you'll want full_timestamp to contain 1493741082289. You can use a ruby filter to divide event-microsecond by 1000 to turn it into milliseconds, then use a mutate filter to simply concatenate event-second and event-microsecond (with no intervening decimal point). Well, the concatenation could of course also be done with the same ruby filter.

1 Like
3

I don't know much ruby but have tried this which seems to work:
(advice on something neater more than welcome)

filter {
    ruby {
      code => "s = sprintf('%03d', (event.get('[event][event-microsecond]').to_i/1000)); s = s + '000'; event.set('milliseconds', s[0..2])"
      add_tag => ["ran the code"]
    }

    mutate { add_field =>
      { "full_timestamp" => "%{[event][event-second]}%{milliseconds}" }
    }
    mutate { convert => { "full_timestamp" => "integer" } }
    date  {
        locale => "en"
        match => ["full_timestamp", "UNIX_MS" ]
        target => "@timestamp"
    }
}

giving:
"milliseconds" => "289",
"full_timestamp" => 1493741082289,
"@timestamp" => 2017-05-02T16:04:42.289Z,

4

Shorter:

event.set('full_timestamp', event.get('[event][event-second]').to_s + (event.get('[event][event-microsecond]') / 1000).to_s)
1 Like
5

Still leaving the mutate to make it an integer?

6

No, you can drop that if you want.

7

Perfect.
Is the date plugin treating it as an integer, so the ruby?

8

The date filter parses strings. I'm sure it's fine with parsing an integer field but it'll start with converting the integer into a string.

1 Like
9

One final observation,

I suspect I need
'[event][event-second]').to_s + (event.get('[event][event-microsecond]') / 1000000).to_s)
to change microseconds (I had /1000 before, which is milliseconds)

10

The desired result is milliseconds so it's correct to divide by 1000. The input is 1493741082 and 289367 and you want 1493741082289.

11

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.