Howto convert UNIX timestamp json field into ISO8601 format

Elastic Stack Logstash
1

Hi,
I've allowed filebeat to parse a json log, but the Unix timestamp fields like this
"EdgeStartTimestamp": 1534732311104000000,
on Kibana is converted in this way
EdgeStartTimestamp 1,534,732,311,104,000,000.
instead of ISO8601.

Could someone help me to convert it in ISO8601 format?
thanks

2

Use a mutate filter to remove the last three digits from the number to turn the microseconds into milliseconds (use the gsub option), then feed the result to a date filter. Use the UNIX_MS date pattern.

3

Hi Magnus,
I've applied this filter

filter {
mutate {
remove_field => [ "host", "tags", "count", "source" ]
}
mutate {
gsub => ["EdgeStartTimestamp", "\d{6}$", ""]
}
date {
match => [ "EdgeStartTimestamp", "UNIX_MS" ]
target => "EdgeStartTime"
}
}

but the result was not what I expected:
|t EdgeStartTime| |48638255-11-25T05:53:19.872Z|
|# EdgeStartTimestamp| |1,534,812,939,553,999,872|

this is the json view on kibana:
"EdgeStartTimestamp": 1534812939554000000,
"ClientRequestMethod": "POST",
"EdgeEndTimestamp": 1534812939700000000,
"input_type": "log",
"WAFRuleID": "",
"EdgeStartTime": "48638255-11-25T05:53:19.872Z"

Where is my mistake?
Thanks.

4

It appears the gsub didn't work. Perhaps you need to convert the field into a string first?

Also, when you're removing the last six digits you're down to seconds so UNIX is the correct date pattern.

5

Yes, this is the trick.
The timestamp is in milliseconds so I've to use UNIX_MS

mutate {
    convert => { "EdgeStartTimestamp" => "string" }
    convert => { "EdgeEndTimestamp" => "string" }
}
mutate {
    gsub => ["EdgeStartTimestamp", "\d{6}$", ""]
    gsub => ["EdgeEndTimestamp", "\d{6}$", ""]
}
mutate {
    convert => { "EdgeStartTimestamp" => "integer" }
    convert => { "EdgeEndTimestamp" => "integer" }
}
date {
    match => [ "EdgeStartTimestamp", "UNIX_MS" ]
    target => "EdgeStartTime"
    remove_field => [ "EdgeStartTimestamp" ]
}
date {
    match => [ "EdgeEndTimestamp", "UNIX_MS" ]
    target => "EdgeEndTime"
    remove_field => [ "EdgeEndTimestamp" ]
}

Can be optimized this flow?
Thank you for your input.

6

The second conversion back to integer serves no purpose. Otherwise this is probably as good as it gets.

7

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.