Best Method to Manipulate Date & TimeStamp

[londonx]

Hi all, looking for a way to efficiently convert an ISO8601 date/time stamp into the full ISO8601 format containing the "T" and "Z"

The log contains a date timestamp in the following format:

  2018-10-16 00:10:01.7764

ISO8601 grok pattern recognises the date & time stamp fields correctly:

  %{TIMESTAMP_ISO8601:timestamp}

{
"timestamp": [
 [
"2018-10-16 00:10:01.7764"
]
],
"YEAR": [
[
"2018"
]
],
"MONTHNUM": [
 [
"10"
]
],
"MONTHDAY": [
[
"16"
]
],
"HOUR": [
[
"00",
null
]
],
"MINUTE": [
[
"10",
null
]
],
"SECOND": [
[
"01.7764"
]
],
"ISO8601_TIMEZONE": [
[
null
]
]
}

So the captured field looks fine:

 "timestamp": "2018-09-13 16:00:52.1074",

Unfortunately the output demands the full ISO8601 format with 'T' separator and 'Z' timezone
e.g.

 2018-10-16T00:10:01.7764Z

what would the most efficient way be of trying to massage the T and Z into the timestamp field? I am assuming I would still want Logstash to know it is a timestamp rather than end up just being a string!

There are two ways I can think of,

1. Use a filter > mutate (will it still recognise the internal Grok variables from the previous step? Or would I have to split up the initial Grok pattern %{TIMESTAMP_ISO8601:timestamp}" into something like %{YEAR}-%{MONTHNUM}-%{MONTHDAY} %{HOUR}:%{MINUTE}:%{SECOND} "

then I could mutate and add the T & Z:

 mutate {
    update => {
         "timestamp" => "%{YEAR}-%{MONTHNUM}-%{MONTHDAY}T%{HOUR}:%{MINUTE}:%{SECOND}Z"

Seems a tad convoluted and expensive though!

2. Use the dedicated "date" filter but there doesn't seem to be a way to convert it once captured unless I am missing something!

 date {
     match => [ "timestamp", "yyyy-MM-dd HH:mm:ss.SSSS" ]
  target => "timestamp2"

Thanks for reading!

[Jenni]

Use the date filter to convert is to a Logstash Timestamp object. If you want it to be an object in the output, you are done at this point. If you want it to be a String, just use

mutate { convert => { "timestamp" => string}}

The default string representation of timestamps in Logstash meets the ISO8601 criteria. (But it uses 3 digits for the fractional seconds as Logstash doesn't support any more precision. If that fourth digit is important to you, you'll have to stick to string manipulations, I guess.)

[londonx]

Thanks yeah that's an interesting point, so I have gone with keeping the simple ISO8601 Grok pattern

%{TIMESTAMP_ISO8601:timestamp}

And used the date filter to convert it into the full ISO8601 output

filter {
  date {
    match => [ "timestamp", "yyyy-MM-dd HH:mm:ss.SSSS" ]
    target => "@timestamp"

So I'll keep both the "@timestamp" full ISO8601 field and retain the "timestamp" with its 4 digit fractional output just in case!

Seems fairly clean and not too much mutating!

Output:

"@timestamp":"2018-07-01T02:13:35.030Z"
"timestamp":"2018-07-01 02:13:35.0304"