How I can modify timestamp in log message to UTC

Elastic Stack Logstash
#1

Hi,

Our log messages contain timestamp in EST format. How I can modify the timestamp to UTC before displaying logs into Kibana.

Example message:

2016-03-26T01:55:47.78-0500 [App/0] OUT [AUDIT ] CWWKE0001I: The server defaultServer has been launched

Looking forward for help on this.

Thanks,
Nagesh Mandava.

#2

Use the grok filter to parse the log lines and extract the timestamp to a separate field. Feed that field to the date filter. It'll parse and store the timestamp in the @timestamp field which always is UTC.

#3

We have below pattern defined to parse the log message
CF_DIRECT_PULL_FORMAT %{TIMESTAMP_ISO8601:timestamp} [%{CF_LOG_TYPE}]%{SPACE}%{WORD}%{SPACE}([%{CF_WLP_LOG_LEVEL:loglevel}%{SPACE}])?%{GREEDYDATA:details}

We have two fields in the viewlet to display the logs in Kibana dashboard.
First field is @timestamp and second is message.
I want to modify the timestamp in message filed to UTC format.

#4

Hello
I think you can do something like this:

  1. Split original message field to timestamp and sub_message with grok:
    grok {
    match => [ "message", "%{TIMESTAMP_ISO8601:timestamp} %{GREEDYDATA:sub_message}" ]
    remove_field => [ "message" ]
    }

  2. Parse timestamp field with date filter.

  3. Create new field message from @timestamp and sub_message:
    if [sub_message] {
    mutate {
    add_field => { "message" => "%{@timestamp} %{sub_message}" }
    }
    }

#5

Can we have multiple grok statements in conf file, because I already have below
grok {
match => {"details" => "[DWPerf-%{WORD}][UN-%{GREEDYDATA:userid}][SI-%{GREEDYDATA:sessionid}]%{GREEDYDATA:details}%{NUMBER:elapsed_time}s"}
overwrite => [ "details" ]
}
mutate {
convert => { "elapsed_time" => "float" }
}

#6

Can we have multiple grok statements in conf file

Yes, of course.

#7

Can you please provide me an example ?

Thanks,
Nagesh Mandava.

#8

There's nothing special about having multiple grok filters.

grok {
  match => ["field1", "expression1"]
}
grok {
  match => ["field2", "expression2"]
}
#9

Thanks for the example.

I have updated my conf file as below
grok {
match => {"details" => "[DWPerf-%{WORD}][UN-%{GREEDYDATA:userid}][SI-%{GREEDYDATA:sessionid}]%{GREEDYDATA:details}%{NUMBER:elapsed_time}s"}
overwrite => [ "details" ]
}
mutate {
convert => { "elapsed_time" => "float" }
}
grok{
match => [ "message", "%{TIMESTAMP_ISO8601:timestamp} [%{CF_LOG_TYPE}]%{SPACE}%{WORD}%{SPACE}([%{CF_WLP_LOG_LEVEL:loglevel}%{SPACE}])?%{GREEDYDATA:details}" ]
remove_field => [ "message" ]
}

    if[CF_DIRECT_PULL_FORMAT2]{

            mutate {
                    add_field => { "message" => "%{@timestamp} %{CF_DIRECT_PULL_FORMAT2}" }
            }

    }

Patterns file updated as below

CF_DIRECT_PULL_FORMAT2 [%{CF_LOG_TYPE}]%{SPACE}%{WORD}%{SPACE}([%{CF_WLP_LOG_LEVEL:loglevel}%{SPACE}])?%{GREEDYDATA:details}

I am not seeing any difference after above changes for the timestamp in message.

message_in_dashboard.jpg1538×105 42.3 KB

#10

Hi Kirill,
Can you please check my previous message and let me know where I am wrong.

Thanks,
Nagesh.

#11

Comments:

  • Where's your date filter? That's what converts your UTC-5 timestamp into UTC so that you can put it back into message.
  • CF_DIRECT_PULL_FORMAT2 is the name of a grok pattern but you're treating it like the name of a field.
  • Don't post screenshots from Kibana. Post the output from a stdout { codec => rubydebug } output. ES and Kibana will only distract you. Reintroduce them when you've verified that the events looks as expected.
  • When you use add_field on the message field you'll actually turn that field into an array that'll contain both the original value and the new value. Use replace instead.

Follow Kirill's example more closely. Except for the add_field thing his example looks perfectly correct.

#12

Thanks Magnus.

I have updated as below based on your review comments.

grok{
patterns_dir => "./patterns"
remove_field => [ "message" ]
match => {"sub_message" => "%{CF_DIRECT_PULL_FORMAT2}" }
}

    if[sub_message]{

        date {
            match => [ "timestamp", "yyyy-MM-dd'T'HH:mm:ss.SSSSSSZZ", "ISO8601" ]
            target => "@timestamp"
        }

        mutate {
            replace => { "message" => "%{@timestamp} %{sub_message}" }
        }
    }

There is still some issue in above snippet. Please help me to fix it.

#13

I have updated as below based on your review comments.

No you haven't. What you're trying is quite far from Kirill's example.

#14

Earlier I was trying to add_filed and remove_filed for the message. I think it is not correct, so updated as below

grok {
patterns_dir => "./patterns"
match => {"sub_message" => "%{CF_DIRECT_PULL_FORMAT2}" }
}
if[sub_message]{

    date {
        match => [ "timestamp", "yyyy-MM-dd'T'HH:mm:ss.SSSSSSZZ", "ISO8601" ]
        target => "@timestamp"
    }

    mutate {
        add_field => { "message" => "%{@timestamp} %{sub_message}" }
    }
}

Is this fine ?

Thanks,
Nagesh Mandava.

#15

I have twice asked you to follow the previously posted example, and yet you are not doing that. I don't have time for this anymore.

#16

I have made following changes and it worked for me.

grok {
patterns_dir => "./patterns"
match => {"message" => "%{CF_DIRECT_PULL_FORMAT3}" }
}
mutate{
convert => ["timestamp", "string"]
}
date {
locale => "en"
match => ["timestamp", "ISO8601"]
timezone => "Europe/Vienna"
target => "timestamp"
}
grok {
patterns_dir => "./patterns"
match => {"sub_message" => "%{CF_DIRECT_PULL_FORMAT2}" }
}
mutate {
replace => { "message" => "%{timestamp} %{sub_message}" }
}

#17