How I can modify timestamp in log message to UTC

nagesh · March 30, 2016, 4:36pm

Hi,

Our log messages contain timestamp in EST format. How I can modify the timestamp to UTC before displaying logs into Kibana.

Example message:

2016-03-26T01:55:47.78-0500 [App/0] OUT [AUDIT ] CWWKE0001I: The server defaultServer has been launched

Looking forward for help on this.

Thanks,
Nagesh Mandava.

magnusbaeck · March 30, 2016, 4:53pm

Use the grok filter to parse the log lines and extract the timestamp to a separate field. Feed that field to the date filter. It'll parse and store the timestamp in the @timestamp field which always is UTC.

nagesh · March 30, 2016, 5:05pm

We have below pattern defined to parse the log message
CF_DIRECT_PULL_FORMAT %{TIMESTAMP_ISO8601:timestamp} [%{CF_LOG_TYPE}]%{SPACE}%{WORD}%{SPACE}([%{CF_WLP_LOG_LEVEL:loglevel}%{SPACE}])?%{GREEDYDATA:details}

We have two fields in the viewlet to display the logs in Kibana dashboard.
First field is @timestamp and second is message.
I want to modify the timestamp in message filed to UTC format.

kirill_polishchuk · March 30, 2016, 7:23pm

Hello
I think you can do something like this:

Split original message field to timestamp and sub_message with grok:
grok {
match => [ "message", "%{TIMESTAMP_ISO8601:timestamp} %{GREEDYDATA:sub_message}" ]
remove_field => [ "message" ]
}
Parse timestamp field with date filter.
Create new field message from @timestamp and sub_message:
if [sub_message] {
mutate {
add_field => { "message" => "%{@timestamp} %{sub_message}" }
}
}

nagesh · March 30, 2016, 10:07pm

Can we have multiple grok statements in conf file, because I already have below
grok {
match => {"details" => "[DWPerf-%{WORD}][UN-%{GREEDYDATA:userid}][SI-%{GREEDYDATA:sessionid}]%{GREEDYDATA:details}%{NUMBER:elapsed_time}s"}
overwrite => [ "details" ]
}
mutate {
convert => { "elapsed_time" => "float" }
}

magnusbaeck · March 31, 2016, 5:31am

Can we have multiple grok statements in conf file

Yes, of course.

nagesh · March 31, 2016, 1:43pm

Can you please provide me an example ?

Thanks,
Nagesh Mandava.

magnusbaeck · March 31, 2016, 1:56pm

There's nothing special about having multiple grok filters.

grok {
  match => ["field1", "expression1"]
}
grok {
  match => ["field2", "expression2"]
}

nagesh · March 31, 2016, 4:57pm

Thanks for the example.

I have updated my conf file as below
grok {
match => {"details" => "[DWPerf-%{WORD}][UN-%{GREEDYDATA:userid}][SI-%{GREEDYDATA:sessionid}]%{GREEDYDATA:details}%{NUMBER:elapsed_time}s"}
overwrite => [ "details" ]
}
mutate {
convert => { "elapsed_time" => "float" }
}
grok{
match => [ "message", "%{TIMESTAMP_ISO8601:timestamp} [%{CF_LOG_TYPE}]%{SPACE}%{WORD}%{SPACE}([%{CF_WLP_LOG_LEVEL:loglevel}%{SPACE}])?%{GREEDYDATA:details}" ]
remove_field => [ "message" ]
}

    if[CF_DIRECT_PULL_FORMAT2]{

            mutate {
                    add_field => { "message" => "%{@timestamp} %{CF_DIRECT_PULL_FORMAT2}" }
            }

    }

Patterns file updated as below

CF_DIRECT_PULL_FORMAT2 [%{CF_LOG_TYPE}]%{SPACE}%{WORD}%{SPACE}([%{CF_WLP_LOG_LEVEL:loglevel}%{SPACE}])?%{GREEDYDATA:details}

I am not seeing any difference after above changes for the timestamp in message.

nagesh · March 31, 2016, 5:40pm

Hi Kirill,
Can you please check my previous message and let me know where I am wrong.

Thanks,
Nagesh.

magnusbaeck · March 31, 2016, 6:01pm

Comments:

Where's your date filter? That's what converts your UTC-5 timestamp into UTC so that you can put it back into message.
CF_DIRECT_PULL_FORMAT2 is the name of a grok pattern but you're treating it like the name of a field.
Don't post screenshots from Kibana. Post the output from a stdout { codec => rubydebug } output. ES and Kibana will only distract you. Reintroduce them when you've verified that the events looks as expected.
When you use add_field on the message field you'll actually turn that field into an array that'll contain both the original value and the new value. Use replace instead.

Follow Kirill's example more closely. Except for the add_field thing his example looks perfectly correct.

nagesh · March 31, 2016, 6:32pm

Thanks Magnus.

I have updated as below based on your review comments.

grok{
patterns_dir => "./patterns"
remove_field => [ "message" ]
match => {"sub_message" => "%{CF_DIRECT_PULL_FORMAT2}" }
}

    if[sub_message]{

        date {
            match => [ "timestamp", "yyyy-MM-dd'T'HH:mm:ss.SSSSSSZZ", "ISO8601" ]
            target => "@timestamp"
        }

        mutate {
            replace => { "message" => "%{@timestamp} %{sub_message}" }
        }
    }

There is still some issue in above snippet. Please help me to fix it.

magnusbaeck · March 31, 2016, 6:42pm

I have updated as below based on your review comments.

No you haven't. What you're trying is quite far from Kirill's example.

nagesh · March 31, 2016, 7:02pm

Earlier I was trying to add_filed and remove_filed for the message. I think it is not correct, so updated as below

grok {
patterns_dir => "./patterns"
match => {"sub_message" => "%{CF_DIRECT_PULL_FORMAT2}" }
}
if[sub_message]{

    date {
        match => [ "timestamp", "yyyy-MM-dd'T'HH:mm:ss.SSSSSSZZ", "ISO8601" ]
        target => "@timestamp"
    }

    mutate {
        add_field => { "message" => "%{@timestamp} %{sub_message}" }
    }
}

Is this fine ?

Thanks,
Nagesh Mandava.

magnusbaeck · March 31, 2016, 8:31pm

I have twice asked you to follow the previously posted example, and yet you are not doing that. I don't have time for this anymore.

nagesh · April 6, 2016, 4:37pm

I have made following changes and it worked for me.

grok {
patterns_dir => "./patterns"
match => {"message" => "%{CF_DIRECT_PULL_FORMAT3}" }
}
mutate{
convert => ["timestamp", "string"]
}
date {
locale => "en"
match => ["timestamp", "ISO8601"]
timezone => "Europe/Vienna"
target => "timestamp"
}
grok {
patterns_dir => "./patterns"
match => {"sub_message" => "%{CF_DIRECT_PULL_FORMAT2}" }
}
mutate {
replace => { "message" => "%{timestamp} %{sub_message}" }
}