Logstash custom DATE fields (extracted by regex or custom patterns) how to convert it to DATE field

Elastic Stack Elasticsearch
1

Hello everyone. Currently, I'm in the stage of writing custom Grok filters in Logstash. I have many different logs - some of them have a date format that fits ISO8601 format. But unfortunately when I use this format in my first grok filter in Logstash sometimes I get the date parsed not from the beginning of the file but from the message field (for example 11-02-1969 or whatever fits this format).
To have full control over what date I want to get from the log I'm using regex with %{YEAR} (day, time, etc.) ready patterns and also I use custom-defined patterns like (?<field_name>\w{24}\s).

The thing is that when I create those fields they are not sortable in Kibana they are Text format not date. Is there any way to make those fields DATE format so they can be recognized and sorted in Kibana column similar to @timestamp?
Or anybody have other ideas?
Regards

2

Hello,

You didn't share any sample message nor the grok pattern you are trying.

Please share some sample message and your current configuration.

3

Sorry.
grok {
match => { "message" => "^[%{MONTHNUM:m}\s*/%{MONTHDAY:d}\s*/%{YEAR:y}\s*%{TIME:t}\s*%{WORD:tz}]" }
add_field => { "real.timestamp" => "%{d}/%{m}/%{y} %{t} %{tz}" }
}
How to convert real.timestamp to DATE type field ?
When I use
grok {
match => { "message" => "%{TIMESTAMP_ISO8601:real.timestamp}" }
}

For example in this message [11/13/23 11:12:23:923 CET] some text value=1977-11-21 01:22:11.0
it takes not the date from the beginning of the new line in the file but from inside message

4

Hey. Anybody have some ideas ?

5

You can use something like this:
\[%{DATESTAMP:date}%{SPACE}%{WORD:tz}\]%{SPACE}

If you need string to date conversion,use the date plugin, something like this:

    date {
        match => ["date", "MM/dd/yy HH:mm:ss,SSS"]
        timezone => "CET" # or %{tz}
    }
6

Still my field real.timestamp is keyword type. I can't sort it (old new ) like @timestamp field :frowning:

Best option for me would be using this patterns and convert the result to timestamp field type.

grok {
match => { "message" => "^[%{MONTHNUM:m}\s*/%{MONTHDAY:d}\s*/%{YEAR:y}\s*%{TIME:t}\s*%{WORD:tz}]" }
add_field => { "real.timestamp" => "%{d}/%{m}/%{y} %{t} %{tz}" }
}

But how to do it ? I need to keep for example format day/month/year hour:minutes:seconds:miliseconds
My logs comes from many systems I need to unify date format in kibana across whole system.

7

If you like [real][timestamp] as the date type, you have to convert. Also you have to delete data view and index or do reindex. By the way, I have corrected the time format to HH:mm:ss:SSS

  date {
        match => ["date", "MM/dd/yy HH:mm:ss:SSS"]
        timezone => "CET" # or %{tz}
        target => "[real][timestamp]"
    }
8

Thanks ! i mixed it up a bit with my config and it worked.

9

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.