# Logstash custom DATE fields (extracted by regex or custom patterns) how to convert it to DATE field

**URL:** <https://discuss.elastic.co/t/logstash-custom-date-fields-extracted-by-regex-or-custom-patterns-how-to-convert-it-to-date-field/348419>\
**Category:** Elasticsearch\
**Created:** [December 1, 2023, 10:10am UTC](https://discuss.elastic.co/t/logstash-custom-date-fields-extracted-by-regex-or-custom-patterns-how-to-convert-it-to-date-field/348419 "2023-12-01T10:10:55Z")\
**Posts on this page:** 9\
**Page:** 1

<div class="post-metadata">

**Author:** ![elk1985](https://avatars.discourse-cdn.com/v4/letter/e/58956e/32.png) [@elk1985](https://discuss.elastic.co/u/elk1985)\
**Post date:** [December 1, 2023, 10:10am UTC](https://discuss.elastic.co/t/logstash-custom-date-fields-extracted-by-regex-or-custom-patterns-how-to-convert-it-to-date-field/348419/1 "2023-12-01T10:10:55Z")

</div>

Hello everyone. Currently, I'm in the stage of writing custom Grok filters in Logstash. I have many different logs - some of them have a date format that fits ISO8601 format. But unfortunately when I use this format in my first grok filter in Logstash sometimes I get the date parsed not from the beginning of the file but from the message field (for example 11-02-1969 or whatever fits this format).  
To have full control over what date I want to get from the log I'm using regex with %{YEAR} (day, time, etc.) ready patterns and also I use custom-defined patterns like (?\<field\_name\>\w{24}\s).

The thing is that when I create those fields they are not sortable in Kibana they are Text format not date. Is there any way to make those fields DATE format so they can be recognized and sorted in Kibana column similar to @timestamp?  
Or anybody have other ideas?  
Regards

---

<div class="post-metadata">

**Author:** ![leandrojmp](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/leandrojmp/32/107231_2.png) [@leandrojmp](https://discuss.elastic.co/u/leandrojmp)\
**Post date:** [December 1, 2023, 12:34pm UTC](https://discuss.elastic.co/t/logstash-custom-date-fields-extracted-by-regex-or-custom-patterns-how-to-convert-it-to-date-field/348419/2 "2023-12-01T12:34:05Z")

</div>

Hello,

You didn't share any sample message nor the grok pattern you are trying.

Please share some sample message and your current configuration.

---

<div class="post-metadata">

**Author:** ![elk1985](https://avatars.discourse-cdn.com/v4/letter/e/58956e/32.png) [@elk1985](https://discuss.elastic.co/u/elk1985)\
**Post date:** [December 1, 2023, 12:51pm UTC](https://discuss.elastic.co/t/logstash-custom-date-fields-extracted-by-regex-or-custom-patterns-how-to-convert-it-to-date-field/348419/3 "2023-12-01T12:51:06Z")

</div>

Sorry.  
grok {  
match =\> { "message" =\> "^[%{MONTHNUM:m}\s\*/%{MONTHDAY:d}\s\*/%{YEAR:y}\s\*%{TIME:t}\s\*%{WORD:tz}]" }  
add\_field =\> { "real.timestamp" =\> "%{d}/%{m}/%{y} %{t} %{tz}" }  
}  
How to convert real.timestamp to DATE type field ?  
When I use  
grok {  
match =\> { "message" =\> "%{TIMESTAMP\_ISO8601:real.timestamp}" }  
}

For example in this message [11/13/23 11:12:23:923 CET] some text value=1977-11-21 01:22:11.0  
it takes not the date from the beginning of the new line in the file but from inside message

---

<div class="post-metadata">

**Author:** ![elk1985](https://avatars.discourse-cdn.com/v4/letter/e/58956e/32.png) [@elk1985](https://discuss.elastic.co/u/elk1985)\
**Post date:** [December 5, 2023, 7:49am UTC](https://discuss.elastic.co/t/logstash-custom-date-fields-extracted-by-regex-or-custom-patterns-how-to-convert-it-to-date-field/348419/4 "2023-12-05T07:49:00Z")

</div>

Hey. Anybody have some ideas ?

---

<div class="post-metadata">

**Author:** ![Rios](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/rios/32/95745_2.png) [@Rios](https://discuss.elastic.co/u/Rios)\
**Post date:** [December 5, 2023, 8:14am UTC](https://discuss.elastic.co/t/logstash-custom-date-fields-extracted-by-regex-or-custom-patterns-how-to-convert-it-to-date-field/348419/5 "2023-12-05T08:14:41Z")

</div>

You can use something like this:  
`\[%{DATESTAMP:date}%{SPACE}%{WORD:tz}\]%{SPACE}`

If you need string to date conversion,use the date plugin, something like this:

```auto
    date {
        match => ["date", "MM/dd/yy HH:mm:ss,SSS"]
        timezone => "CET" # or %{tz}
    }

```

---

<div class="post-metadata">

**Author:** ![elk1985](https://avatars.discourse-cdn.com/v4/letter/e/58956e/32.png) [@elk1985](https://discuss.elastic.co/u/elk1985)\
**Post date:** [December 5, 2023, 11:21am UTC](https://discuss.elastic.co/t/logstash-custom-date-fields-extracted-by-regex-or-custom-patterns-how-to-convert-it-to-date-field/348419/6 "2023-12-05T11:21:06Z")

</div>

Still my field real.timestamp is keyword type. I can't sort it (old new ) like @timestamp field ☹

Best option for me would be using this patterns and convert the result to timestamp field type.

```auto
grok {
match => { "message" => "^[%{MONTHNUM:m}\s*/%{MONTHDAY:d}\s*/%{YEAR:y}\s*%{TIME:t}\s*%{WORD:tz}]" }
add_field => { "real.timestamp" => "%{d}/%{m}/%{y} %{t} %{tz}" }
}

```

But how to do it ? I need to keep for example format day/month/year hour:minutes:seconds:miliseconds  
My logs comes from many systems I need to unify date format in kibana across whole system.

---

<div class="post-metadata">

**Author:** ![Rios](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/rios/32/95745_2.png) [@Rios](https://discuss.elastic.co/u/Rios)\
**Post date:** [December 5, 2023, 11:31am UTC](https://discuss.elastic.co/t/logstash-custom-date-fields-extracted-by-regex-or-custom-patterns-how-to-convert-it-to-date-field/348419/7 "2023-12-05T11:31:09Z")

</div>

If you like [real][timestamp] as the date type, you have to convert. Also you have to delete data view and index or do reindex. By the way, I have corrected the time format to HH:mm:ss:SSS

```auto
  date {
        match => ["date", "MM/dd/yy HH:mm:ss:SSS"]
        timezone => "CET" # or %{tz}
        target => "[real][timestamp]"
    }

```

---

<div class="post-metadata">

**Author:** ![elk1985](https://avatars.discourse-cdn.com/v4/letter/e/58956e/32.png) [@elk1985](https://discuss.elastic.co/u/elk1985)\
**Post date:** [December 7, 2023, 12:45pm UTC](https://discuss.elastic.co/t/logstash-custom-date-fields-extracted-by-regex-or-custom-patterns-how-to-convert-it-to-date-field/348419/8 "2023-12-07T12:45:28Z")

</div>

Thanks ! i mixed it up a bit with my config and it worked.

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\
**Post date:** [January 4, 2024, 12:46pm UTC](https://discuss.elastic.co/t/logstash-custom-date-fields-extracted-by-regex-or-custom-patterns-how-to-convert-it-to-date-field/348419/9 "2024-01-04T12:46:16Z")

</div>

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.