# Logstash date filter don't parse time

**URL:** <https://discuss.elastic.co/t/logstash-date-filter-dont-parse-time/286064>\
**Category:** Logstash\
**Created:** [October 6, 2021, 6:58pm UTC](https://discuss.elastic.co/t/logstash-date-filter-dont-parse-time/286064 "2021-10-06T18:58:27Z")\
**Posts on this page:** 4\
**Page:** 1

<div class="post-metadata">

**Author:** ![AlexG](https://avatars.discourse-cdn.com/v4/letter/a/ecc23a/32.png) [@AlexG](https://discuss.elastic.co/u/AlexG)\
**Post date:** [October 6, 2021, 6:58pm UTC](https://discuss.elastic.co/t/logstash-date-filter-dont-parse-time/286064/1 "2021-10-06T18:58:27Z")

</div>

I have a log in msi installer format (time only, without date):

```auto
MSI (c) (F0:8C) [09:00:05:007]: Client-side and UI is none or basic: Running entire install on the server.
MSI (c) (F0:8C) [09:00:05:007]: Grabbed execution mutex.
MSI (c) (F0:8C) [09:00:05:007]: Cloaking enabled.
MSI (c) (F0:8C) [09:00:05:007]: Attempting to enable all disabled privileges before calling Install on Server
MSI (c) (F0:8C) [09:00:05:007]: Incrementing counter to disable shutdown. Counter after increment: 0

```

I want to get a time substring from the log and merge it with the current date.  
This is my filters config:

```auto
filter {
  if "curl" in [tags] {
    date {
      locale => "en"
      match => ["message", "HH:mm:ss:SSS"]
      timezone => "+0200"
      target => "logtime"
    }
    mutate {
         add_field => {"logdate" => "%{+YYYY-MM-dd}"}
         add_field => {"tmp_field" => "%{logdate}_%{logtime}"}
    }
    date {
      locale => "en"
      match => ["tmp_field", "YYYY-MM-dd_HH:mm:ss:SSS"]
      timezone => "+0200"
      remove_field => ["logdate","logtime","tmp_field"]
    }
  }
}

```

When loading logs into logstash, the filter does not work and I get the \_dateparsefailure tag for each event.  
What am I doing wrong?

---

<div class="post-metadata">

**Author:** ![Badger](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/badger/32/25190_2.png) [@Badger](https://discuss.elastic.co/u/Badger)\
**Post date:** [October 6, 2021, 7:57pm UTC](https://discuss.elastic.co/t/logstash-date-filter-dont-parse-time/286064/2 "2021-10-06T19:57:17Z")

</div>

> [@AlexG](#):
>
> What am I doing wrong?

Firstly, the pattern in a date filter has to match the entire field. In your first date filter you are only matching a small sub-string of [message], so it will not work and will add a \_dateparsefailure tag. You could try

```
dissect { mapping => { "message" => "%{}[%{logtime}]:%{}" } }

```

or you could pick it out using grok.

Secondly, when you supply an option to a filter more than once they are merged. The order in which they are evaluated depends on the logstash version. In some version [logdate] will be inserted into [tmp\_field] before it is set, and in other versions it will be referenced after it is set. If you want to control the order use two mutate filters.

```
mutate { add_field => {"logdate" => "%{+YYYY-MM-dd}"} }
mutate { add_field => {"tmp_field" => "%{logdate}_%{logtime}"} }

```

However, you do not need two mutates, you can do both sprintf references in one

```
mutate { add_field => {"tmp_field" => "%{+YYYY-MM-dd}_%{logtime}"} }

```

Note that you can create temporary fields in the [@metadata] field. These are available to the logstash pipeline but ignored by the output

```
mutate { add_field => {"[@metadata][timestamp]" => "%{+YYYY-MM-dd}_%{logtime}"} }

```

Then you do not need to use remove\_field in the date filter.

---

<div class="post-metadata">

**Author:** ![AlexG](https://avatars.discourse-cdn.com/v4/letter/a/ecc23a/32.png) [@AlexG](https://discuss.elastic.co/u/AlexG)\
**Post date:** [October 7, 2021, 10:17am UTC](https://discuss.elastic.co/t/logstash-date-filter-dont-parse-time/286064/3 "2021-10-07T10:17:19Z")

</div>

> [@Badger](#):
>
> so it will not work and will add a \_dateparsefailure tag. You could try

Thanks for the answer!  
I edited the config according to the recommendations:

```auto
if "curl" in [tags] {
    dissect {
      mapping => { "message" => "%{}[%{logtime}]:%{}" }
    }
    mutate {
      add_field => {"[@metadata][timestamp]" => "%{+YYYY-MM-dd}_%{logtime}"}
    }
    date {
      locale => "en"
      match => ["[@metadata][timestamp]", "YYYY-MM-dd_HH:mm:ss:SSS"]
      timezone => "+0200"
    }
  }

```

After that, a logtime field was created in the logs with a time substring from message.  
But the @timestamp field still contains the loading times of the files in logstash, the date filter did not work.  
Also, after adding dissect, the logs began to be loaded into elasticsearch for about an hour (previously it took a few seconds).

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\
**Post date:** [November 4, 2021, 10:18am UTC](https://discuss.elastic.co/t/logstash-date-filter-dont-parse-time/286064/4 "2021-11-04T10:18:15Z")

</div>

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.