# Logstash filter date "atypical" failure

**URL:** <https://discuss.elastic.co/t/logstash-filter-date-atypical-failure/215039>\
**Category:** Logstash\
**Created:** [January 14, 2020, 7:49pm UTC](https://discuss.elastic.co/t/logstash-filter-date-atypical-failure/215039 "2020-01-14T19:49:26Z")\
**Posts on this page:** 6\
**Page:** 1

<div class="post-metadata">

**Author:** ![blenolopes](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/blenolopes/32/60799_2.png) [@blenolopes](https://discuss.elastic.co/u/blenolopes)\
**Post date:** [January 14, 2020, 7:49pm UTC](https://discuss.elastic.co/t/logstash-filter-date-atypical-failure/215039/1 "2020-01-14T19:49:26Z")

</div>

Hello logstashers,

I have an "atypical" problem with a particular log format using date filter. Follow the tests.  
`
bin/logstash --version
logstash 7.5.0
`  
**Success**  
`
~$ echo "14/01/2020 16:03:00,143" | bin/logstash -e 'input { stdin {} } filter { date { match => ["message", "dd/MM/yyyy HH:mm:ss,SSS"] } }'
`  
`
{
"message" => "14/01/2020 16:03:00,143",
"@version" => "1",
"@timestamp" => 2020-01-14T19:03:00.143Z,
"host" => "myhost"
}
`

**Failure**  
`
~$ echo "[14/01/2020 16:03:00,143 XXX] [XXX] MSG" | bin/logstash -e 'input { stdin {} } filter { date { match => ["message", "dd/MM/yyyy HH:mm:ss,SSS"] } }'
`  
`
{
"@timestamp" => 2020-01-14T19:42:51.585Z,
"@version" => "1",
"message" => "[14/01/2020 16:03:00,143 XXX] [XXX] MSG",
"tags" => [
[0] "_dateparsefailure"
],
"host" => "myhost"
}
`  
Where am i going wrong?

Grateful for the help and attention. 😉

---

<div class="post-metadata">

**Author:** ![Badger](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/badger/32/25190_2.png) [@Badger](https://discuss.elastic.co/u/Badger)\
**Post date:** [January 14, 2020, 8:10pm UTC](https://discuss.elastic.co/t/logstash-filter-date-atypical-failure/215039/2 "2020-01-14T20:10:53Z")

</div>

> [@blenolopes](#):
>
> Where am i going wrong?

The pattern in the date filter has to match the entire field. Your [message] field in the second example includes much more stuff that your pattern does not match. I would use dissect to extract the date from [message]

```
dissect { mapping => { "message" => "[%{[@metadata][timestamp]} %{+%{[@metadata][timestamp]} %{}" } }
date { match => ["[@metadata][timestamp]", "dd/MM/yyyy HH:mm:ss,SSS" ] }

```

---

<div class="post-metadata">

**Author:** ![blenolopes](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/blenolopes/32/60799_2.png) [@blenolopes](https://discuss.elastic.co/u/blenolopes)\
**Post date:** [January 15, 2020, 4:31pm UTC](https://discuss.elastic.co/t/logstash-filter-date-atypical-failure/215039/3 "2020-01-15T16:31:23Z")

</div>

Hello Badger,

Thanks for the answer. But I still have the problem. Testing logstash generates an exception because it cannot identify the field passed in mapping dissect.

```auto
[ERROR] 2020-01-15 13:20:38.686 [[main]>worker0] Dissector - Dissect threw an exception {exception=org.logstash.FieldReference$IllegalSyntaxException: Invalid FieldReference: `%{[@metadata][timestamp]`, backtrace=org.logstash.FieldReference$StrictTokenizer.tokenize(FieldReference.java:283)
   org.logstash.FieldReference.parse(FieldReference.java:184)
   org.logstash.FieldReference.parseToCache(FieldReference.java:175)
   org.logstash.FieldReference.from(FieldReference.java:107)
   org.logstash.Event.includes(Event.java:194)
   org.logstash.dissect.fields.AppendField.append(AppendField.java:42)
   org.logstash.dissect.Dissector.dissect(Dissector.java:142)
   org.logstash.dissect.JavaDissectorLibrary$RubyDissect.invokeDissection(JavaDissectorLibrary.java:180)
   org.logstash.dissect.JavaDissectorLibrary$RubyDissect.dissect(JavaDissectorLibrary.java:129)
   org.logstash.dissect.JavaDissectorLibrary$RubyDissect.lambda$dissectMulti$0(JavaDissectorLibrary.java:148)
   java.base/java.util.Spliterators$ArraySpliterator.forEachRemaining(Spliterators.java:948)
   java.base/java.util.stream.ReferencePipeline$Head.forEach(ReferencePipeline.java:658)}
{
    "@timestamp" => 2020-01-15T16:20:37.160Z,
      "@version" => "1",
       "message" => "[14/01/2020 16:03:00,143 XXX] [XXX] MSG",
          "host" => "myhost",
          "tags" => [
        [0] "_dissectfailure",
        [1] "_dateparsefailure"
    ]
}

```

I tried to remove [@metadata], but the problem persists. Another question, why use [@metadata] before the name/alias of the [timestamp] field?

My test line:

```auto
echo "[14/01/2020 16:03:00,143 XXX] [XXX] MSG" | bin/logstash -e 'input { stdin {} } filter { dissect { mapping => { "message" => "[%{[@metadata][timestamp]} %{+%{[@metadata][timestamp]} %{}" } } date { match => ["[@metadata][timestamp]", "dd/MM/yyyy HH:mm:ss,SSS" ] } }'

```

Grateful for the attention.

---

<div class="post-metadata">

**Author:** ![blenolopes](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/blenolopes/32/60799_2.png) [@blenolopes](https://discuss.elastic.co/u/blenolopes)\
**Post date:** [January 15, 2020, 5:27pm UTC](https://discuss.elastic.co/t/logstash-filter-date-atypical-failure/215039/4 "2020-01-15T17:27:32Z")

</div>

Hi Badger,

I managed to solve. Following solution. Thank you for your help.

```auto
dissect {
    mapping => {
      "message" => "[%{timestamp} %{+timestamp} %{}"
    }
  }

  date {
    match => ["timestamp", "dd/MM/yyyy HH:mm:ss,SSS"]
    target => "@timestamp"
  }

```

---

<div class="post-metadata">

**Author:** ![Badger](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/badger/32/25190_2.png) [@Badger](https://discuss.elastic.co/u/Badger)\
**Post date:** [January 15, 2020, 5:38pm UTC](https://discuss.elastic.co/t/logstash-filter-date-atypical-failure/215039/5 "2020-01-15T17:38:41Z")

</div>

Yeah, I had an extract %{ in there. It should have been

```
dissect { mapping => { "message" => "[%{[@metadata][timestamp]} %{+[@metadata][timestamp]} %{}" } }

```

I use [@metadata] since that can be referenced in the filters, but it not included in the output. It probably does not add any value to have a [timestamp] field in your final documents.

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\
**Post date:** [February 12, 2020, 5:38pm UTC](https://discuss.elastic.co/t/logstash-filter-date-atypical-failure/215039/6 "2020-02-12T17:38:43Z")

</div>

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.