# Problem with date filter to replace timestamp

**URL:** <https://discuss.elastic.co/t/problem-with-date-filter-to-replace-timestamp/169362>\
**Category:** Logstash\
**Created:** [February 21, 2019, 9:49am UTC](https://discuss.elastic.co/t/problem-with-date-filter-to-replace-timestamp/169362 "2019-02-21T09:49:52Z")\
**Posts on this page:** 8\
**Page:** 1

<div class="post-metadata">

**Author:** ![hackman61](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/hackman61/32/41095_2.png) [@hackman61](https://discuss.elastic.co/u/hackman61)\
**Post date:** [February 21, 2019, 9:49am UTC](https://discuss.elastic.co/t/problem-with-date-filter-to-replace-timestamp/169362/1 "2019-02-21T09:49:52Z")

</div>

Hello,

I have latest ELK docker image, and I send application logs from windows with filebeat, everything is almost fine 😉

I have this kinf of message :

```
2019-02-21 10:35:00,085 [3140] INFO - Début du batch EDNotif 
2019-02-21 10:35:00,993 [3140] INFO - Récupération des cheptels à traiter... 
2019-02-21 10:35:01,005 [3140] INFO - Récupération des cheptels à traiter... OK ! (15ms) 
2019-02-21 10:35:01,006 [3140] INFO - Récupération des événements à traiter par cheptels... 
2019-02-21 10:35:01,006 [3140] INFO - Récupération des événements à traiter par cheptels... OK ! (0ms) 
2019-02-21 10:35:01,007 [3140] INFO - Fin du batch EDNotif (Code retour : 0) 

```

I work with this Logstash config :

```
input {
  beats {
    port => 5044
    codec => plain {
     charset => "UTF-8"
     }
  }
}

    filter {
  if [fields][infra] {

    date {
        match => ["timestamp", "yyyy-MM-dd HH:mm:ss,SSS"]
        target => "@timestamp"
     }

    grok {
      match => { "message" => "%{TIMESTAMP_ISO8601}%{SPACE}%{EXIM_PID:PID} %{WORD:severity} %{GREEDYDATA:message}" }
      overwrite => "message"
    }

  }
}
output {
  if [fields][infra] {

    elasticsearch {
      hosts => ["localhost"]
      manage_template => false
      index => "osmos-%{+YYYY.MM.dd}"
    }
# file {
# path => "/etc/logstash/conf.d/osmos.log"
# }
  }
}

```

The message is correctly sended to ES, and in Kibana I can see my messages with fields.  
BUT I have a problem with Timestamp. My date filter is not applied. Kibana shows me this data :

 ![image](https://us1.discourse-cdn.com/elastic/original/3X/0/c/0c00980812b42ef4c756c1a55bea79c8da6c51b7.png)

The timestamp is not correct, so the sequence of events is not in the correct order. Can anybody helps me ?

Thank you 🙂

---

<div class="post-metadata">

**Author:** ![Badger](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/badger/32/25190_2.png) [@Badger](https://discuss.elastic.co/u/Badger)\
**Post date:** [February 21, 2019, 1:58pm UTC](https://discuss.elastic.co/t/problem-with-date-filter-to-replace-timestamp/169362/2 "2019-02-21T13:58:59Z")

</div>

The grok that parses the timestamp has to come before the date filter that is meant to parse it.

The grok filter does not capture the timestamp. Change %{TIMESTAMP\_ISO8601} to %{TIMESTAMP\_ISO8601:timestamp}. That said, that timestamp does not match TIMESTAMP\_ISO8601 so you need a different pattern.

It appears that your input is not UTF-8. Maybe ISO-8859-1?

---

<div class="post-metadata">

**Author:** ![hackman61](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/hackman61/32/41095_2.png) [@hackman61](https://discuss.elastic.co/u/hackman61)\
**Post date:** [February 21, 2019, 2:20pm UTC](https://discuss.elastic.co/t/problem-with-date-filter-to-replace-timestamp/169362/3 "2019-02-21T14:20:28Z")

</div>

Hello @Badger, thank you for your response !

You're right, I have modified filter like that :

```
  filter {
    if [fields][infra] {
        grok {
          match => { "message" => "%{TIMESTAMP_ISO8601:date_batch}%{SPACE}%{EXIM_PID:PID} %{WORD:severity} %{GREEDYDATA:message}" }
  overwrite => "message"
}

date {
    match => ["date_batch", "yyyy-MM-dd HH:mm:ss,SSS"]
# target => "@timestamp"
    add_field => ["date_OK"]
    }
  }
}

```

And now it works !! My message comes in the correct order 😉

For the encoding I have done this :

In filebeat :

```
encoding: ISO-8859-1

```

In Logstash beats input :

```
input {
  beats {
    port => 5044
    codec => plain {
     charset => "UTF-8"
     }
  }
}

```

> And everything is fine, thank you !!

---

<div class="post-metadata">

**Author:** ![hackman61](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/hackman61/32/41095_2.png) [@hackman61](https://discuss.elastic.co/u/hackman61)\
**Post date:** [February 21, 2019, 2:28pm UTC](https://discuss.elastic.co/t/problem-with-date-filter-to-replace-timestamp/169362/4 "2019-02-21T14:28:45Z")

</div>

Ooops, in fact it doesn't work, I'm not able to reproduce a working situation ☹

So my filter is like that now :

```
filter {
  if [fields][infra] {
    grok {
      match => { "message" => "%{TIMESTAMP_ISO8601:date_batch}%{SPACE}%{EXIM_PID:PID} %{WORD:severity} %{GREEDYDATA:message}" }
      overwrite => "message"
    }

    date {
        match => ["date_batch", "yyyy-MM-dd HH:mm:ss,SSS"]
        target => "@timestamp"

    }
  }
}

```

You said my timestamp is not ISO8601, is it because of milliseconds, wich are with a coma , and not a dot ?

---

<div class="post-metadata">

**Author:** ![Badger](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/badger/32/25190_2.png) [@Badger](https://discuss.elastic.co/u/Badger)\
**Post date:** [February 21, 2019, 2:58pm UTC](https://discuss.elastic.co/t/problem-with-date-filter-to-replace-timestamp/169362/5 "2019-02-21T14:58:13Z")

</div>

That filter works for me.

I was mistaken when I said the timestamp does not match ISO8601. It does match.

---

<div class="post-metadata">

**Author:** ![hackman61](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/hackman61/32/41095_2.png) [@hackman61](https://discuss.elastic.co/u/hackman61)\
**Post date:** [February 21, 2019, 3:04pm UTC](https://discuss.elastic.co/t/problem-with-date-filter-to-replace-timestamp/169362/6 "2019-02-21T15:04:31Z")

</div>

It's strange, it works, with one hour of delay 😃

 ![image](https://us1.discourse-cdn.com/elastic/original/3X/d/9/d989fcb93243c7f6945e93158d8fd28de57d4a96.png)

I think I must define a timezone ?

---

<div class="post-metadata">

**Author:** ![hackman61](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/hackman61/32/41095_2.png) [@hackman61](https://discuss.elastic.co/u/hackman61)\
**Post date:** [February 21, 2019, 3:10pm UTC](https://discuss.elastic.co/t/problem-with-date-filter-to-replace-timestamp/169362/7 "2019-02-21T15:10:42Z")

</div>

It's OK !!

With this :

```
date {
    match => ["date_batch", "yyyy-MM-dd HH:mm:ss,SSS"]
    timezone => "Europe/Paris"
    target => "@timestamp"

}

```

Every thing is fine 😃

Thank you very much !!

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\
**Post date:** [March 21, 2019, 3:10pm UTC](https://discuss.elastic.co/t/problem-with-date-filter-to-replace-timestamp/169362/8 "2019-03-21T15:10:43Z")

</div>

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.