# Set timestamp from message

**URL:** https://discuss.elastic.co/t/set-timestamp-from-message/269073
**Category:** Logstash
**Created:** [April 1, 2021, 6:48pm UTC](https://discuss.elastic.co/t/set-timestamp-from-message/269073 "2021-04-01T18:48:34Z")
**Posts on this page:** 6
**Page:** 1

<div class="post-metadata">

### Author: ![Andrey\_RF](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/andrey_rf/32/86484_2.png) [@Andrey\_RF](https://discuss.elastic.co/u/Andrey_RF)
#### Post date: [April 1, 2021, 6:48pm UTC](https://discuss.elastic.co/t/set-timestamp-from-message/269073/1 "2021-04-01T18:48:34Z")

</div>

I have a pipeline to receive logs from different programs with different timestamp formats.

```auto
    input {
      beats {
        port => 5000
      }
    }

    filter {
      date {
        match => ["message", "yyyy-MM-dd HH:mm:ss,SSS", "yyyy-MM-dd HH:mm:ss", "yyyy-MM-dd HH:mm:ss.SSS", "ISO8601"]
      }
    }

    output {
      elasticsearch {
        hosts => ["elasticsearch:9200"]
        index => "logstash-%{[host][hostname]}"
      }
    }

```

It puts log in elasticsearch but logs in ES have @timestamp equal to the time when they were saved in US, but not the creation date.

I have some logs examples:

```auto
[2020-08-28 14:00:02,940: ERROR/MainProcess] consumer: Cannot connect to ...

```

```auto
2021-04-01 15:06:49 =SUPERVISOR REPORT====
	Some other text

```

---

<div class="post-metadata">

### Author: ![stephenb](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/stephenb/32/40856_2.png) [@stephenb](https://discuss.elastic.co/u/stephenb)
#### Post date: [April 1, 2021, 7:14pm UTC](https://discuss.elastic.co/t/set-timestamp-from-message/269073/2 "2021-04-01T19:14:17Z")

</div>

Hi @Andrey_RF Welcome to the community.

I suspect you need to add the [timezone](https://www.elastic.co/guide/en/logstash/current/plugins-filters-date.html#plugins-filters-date-timezone) setting as part of your `date` filter, otherwise it is making an assumption.

Also please keep in mind all dates in elasticsearch are stored as UTC. If you look at them in Kibana Apps they will be displayed in your local timezone, if you display them directly via curl / API they will show in UTC

---

<div class="post-metadata">

### Author: ![Andrey\_RF](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/andrey_rf/32/86484_2.png) [@Andrey\_RF](https://discuss.elastic.co/u/Andrey_RF)
#### Post date: [April 1, 2021, 7:44pm UTC](https://discuss.elastic.co/t/set-timestamp-from-message/269073/3 "2021-04-01T19:44:32Z")

</div>

Hello. Thanks 🙂

I added timezone like this

```auto
filter {
  date {
    match => ["message", "yyyy-MM-dd HH:mm:ss,SSS", "yyyy-MM-dd HH:mm:ss", "yyyy-MM-dd HH:mm:ss.SSS", "ISO8601"]
    timezone => "Europe/Moscow"
  }
}

```

and it doesn't help.

Yeah, I keep in my mind that ES are stored as UTC. But I had logs for 2020 and they have `timestamp` like April 1st 2021.

I can show you full log from ES if that help you.

---

<div class="post-metadata">

### Author: ![stephenb](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/stephenb/32/40856_2.png) [@stephenb](https://discuss.elastic.co/u/stephenb)
#### Post date: [April 1, 2021, 8:14pm UTC](https://discuss.elastic.co/t/set-timestamp-from-message/269073/4 "2021-04-01T20:14:16Z")

</div>

I think I missed the obvious.

The code below will not extract the time out of your `message` field, you will need to parse the `message` first with a [grok filter](https://www.elastic.co/guide/en/logstash/current/plugins-filters-grok.html) and then pass the field with the timestamp into it the date filter. So what is mostly likely happening that date filter is completely failing there is probably a tag in the document in elastic something like `_dateparsefailure` as so since it is failing it is just inserting the "now" time.

This wont work... because you are passing in a whole `message` not just the `timestamp` field

```auto
filter {
  date {
    match => ["message", "yyyy-MM-dd HH:mm:ss,SSS", "yyyy-MM-dd HH:mm:ss", "yyyy-MM-dd HH:mm:ss.SSS", "ISO8601"]
    timezone => "Europe/Moscow"
  }
}

```

So your filter should look something like

```auto

filter {
  grok {
      match => {
      "message" => [
        # parse the message <!---- THESE ARE JUST EXAMPLES there can me multiple
      "%{TIMESTAMP_ISO8601:timestamp} %{LOGLEVEL:log-level} \[%{DATA:class}\]:%{GREEDYDATA:message}",
      "%{TIMESTAMP_ISO8601:timestamp} %{LOGLEVEL:log-level} \[%{DATA:class}\]:%{GREEDYDATA:message}"
     ]
    }
  }

  # Now you can parse the timestamp
 date {
    match => ["timestamp", "yyyy-MM-dd HH:mm:ss,SSS", "yyyy-MM-dd HH:mm:ss", "yyyy-MM-dd HH:mm:ss.SSS", "ISO8601"]
    timezone => "Europe/Moscow"
  }
}

```

There are some good examples [Here](https://www.elastic.co/guide/en/logstash/current/config-examples.html#_processing_syslog_messages) also lots of [good articles on line](https://logz.io/blog/logstash-grok/)

---

<div class="post-metadata">

### Author: ![Andrey\_RF](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/andrey_rf/32/86484_2.png) [@Andrey\_RF](https://discuss.elastic.co/u/Andrey_RF)
#### Post date: [April 2, 2021, 11:12am UTC](https://discuss.elastic.co/t/set-timestamp-from-message/269073/5 "2021-04-02T11:12:38Z")

</div>

> [@stephenb](#):
>
> `HESE ARE JUST EXAMPLES there can me multiple`

Thanks for your quick reply. I realized my mistake and now it's working.

---

<div class="post-metadata">

### Author: ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)
#### Post date: [April 30, 2021, 11:12am UTC](https://discuss.elastic.co/t/set-timestamp-from-message/269073/6 "2021-04-30T11:12:57Z")

</div>

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.
