# Logstash timestamp error when CEST is at the end

**URL:** https://discuss.elastic.co/t/logstash-timestamp-error-when-cest-is-at-the-end/27843
**Category:** Logstash
**Created:** [August 21, 2015, 3:22pm UTC](https://discuss.elastic.co/t/logstash-timestamp-error-when-cest-is-at-the-end/27843 "2015-08-21T15:22:15Z")
**Posts on this page:** 7
**Page:** 1

<div class="post-metadata">

### Author: ![michaelzoet](https://avatars.discourse-cdn.com/v4/letter/m/dbc845/32.png) [@michaelzoet](https://discuss.elastic.co/u/michaelzoet)
#### Post date: [August 21, 2015, 3:22pm UTC](https://discuss.elastic.co/t/logstash-timestamp-error-when-cest-is-at-the-end/27843/1 "2015-08-21T15:22:16Z")

</div>

Hi all,

I have a weird timestamp problem and maybe someone can help on this:

The PostgreSQL logs are in CSV format and I get a fixed date string from Postgres with CEST at the end.  
How do I need to set the timestamp expression in the date filter to catch/accept CEST?

An example log message:  
2015-08-21 17:02:22.753 CEST,"demo","demo",18372,"XXX.XXX.YYY.ZZZ:51716",55d73c07.47c4,3,"idle",2015-08-21 16:56:07 CEST,,0,LOG,00000,"Verbindungsende: Sitzungszeit: 0:06:15.000 Benutzer=demo Datenbank=demo Host=XXX.XXX.YYY.YYY port=51716",,,,,,,,,""

An example error message from Logstash (does not correspond to the message above but it would have the same meaning):  
{:timestamp=\>"2015-08-21T16:41:02.152000+0200", :message=\>"Failed parsing date from field", :field=\>"timestamp", :value=\>"2015-08-21 16:41:01.373 CEST", :exception=\>"Invalid format: "2015-08-21 16:41:01.373 CEST" is malformed at "CEST"", :config\_parsers=\>"YYYY-MM-DD HH:mm:ss.SSS z", :config\_locale=\>"default=en\_US", :level=\>:warn}

The corresponding logstash.conf snippet:  
...  
filter {  
...  
if [type] =~ "postgres" {  
csv {  
columns =\> ["timestamp", "postgres\_username", "postgres\_databasename", "postgres\_process\_id", "postgres\_connection\_from", "postgres\_session\_id", "postgres\_session\_line\_number", "postgres\_command\_tag", "postgres\_session\_start\_time", "postgres\_virtual\_transaction\_id", "postgres\_transaction\_id", "postgres\_error\_severity", "postgres\_sql\_state\_code", "postgres\_message", "postgres\_detail", "postgres\_hint", "postgres\_internal\_query", "postgres\_internal\_query\_pos", "postgres\_context", "postgres\_query", "postgres\_query\_pos", "postgres\_location", "postgres\_application\_name"]  
}  
date {  
match =\> ["timestamp", "YYYY-MM-DD HH:mm:ss.SSS z"]  
}  
}  
...

Thx in advance,  
Michael

---

<div class="post-metadata">

### Author: ![magnusbaeck](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/magnusbaeck/32/44943_2.png) [@magnusbaeck](https://discuss.elastic.co/u/magnusbaeck)
#### Post date: [August 21, 2015, 5:26pm UTC](https://discuss.elastic.co/t/logstash-timestamp-error-when-cest-is-at-the-end/27843/2 "2015-08-21T17:26:53Z")

</div>

Quoting the [Joda-Time documentation](http://joda-time.sourceforge.net/apidocs/org/joda/time/format/DateTimeFormat.html):

> Zone names: Time zone names ('z') cannot be parsed

So, unless you can hardcode the timezone in the date filter configuration I think you need to translate the timezone names into offsets and include the offset in the timestamp to be parsed.

---

<div class="post-metadata">

### Author: ![michaelzoet](https://avatars.discourse-cdn.com/v4/letter/m/dbc845/32.png) [@michaelzoet](https://discuss.elastic.co/u/michaelzoet)
#### Post date: [August 24, 2015, 12:29pm UTC](https://discuss.elastic.co/t/logstash-timestamp-error-when-cest-is-at-the-end/27843/3 "2015-08-24T12:29:57Z")

</div>

THX for the info. I will think about a solution later. At the moment the entries are there with two seconds difference and we can live with that at the moment.

---

<div class="post-metadata">

### Author: ![zappe](https://avatars.discourse-cdn.com/v4/letter/z/439d5e/32.png) [@zappe](https://discuss.elastic.co/u/zappe)
#### Post date: [October 28, 2015, 9:17am UTC](https://discuss.elastic.co/t/logstash-timestamp-error-when-cest-is-at-the-end/27843/4 "2015-10-28T09:17:53Z")

</div>

There seems to be some kind of fix for this, [http://stackoverflow.com/questions/13181970/joda-datetime-parse-date-with-cest-in-string/13182191#13182191](http://stackoverflow.com/questions/13181970/joda-datetime-parse-date-with-cest-in-string/13182191#13182191)

But I'm not sure which version of joda-time Logstash uses?

---

<div class="post-metadata">

### Author: ![rtoma](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/rtoma/32/44812_2.png) [@rtoma](https://discuss.elastic.co/u/rtoma)
#### Post date: [February 2, 2016, 11:10am UTC](https://discuss.elastic.co/t/logstash-timestamp-error-when-cest-is-at-the-end/27843/5 "2016-02-02T11:10:43Z")

</div>

Here is my fix:

First I grok the logline and separate the datetime, timezone and the rest.

```
grok {
  match => ['message', "%{DATA:timestamp} %{WORD:tz} ........"]
}

```

In my zone the 'tz' variable now contains 'CET' or 'CEST'.

With the translate filter I lookup the corresponding offset.  
Note: you need to extend the dictionary with your own relevant zones + offsets.

```
translate {
  field => 'tz'
  destination => 'tz_num'
  dictionary => [
    'CET', '+0100',
    'CEST', '+0200'
  ]
}

```

Append the offset to the grokked timestamp:

```
mutate {
  replace => ['timestamp', '%{timestamp} %{tz_num}']
}

```

And now parse it as usual:

```
date {
  match => [ 'timestamp', 'yyyy-MM-dd HH:mm:ss.SSS Z' 
}

```

And as bonus:

```
mutate {
  remove => ['tz', 'tz_num']
}

```

Profit.

Hope this helps?

---

<div class="post-metadata">

### Author: ![matejzero](https://avatars.discourse-cdn.com/v4/letter/m/e9bcb4/32.png) [@matejzero](https://discuss.elastic.co/u/matejzero)
#### Post date: [August 2, 2016, 12:14pm UTC](https://discuss.elastic.co/t/logstash-timestamp-error-when-cest-is-at-the-end/27843/6 "2016-08-02T12:14:13Z")

</div>

This helped me a lot when parsing Cisco logs.

Unfortunately Cisco doesn't have support for numbered timezones:S

---

<div class="post-metadata">

### Author: ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)
#### Post date: [July 6, 2017, 4:45am UTC](https://discuss.elastic.co/t/logstash-timestamp-error-when-cest-is-at-the-end/27843/7 "2017-07-06T04:45:20Z")

</div>


