# Parsing different syslog date formats

**URL:** <https://discuss.elastic.co/t/parsing-different-syslog-date-formats/63431>\
**Category:** Logstash\
**Created:** [October 19, 2016, 6:38pm UTC](https://discuss.elastic.co/t/parsing-different-syslog-date-formats/63431 "2016-10-19T18:38:40Z")\
**Posts on this page:** 6\
**Page:** 1

<div class="post-metadata">

**Author:** ![pixelrebel](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/pixelrebel/32/14056_2.png) [@pixelrebel](https://discuss.elastic.co/u/pixelrebel)\
**Post date:** [October 19, 2016, 6:38pm UTC](https://discuss.elastic.co/t/parsing-different-syslog-date-formats/63431/1 "2016-10-19T18:38:40Z")

</div>

I have two types of timestamps coming into my logstash syslog input:

```auto
SYSLOGTIMESTAMP - "Oct 19 11:29:00"
TIMESTAMP_ISO8601 - "2016-10-19T18:31:52.519Z"

```

My grok below works for both:

```auto
        grok {
            match => { "message" => "(<%{NUMBER:syslog_event_id}>)?%{SYSLOGTIMESTAMP:syslog_timestamp} (%{SYSLOGHOST:syslog_hostname} )?%{DATA:syslog_program}(?:\[%{POSINT:syslog_pid}\])?:%{GREEDYDATA:syslog_message}" }
            match => { "message" => "(<%{NUMBER:syslog_event_id}>)?%{TIMESTAMP_ISO8601:syslog_timestamp} (%{SYSLOGHOST:syslog_hostname} )?%{DATA:syslog_program}(?:\[%{POSINT:syslog_pid}\])?:%{GREEDYDATA:syslog_message}" }
            add_field => ["received_at", "%{@timestamp}"]
            add_field => ["received_from", "%{host}"]
        }

```

And here's the date stanza which I think is where it's failing:

```auto
        date {
            match => ["syslog_timestamp", "MMM d HH:mm:ss", "MMM dd HH:mm:ss", "ISO8601"]
        }

```

The problem is that only one type makes it into the ES index. Whichever system is the first in the index wins. In today's index, the TIMESTAMP\_ISO8601 won, so here's the subsequent error for SYSLOGTIMESTAMP:

```auto
response=>{"create"=>{"_index"=>"syslog-2016.10.19", "_type"=>"syslog", "_id"=>"AVfeNnZTkUTjKMipILXD", "status"=>400, "error"=>{"type"=>"mapper_parsing_exception", "reason"=>"failed to parse [syslog_timestamp]", "caused_by"=>{"type"=>"illegal_argument_exception", "reason"=>"Invalid format: \"Oct 19 11:31:32\""}}}}, :level=>:warn}

```

What am I doing wrong here?

---

<div class="post-metadata">

**Author:** ![magnusbaeck](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/magnusbaeck/32/44943_2.png) [@magnusbaeck](https://discuss.elastic.co/u/magnusbaeck)\
**Post date:** [October 19, 2016, 8:01pm UTC](https://discuss.elastic.co/t/parsing-different-syslog-date-formats/63431/2 "2016-10-19T20:01:03Z")

</div>

I suggest you remove the `syslog_timestamp` field after you're done parsing it. You're storing the results into the `@timestamp` field so the unparsed field is hardly useful to keep around.

---

<div class="post-metadata">

**Author:** ![pixelrebel](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/pixelrebel/32/14056_2.png) [@pixelrebel](https://discuss.elastic.co/u/pixelrebel)\
**Post date:** [October 19, 2016, 8:59pm UTC](https://discuss.elastic.co/t/parsing-different-syslog-date-formats/63431/3 "2016-10-19T20:59:01Z")

</div>

Well, that solves the problem. I was keeping the field there while I am still learning to make sure the parse worked.

Curious, any idea why it was failing to insert into ES? It's just a string field, not sure why formatting matters.

---

<div class="post-metadata">

**Author:** ![magnusbaeck](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/magnusbaeck/32/44943_2.png) [@magnusbaeck](https://discuss.elastic.co/u/magnusbaeck)\
**Post date:** [October 20, 2016, 4:02am UTC](https://discuss.elastic.co/t/parsing-different-syslog-date-formats/63431/4 "2016-10-20T04:02:53Z")

</div>

I suspect ES was trying to parse it as a date. I think the dynamic mapper will map a field as a date if the first sample it sees looks like a date, which it perhaps did in your case, but when the same field in following documents can't be parsed the same way this is what you get. What's the current mapping of the field?

---

<div class="post-metadata">

**Author:** ![pixelrebel](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/pixelrebel/32/14056_2.png) [@pixelrebel](https://discuss.elastic.co/u/pixelrebel)\
**Post date:** [October 20, 2016, 7:13am UTC](https://discuss.elastic.co/t/parsing-different-syslog-date-formats/63431/5 "2016-10-20T07:13:21Z")

</div>

Okay, that makes sense. That explains why the first entry wins. I don't have a template for my syslog index, so I guess this is ES default behavior. Thanks @magnusbaeck

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\
**Post date:** [July 6, 2017, 4:33am UTC](https://discuss.elastic.co/t/parsing-different-syslog-date-formats/63431/6 "2017-07-06T04:33:28Z")

</div>