Parsing different syslog date formats

Elastic Stack Logstash
1

I have two types of timestamps coming into my logstash syslog input:

SYSLOGTIMESTAMP   - "Oct 19 11:29:00"
TIMESTAMP_ISO8601 - "2016-10-19T18:31:52.519Z"

My grok below works for both:

        grok {
            match => { "message" => "(<%{NUMBER:syslog_event_id}>)?%{SYSLOGTIMESTAMP:syslog_timestamp} (%{SYSLOGHOST:syslog_hostname} )?%{DATA:syslog_program}(?:\[%{POSINT:syslog_pid}\])?:%{GREEDYDATA:syslog_message}" }
            match => { "message" => "(<%{NUMBER:syslog_event_id}>)?%{TIMESTAMP_ISO8601:syslog_timestamp} (%{SYSLOGHOST:syslog_hostname} )?%{DATA:syslog_program}(?:\[%{POSINT:syslog_pid}\])?:%{GREEDYDATA:syslog_message}" }
            add_field => [ "received_at", "%{@timestamp}" ]
            add_field => [ "received_from", "%{host}" ]
        }

And here's the date stanza which I think is where it's failing:

        date {
            match => [ "syslog_timestamp",  "MMM  d HH:mm:ss", "MMM dd HH:mm:ss", "ISO8601" ]
        }

The problem is that only one type makes it into the ES index. Whichever system is the first in the index wins. In today's index, the TIMESTAMP_ISO8601 won, so here's the subsequent error for SYSLOGTIMESTAMP:

response=>{"create"=>{"_index"=>"syslog-2016.10.19", "_type"=>"syslog", "_id"=>"AVfeNnZTkUTjKMipILXD", "status"=>400, "error"=>{"type"=>"mapper_parsing_exception", "reason"=>"failed to parse [syslog_timestamp]", "caused_by"=>{"type"=>"illegal_argument_exception", "reason"=>"Invalid format: \"Oct 19 11:31:32\""}}}}, :level=>:warn}

What am I doing wrong here?

2

I suggest you remove the syslog_timestamp field after you're done parsing it. You're storing the results into the @timestamp field so the unparsed field is hardly useful to keep around.

3

Well, that solves the problem. I was keeping the field there while I am still learning to make sure the parse worked.

Curious, any idea why it was failing to insert into ES? It's just a string field, not sure why formatting matters.

4

I suspect ES was trying to parse it as a date. I think the dynamic mapper will map a field as a date if the first sample it sees looks like a date, which it perhaps did in your case, but when the same field in following documents can't be parsed the same way this is what you get. What's the current mapping of the field?

5

Okay, that makes sense. That explains why the first entry wins. I don't have a template for my syslog index, so I guess this is ES default behavior. Thanks @magnusbaeck