Trouble parsing date with date filter

[jclose (Jason)]

I'm having trouble with some date parsing. The entry I'm getting doesn't have a year in the datestamp, so I'm trying to pull the year out of @timestamp, and then put the proper timestamp back into @timestamp.

So my date format looks like:
Mar 22 15:09:49

What I have done is this:

filter {
dissect {
mapping => [ "@timestamp", "%{year}-%{?MON}-%{?DAY}T%{?HOUR}:%{?SEC}:%{?MIN}%{?REMAINDER}" ]
}
grok {
match => ["message", "%{MONTH:month} %{MONTHDAY:day} %{NOTSPACE:time} %{GREEYDATA:msg}" ]
}
mutate {
add_field => {
"gen_datetime" => "%{year}-%{month}-%{day} %{time}"
}
remove_field => [ "year", "month", "day", "time" ]
}
date {
match => [ "gen_datetime", "yyyy-MMM-dd HH:mm:ss" ]
target => "@timestamp"
}
}

The idea is to put the log values into gen_datetime, and then use that to move back into @timestamp.

The error I am getting is: "reason"=>"failed to parse [gen_datetime]", "caused_by"=>{"type"=>"illegal_argument_exception", "reason"=>"Invalid format: "2017-Mar-22 15:09:49" is malformed at "-Mar-22 15:09:49""}

To me, it looks right. Any clue on what I'm doing wrong?

[magnusbaeck (Magnus Bäck)]

It looks like Elasticsearch has mapped gen_datetime as a timestamp field but isn't able to parse "2017-Mar-22 15:09:49" as a timestamp. Since you probably don't want to keep that field anyway I suggest you add remove_field => ["gen_datetime"] to your date filter so that the field is deleted after a successful parsing operation.

[jclose (Jason)]

I'm creating the "gen_datetime" field with the following

mutate {
add_field => { "gen_datetime" => "%{year}-%{month}-%{day} %{time}"
}

I'm creating that field, and filling it with the values from other fields.

[magnusbaeck (Magnus Bäck)]

Yes, that was clear from your previous post.

[jclose (Jason)]

So after playing with this for a long while, I figured out the issue.

I don't quite know why the following works, but it does...

To create the gen_datetime, I used:

add_field => { "gen_datetime" => "%{year}-%{month}-%{day} %{time} America/Chicago" }

And then for the date creation, I used:

match => [ "gen_datetime", "yyyy-MMM-dd HH:mm:ss ZZZ" ]
target => "@timestamp"

This works.