java.lang.IllegalArgumentException: Invalid format - date problem

[rromanovic (Roman)]

Hi,
I'm having problems with storing date in "yyyy-MM-dd HH:mm:ss,SSS" format in index.

The format I'm reading from log file is "yyyy-MM-dd HH:mm:ss.SSSS", for example "2017-10-31 05:00:00.4216".

First, I'm creating field in which I store date in format "yyyy-MM-dd HH:mm:ss.SSS". After that I'm using gsub filter to change "." to "," in date to get format "yyyy-MM-dd HH:mm:ss,SSS", and lastly I'm using date filter to store field as a date.

Here is the logstash conf:
input {
beats {
port => "5043"
}
}

filter {
grok {
match => {"message" => "(?<datumGNBR>(\d{4}(-\d{2}){2} \d{2}:\d{2}:\d{2}.\d{3}))\d"}
}

mutate {
  gsub => ["datumGNBR", "\.", ","]
}

date {
  match => ["datumGNBR", "yyyy-MM-dd HH:mm:ss,SSS"]
}

}

output {
elasticsearch {
hosts => ["localhost:9200"]
index => "someindex"
}
}

datumGNBR is defined as follows in elasticsearch:
"datumGNBR": {
"type": "date",
"format": ["yyyy-MM-dd HH:mm:ss,SSS"]
}

And after starting logstash and filebeat I can't get my someindex populated. There are no errors in filebeat nor logstash logs, but in elasticsearch log I'm finding errors like this:

org.elasticsearch.index.mapper.MapperParsingException: failed to parse [datumGNBR]
...
Caused by: java.lang.IllegalArgumentException: Invalid format: "2017-10-31 23:00:46,519"

Does anybody have any idea where I'm wrong?
Thank you!

[rromanovic (Roman)]

Once I defined datumGNBR as follows:

"datumGNBR": {
  "type": "date",
  "format": "yyyy-MM-dd HH:mm:ss.SSS||yyyy-MM-dd'T'HH:mm:ss.SSSZ"
}

everything sorted out.

Link I found very useful: https://github.com/elastic/kibana/issues/66