Hi,
for some reason, logstash (version 1.5) can't process logs with this exception:
{:timestamp=>"2016-04-26T09:20:12.141000-0400", :message=>"Failed parsing date from field", :field=>"time", :value=>"2016-04-26T09:20:03.520-04:00", :exception=>java.lang.IllegalArgumentException: Invalid format: "2016-04-26T09:20:03.520-04:00" is malformed at "T09:20:03.520-04:00", :level=>:warn}
My Time field in json is:
"time":"25-04-2016 04:21:06.786"
my logstash configuration is:
filter {
if [type] == "json" {
json {
source => "message"
}
date {
match => [ "time", "dd-MM-yyyy HH:mm:ss", "dd-MM-yyyy HH:mm:ss:SSS", "dd-MM-yyyy HH:mm:ss.SSS", "yyyy-MM-dd HH:mm:ss,SSS" ]
}
}
}
On Elasticsearch side I see this exception:
failed to parse date field [25-04-2016 04:48:14.305], tried both date format [dateOptionalTime], and timestamp number with locale [] java.lang.IllegalArgumentException: Invalid format: "25-04-2016 04:48:14.305" is malformed at "16 04:48:14.305"
How do I fix this? Isn't my "time" field suppose to replace "@timestamp" ?
Judging by the first error message the time field contains "2016-04-26T09:20:12.141000-0400" and not "25-04-2016 04:21:06.786", and none of the date patterns you've supplied matches that. The last one is close, but doesn't expect the "T" separator between the date and the time (which the error message also indicates in the "... is malformed at ..." part).
Almost. There's also the timezone. Something like yyyy-MM-dd'T'HH:mm:ss,SSSZZZ should be okay. Or, come to thing of it, the ISO8601 special case should also work.
For some reason it doesn't work..
This is my configuration:
date {
match => [ "time", "yyyy-MM-dd'T'HH:mm:ss,SSSZZZ", "ISO8601" ]
}
Still getting these exceptions:
{:timestamp=>"2016-04-27T04:06:36.194000-0400", :message=>"Failed parsing date from field", :field=>"time", :value=>"25-04-2016 09:00:36.159", :exception=>java.lang.IllegalArgumentException: Invalid format: "25-04-2016 09:00:36.159" is malformed at "16 09:00:36.159", :level=>:warn}
Your time apparently contains multiple date formats. With the current configuration you only support ISO8601 timestamp but you also need the ones you had previously.
Apache, Apache Lucene, Apache Hadoop, Hadoop, HDFS and the yellow elephant
logo are trademarks of the
Apache Software Foundation
in the United States and/or other countries.