Hello there at Elastic S. and Filebeat, any help here is appreciated.
I send app logs to elasticsearch using filebeat, this can work fine for weeks, then fails, saying "unable to parse date". My env is like this:
Filebeat: 5.3.1
Elastic search: 5.5.1 (must be that version, but did not set up so am not sure 100%, but I know it is 5.x)
Kibana: 5.51 (must be that version too).
Note that simulation API for the pipeline and the messages from the logs works fine, and as mentioned, even file beat pushes the files for some weeks, all of a sudden it starts failing, nothing goes in the ES.
The errors I get are like this:
++++++++++++++++++
2017-09-18T16:27:41-04:00 INFO Bulk item insert failed (i=1, status=500): {"type":"exception","reason":"java.lang.IllegalArgumentException: java.lang.IllegalArgumentException: unable to parse date [2017-09-18T20:27:22.357Z]","caused_by":{"type":"illegal_argument_exception","reason":"java.lang.IllegalArgumentException: unable to parse date [2017-09-18T20:27:22.357Z]","caused_by":{"type":"illegal_argument_exception","reason":"unable to parse date [2017-09-18T20:27:22.357Z]","caused_by":{"type":"illegal_argument_exception","reason":"Invalid format: "2017-09-18T20:27:22.357Z" is malformed at "17-09-18T20:27:22.357Z""}}},"header":{"processor_type":"date"}}
++++++++++++++++++
My pipeline looks like:
"processors": [
{
"gsub": {
"field": "message",
"pattern": "\n",
"replacement": "\\n"
}
},
{
"grok": {
"field": "message",
"patterns": [
"%{IPORHOST:log.clientip} %{USER:log.ident} %{USER:log.auth} %{HTTPDATE:log.rawdatetime} "(?:%{WORD:log.verb} %{NOTSPACE:log.request}(?: HTTP/%{NUMBER:log.httpversion})?|%{DATA:log.rawrequest})" %{NUMBER:log.response} (?:%{NUMBER:log.bytes}|-)",
"%{AEM_DATE_FORMAT:log.datetime} %{DATA:log.level} %{DATA:log.module} %{GREEDYDATA:log.message}"
],
"pattern_definitions": {
"AEM_DATE_FORMAT": "%{DATE_EU} %{TIME}"
},
"on_failure": [
{
"set": {
"field": "ingestError",
"value": "{{ _ingest.on_failure_message }}"
}
},
{
"set": {
"field": "log.datetime",
"value": "{{ @timestamp }}"
}
}
]
}
},
{
"date": {
"field": "log.datetime",
"target_field": "log.datetime",
"formats": [
"dd.MM.yyyy HH:mm:ss.SSS"
],
"timezone": "America/New_York"
}
}
]
Thanks