I have written a module for filebeat to parse a log from a application I have written
The pipeline mostly works, except in kibana i get the error "Text '19/May/2020 15:25:17' could not be parsed at index 12" for each record
And example log entry looks like
[19/May/2020 15:25:17] INFO [module name] the log message
My pipeline file looks like
{
"description": "Pipeline for parsing mylogs.",
"processors": [
{
"grok": {
"field": "message",
"ignore_missing": true,
"patterns": [
"\\[%{DATETIME:my.datetime}\\] %{LOGLEVEL:level} \\[%{GREEDYDATA:module}\\] %{GREEDYDATA:message}"
],
"pattern_definitions": {
"DATETIME": "%{MONTHDAY:day}/%{MONTH:month}/%{YEAR:year} %{TIME:time}",
"GREEDYDATA": "(.|\n|\t)*"
}
}
},
{
"date": {
"field": "my.datetime",
"target_field": "@timestamp",
"formats": ["dd/MMM/yyyy HH:mm:ss"]
}
}
],
"on_failure": [
{
"set": {
"field": "error.message",
"value": "{{ _ingest.on_failure_message }}"
}
}
]
}
In kibanna the @timestamp is displayed as when the log was parsed by filebeat, not the timestamp of the entry.
The other fields are all correctly shown in the record (ie day, month, year.. and the full datetime)
In the record there is the error.message filed with Text '19/May/2020 15:25:17' could not be parsed at index 12
I think that error message means index 12 of my format string causes an error when parsing the input? So this would be the hour?