Hi,
Our applications emit logs in ECS format, a log entry per line. In Filebeat config we have a parser defined:
parsers:
- ndjson:
target: ''
expand_keys: true
overwrite_keys: true
add_error_key: true
Then there is a processor in the ingest pipeline which converts some of the fields into lower case:
processors:
- lowercase:
field: 'log.level'
on_failure:
- set:
field: error.message
value: '{{ _ingest.on_failure_message }}'
Sometimes we have issues with free disk space which lead to incorrect log entries in the file, where one log entry is concatenated with previous incompletely written log entry, e.g.:
{"@timestamp":"2024-10-17T10:35:59.1814105+02:00","log.level":"Information","message":"Requ{"@timestamp":"2024-10-17T12:00:01.0603052+02:00","log.level":"Information","message":"Ok"}
Notice how the first message property is not written completely and then the second log entry is immediately appended to it.
When Filebeat processes such lines, it emits an error like "Error decoding JSON: invalid character '@' after object key:value pair" (error message varies a bit depending on where exactly previous log line was abrupted).
Finally, broken log lines are delivered into Elastic where error.message is set to "field [level] not present as part of path [log.level]".
We would like to just completely ignore such broken log lines. How to implement it properly? ignore_decoding_error option of ndjson parser seems to just control whether error log is emitted or not.