filebeat log is filled with errors "Error decoding JSON: invalid character"
filebeat.yml
---
filebeat.inputs:
- type: filestream
id: 0
paths:
- '/data/mixed_json_plain.log'
exclude_lines: [ "^PLAIN" ]
parsers:
- ndjson:
add_error_key: true
keys_under_root: true
overwrite_keys: true
message_key: message
expand_keys: true
output.file:
path: /data/filebeat_output
rotate_every_kb: 100000
rotate_on_startup: false
codec.json:
pretty: true
log sample file mixed_json_plain.log
{"id":1,"message":"lorem ipsum"}
PLAIN first line
PLAIN second line
{"id":2,"message":"dolor sit amet"}
{"id":3,"message":"consectetur adipiscing elit"}
To reproduce errors
docker run --rm -v $PWD:/data -v $PWD/filebeat.yml:/usr/share/filebeat/filebeat.yml docker.elastic.co/beats/filebeat:8.2.0
When run with -e -d '*' it is visible on debug level, the line is exluded correctly.
Drop line as it does match one of the exclude patterns%sPLAIN first line