Preprocessor decode_json_fields does not include error key when add_error_key true

I have connected filebeat to logstash and process logs from my docker app. Filebeat, logstash, elsticsearch and kibana are dockerized. I would like to harvest docker logs so I implement python app, that generate this logs:

This is just message at: 2020-06-01 14:49:41.287697
{"time": "2020-06-01 14:49:42.288979", "text": "Random text"}
This is just message at: 2020-06-01 14:49:45.291579
{"time": "2020-06-01 14:49:46.292781", "text": "Random text"}
This is just message at: 2020-06-01 14:49:49.295620
{"time": "2020-06-01 14:49:50.296886", "text": "Random text"}
This is just message at: 2020-06-01 14:49:53.299630

Filebeat process this logs, that works fine. I am trying to parse json messages with decode_json_fields. I set add_error_key: true. Messages that are json are parsed properly, but when there is just text, message is not parsed - expected, but there is also no error message. From documentation for filebeat 7.7:
If it set to true, in case of error while decoding json keys error field is going to be part of event with error message..

This is my filebeat configuration:

filebeat.inputs:
- type: container
  paths:
    - '/var/lib/docker/containers/*/*.log'

processors:
- add_docker_metadata:
  host: "unix:///var/run/docker.sock"

- decode_json_fields:
  fields: ["message"]
  target: ""
  add_error_key: true
  overwrite_keys: true

output.logstash:
  hosts: ["logstash:5000"]

logging.level: info
logging.json: true
logging.metrics.enabled: false

Expectation:

If messages can be parsed, it will be parsed and moved to root path. In case there is just string, I would expected some error message, like logstash when cant parse json message with json filter, it append tag _jsonparsefailure.

Why does not error message is not included when parsing messages like this: This is just message at: 2020-06-01 14:49:41.287697?

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.