Add original message to documents logged with decode_json_fields

[Cylindric (Mark Hanford)]

Hi folks. I am using Filebeat to log a bunch of JSON-formatted log-files into ES, and occasionally I get records with a json error such as

Error decoding JSON: invalid character ',' in numeric literal

Can I add the original message field to each document in some way, so I can see what the raw json line was that caused this error? I suspect it's just something not formatted correctly in my log file, but I can't just search for ,

-
  type: log
  document_type: varnish
  close_inactive: 1m
  ignore_older: 15m
  paths:
    - /var/log/varnish/varnishncsa.log
  json.keys_under_root: true
  json.add_error_key: true
  json.overwrite_keys: true

[ruflin]

This is currently not possible but I can see that it could be useful to have in the error message as the line that could not be decoded. Could you open a feature request on Github for this?

[Cylindric (Mark Hanford)]

Done, thank you for confirming so I don't chase my own tail

Issue: Add a `message` field to messages that have a JSON error_key

[andrewkroh]

Maybe you could use the decode_json_fields processor. It does not delete the source field. Then you could use a drop_fields processor to delete the message field when the json_error key is not present. Just a thought, haven't tried it.

https://www.elastic.co/guide/en/beats/filebeat/current/decode-json-fields.html