Elastic agent filebeat processors' decode_json_fields error

Elastic Stack Elastic Agent
1

Hi,

  1. I am testing out elasticsearch logging via agent enrolled in fleet
  2. custom logs integration with logpath /var/lib/docker/containers//-json.log
  3. testing out k8s container logs with similar setup
  4. setup that functioned properly in 8.3.2 stops working on 8.4.0

custom configurations:

processors:
  - decode_json_fields:
      fields: ["message"]
      process_array: true
      max_depth: 3
      target: "messages"
      overwrite_keys: false
      add_error_key: true

I'm trying to decode a multi level json field via the configuration above, which worked fine in 8.3.2, but spits out the following error in 8.4.0.

Preview of field's value: '{filename=hello, func=hello.py, level=INFO, line=16, message={res_body={"detail": "user \"77777\" not found."}, path=/api/v1.0/hello/, method=GET, req_body=null, req_params=b'name=77777', latency=1, host=[-.-.-.-, port], client=[-.-.-.-, port], status=400}, loggerName=name, timestamp=2022-08-30T06:41:16.855Z}'","caused_by":{"type":"illegal_state_exception","reason":"Can't get text on a START_OBJECT at 1:514"}}, dropping event!

example message field:
{"log":"{"timestamp": "2022-08-30T06:42:25.511Z", "level": "INFO", "loggerName": "name", "filename": "hello", "line": 16, "func": "hello.py", "message": "{\"host\": [\"ip_adr\", \"port\"], \"path\": \"/api/v1.0/hello/\", \"client\": [\"ip_addr\", \"port\"], \"method\": \"GET\", \"status\": 400, \"req_body\": null, \"req_params\": \"b'name=77777'\", \"res_body\": \"{\\\"detail\\\": \\\"user \\\\\\\"77777\\\\\\\" not found.\\\"}\", \"latency\": 1}"}\n","stream":"stderr","time":"2022-08-30T06:42:25.511614485Z"}

decode_json_fields works if max_depth is 1, but stops working from 2. Using pipeline to decode the field still works fortunately, but I'm trying to reduce the elasticsearch node's performance hit minimal via filebeat's processors. I have tried using a different index to no avail. Any idea why this processor fails after the upgrade?

thanks

2

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.