Logstash - _jsonparsefailure on valid json in udp input

[mybyte (Michael)]

I'm using Logstash 8.0.0 with a fairly straightforward udp pipeline:

input {
  udp {
    codec => json {
      charset => "UTF-8"
    }
	  port => 12200
  }
}

output {
  stdout {}
}

The data is being sent from a custom Log4j1 Appender, but I'm fairly sure it's valid JSON. It's being generated using json-simple. All messages seem to fail, showing _jsonparsefailure during processing:

{
    "@timestamp" => 2022-02-21T12:32:33.240128Z,
          "host" => {
        "ip" => "192.168.140.193"
    },
       "message" => "{\"message\":\"Camel-Script activator stopped\",\"logger_name\":\"org.apache.camel.script.osgi.Activator\",\"level_value\":20000,\"level\":\"INFO\",\"thread_name\":\"FelixStartLevel\",\"application\":\"fuse-scan\",\"environment\":\"local-docker\",\"service\":\"Cockpit\",\"caller_class_name\":\"org.apache.camel.script.osgi.Activator\",\"caller_line_number\":\"82\",\"caller_method_name\":\"stop\",\"caller_file_name\":\"Activator.java\",\"host\":\"zd3402\"}",
      "@version" => "1",
          "tags" => [
        [0] "_jsonparsefailure"
    ]
}

I've looked at a whole bunch of messages and all of them seem to be valid JSON objects... Any ideas what might be causing this behavior?

[Rios]

It seems this is not JSON message. Use KV plugin instead.

[mybyte (Michael)]

How is this not a valid message? Take a look a "message". This is literally the message content received by Logstash.

[Rios]

Yes, you have right it is JSON format. My mistake
Try not to use UTF-8 as codec, but most likely backslash is a problem.
Use gsub to remove backslash then apply JSON codec in filter
gsub => [ "message", '' , '"' ]

[Rios]

From the documentation:

If you are streaming JSON messages delimited by \n then see the **json_lines** codec.

Encoding will result in a compact JSON representation (no line terminators or indentation)

If this codec recieves a payload from an input that is not valid JSON, then it will fall back to plain text and add a tag _jsonparsefailure . Upon a JSON failure, the payload will be stored in the message field.

So,
a)

input {
  udp {
    codec => json_lines{
      charset => "UTF-8"
    }
	  port => 12200
  }
}

b) do no use the codec in input. Instead do replacement in the filter. But I'm sure a) option will help

[mybyte (Michael)]

Found the issue - I think... I'm not sure why, but udp doesn't seem compatible with ecs_compatibility => "v8" (global setting in our case).

I disabled it and now everything is being parsed correctly:

udp {
    ecs_compatibility => "disabled"
    codec => "json"
    port => 12200
}

[Rios]

AFAIK, your source fields must be according ECS names.