Json_lines codec is not able to parse certain jsons correctly

Elastic Stack Logstash
1

I am using json_lines codec along with http_input in my pipeline. One of the elements in my JSON is querystring. Usually, this codec is able to handle all the variants of valid jsons except below

   "querystring": {
      "a": true,
      "p[]": [
        "q",
        "l"
      ],
      "z": "x"
    }

When I use this event, logstash writes one line on log saying:

[2020-07-17T07:17:10,052][WARN ][logstash.codecs.jsonlines] JSON parse error, original data now in message field {:error=>#<LogStash::Json::ParserError: Invalid FieldReference: `p[]`>, :data=>"

When I tried using same json without '' after letter 'p', the codec is able to parse it correctly. So below json, works fine :

   "querystring": {
      "a": true,
      "p": [
        "q",
        "l"
      ],
      "z": "x"
    }

Does anyone know how to fix this behaviour or the root cause of this?

2

I have parsed both jsons with plain ruby and jackson( which Logstash's jruby also uses) without any issues!

3

Yes, it is a known issue. The JSON is parsed OK, but when time comes to event.set the key it is treated as a field reference, which breaks. I believe you can work around this by using a json filter instead of a codec and setting the target option, or else by using mutate+gsub to strip the square brackets out.

4

Thanks Badger!

The thing is, we are using json_lines codec along with http_input plugin, which is the first the thing running in pipeline. so unfortunately, we cant use mutate+gsub. Having said this, we tried using json filter but it cant handle mutiple jsons separated by new lines.

5

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.