Removing square brackets from json key in http input

Hi Community!

I've been having the following problem. I'm receiving, through http input filter, a JSON that has bracket square in the keys so Logstash isn't able to parse it, as result it returns a _jsonparsefailure.

An example of the JSON that I receive is the following:

{
  "alertData": {
        "name": "foo - bar",
        "url": "manage",
        "changes": {
            "ssid[bar]": {
                "label": "Splash page",
                "newText": "None",
                "oldText": "Click-through",
                "changedBy": "foo",
                "ssidId": null
            }
        },
        "userId": 123
    }
}

my pipeline is the following:

input {
    http {
        host => "0.0.0.0"
        port => "8080"
        codec => "json"
    }
}
filter {
    mutate {
        remove_field => [ "event", "http", "url", "@version"]
    }
}
output {
  elasticsearch {
    hosts => "elasticsearch:9200"
    user => "foo"
    password => "bar"
    index => "example"
    codec => json_lines
  }
}

I've found some discussions about it, like this or this other one, but they don't seem to work.

Thank you in advance!

You can work around this by removing the codec and using a json filter with the target option set.

That's likely not the problem, as "ssid[bar]" is actually a valid field name.
You can try debugging it by using an output to file{} or logshark.

Yes, it is valid, but until very recently, although the JSON could be parsed, when the codec or filter tried to do an event.set("ssid[bar]", ...) it would get an invalid field reference, and there were corner cases where it would crash logstash completely.

1 Like

@Badger thats true indeed, thanks! OP must be using a version lower than 8.3 then.
@Felipe_Fuller I have just tested it with Logstash 8.3.2 and it works, so you should just upgrade

1 Like

That's correct, I'm using version 8.2. I think the easiest work around is the update because in my case, that field is dynamic meaning that I can receive different values inside of the square brackets. As conclusion, I updated my Logstash to 8.3.3 to test it and it worked as a charm!

Thank you very much @Badger and @Ugo_Sangiorgi for the help!