Removing square brackets from json key in http input

[Felipe_Fuller]

Hi Community!

I've been having the following problem. I'm receiving, through http input filter, a JSON that has bracket square in the keys so Logstash isn't able to parse it, as result it returns a _jsonparsefailure.

An example of the JSON that I receive is the following:

{
  "alertData": {
        "name": "foo - bar",
        "url": "manage",
        "changes": {
            "ssid[bar]": {
                "label": "Splash page",
                "newText": "None",
                "oldText": "Click-through",
                "changedBy": "foo",
                "ssidId": null
            }
        },
        "userId": 123
    }
}

my pipeline is the following:

input {
    http {
        host => "0.0.0.0"
        port => "8080"
        codec => "json"
    }
}
filter {
    mutate {
        remove_field => [ "event", "http", "url", "@version"]
    }
}
output {
  elasticsearch {
    hosts => "elasticsearch:9200"
    user => "foo"
    password => "bar"
    index => "example"
    codec => json_lines
  }
}

I've found some discussions about it, like this or this other one, but they don't seem to work.

Thank you in advance!

[Badger]

@Felipe_Fuller

I've found some discussions about it, like this or this other one, but they don't seem to work.

You can work around this by removing the codec and using a json filter with the target option set.

[Ugo_Sangiorgi]

That's likely not the problem, as "ssid[bar]" is actually a valid field name.
You can try debugging it by using an output to file{} or logshark.

[Badger]

@Ugo_Sangiorgi

That's likely not the problem, as "ssid[bar]" is actually a valid field name.

Yes, it is valid, but until very recently, although the JSON could be parsed, when the codec or filter tried to do an event.set("ssid[bar]", ...) it would get an invalid field reference, and there were corner cases where it would crash logstash completely.

[Ugo_Sangiorgi]

@Badger thats true indeed, thanks! OP must be using a version lower than 8.3 then.
@Felipe_Fuller I have just tested it with Logstash 8.3.2 and it works, so you should just upgrade

[Felipe_Fuller]

That's correct, I'm using version 8.2. I think the easiest work around is the update because in my case, that field is dynamic meaning that I can receive different values inside of the square brackets. As conclusion, I updated my Logstash to 8.3.3 to test it and it worked as a charm!

Thank you very much @Badger and @Ugo_Sangiorgi for the help!