JSON Parsing Issue

[wwalker (Walker)]

Running Logstash 7.10 and having an issue with parsing a JSON-ish field. Here's my filter config:

    if [vpn][tunnel_type] {
      mutate {
        gsub => [
          "[vpn][tunnel_type]", '\=\>', ':'
        ]
      }
      json {
        source => "[vpn][tunnel_type]"
      }
  }

The error message I get in the pipeline logs is:

Error parsing json {:source=>"[vpn][tunnel_type]", :raw=>{"tunnel_id"=>"1087764505", "tunnel_type"=>"ssl-tunnel", "tunnel_ip"=>"192.168.1.1"}, :exception=>java.lang.ClassCastException}

Am I reading it right that the raw JSON coming in is formatted with => instead of :? I've tried both the gsub regex as shown above as well as just =>. Is there a different syntax I should be using with gsub? Or maybe I'm reading the log entry wrong and the raw message is also missing the opening and close script blocks {}?

[Badger]

Your [vpn][tunnel_type] is not a string, it is an object. So your mutate filter does not do anything. Trying to parse this object with a json filter

    "tunnel_type" => {
          "tunnel_id" => "1087764505",
        "tunnel_type" => "ssl-tunnel",
          "tunnel_ip" => "192.168.1.1"
    }

gets the error message

 Error parsing json {:source=>"[vpn][tunnel_type]", :raw=>{"tunnel_id"=>"1087764505", "tunnel_ip"=>"192.168.1.1", "tunnel_type"=>"ssl-tunnel"}, :exception=>java.lang.ClassCastException: ...

If it were a string then you would get the error

Error parsing json {:source=>"[vpn][tunnel_type]", :raw=>"{\"tunnel_id\"=>\"1087764505\", \"tunnel_type\"=>\"ssl-tunnel\", \"tunnel_ip\"=>\"192.168.1.1\"}", :exception=>#<LogStash::Json::ParserError: Unexpected character ('=' (code 61)): ...

If you need to move the contents of [vpn][tunnel_type] to the top level use ruby.

[wwalker (Walker)]

Pulling code from where I used ruby last...because I don't know ruby that well, would this be the proper usage?

ruby {
  code => "event.set('[vpn][tunnel_type]', event.get('[vpn][tunnel_type][tunnel_type]'))"
}
ruby {
  code => "event.set('[vpn][tunnel_id]', event.get('[vpn][tunnel_type][tunnel_id]'))"
}
ruby {
  code => "event.set('[vpn][tunnel_ip]', event.get('[vpn][tunnel_type][tunnel_ip]'))"
}

or can I condense them all into a single call of the ruby filter with three codes specified:

ruby {
  code => "event.set('[vpn][tunnel_type]', event.get('[vpn][tunnel_type][tunnel_type]'))"
  code => "event.set('[vpn][tunnel_id]', event.get('[vpn][tunnel_type][tunnel_id]'))"
  code => "event.set('[vpn][tunnel_ip]', event.get('[vpn][tunnel_type][tunnel_ip]'))"
}

[Badger]

@wwalker

code => "event.set('[vpn][tunnel_type]', event.get('[vpn][tunnel_type][tunnel_type]'))"

Once you overwrite [vpn][tunnel_type] you can no longer reference the other fields in it. If the names of the fields are fixed you can use mutate

mutate {
    rename => {
        "[vpn][tunnel_type][tunnel_ip]" => "[vpn][tunnel_ip]"
        "[vpn][tunnel_type][tunnel_id]" => "[vpn][tunnel_id]"
    }
}
mutate {
    rename => {
        "[vpn][tunnel_type][tunnel_type]" => "[vpn][tunnel_type]"
    }
}

I believe it would work if you merged those two, provided "[vpn][tunnel_type][tunnel_type]" was listed last, but I do not think logstash provides a guarantee of that. Doing it in two filters should guarantee it.

[wwalker (Walker)]

I scrapped everything that was in the pipeline for this event and started from scratch. I don't know why JSON filter was ever used as the data comes in structured as key/value pairs. It also looks like mutate's rename filter was being invoked to rename the nested duplicate tunnel_type up a level...but that wasn't working for some reason. What I got to work was using ruby to pull the information up a level and then delete the original nested field:

ruby {
        code => "event.set('[vpn][tunnel_id]', event.get('[vpn][tunnel_type][tunnel_id]'))"
          remove_field => [
            "[vpn][tunnel_type][tunnel_id]"
          ]
      }
      ruby {
        code => "event.set('[vpn][tunnel_ip]', event.get('[vpn][tunnel_type][tunnel_ip]'))"
          remove_field => [
            "[vpn][tunnel_type][tunnel_ip]"
          ]
      }
      ruby {
        code => "event.set('[vpn][tunnel_type]', event.get('[vpn][tunnel_type][tunnel_type]'))"
          remove_field => [
            "[vpn][tunnel_type][tunnel_type]"
          ]
      }

I probably need to put some logic in to ensure that a given field is present before executing one of these ruby filters. However, in the absence of a value, the key/value pair is still present as tunnel_type=.

Regardless, you helped put me on the right path, so much appreciated for the help @Badger