Avoiding Field Reference Grammer in JSON Parsing

Elastic Stack Logstash
4

Not much movement on this :frowning:

Since it seems to be an issue for others, I thought I would post a temporary fix using a ruby filter (ala option 1 and 2 above). This is a temporary strategy, that may or may not work depending on your use case. At least it will stop pipeline crashes in logstash 7.0.0 with forced strict FR Grammar.

This filter will JSON parse a JSON encoded string in the [data] field and strip [ or ] from any Hash keys and set the parsed object back to the [data] field:

filter {      
  ruby {
    code => "
      def sanitize_field_reference(item)
        case item
        when Hash
          item.keys.each{ |k| item[k.gsub(/[\[\]]/, '')] = sanitize_field_reference(item.delete(k)) }
          return item
        when Array
          return item.map { |e| sanitize_field_reference(e) }
        else
          return item
        end
      end

      event.set('[data]', sanitize_field_reference(JSON.parse(event.get('[data]'))))"
  }
}

Would still welcome a more maintainable or production worthy answer regarding decoding valid JSON objects that contain Field Reference Grammar tokens.

Cheers,

Hal