Removing nested fields with empty field names

[Mugen (Lars)]

Hi guys,
I've got the following problem:
Some of my events somehow get sent to logstash with a field consisiting of an empty name and an empty value ([records][conditions][""]). Of course Elasticsearch doesn't like this. Is there a way to delete all fields with an empty name ? I just think deleting all fields with an empty name would be safer, if this happens to another field. I allready looked into the prune filter. but the doc says its only for top level fields.

{
    "records": {
        "systemId": "XXXX",
        "resourceId": "XXXX",
        "time": "XXXX",
        "operationName": "XXX",
        "properties": {
            "subnetPrefix": "XXXX",
            "ruleName": "XXXX",
            "type": "allow",
            "primaryIPv4Address": "XXXX",
            "macAddress": "XXXXX",
            "conditions": {
                "": "",
                "destinationIP": "0.0.0.0/0",
                "sourcePortRange": "0-65535",
                "destinationPortRange": "0-65535"
            },
            "priority": 65001,
            "direction": "In",
            "vnetResourceGuid": "XXX",
            "category": "XXX",
            "@version": "1",
            "resource_type_1": "XXX",
            "resource_provider_namespace": "MICROSOFT.NETWORK",
            "@timestamp": "XXX",
            "logstash_instance": "XXX",
            "customer": "XXX",
            "resource_name_1": "XXX",
            "subscription_id": "XXX",
            "eventhub": "XXX",
            "resource_group_name": "XXX"
        }
    }
}

Thanks in advance

[magnusbaeck (Magnus Bäck)]

You'll have to use a ruby filter with some custom Ruby code.

[Mugen (Lars)]

Found this yesterday :

filter {
  # remove fields with empty values
  ruby {
    code => "
      def walk_hash(parent, path, hash)
        path << parent if parent
        hash.each do |key, value|
          walk_hash(key, path, value) if value.is_a?(Hash)
          @paths << (path + [key]).map {|p| '[' + p + ']' }.join('')
        end
        path.pop
      end

      @paths = []
      walk_hash(nil, [], event.to_hash)

      @paths.each do |path|
        value = event.get(path)
        event.remove(path) if value.nil? || (value.respond_to?(:empty?) && value.empty?)
      end
    "
  }
}

source: https://www.endpoint.com/blog/2017/11/22/logstash-remove-fields-empty-values

Do you think this will still work?
Will test it tomorrow. Hope it won't impact the performance a lot.

[Mugen (Lars)]

Sadly this ruby code didn't work with logstash 6.2.4.

UPDATE:

Found a solutinon

This Script by Colin Surprenant

Only the alternate_compact_event.rb worked for me.

# this is an alternate compact function implementation which also removes keys with empty string values
# writing the tests for this is left as an excercise to the reader :D

def compact(h)
  h.inject({}) do |result, (k, v)|
    if v.is_a?(Hash)
      result[k] = compact(v)
    elsif v.is_a?(String)
      result[k] = v unless v.empty?
    elsif !v.nil?
      result[k] = v
    end
    result
  end
end

def filter(event)
  return [LogStash::Event.new(compact(event.to_hash_with_metadata))]
end

Don't forget to add the filter method since it is missing in the gist of the alternate vesion.