Removing nested fields with empty field names

Hi guys,
I've got the following problem:
Some of my events somehow get sent to logstash with a field consisiting of an empty name and an empty value ([records][conditions][""]). Of course Elasticsearch doesn't like this. Is there a way to delete all fields with an empty name ? I just think deleting all fields with an empty name would be safer, if this happens to another field. I allready looked into the prune filter. but the doc says its only for top level fields.

{
    "records": {
        "systemId": "XXXX",
        "resourceId": "XXXX",
        "time": "XXXX",
        "operationName": "XXX",
        "properties": {
            "subnetPrefix": "XXXX",
            "ruleName": "XXXX",
            "type": "allow",
            "primaryIPv4Address": "XXXX",
            "macAddress": "XXXXX",
            "conditions": {
                "": "",
                "destinationIP": "0.0.0.0/0",
                "sourcePortRange": "0-65535",
                "destinationPortRange": "0-65535"
            },
            "priority": 65001,
            "direction": "In",
            "vnetResourceGuid": "XXX",
            "category": "XXX",
            "@version": "1",
            "resource_type_1": "XXX",
            "resource_provider_namespace": "MICROSOFT.NETWORK",
            "@timestamp": "XXX",
            "logstash_instance": "XXX",
            "customer": "XXX",
            "resource_name_1": "XXX",
            "subscription_id": "XXX",
            "eventhub": "XXX",
            "resource_group_name": "XXX"
        }
    }
}

Thanks in advance

You'll have to use a ruby filter with some custom Ruby code.

Found this yesterday :

filter {
  # remove fields with empty values
  ruby {
    code => "
      def walk_hash(parent, path, hash)
        path << parent if parent
        hash.each do |key, value|
          walk_hash(key, path, value) if value.is_a?(Hash)
          @paths << (path + [key]).map {|p| '[' + p + ']' }.join('')
        end
        path.pop
      end

      @paths = []
      walk_hash(nil, [], event.to_hash)

      @paths.each do |path|
        value = event.get(path)
        event.remove(path) if value.nil? || (value.respond_to?(:empty?) && value.empty?)
      end
    "
  }
}

source: https://www.endpoint.com/blog/2017/11/22/logstash-remove-fields-empty-values

Do you think this will still work?
Will test it tomorrow. Hope it won't impact the performance a lot. :slight_smile:

Sadly this ruby code didn't work with logstash 6.2.4.

UPDATE:

Found a solutinon

This Script by Colin Surprenant

Only the alternate_compact_event.rb worked for me.

# this is an alternate compact function implementation which also removes keys with empty string values
# writing the tests for this is left as an excercise to the reader :D

def compact(h)
  h.inject({}) do |result, (k, v)|
    if v.is_a?(Hash)
      result[k] = compact(v)
    elsif v.is_a?(String)
      result[k] = v unless v.empty?
    elsif !v.nil?
      result[k] = v
    end
    result
  end
end

def filter(event)
  return [LogStash::Event.new(compact(event.to_hash_with_metadata))]
end

Don't forget to add the filter method since it is missing in the gist of the alternate vesion.

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.