Removing empty fields from an event (2024 edition!)

[Supermathie (Michael Brown)]

Continuing the discussion from Never ending story: how to check and remove empty fields, arrays etc:

There's no simple way to (recursively) get all field names in an event, but one can do this:

filter {
    ruby {
       init => "
       def remove_empty_fields(event)
         _find_empty_fields(event.to_hash) { |path| event.remove('[' + path.join('][') + ']') }
       end

       def _find_empty_fields(event, path = [], &blk)
         event.each do |k, v|
           curpath = path + [k]
           case v
           when nil
             yield curpath
           when ''
             yield curpath
           when []
             yield curpath
           when {}
             yield curpath
           when Hash
             _find_empty_fields(v, curpath, &blk)
           end
         end
       end
       "
       code => "remove_empty_fields(event)"
    }
}

[leandrojmp (Leandro Pereira)]

Another option is to use the Elastic approach with Elastic Agent integrations and remove those fields using a script processor.

If I'm not wrong all Elastic Agent integrations have this processor to remove empty/null fields.

  - script:
      description: Drops null/empty values recursively
      lang: painless
      source: |
        boolean drop(Object o) {
          if (o == null || o == "") {
            return true;
          } else if (o instanceof Map) {
            ((Map) o).values().removeIf(v -> drop(v));
            return (((Map) o).size() == 0);
          } else if (o instanceof List) {
            ((List) o).removeIf(v -> drop(v));
            return (((List) o).length == 0);
          }
          return false;
        }
        drop(ctx);

I created an ingest pipeline with only this processor and configured it to be the executed on all my data that is ingested using Logstash by setting an index.final_pipeline in the templates.