How to remove empty fields from filebeat-8.15.1

Elastic Stack Beats
1

I have installed and configured filebeat-8.15 on Windows 2019 host, and in the Kibana 'Discover' section in the , customized 'Data Views' which I have created by editing "Index Template" , and dropping the unwanted empty fields , I am getting

image
image1895×697 73.4 KB

BTW, I have also created the an ingest pipeline -->

[
  {
    "script": {
      "source": "           for (entry in ctx.entrySet()) {             if (entry.value instanceof String && entry.value.trim().length() == 0) {               ctx.remove(entry.key);             }           }         "
    }
  }
]

And load it in filebeat.yml -->

output.elasticsearch:
  hosts: ["https://<host_ip>:9200"]
  username: "elastic"
  password: "***"
  pipeline: "remove_empty_fields"
  ssl:
    enabled: true
    verification_mode: none

But still not able to get rid of the empty fields, is there really a way to omit 6558 number of empty fields? Please let me know

3

Does this Kibana data view only match this index?

Do the templates that defined this index define these fields?

How old is this index?

4

Hi @rugenl
Thanks for your response.
On your question,

  1. Yes
    2)Yes
    3)Contains last 180 days data.
5

If your template defines these fields, but they don't contain data, (I think in any events in the Kibana range, in this case last 15 minutes) they appear as "empty fields".

If the fields never exist, don't map them.

Maybe similar to this: Mapping explosion | Elasticsearch Guide [8.15] | Elastic