Automatically excluding fields in new indexes



we have setup different indexes by day on our data and now we would like to exclude some fields from being indexed.
How should we proceed, so that every newly created index will automatically have those fields excluded ?

Thank you,

(Willemdh) #2

Hey Braoul,

It depends a bit how you ingest the data to decide what your options are. Are you using Beats agents to ingest the data? Do you ingest data into Elasticsearch directly or do you ingest the data into Logstash?

When using Beats, you could the drop_fields processor, for example:

  - drop_fields:
      fields: ["kubernetes.pod.start_time"]
        kubernetes.pod.start_time: ""

When not using Beats, you could make use of an Elasticsearch remove processor:

When you use Logstash, you can create a remove_field mutate filter:




Thank you Willem.

I am using Logstash, and I am already using blacklists or whitelists to filter some data.

What I would like to do is to store the fields, to be able to see them when needed, but not indexing them, to decrease the processing power needed. For that I thought about putting "index" : "no" in the index definition.
Is what I would like actually possible? or do I have to remove the fields in Logstash like you propose to decrease the cpu needed?

(Willemdh) #4


You can use the "enabled": false mapping parameter in your template for that I think:

Let me know if that works for you, I never used that actually.




Thank you very much! it seems like what I mean.

Now, do you know if it is possible to apply "enabled":false to the new indexes that will be created everyday?

(Willemdh) #6

As I said, you can define this in the template for your index. Template settings get applied to the corresponding indices the moment they are created.


Ok great, I was not aware of the index templates in Kibana. Thank you for your help

(system) #8

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.