Automatically excluding fields in new indexes

Elastic Stack Elasticsearch
1

Hi,

we have setup different indexes by day on our data and now we would like to exclude some fields from being indexed.
How should we proceed, so that every newly created index will automatically have those fields excluded ?

Thank you,
Boris

2

Hey Braoul,

It depends a bit how you ingest the data to decide what your options are. Are you using Beats agents to ingest the data? Do you ingest data into Elasticsearch directly or do you ingest the data into Logstash?

When using Beats, you could the drop_fields processor, for example:

  - drop_fields:
      fields: ["kubernetes.pod.start_time"]
      when.equals:
        kubernetes.pod.start_time: ""

https://www.elastic.co/guide/en/beats/filebeat/current/drop-fields.html

When not using Beats, you could make use of an Elasticsearch remove processor:

https://www.elastic.co/guide/en/elasticsearch/reference/current/remove-processor.html

When you use Logstash, you can create a remove_field mutate filter:

https://www.elastic.co/guide/en/logstash/current/plugins-filters-mutate.html#plugins-filters-mutate-remove_field

Grtz

Willem

3

Thank you Willem.

I am using Logstash, and I am already using blacklists or whitelists to filter some data.

What I would like to do is to store the fields, to be able to see them when needed, but not indexing them, to decrease the processing power needed. For that I thought about putting "index" : "no" in the index definition.
Is what I would like actually possible? or do I have to remove the fields in Logstash like you propose to decrease the cpu needed?

4

Braoul,

You can use the "enabled": false mapping parameter in your template for that I think:

https://www.elastic.co/guide/en/elasticsearch/reference/current/enabled.html

Let me know if that works for you, I never used that actually.

Grtz

Willem

5

Thank you very much! it seems like what I mean.

Now, do you know if it is possible to apply "enabled":false to the new indexes that will be created everyday?

6

As I said, you can define this in the template for your index. Template settings get applied to the corresponding indices the moment they are created.

7

Ok great, I was not aware of the index templates in Kibana. Thank you for your help

8

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.