How static mapping works at storage level

Elastic Stack Elasticsearch
1

Team,

I am using static mappings for elastic index with 10 fields as indexed fields, but application is sending 20 fields.

out of 20 fields only 10 fields will be available for searching and querying

does non indexed fields consumer storage at elastic?

Thanks,

2

Hi @prabhakar_talari

Why don't you drop those fields via logstash so they are not stored? Any data store takes up storage space.

3

Hi,

Thanks for the reply.

I will be dropping the fields if i know but consumers are generating some additional fields which we are not aware so in those cases would like to understand if elastic use the storage for non-indexed fields or not.

4

Yes, as they are still stored.

5

In most cases, the answer is yes. Unless you explicitly turn off _source option, the entire document you entered is stored in the _source field.

I'm not sure it is possible, you may delete unnecessary fields by script processor of ingest pipeline.

6

Thank you..

If we know the unnecessary field names we can drop easily but we don't have this list that's the big challenge here.

I am looking for the way on how to accept only specific fields from document and drop rest all, as i could not find a way as off now.

Thanks,

7

Dynamic field mapping | Elasticsearch Guide [8.2] | Elastic, sepcifically;

You can disable dynamic mapping, both at the document and at the object level. Setting the dynamic parameter to false ignores new fields, and strict rejects the document if Elasticsearch encounters an unknown field.

8

Could you let me know how you drop the known unnecessary fields? I suppose it is nearly as difficult as to drop unknown fields..

As you said using static mappings, I thought you are looking for the way to delete unnecessary fields from _source. Is it right?

9

I am leaving a reply here as I am also interested in getting the answer of this question.

10

Know unnecessary fields are just fields that exist in your document but you know that you won't use it for search or aggregation so you can drop it before or during ingestion.

For a silly example, suppose you have database of people data and you have a field called species where all the valuse are human, this is an unnecessary field so you can drop it.

1 Like
11

Thank you. I just remembered the remove processor to drop the field completely from _source. I'd thought both (drop known fields or drop unknown fields) need Ruby script processor. Thanks!

1 Like
12

Thanks for the response.

I pretty new to coding so not sure how to create Ruby script, but even though if we create and implement at logstash or ingestion pipeline layer the ingestion might slow i feel.

I have approach, but it is not going to delete the fields but it will hide the field from _source and storage consumption also less.

As part of template creation disable dynamic mapping and what ever the fields we want to index mention them at properties and include area

This way we will avoid showing non indexed fields at discover tab in kibana

Thanks

1 Like
13

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.