Implications and reasons of field mapping limit

Elastic Stack Elasticsearch
1

There are many blogs stating that a high number of fields in the mapping leads to a mapping explosion which may cause out-of-memory errors. However, I am unable to understand what exact data structure is causing this memory bloat up and if it is just a memory concern that can be fixed by more RAM, or if it is a fundamental limitation in the system.

We have a use case for storing custom fields for many documents. So each document would have 100s of custom fields and this number can keep increasing. So we aimed to store them as nested documents like so:

{
  "custom_fields": [
    {
      "field_name": "priority",
      "keyword_field_value": "p1"
    },
    {
      "field_name": "ingested_at",
      "date_field_value": "2024..."
    }
  ]
}

However, this leads to too many documents(as in nested doc is a lucene doc) and hence impacts search latency.
Would it be feasible to increase the field limit to 10K, given we only use static mapping and every addition is intended? Plus, we increase the memory of the appropriate nodes (I believe it is the data node memory that will be impacted).

2

Hi @Devanshu_Shah :

The default limit is there to prevent a mapping explosion when using in combination with dynamic mapping.

Adding fields to mappings has some overhead, both in terms of the mapping information in the data nodes, as well as propagating the mapping changes to the cluster. Also, search performance will be impacted when targeting more fields.

If you use static mapping and thus won't be susceptible to uncontrolled field growth, you can raise the default index.mapping.total_fields.limit mapping limit setting and check the performance and memory usage for your nodes.

In case you don't need to search over all the fields, you could consider using the flattened field type to prevent a high number of fields in your mappings that is not being used for search.

Hope that helps!

1 Like