Implications of large number of fields in a document

Elastic Stack Elasticsearch
1

Hi,

I read that in 6.x version the default limit of index.mapping.total_fields.limit is 1000. We have an index which has fields around 3000 in number. I would like to understand the implications of having so many fields in one index in Elastic Search.

2

Hi @anshulagrawal

With several fields within a mapping you can suffer the consequences of memory errors and situations that will be difficult to recover, this problem is very common, but we managed to avoid.
What I recommend is that the mapping contains only the fields that will actually be used. If even then, the mapping contains 3000 fields, I recommend that this be handled via index. Split these fields into different indexes.

What can also be done, is to increase thshould index.mapping.total_fields.limit to the amount of field the mapping will contain.
You can also leave the mapping pre-defined, and add the option dynamic: true, in this way, if any field is not mapping, it will be created automatically. Or, if you are sure that no fields will be inserted other than the ones you have set, you can add dynamic: false.

Hope this helps.

3

Hi,

We use dynamic templates only and we are currently migrating from version 1.7 to 6.x. Since 1.7 didnt have any such limits we never faced any issue.
I just wanted to understand what impact could a document with growing fields create and how does elastic search decide default limit for such values.

4

@anshulagrawal

The increase of this parameter can bring some problems, for example in the cluster state. For each update of the indexes the cluster state is changed and sent to all nodes, imagine that with each insert a new field is inserted causing the CS to grow, and this will cause latency in the replication of the state information. Another concern is the overhead of the Lucene, having thousands of field could cause it. Using 1000 fields is the recommendation of elastic, but it is still recommended to decrease this linear.

5

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.