What is the downside of increasing index.mapping.total_fields.limit dramatically?

In my use case, I have dynamic fields in my documents which need to be searchable. Each document will only have a couple of these fields (so the total document size will never be massive). I will also only ever be searching/filtering on a couple of them at a time....so even tho I could have thousands of these fields in the index, there will only be 2 or 3 of them present in any document or query. Is there a downside to increasing the limit to, say, 10000 or more?

In reality, the real number of active unique fields would likely hover somewhere around or below 1000...but there is no way (afaik) to remove them from the mapping once they are no longer in use, so the number of fields in the index will keep growing over time as these dynamic fields keep getting created. I will have to periodically regenerate the entire index with a "fresh" mapping to keep them from hitting the limit, so....I'm just trying to figure out what is a safe upper limit to minimize how frequently I need to reindex everything.

Too many fields will casuse Mapping too large. Why your field not a fixed set? The number of fields keep growing is not a good idea.

What do you mean by "mapping too large"? Is that some an upper limit to the total_fields.limit setting?

The reason my fields are not a fixed set is because they are truly dynamic - in my app the user can create their own fields dynamically ("user defined fields"), enter values for those fields, and then search based on them.

Mapping too large will cause memory issue and performance degradation. The default maximum number of fields in an index is 1000.

The field count is not the only problem if you have dynamic content as you are likely to eventually encounter mapping conflicts which will prevent data from being indexed, e.g. if one document had a field that contains a string and another has it contain an object.

No worries there, my app guarantees a unique field name (this is abstracted away from the user)

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.