Effect of increasing index.mapping.nested_objects.limit

[csdev]

Hi Everyone,

As per Elasticsearch documentation available at link Mapping limit settings | Elasticsearch Guide [8.1] | Elastic, the limit of 10k on index.mapping.nested_objects.limit is kept to prevent out of memory errors.

In my use case this value is expected to be 1000k or similar and I tried to keep 1000k as value to the index setting index.mapping.nested_objects.limit and it is not causing any memory footprint increase. I mean to say that this number does not show any impact on memory until the nested objects in a document are reaching the limit.

So, my question is, is it safe to set index.mapping.nested_objects.limit setting to a higher number like 1000k in advance?

Also, if the document is large enough to hold these many nested object, I think heap size would be playing a part here. So, are there any metrices like per GB heap space, x number of nested object are possible?

Thanks & Regards

[DineshNaik (Dinesh Kumar Naik)]

Welcome to the community !

Yes, The value for index.mapping.nested_objects.limit can be changed based on your use case and the resources you have.

The limit of 10K is kept in place to prevent mappings and searches from becoming too large.

Keeping the value for index.mapping.nested_objects.limit too large can lead to performance degradations and memory issues, especially in clusters with a high load or few resources.

@csdev

Also, if the document is large enough to hold these many nested object, I think heap size would be playing a part here.

yes, the heap will be crucial here. The larger the heap the better it will be for you.
But do note, the upper limit can cause out of memory for large heap size as well.

@csdev

So, are there any metrices like per GB heap space, x number of nested object are possible?

i am not sure if there is any metric to monitor that because that would purely depend on the size of the nested docs which can differ for every use case.

[csdev]

Hi @DineshNaik

Thanks for quick response.

In my case the nested object has 4-5 fields at max which I think would have lesser footprint on memory.

So, It looks like setting a higher value to index.mapping.nested_objects.limit at the time of creation of index, won’t have any issue considering the hardware resources are ready to take care of document of given size.

This setting is there, to prevent smaller clusters with limited resources from heavily loaded documents i.e. document with a lot of nested objects.

Regards

[DineshNaik (Dinesh Kumar Naik)]

Yes, I would still suggest you do a thorough load test to know the breaking points and make sure you don't get any surprises in your production instance.