In my use case, I have dynamic fields in my documents which need to be searchable. Each document will only have a couple of these fields (so the total document size will never be massive). I will also only ever be searching/filtering on a couple of them at a time....so even tho I could have thousands of these fields in the index, there will only be 2 or 3 of them present in any document or query. Is there a downside to increasing the limit to, say, 10000 or more?
In reality, the real number of active unique fields would likely hover somewhere around or below 1000...but there is no way (afaik) to remove them from the mapping once they are no longer in use, so the number of fields in the index will keep growing over time as these dynamic fields keep getting created. I will have to periodically regenerate the entire index with a "fresh" mapping to keep them from hitting the limit, so....I'm just trying to figure out what is a safe upper limit to minimize how frequently I need to reindex everything.
What do you mean by "mapping too large"? Is that some an upper limit to the total_fields.limit setting?
The reason my fields are not a fixed set is because they are truly dynamic - in my app the user can create their own fields dynamically ("user defined fields"), enter values for those fields, and then search based on them.
The field count is not the only problem if you have dynamic content as you are likely to eventually encounter mapping conflicts which will prevent data from being indexed, e.g. if one document had a field that contains a string and another has it contain an object.
Apache, Apache Lucene, Apache Hadoop, Hadoop, HDFS and the yellow elephant
logo are trademarks of the
Apache Software Foundation
in the United States and/or other countries.