Having read the various posts on multiple indexes with small number of document vs single index with all the documents, I still am not sure what to do in my situation...
We have a approximately 2-3 million documents spread across what would be 300-400 'types'. When I say type, each type is the same fundamental document but each one can have an unlimited number of associated data attributes. To index them my preference was to put them 1 index per type (300-400 index), each with a common alias. This would ensure that a single index would not risk type clashes on those data attributes, or potential mapping explosion.
But the overhead of each shard has made me think that this is not advisable. If I am wrong about this and I can have several hundred indexes, that would be great as all the other problems go away.
Assuming I cannot have this many indexes on a 3 node cluster, the alternative is to put them in the same index. We are happy not to index (index: false) the associated data attributes, which removes the risk of a type clashes but still persists them into _source. This is fine, however I still want those attributes added in the mappings, just with indexing disabled. However, we are hitting a configurable limit of 1000 fields per mapping.
Can I have several hundred indexes running on a 3 node cluster? Each index would have between 100-100,000 documents in it.
If 99% of my mappings have indexing disabled, is there any overhead in leaving dynamic turned on so the mappings are updated? I would need to increase the max fields per index to the region of 100,000. I know this is way above the recommended number of fields, but if indexing is disabled on the majority of those fields, does this recommended limit still stand?