We are developing an enterprise e-commerce b-to-b solution with potentially hundreds or maybe even thousands of clients.
Each client has product catalog (with dynamic fields) which means each product catalog can have different fields. We wonder if there is a standard approach to deal with that problem by means of Elasticsearch? Some context (a bit of pre-history):
- We had a separate index per dynamic document first, we had a lot of them so we reached es limit for number of indices (1000).
After that we re-designed the system such that: - We now have index/300 GB of data - it worked fine for some time, but now we are hitting es total limit of fields on index: (Type: illegal_argument_exception Reason: "Limit of total fields [1000] has been exceeded while adding new fields [14]" As a temporal solution we increased the limit to be 1500, but as we understand we cannot increase it endlessly as it will start degrading the performance of our index) We are now thinking how can we approach that problem again, and there are two options we are considering at the moment: 1. The first one to have index/company (but then we are limited to 1000 companies and after that we need to look for a new solution + managing 1000 indices is way harder then just 1 ). 2. The second is using flattened field type (Flattened field type | Elasticsearch Guide [8.17] | Elastic), which has some limitations, specifically since it treats all values as keywords, they are not analysed by default, which means that Elasticsearch doesn't break them down into tokens (like it would with a text field), so searching for parts of words or fuzzy matching requires a more manual approach (thats how I understand it). As you can see both of them have its caveats.
Feels like we are missing something in our understanding of how Elasticsearch indices should be used. We would appreciate any thoughts or help!