I use ES for searching of my basic CRUD constructs. But now we need to expand to help us search, sort, paginate our no-code constructs. These are json documents that have 100% dynamic fields. Some rough numbers:
- We have thousands of organizations
- Each organization has an average of 20 groups but some have several 100's of groups
- Each group has a single no-code data definition. The documents for each group will share the same attributes but all other documents in other groups are guaranteed not to share any of the same attributes.
- Documents are small, average 252 bytes up to 14 kb
- The number of attributes in each document could be from 1-100 but averages around 20
The question is how can I index the no-code data? I can't throw them all in a single index obviously as it would have an enormous number of attributes.
Can I create an index per group?
That would result in 1000 orgs X 20 groups = 20000 indices and growing rapidly.
Is that within reason or far-fetched? What if it's 10x orgs or 100x orgs? Could it scale? Basically many small indices. I'm guessing not well.
My instinct is that we should map all dynamic no-code data down to static fields. Since we understand the data types we could map the data to something like:
{ textField1: string, textField2: string, textFieldN: string, dateField1: Date, dateFieldX:Date etc..}
We are willing to limit the number of attributes that can be mapped into ES so this approach seems doable but just clunky.
Is this the only option?
