Dynamic data (no code) scenario strategy

I use ES for searching of my basic CRUD constructs. But now we need to expand to help us search, sort, paginate our no-code constructs. These are json documents that have 100% dynamic fields. Some rough numbers:

  • We have thousands of organizations
  • Each organization has an average of 20 groups but some have several 100's of groups
  • Each group has a single no-code data definition. The documents for each group will share the same attributes but all other documents in other groups are guaranteed not to share any of the same attributes.
  • Documents are small, average 252 bytes up to 14 kb
  • The number of attributes in each document could be from 1-100 but averages around 20

The question is how can I index the no-code data? I can't throw them all in a single index obviously as it would have an enormous number of attributes.

Can I create an index per group?

That would result in 1000 orgs X 20 groups = 20000 indices and growing rapidly.

Is that within reason or far-fetched? What if it's 10x orgs or 100x orgs? Could it scale? Basically many small indices. I'm guessing not well.

My instinct is that we should map all dynamic no-code data down to static fields. Since we understand the data types we could map the data to something like:

{ textField1: string, textField2: string, textFieldN: string, dateField1: Date, dateFieldX:Date etc..}

We are willing to limit the number of attributes that can be mapped into ES so this approach seems doable but just clunky.

Is this the only option?

Welcome to our community! :smiley:

It's the only sensible option, yes.

The others are the ones you pointed out, specific indices, which don't really scale.

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.